ó 8Þ Pc@s8dgZddlZddlZddlZddlZddlZddlmZmZm Z ddl m Z m Z dZ dZdZejƒZd„Zefed „Zdefd „ƒYZd efd „ƒYZd efd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZdS(tPooliÿÿÿÿN(tProcesst cpu_countt TimeoutError(tFinalizetdebugiiicCs t|ŒS(N(tmap(targs((s*/usr/lib/python2.7/multiprocessing/pool.pytmapstar@sc Cs}|dks0t|ƒtkr*|dks0t‚|j}|j}t|dƒrn|jjƒ|j jƒn|dk r‡||Œnd}xÛ|dks®|rj||krjy |ƒ}Wn"t t fk rßt dƒPnX|dkrút dƒPn|\} } } } } yt | | | Žf}Wntk rI}t|f}nX|| | |fƒ|d7}qWt d|ƒdS(Nit_writers)worker got EOFError or IOError -- exitingsworker got sentinel -- exitingisworker exiting after %d tasks(tNonettypetinttAssertionErrortputtgetthasattrR tcloset_readertEOFErrortIOErrorRtTruet ExceptiontFalse(tinqueuetoutqueuet initializertinitargstmaxtasksRRt completedttasktjobtitfuncRtkwdstresultte((s*/usr/lib/python2.7/multiprocessing/pool.pytworkerGs40     !    cBseZdZeZddddd„Zd„Zd„Zd„Zd„Z did„Z dd„Z dd „Z dd „Z didd „Zddd „Zed „ƒZed„ƒZed„ƒZed„ƒZd„Zd„Zd„Zd„Zed„ƒZed„ƒZRS(sH Class which supports an async version of the `apply()` builtin c Cs)|jƒtjƒ|_i|_t|_||_||_||_|dkr|y t ƒ}Wq|t k rxd}q|Xn|dkr—t dƒ‚n|dk rÂt |dƒ rÂtdƒ‚n||_g|_|jƒtjdtjd|fƒ|_t|j_t|j_|jjƒtjdtjd|j|j|j|jfƒ|_t|j_t|j_|jjƒtjdtjd|j|j|jfƒ|_ t|j _t|j _|j jƒt!||j"d|j|j#|j|j|j|j|j |jfddƒ|_$dS( Nis&Number of processes must be at least 1t__call__sinitializer must be a callablettargetRt exitpriorityi(%t _setup_queuestQueuet _taskqueuet_cachetRUNt_statet_maxtasksperchildt _initializert _initargsR RtNotImplementedErrort ValueErrorRt TypeErrort _processest_poolt_repopulate_poolt threadingtThreadRt_handle_workerst_worker_handlerRtdaemontstartt _handle_taskst _quick_putt _outqueuet _task_handlert_handle_resultst _quick_gett_result_handlerRt_terminate_poolt_inqueuet _terminate(tselft processesRRtmaxtasksperchild((s*/usr/lib/python2.7/multiprocessing/pool.pyt__init__qsV                     $          cCswt}xjttt|jƒƒƒD]M}|j|}|jdk r"td|ƒ|jƒt }|j|=q"q"W|S(sCleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. scleaning up worker %dN( RtreversedtrangetlenR6texitcodeR RtjoinR(RHtcleanedR R%((s*/usr/lib/python2.7/multiprocessing/pool.pyt_join_exited_workers«s"  c Cs¦xŸt|jt|jƒƒD]}|jdtd|j|j|j|j |j fƒ}|jj |ƒ|j j ddƒ|_ t|_|jƒtdƒqWdS(s€Bring the number of pool processes up to the specified number, for use after reaping workers which have exited. R'RRt PoolWorkers added workerN(RMR5RNR6RR%RFR@R0R1R/tappendtnametreplaceRR<R=R(RHR tw((s*/usr/lib/python2.7/multiprocessing/pool.pyR7ºs#   cCs|jƒr|jƒndS(sEClean up any exited workers and start replacements for them. N(RRR7(RH((s*/usr/lib/python2.7/multiprocessing/pool.pyt_maintain_poolÊs cCsPddlm}|ƒ|_|ƒ|_|jjj|_|jjj|_ dS(Ni(t SimpleQueue( tqueuesRYRFR@R tsendR?RtrecvRC(RHRY((s*/usr/lib/python2.7/multiprocessing/pool.pyR)Ðs   cCs.|jtkst‚|j|||ƒjƒS(s1 Equivalent of `apply()` builtin (R.R-R t apply_asyncR(RHR!RR"((s*/usr/lib/python2.7/multiprocessing/pool.pytapply×scCs.|jtkst‚|j|||ƒjƒS(s/ Equivalent of `map()` builtin (R.R-R t map_asyncR(RHR!titerablet chunksize((s*/usr/lib/python2.7/multiprocessing/pool.pyRÞsicsÝ|jtkst‚|dkrft|jƒ‰|jj‡‡fd†t|ƒDƒˆjfƒˆS|dksxt‚t j ˆ||ƒ}t|jƒ‰|jj‡fd†t|ƒDƒˆjfƒd„ˆDƒSdS(sZ Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()` ic3s0|]&\}}ˆj|ˆ|fifVqdS(N(t_job(t.0R tx(R#R!(s*/usr/lib/python2.7/multiprocessing/pool.pys ìsc3s0|]&\}}ˆj|t|fifVqdS(N(RbR(RcR Rd(R#(s*/usr/lib/python2.7/multiprocessing/pool.pys óscss"|]}|D] }|Vq qdS(N((Rctchunktitem((s*/usr/lib/python2.7/multiprocessing/pool.pys õsN( R.R-R t IMapIteratorR,R+Rt enumeratet _set_lengthRt _get_tasks(RHR!R`Rat task_batches((R#R!s*/usr/lib/python2.7/multiprocessing/pool.pytimapås csÝ|jtkst‚|dkrft|jƒ‰|jj‡‡fd†t|ƒDƒˆjfƒˆS|dksxt‚t j ˆ||ƒ}t|jƒ‰|jj‡fd†t|ƒDƒˆjfƒd„ˆDƒSdS(sK Like `imap()` method but ordering of results is arbitrary ic3s0|]&\}}ˆj|ˆ|fifVqdS(N(Rb(RcR Rd(R#R!(s*/usr/lib/python2.7/multiprocessing/pool.pys þsc3s0|]&\}}ˆj|t|fifVqdS(N(RbR(RcR Rd(R#(s*/usr/lib/python2.7/multiprocessing/pool.pys scss"|]}|D] }|Vq qdS(N((RcReRf((s*/usr/lib/python2.7/multiprocessing/pool.pys sN( R.R-R tIMapUnorderedIteratorR,R+RRhRiRRj(RHR!R`RaRk((R#R!s*/usr/lib/python2.7/multiprocessing/pool.pytimap_unordered÷s cCsV|jtkst‚t|j|ƒ}|jj|jd|||fgdfƒ|S(s> Asynchronous equivalent of `apply()` builtin N( R.R-R t ApplyResultR,R+RRbR (RHR!RR"tcallbackR#((s*/usr/lib/python2.7/multiprocessing/pool.pyR] s+csû|jtkst‚t|dƒs3t|ƒ}n|dkr}tt|ƒt|jƒdƒ\}}|r}|d7}q}nt|ƒdkr˜d}nt j |||ƒ}t |j |t|ƒ|ƒ‰|j j‡fd†t|ƒDƒdfƒˆS(s< Asynchronous equivalent of `map()` builtin t__len__iiic3s0|]&\}}ˆj|t|fifVqdS(N(RbR(RcR Rd(R#(s*/usr/lib/python2.7/multiprocessing/pool.pys #sN(R.R-R RtlistR tdivmodRNR6RRjt MapResultR,R+RRh(RHR!R`RaRptextraRk((R#s*/usr/lib/python2.7/multiprocessing/pool.pyR_s ( cCs]x<|jjtkr>|jtkr>|jƒtjdƒqW|jjdƒt dƒdS(Ngš™™™™™¹?sworker handler exiting( R;R.R-RXttimetsleepR+RR R(tpool((s*/usr/lib/python2.7/multiprocessing/pool.pyR:'s $ c Cs1tjƒ}x¶t|jdƒD]˜\}}d}x‚t|ƒD]P\}}|jratdƒPny||ƒWq>tk rtdƒPq>Xq>W|rtdƒ||dƒqqPqWtdƒy@tdƒ|j dƒtdƒx|D]} |dƒqðWWntk r"td ƒnXtd ƒdS( Niÿÿÿÿs'task handler found thread._state != RUNscould not put task on queuesdoing set_length()istask handler got sentinels/task handler sending sentinel to result handlers(task handler sending sentinel to workerss/task handler got IOError when sending sentinelsstask handler exiting( R8tcurrent_threadtiterRR RhR.RRR( t taskqueueRRRxtthreadttaskseqt set_lengthR Rtp((s*/usr/lib/python2.7/multiprocessing/pool.pyR>0s6             cCsòtjƒ}x¶y |ƒ}Wn"ttfk r@tdƒdSX|jrm|jtks_t‚tdƒPn|dkr‡tdƒPn|\}}}y||j ||ƒWqt k rÁqXqx¢|ri|jtkriy |ƒ}Wn"ttfk rtdƒdSX|dkr+tdƒqÈn|\}}}y||j ||ƒWqÈt k reqÈXqÈWt |dƒrÕtdƒy5x.t dƒD] }|j jƒs¬Pn|ƒq“WWqÕttfk rÑqÕXntdt|ƒ|jƒdS( Ns.result handler got EOFError/IOError -- exitings,result handler found thread._state=TERMINATEsresult handler got sentinels&result handler ignoring extra sentinelRs"ensuring that outqueue is not fulli s7result handler exiting: len(cache)=%s, thread._state=%s(R8RyRRRR.t TERMINATER R t_settKeyErrorRRMRtpollRN(RRtcacheR|RRR tobj((s*/usr/lib/python2.7/multiprocessing/pool.pyRBWsX              ccsCt|ƒ}x0ttj||ƒƒ}|s1dS||fVqdS(N(Rzttuplet itertoolstislice(R!tittsizeRd((s*/usr/lib/python2.7/multiprocessing/pool.pyRj‘s  cCstdƒ‚dS(Ns:pool objects cannot be passed between processes or pickled(R2(RH((s*/usr/lib/python2.7/multiprocessing/pool.pyt __reduce__šscCs5tdƒ|jtkr1t|_t|j_ndS(Ns closing pool(RR.R-tCLOSER;(RH((s*/usr/lib/python2.7/multiprocessing/pool.pyRŸs  cCs-tdƒt|_t|j_|jƒdS(Nsterminating pool(RR€R.R;RG(RH((s*/usr/lib/python2.7/multiprocessing/pool.pyt terminate¥s   cCsntdƒ|jttfks%t‚|jjƒ|jjƒ|jjƒx|j D]}|jƒqVWdS(Ns joining pool( RR.RŒR€R R;RPRARDR6(RHR((s*/usr/lib/python2.7/multiprocessing/pool.pyRP«s    cCsWtdƒ|jjƒx9|jƒrR|jjƒrR|jjƒtjdƒqWdS(Ns7removing tasks from inqueue until task handler finishedi( Rt_rlocktacquiretis_aliveRRƒR\RvRw(Rt task_handlerRŠ((s*/usr/lib/python2.7/multiprocessing/pool.pyt_help_stuff_finish´s    c Csstdƒt|_t|_tdƒ|j||t|ƒƒ|jƒsct|ƒdksct‚t|_|jdƒtdƒ|j ƒ|ràt |ddƒràtdƒx-|D]"} | j dkr·| j ƒq·q·Wntdƒ|j dƒtd ƒ|j dƒ|rot |ddƒrotd ƒx;|D]0} | jƒr8td | j ƒ| j ƒq8q8WndS( Nsfinalizing pools&helping task handler/workers to finishisjoining worker handlerRsterminating workerssjoining task handlerg}Ô%­I²Tsjoining result handlersjoining pool workersscleaning up worker %d(RR€R.R’RNRR RR RPRRORtpid( tclsR{RRRxtworker_handlerR‘tresult_handlerR„R((s*/usr/lib/python2.7/multiprocessing/pool.pyRE½s2    $             N((((t__name__t __module__t__doc__RR RKRRR7RXR)R^RRlRnR]R_t staticmethodR:R>RBRjR‹RRRPR’t classmethodRE(((s*/usr/lib/python2.7/multiprocessing/pool.pyRks0  9         ':     RocBsDeZd„Zd„Zd„Zdd„Zdd„Zd„ZRS(cCsStjtjƒƒ|_tjƒ|_||_t|_ ||_ |||jeZd„Zd„Zdd„ZeZd„Zd„ZRS(cCsktjtjƒƒ|_tjƒ|_||_tj ƒ|_ d|_ d|_ i|_|||j#s(        $ÿ‚2,I