多重处理和莳萝可以一起做什么？

多重处理和莳萝可以一起做什么？

multiprocessing

对酸洗有一些不好的选择。别误会，它做出了一些不错的选择，使它可以对某些类型进行腌制，以便可以在池的地图功能中使用它们。但是，由于我们有

dill

能力进行酸洗，因此多处理程序本身的酸洗变得有些局限。实际上，如果

multiprocessing

使用

pickle

代替

cPickle

…并删除其自身的某些酸洗覆盖，则

dill

可以接管并为给出更多的完整序列化

multiprocessing

。

在此之前，有一个

multiprocessing

名为“
pathos”的分支（不幸的是，发行版本有些陈旧），它消除了上述限制。Pathos还添加了一些多处理所没有的不错的功能，例如map函数中的multi-
args。经过一些轻微的更新后，Pathos即将发布，主要是转换为python3.x。

Python 2.7.5 (default, Sep 30 2013, 20:15:49) [GCC 4.2.1 (Apple Inc. build 5566)] on darwinType "help", "copyright", "credits" or "license" for more information.>>> import dill>>> from pathos.multiprocessing import ProcessingPool    >>> pool = ProcessingPool(nodes=4)>>> result = pool.map(lambda x: x**2, range(10))>>> result[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

只是炫耀一下

pathos.multiprocessing

可以做什么…

>>> def busy_add(x,y, delay=0.01):...     for n in range(x):...        x += n...     for n in range(y):...        y -= n...     import time...     time.sleep(delay)...     return x + y... >>> def busy_squared(x):...     import time, random...     time.sleep(2*random.random())...     return x*x... >>> def squared(x):...     return x*x... >>> def quad_factory(a=1, b=1, c=0):...     def quad(x):...         return a*x**2 + b*x + c...     return quad... >>> square_plus_one = quad_factory(2,0,1)>>> >>> def test1(pool):...     print pool...     print "x: %sn" % str(x)...     print pool.map.__name__...     start = time.time()...     res = pool.map(squared, x)...     print "time to results:", time.time() - start...     print "y: %sn" % str(res)...     print pool.imap.__name__...     start = time.time()...     res = pool.imap(squared, x)...     print "time to queue:", time.time() - start...     start = time.time()...     res = list(res)...     print "time to results:", time.time() - start...     print "y: %sn" % str(res)...     print pool.amap.__name__...     start = time.time()...     res = pool.amap(squared, x)...     print "time to queue:", time.time() - start...     start = time.time()...     res = res.get()...     print "time to results:", time.time() - start...     print "y: %sn" % str(res)... >>> def test2(pool, items=4, delay=0):...     _x = range(-items/2,items/2,2)...     _y = range(len(_x))...     _d = [delay]*len(_x)...     print map...     res1 = map(busy_squared, _x)...     res2 = map(busy_add, _x, _y, _d)...     print pool.map...     _res1 = pool.map(busy_squared, _x)...     _res2 = pool.map(busy_add, _x, _y, _d)...     assert _res1 == res1...     assert _res2 == res2...     print pool.imap...     _res1 = pool.imap(busy_squared, _x)...     _res2 = pool.imap(busy_add, _x, _y, _d)...     assert list(_res1) == res1...     assert list(_res2) == res2...     print pool.amap...     _res1 = pool.amap(busy_squared, _x)...     _res2 = pool.amap(busy_add, _x, _y, _d)...     assert _res1.get() == res1...     assert _res2.get() == res2...     print ""... >>> def test3(pool): # test against a function that should fail in pickle...     print pool...     print "x: %sn" % str(x)...     print pool.map.__name__...     start = time.time()...     res = pool.map(square_plus_one, x)...     print "time to results:", time.time() - start...     print "y: %sn" % str(res)... >>> def test4(pool, maxtries, delay):...     print pool...     m = pool.amap(busy_add, x, x)...     tries = 0...     while not m.ready():...         time.sleep(delay)...         tries += 1...         print "TRY: %s" % tries...         if tries >= maxtries:...  print "TIMEOUT"...  break...     print m.get()... >>> import time>>> x = range(18)>>> delay = 0.01>>> items = 20>>> maxtries = 20>>> from pathos.multiprocessing import ProcessingPool as Pool>>> pool = Pool(nodes=4)>>> test1(pool)<pool ProcessingPool(ncpus=4)>x: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]maptime to results: 0.0553691387177y: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289]imaptime to queue: 7.91549682617e-05time to results: 0.102381229401y: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289]amaptime to queue: 7.08103179932e-05time to results: 0.0489699840546y: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289]>>> test2(pool, items, delay)<built-in function map><bound method ProcessingPool.map of <pool ProcessingPool(ncpus=4)>><bound method ProcessingPool.imap of <pool ProcessingPool(ncpus=4)>><bound method ProcessingPool.amap of <pool ProcessingPool(ncpus=4)>>>>> test3(pool)<pool ProcessingPool(ncpus=4)>x: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]maptime to results: 0.0523059368134y: [1, 3, 9, 19, 33, 51, 73, 99, 129, 163, 201, 243, 289, 339, 393, 451, 513, 579]>>> test4(pool, maxtries, delay)<pool ProcessingPool(ncpus=4)>TRY: 1TRY: 2TRY: 3TRY: 4TRY: 5TRY: 6TRY: 7[0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34]