并行的Python子进程

并行的Python子进程

您可以在一个线程中完成。

假设您有一个脚本可以随机打印行：

#!/usr/bin/env python#file: child.pyimport osimport randomimport sysimport timefor i in range(10):    print("%2d %s %s" % (int(sys.argv[1]), os.getpid(), i))    sys.stdout.flush()    time.sleep(random.random())

而且您想在输出可用后立即收集它，您可以

select

按照@zigg的建议在POSIX系统上 使用：

#!/usr/bin/env pythonfrom __future__ import print_functionfrom select     import selectfrom subprocess import Popen, PIPE# start several subprocessesprocesses = [Popen(['./child.py', str(i)], stdout=PIPE,        bufsize=1, close_fds=True,        universal_newlines=True)  for i in range(5)]# read outputtimeout = 0.1 # secondswhile processes:    # remove finished processes from the list (O(N**2))    for p in processes[:]:        if p.poll() is not None: # process ended print(p.stdout.read(), end='') # read the rest p.stdout.close() processes.remove(p)    # wait until there is something to read    rlist = select([p.stdout for p in processes], [],[], timeout)[0]    # read a line from each process that has output ready    for f in rlist:        print(f.readline(), end='') #NOTE: it can block

更具可移植性的解决方案（应在Windows，Linux，OSX上运行）可以为每个进程使用读取器线程，请参阅python中的对子进程的非阻塞读取。

这

os.pipe()

是适用于Unix和Windows的基于解决方案：

#!/usr/bin/env pythonfrom __future__ import print_functionimport ioimport osimport sysfrom subprocess import PopenON_POSIX = 'posix' in sys.builtin_module_names# create a pipe to get datainput_fd, output_fd = os.pipe()# start several subprocessesprocesses = [Popen([sys.executable, 'child.py', str(i)], stdout=output_fd,        close_fds=ON_POSIX) # close input_fd in children  for i in range(5)]os.close(output_fd) # close unused end of the pipe# read output line by line as soon as it is availablewith io.open(input_fd, 'r', buffering=1) as file:    for line in file:        print(line, end='')#for p in processes:    p.wait()