如果任务失败，我该如何将任务放回队列中？

Question

我有一个脚本，看起来像这样：

#!/usr/bin/env python 
# encoding: utf-8 

import time, random, os, multiprocessing 

def main(): 
    NPROCESSES = 5 
    pool = multiprocessing.Pool(processes=NPROCESSES) 

    a = [1,2,3,4,5,6,7,8,9,0] 
    for _ in pool.imap_unordered(do_task, a): 
     pass 

def do_task(n): 
    try: 
     might_crash(n) 
    except Hell, e: 
     print e, " crashed." 

def might_crash(n): 
    time.sleep(3*random.random()) 
    if random.randrange(3) == 0: 
     raise Hell(n) 
    print n 

class Hell(Exception): 
    pass 

if __name__=="__main__":  
    main() 

这个脚本，通常会“A”打印值，但might_crash（）会随机产生一个异常。

我想捕获这些异常，并将当前的do_task（）放回队列中以便稍后重试。

如何将当前任务返回队列中，如果它应该失败？

Answer 1

你可以从do_task收集的结果中，检查结果是Hell情况下，这些东西到任务列表new_tasks，并循环，直到没有new_tasks：

import time 
import random 
import os 
import multiprocessing as mp 

def main(): 
    NPROCESSES = 5 
    pool=mp.Pool(NPROCESSES) 
    a = [1,2,3,4,5,6,7,8,9,0] 
    new_tasks=a 
    while new_tasks: 
     a=new_tasks 
     new_tasks=[] 
     for result in pool.imap_unordered(do_task, a): 
      if isinstance(result,Hell): 
       new_tasks.append(result.args[0]) 
      else: 
       print(result) 

def do_task(n): 
    try: 
     result=might_crash(n) 
    except Hell as e:   
     print("{0} crashed.".format(e.args[0])) 
     result=e 
    return result 

def might_crash(n): 
    time.sleep(3*random.random()) 
    if random.randrange(3) == 0: 
     raise Hell(n) 
    return '{0} done'.format(n) 

class Hell(Exception): 
    pass 

if __name__=="__main__":  
    main() 

产生

1 done 
6 crashed. 
4 done 
7 crashed. 
5 done 
9 done 
3 done 
2 crashed. 
8 done 
0 crashed. 
0 crashed. 
2 done 
7 crashed. 
6 done 
0 done 
7 done