2016-12-14 59 views
1

我正在使用线程进行远程API调用,不使用联接,以便程序可以在不等待最后一个完成的情况下进行下一个API调用。线程提示是连续运行的,不是平行的吗?

像这样:

def run_single_thread_no_join(function, args): 
thread = Thread(target=function, args=(args,)) 
thread.start() 
return 

的问题是我需要知道,当所有的API调用完成。所以我转移到代码是使用提示&加入。

线程似乎现在串行运行。

我似乎无法弄清楚如何让连接工作,以便线程并行执行。

我在做什么错?

def run_que_block(methods_list, num_worker_threads=10): 
''' 
Runs methods on threads. Stores method returns in a list. Then outputs that list 
after all methods in the list have been completed. 

:param methods_list: example ((method name, args), (method_2, args), (method_3, args) 
:param num_worker_threads: The number of threads to use in the block. 
:return: The full list of returns from each method. 
''' 

method_returns = [] 

# log = StandardLogger(logger_name='run_que_block') 

# lock to serialize console output 
lock = threading.Lock() 

def _output(item): 
    # Make sure the whole print completes or threads can mix up output in one line. 
    with lock: 
     if item: 
      print(item) 
     msg = threading.current_thread().name, item 
     # log.log_debug(msg) 

    return 

# The worker thread pulls an item from the queue and processes it 
def _worker(): 

    while True: 
     item = q.get() 
     if item is None: 
      break 

     method_returns.append(item) 
     _output(item) 

     q.task_done() 

# Create the queue and thread pool. 
q = Queue() 

threads = [] 
# starts worker threads. 
for i in range(num_worker_threads): 
    t = threading.Thread(target=_worker) 
    t.daemon = True # thread dies when main thread (only non-daemon thread) exits. 
    t.start() 
    threads.append(t) 

for method in methods_list: 
    q.put(method[0](*method[1])) 

# block until all tasks are done 
q.join() 

# stop workers 
for i in range(num_worker_threads): 
    q.put(None) 
for t in threads: 
    t.join() 

return method_returns 
+0

的可能的复制[Python的线程不会同时运行(http://stackoverflow.com/questions/19614224/python-threads-dont-run-simultaneously) –

+0

一些你'def's的需要他们缩进的身体。 –

回答

1

你做所有的工作在主线程:

for method in methods_list: 
    q.put(method[0](*method[1])) 

假设在methods_list每个条目是一个可调用和它的参数的顺序,你做了所有的工作,在主线程,然后将每个函数调用的结果放入队列中,除了print ing之外(这通常不足以证明线程/队列开销),并不允许任何并行化。

想必,你想要的线程做各项功能的工作,所以改变环路:

for method in methods_list: 
    q.put(method) # Don't call it, queue it to be called in worker 

并更改_worker功能,所以它调用确实在线程工作中的作用:

def _worker(): 
    while True: 
     method, args = q.get() # Extract and unpack callable and arguments 
     item = method(*args) # Call callable with provided args and store result 
     if item is None: 
      break 

     method_returns.append(item) 
     _output(item) 

     q.task_done()