从pickle文件加载multiprocessing.manager.dictionary时出错

Question

从pickle加载multiprocessing.manager.dictionary时出现错误。我确定pickle文件存在于我运行python的目录中。

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2 
Type "help", "copyright", "credits" or "license" for more information. 
>>> import pickle 
>>> import multiprocessing 
>>> a = open("test.pkl", "rb") 
>>> pickle.load(a) 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/lib/python2.7/pickle.py", line 1378, in load 
    return Unpickler(file).load() 
    File "/usr/lib/python2.7/pickle.py", line 858, in load 
    dispatch[key](self) 
    File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce 
    value = func(*args) 
    File "/usr/lib/python2.7/multiprocessing/managers.py", line 879, in RebuildProxy 
    return func(token, serializer, incref=incref, **kwds) 
    File "/usr/lib/python2.7/multiprocessing/managers.py", line 733, in __init__ 
    self._incref() 
    File "/usr/lib/python2.7/multiprocessing/managers.py", line 783, in _incref 
    conn = self._Client(self._token.address, authkey=self._authkey) 
    File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client 
    c = SocketClient(address) 
    File "/usr/lib/python2.7/multiprocessing/connection.py", line 304, in SocketClient 
    s.connect(address) 
    File "/usr/lib/python2.7/socket.py", line 224, in meth 
    return getattr(self._sock,name)(*args) 
socket.error: [Errno 2] No such file or directory 
>>> pickle.load(a) 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/lib/python2.7/pickle.py", line 1378, in load 
    return Unpickler(file).load() 
    File "/usr/lib/python2.7/pickle.py", line 858, in load 
    dispatch[key](self) 
    File "/usr/lib/python2.7/pickle.py", line 1165, in load_put 
    self.memo[self.readline()[:-1]] = self.stack[-1] 
IndexError: list index out of range 

下面是将代码保存到泡菜中的代码部分。

from multiprocessing import Manager, Pool 
import pickle 
from functools import partial 

def output(dic, s): 
    a = open(s + ".pkl", "wb") 
    pickle.dump(dic, a) 
    a.close() 

data_list = [1,2,3,4] 
pool = Pool(processes = 4) 
m = Manager() 
lock = m.Lock() 
dic1 = m.dict() 
func = partial(f, dic1) # f is a function that takes 3 arguments with the returned result stored in dic1 
pool.map(func, data_list) 
output(dic1, "test") 
pool.close() 
pool.join() 

如果我在输出前打印字典，终端上的结果看起来很好。

Answer 1

Manager.dict()实际上并没有返回一个字典，而是一个代理对象，由一个不同的进程管理的实际字典。当您在该对象上调用方法时，该调用被转发到实际字典所在的管理器。

由于管理器和客户端进程之间的通信是使用pickle协议进行的，所以这些对象是pickleable的，当unpickled再次为生活在manager进程内的实际对象生成一个代理时，这就是为什么你会看到尝试创建一个连接当你试图取消字典。

所以，如果你要腌经理字典的内容，它之前被转化为实际字典：

... 
pickle.dump(dict(dic), a) 
...