使用多处理Pool.map()时不能pickle

I'm trying to use multiprocessing's Pool.map() function to divide out work simultaneously. When I use the following code, it works fine:

我正在尝试使用multiprocessing的Pool.map()函数来同时划分工作。当我使用以下代码时，它工作得很好:

import multiprocessing

def f(x):
    return x*x

def go():
    pool = multiprocessing.Pool(processes=4)        
    print pool.map(f, range(10))


if __name__== '__main__' :
    go()

However, when I use it in a more object-oriented approach, it doesn't work. The error message it gives is:

然而，当我在更面向对象的方法中使用它时，它就不起作用了。它给出的错误信息是:

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed

This occurs when the following is my main program:

这发生在以下是我的主要计划:

import someClass

if __name__== '__main__' :
    sc = someClass.someClass()
    sc.go()

and the following is my someClass class:

下面是我的课程:

import multiprocessing

class someClass(object):
    def __init__(self):
        pass

    def f(self, x):
        return x*x

    def go(self):
        pool = multiprocessing.Pool(processes=4)       
        print pool.map(self.f, range(10))

Anyone know what the problem could be, or an easy way around it?

有人知道问题出在哪里吗?

9 个解决方案

#1

The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method.

问题是，多处理必须将事物pickle到进程之间，并且绑定方法是不可pickle的。解决方案(不管你是否认为它“简单”)是将基础结构添加到程序中，以使这些方法被pickle，并使用copy_reg标准库方法注册它。

For example, Steven Bethard's contribution to this thread (towards the end of the thread) shows one perfectly workable approach to allow method pickling/unpickling via copy_reg.

例如，Steven Bethard对这个线程的贡献(接近线程的末尾)显示了一种完全可行的方法，允许通过copy_reg对方法进行pickle / unpickle。

9 个解决方案

#1

更多相关文章

随机推荐