Python：如何在python中运行嵌套并行进程？

我有交易者交易的数据集df.我有2个级别的for循环,如下所示：smartTrader =[]for asset in range(len(Assets)):df = df[df[Assets] == asset]# I have some more calculations herefor trader in range(len(df[Tra...

我有交易者交易的数据集df.
我有2个级别的for循环,如下所示：

smartTrader =[]

for asset in range(len(Assets)):
    df = df[df['Assets'] == asset]
    # I have some more calculations here
    for trader in range(len(df['TraderID'])):
        # I have some calculations here, If trader is successful, I add his ID  
        # to the list as follows
        smartTrader.append(df['TraderID'][trader])

    # some more calculations here which are related to the first for loop.

我想并行化“资产”中每个资产的计算,并且还希望并行化每个资产的每个交易者的计算.完成所有这些计算后,我想基于smartTrader列表进行其他分析.

这是我第一次尝试并行处理,因此请耐心等待,感谢您的帮助.

解决方法:

如果使用pathos,它提供了多处理的分支,则可以轻松嵌套并行映射. pathos是为轻松测试嵌套并行映射的组合而构建的-嵌套并行映射是嵌套for循环的直接转换.
它提供了阻塞,非阻塞,迭代,异步,串行,并行和分布式的映射选择.

>>> from pathos.pools import ProcessPool, ThreadPool
>>> amap = ProcessPool().amap
>>> tmap = ThreadPool().map
>>> from math import sin, cos
>>> print amap(tmap, [sin,cos], [range(10),range(10)]).get()
[[0.0, 0.8414709848078965, 0.9092974268256817, 0.1411200080598672, -0.7568024953079282, -0.9589242746631385, -0.27941549819892586, 0.6569865987187891, 0.9893582466233818, 0.4121184852417566], [1.0, 0.5403023058681398, -0.4161468365471424, -0.9899924966004454, -0.6536436208636119, 0.2836621854632263, 0.9601702866503661, 0.7539022543433046, -0.14550003380861354, -0.9111302618846769]]

在此示例中,使用了一个处理池和一个线程池,其中线程映射调用处于阻塞状态,而处理映射调用是异步的(请注意最后一行的get).

在这里获取悲伤：https://github.com/uqfoundation
或搭配：

$pip安装git https://github.com/uqfoundation/pathos.git@master