初探 python 中的并行计算,计算时间比较

在一些 python 的高级代码中经常见到并行计算的使用,稍微接触了下,还是挺有趣的。

  • 需要注意的是,并行计算并不一定比传统的循环代码快,只有当循环运算次数非常大时,并行计算才有优势
  • 因此,并行计算涉及到计算任务在处理器不同内核之间的分配,不同内核的计算结果组装在一起返回,这部分处理时间是不可避免的
  • 尝试在动态规划中使用并行计算,极易出错,电脑最后还死机了,最后也没有成功

1. 使用 Pool 的一个例子

  • 下面的代码中使用了 Pool 与 map 来进行并行计算
  • Pool() 小括号里面可以跟数字,表示开启的进程数
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Wed Nov 27 22:20:40 2024

@author: zhenchen

@disp:  test multi processing in python

    
    
"""

from multiprocessing import Pool
import time

def f(x):
    return x*x

if __name__ == '__main__':
    
    loop_nums = [10, 1000, 100000, 1000000, 4000000, 10000000]
    for loop_num in loop_nums:
        input_data = range(loop_num)
        
        print('loop number is {:,}'.format(loop_num))
        start = time.process_time()
        with Pool() as p:        
            p.map(f, input_data)
        end = time.process_time()
        print('running time with parallel computing is %.4fs' % (end - start))
        
        start = time.process_time()
        for i in range(loop_num):
            f(input_data[i])
        end = time.process_time()
        print('running time without parallel computing is %.4fs\n' % (end - start))
    
    

输出结果:

loop number is 10
running time with parallel computing is 0.2728s
running time without parallel computing is 0.0000s

loop number is 1,000
running time with parallel computing is 0.2462s
running time without parallel computing is 0.0002s

loop number is 100,000
running time with parallel computing is 0.2654s
running time without parallel computing is 0.0159s

loop number is 4,000,000
running time with parallel computing is 0.5915s
running time without parallel computing is 0.6335s

loop number is 10,000,000
running time with parallel computing is 1.2106s
running time without parallel computing is 1.6005s

  • 上面的例子中,当循环计算 4 百万以上时,并行计算才比 for 循环有优势

2. 使用 Process 的例子

下面的例子中,使用了 Process 来开启并行计算,

  • 使用 start() 开启子进程
  • 使用 join() 关闭子进程

Process 中的参数:

  • target: 进程执行的函数
  • args: 函数的参数
from multiprocessing import Process
import os

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(name):
    info('function f')
    print('hello', name)

if __name__ == '__main__':
    info('main line')
    for i in range(10):
        p = Process(target=f, args=('bob', ))
        p.start()
        p.join()

输出:

main line
module name: __main__
parent process: 14448
process id: 13432

function f
module name: __mp_main__
parent process: 13432
process id: 4964
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 8236
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 11660
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 12228
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 8356
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 9068
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 15208
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 8884
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 1572
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 1248
hello bob

查看当前电脑的核数:

import multiprocessing

print("Number of cpu : ", multiprocessing.cpu_count())

另外一个例子:

from multiprocessing import Process


def print_func(continent='Asia'):
    print('The name of continent is : ', continent)

if __name__ == "__main__":  # confirms that the code is under main function
    names = ['America', 'Europe', 'Africa']
    procs = []
    proc = Process(target=print_func)  # instantiating without any argument
    procs.append(proc)
    proc.start()

    # instantiating process with arguments
    for name in names:
        # print(name)
        proc = Process(target=print_func, args=(name,))
        procs.append(proc)
        proc.start()

    # complete the processes
    for proc in procs:
        proc.join()

输出:

The name of continent is :  Asia
The name of continent is :  America
The name of continent is :  Europe
The name of continent is :  Africa

3. 使用 Queue 的例子

  • Queue 使用先进先出的队列管理子进程
  • 使用 put() 函数将数据推进队列里
  • 使用 get() 取出队列里的数据
from multiprocessing import Queue

colors = ['red', 'green', 'blue', 'black']
cnt = 1
# instantiating a queue object
queue = Queue()
print('pushing items to queue:')
for color in colors:
    print('item no: ', cnt, ' ', color)
    queue.put(color)
    cnt += 1

print('\npopping items from queue:')
cnt = 0
while not queue.empty():
    print('item no: ', cnt, ' ', queue.get())
    cnt += 1
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

心态与习惯

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值