在一些 python 的高级代码中经常见到并行计算的使用,稍微接触了下,还是挺有趣的。
- 需要注意的是,并行计算并不一定比传统的循环代码快,只有当循环运算次数非常大时,并行计算才有优势
- 因此,并行计算涉及到计算任务在处理器不同内核之间的分配,不同内核的计算结果组装在一起返回,这部分处理时间是不可避免的
- 尝试在动态规划中使用并行计算,极易出错,电脑最后还死机了,最后也没有成功
1. 使用 Pool 的一个例子
- 下面的代码中使用了 Pool 与 map 来进行并行计算
- Pool() 小括号里面可以跟数字,表示开启的进程数
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Wed Nov 27 22:20:40 2024
@author: zhenchen
@disp: test multi processing in python
"""
from multiprocessing import Pool
import time
def f(x):
return x*x
if __name__ == '__main__':
loop_nums = [10, 1000, 100000, 1000000, 4000000, 10000000]
for loop_num in loop_nums:
input_data = range(loop_num)
print('loop number is {:,}'.format(loop_num))
start = time.process_time()
with Pool() as p:
p.map(f, input_data)
end = time.process_time()
print('running time with parallel computing is %.4fs' % (end - start))
start = time.process_time()
for i in range(loop_num):
f(input_data[i])
end = time.process_time()
print('running time without parallel computing is %.4fs\n' % (end - start))
输出结果:
loop number is 10
running time with parallel computing is 0.2728s
running time without parallel computing is 0.0000s
loop number is 1,000
running time with parallel computing is 0.2462s
running time without parallel computing is 0.0002s
loop number is 100,000
running time with parallel computing is 0.2654s
running time without parallel computing is 0.0159s
loop number is 4,000,000
running time with parallel computing is 0.5915s
running time without parallel computing is 0.6335s
loop number is 10,000,000
running time with parallel computing is 1.2106s
running time without parallel computing is 1.6005s
- 上面的例子中,当循环计算 4 百万以上时,并行计算才比 for 循环有优势
2. 使用 Process 的例子
下面的例子中,使用了 Process 来开启并行计算,
- 使用 start() 开启子进程
- 使用 join() 关闭子进程
Process 中的参数:
- target: 进程执行的函数
- args: 函数的参数
from multiprocessing import Process
import os
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
def f(name):
info('function f')
print('hello', name)
if __name__ == '__main__':
info('main line')
for i in range(10):
p = Process(target=f, args=('bob', ))
p.start()
p.join()
输出:
main line
module name: __main__
parent process: 14448
process id: 13432
function f
module name: __mp_main__
parent process: 13432
process id: 4964
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 8236
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 11660
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 12228
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 8356
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 9068
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 15208
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 8884
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 1572
hello bob
function f
module name: __mp_main__
parent process: 13432
process id: 1248
hello bob
查看当前电脑的核数:
import multiprocessing
print("Number of cpu : ", multiprocessing.cpu_count())
另外一个例子:
from multiprocessing import Process
def print_func(continent='Asia'):
print('The name of continent is : ', continent)
if __name__ == "__main__": # confirms that the code is under main function
names = ['America', 'Europe', 'Africa']
procs = []
proc = Process(target=print_func) # instantiating without any argument
procs.append(proc)
proc.start()
# instantiating process with arguments
for name in names:
# print(name)
proc = Process(target=print_func, args=(name,))
procs.append(proc)
proc.start()
# complete the processes
for proc in procs:
proc.join()
输出:
The name of continent is : Asia
The name of continent is : America
The name of continent is : Europe
The name of continent is : Africa
3. 使用 Queue 的例子
- Queue 使用先进先出的队列管理子进程
- 使用 put() 函数将数据推进队列里
- 使用 get() 取出队列里的数据
from multiprocessing import Queue
colors = ['red', 'green', 'blue', 'black']
cnt = 1
# instantiating a queue object
queue = Queue()
print('pushing items to queue:')
for color in colors:
print('item no: ', cnt, ' ', color)
queue.put(color)
cnt += 1
print('\npopping items from queue:')
cnt = 0
while not queue.empty():
print('item no: ', cnt, ' ', queue.get())
cnt += 1