Multiple approaches for parallelizing code (multiprocessing, threading, joblib)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Alexander Wong 59d8f5cc35
sleep 50msec comment fix
2 weeks ago
.vscode trivial map examples 2 weeks ago
util joblib backends, thread/processes pools 2 weeks ago
.flake8 trivial map examples 2 weeks ago
.gitignore Initial commit 2 weeks ago
LICENSE Initial commit 2 weeks ago
README.md joblib backends, thread/processes pools 2 weeks ago
requirements.txt trivial map examples 2 weeks ago
sleep_50msec.py sleep 50msec comment fix 2 weeks ago
times_two.py trivial map examples 2 weeks ago

README.md

python-parallelization-benchmark

Multiple approaches for parallelizing code (multiprocessing, threading, joblib)

Quickstart

python3 -m venv env
source env/bin/activate

Sample Outputs

./sleep_50msec.py 
for loop 5.019243425000241
list comp 5.014174804000504
map 5.012271255998712
multiprocessing.Pool.map 0.9182787979989371
concurrent.futures.ProcessPoolExecutor 0.6891949049986579
multiprocessing.dummy.Pool.map 0.9059246159995382
concurrent.futures.ThreadPoolExecutor 0.16103629900135275
joblib.Parallel (loky) 0.9259599640008673
joblib.Parallel (multiprocessing) 0.7353465940013848
joblib.Parallel (threading) 0.7106083839989878
./times_two.py
for loop 0.03604346900101518
list comp 0.030031919999601087
map 0.017079889999877196
multiprocessing.Pool.map 0.11840330099948915
concurrent.futures.ProcessPoolExecutor 20.37815549700099
multiprocessing.dummy.Pool.map 0.10221909800020512
concurrent.futures.ThreadPoolExecutor 3.1018160650000937
joblib.Parallel (loky) 3.064631451999958
joblib.Parallel (multiprocessing) 0.6765315240008931
joblib.Parallel (threading) 5.437431302998448