Python Multiprocessing

Learning Objectives

By the end of this lesson, learners will be able to:

• Use Python’s multiprocessing library to parallelize a CPU bound problem.

• Set up a pool of workers and delegate tasks to different processes to run concurrently, using the ProcessPoolExecutor class.

IO bound example with Python multithreading

In this simple example we will create an expensive CPU bound recursive function that generates the n-th Fibonacci number several times, and report the amount of time it took to run for benchmarking purposes. Note there are much more efficient ways to implement this function, but this expensive impplimentation is deliberate. First we will do it serially, and then use the multiprocessing library to delegate blocks of webpages to different threads. This problem is CPU bound, as the time it takes for process to complete is dependent on the speed of the CPU.

The serial example

Download complete serial cpu bound example file

The multithreading example

Download complete threaded cpu_bound example file

from jupyterquiz import display_quiz
display_quiz("questions/summary_multithreading.json")