Parallel Computing

Parallel Computing#

Course Description#

This course will provide an introduction to parallel programming. Participants will gain practical experience in writing parallel software, understanding how to decompose problems for efficient execution across multiple processes and threads.

Course Objectives#

On completion of this series of workshops, participants will:

Be able to explain what is meant by distributed and shared-memory parallelism.
Know how to write software that can run across multiple processes using MPI.
Be able to write code that utilizes multithreading for parallel execution.
Be able to identify how a problem can be divided and parallelised effectively.
Gain hands-on experience writing and optimizing parallel code.

Pre-requisite Knowledge#

This course is for participants who already have some programming experience with Python. If you are not familiar with Python, our Introduction to Python course, is available here.

Installation Instructions#

Installing MPI#

The Message Passing Interface (MPI) is a standard for passing messages between multiple networked processes running a parallel program. As MPI is a standard, rather than a piece of software, there is not a single software package that you need to install. There are a few different ‘flavours’ of MPI, and the majority of HPC systems will have their own versions which are tuned for their specific systems. It is highly recommended that you use a system’s built-in MPI libraries if they are available, but if not let’s go through the process of installing an MPI library.

Note

Requirements
For this workshop, you will need a multi-core machine which can run a Unix-based terminal (i.e. Linux/WSL or Mac).

MacOS installation#

MPI can be easily installed with Homebrew. Check your machine has homebrew installed with

$ which brew

If this returns the location of the brew executable, then you can proceed with:

$ brew install open-mpi

Linux/WSL installation#

An easy way to install MPI on Linux or WSL platforms is using the Spack package manager (this can also be done for MacOS but requires some additional steps).

Using Spack package manager (multi-platform)#

In our case, we can install OpenMPI, which is a free and open source MPI implementation. OpenMPI can be installed in a number of different ways, but the recommended way is to use the Spack HPC package manager, which is in a class of its own in the way it handles different MPI implementations.

Spack is really simple to install, all you need to need to do is clone the Spack repository:

$ git clone --depth=100 --branch=releases/v0.21 https://github.com/spack/spack.git ~/spack

and source the included setup script:

$ source ~/spack/share/spack/setup-env.sh

Every time you want to use Spack you will need to source this script, so it may be easier to add this to your shell login script, (i.e. ~/.bashrc, ~/.zshrc, etc.). We need to let Spack find any compilers in our system, which we can do with:

$ spack compiler find

Installing MPI with spack#

Note

This method requires you to have a working set of compilers for C and Fortran. If you don’t have these on your system the simplest way to get them is to install the GNU compiler collection (GCC).

We can use Spack to install an MPI library, which will default to installing OpenMPI. If we run

$ spack spec mpi

we can see what Spack will install, and we can use

$ spack install mpi

to execute the installation. Once this is done, we can load the new mpi module with

$ spack load mpi

and check the installation with

$ which mpirun

This command tells us where the mpirun command has been installed, which is the primary way that we can launch an executable across multiple process with MPI. With the installation complete we are ready to run some programs with MPI.

Installing Python#

Python is required for the first section of the course. If you do now have a working Python installation on your machine (you can check this with which python), you can follow the installation instructions from the CFRR Intro to Python course here.

MPI for Python (MPI4Py)#

The first part of this workshop is focussed on distributed memory parallelism with MPI, making use of the Python programming language. There are many different interfaces to MPI for many different languages, so we’ve chosen Python for the benefits it provides to write examples in an easy-to-understand format. Whilst the specific syntax of the commands learned in this part of the course wont be applicable across different languages, the overall code structures and concepts are highly transferable, so once you have a solid grasp of the fundamentals of MPI you should be able to take those concepts to any language with an MPI interface and write parallel code!

The python package that we will be using in this course to implement MPI command is the MPI4Py package, which can be installed via pip as follows:

pip install mpi4py

Self Study Material Link#

The self-study material for this course is available here.

Acknowledgements#

This course was adapted from the Software Carpentries Programming with Python. It has been developed by the University of Exeter Research Software Engineering Group and a team of generous volunteers who are enthusiastic about sharing their skills with the wider research community.

Its provision is dependent on the time of these volunteers. If you have benefited in any way from this course and want to support its long term sustainability then please take the time to complete our feedback survey, recommend it to your colleagues, and enthuse about it to your senior leadership team!

Developers#

The contributors to this course include:

Ed Hone

Course Delivery Content#

There is currently no additional content that is used outside of the self-study notes to deliver this course.

License Info#

Instructional Material

The instructional material in this course is copyright © 2024 University of Exeter and is made available under the Creative Commons Attribution 4.0 International licence (https://creativecommons.org/licenses/by/4.0/). Instructional material consists of material that is contained within the “individual_modules/parallel_computing” directory, and images folders in this directory, with the exception of code snippets and example programs found in files within these folders. Such code snippets and example programs are considered software for the purposes of this licence.

Software

Except where otherwise noted, software provided in this repository is made available under the MIT licence (https://opensource.org/licenses/MIT).

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The software in this repository is adapted from software that is covered by the following copyright and permission notice:

Copyright © 2024 Software Carpentry

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.