Summary and Schedule

This is a new lesson built with The Carpentries Workbench.

	Setup Instructions	Download files required for the lesson
Duration: 00h 00m	1. Introduction	What problems are we solving, and what are we not discussing? Why do we use Python? What is parallel programming? Why can writing a parallel program be challenging?
Duration: 00h 25m	2. Benchmarking	How do we know whether our program ran faster in parallel? How do we appraise efficiency? :::
Duration: 01h 25m	3. Computing \(\pi\)	How do I parallelize a Python application? What is data parallelism? What is task parallelism? :::
Duration: 02h 55m	4. Threads And Processes	What is the Global Interpreter Lock (GIL)? How do I use multiple threads in Python? :::
Duration: 04h 25m	5. Delayed Evaluation	What abstractions does Dask offer? How can I paralellize existing Python code? :::
Duration: 04h 37m	6. Map and Reduce	Which abstractions does Dask offer? Which programming patterns exist in the parallel universe? :::
Duration: 06h 07m	7. Data Science with Polars	Dask’s dataframes interface looks nice, but can it go any faster? :::
Duration: 07h 07m	8. Exercise with Fractals	Can we tackle a real problem now? :::
Duration: 08h 07m	9. Asyncio	What is Asyncio? When is Asyncio useful? :::
Duration: 08h 47m	10. Calling External C and C++ Libraries from Python	Which options are available to call from Python C and C++ libraries? How does this work together with Numpy arrays? How do I use this in multiple threads while lifting the GIL? :::
Duration: 10h 17m	Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

This is an intermediate level Python course. We expect familiarity with the command-line, and that you are comfortable working with a coding text editor (like for instance, VS Code).

Data Sets

The course uses the following data sets:

• New York taxi data (description).

The instructions below include downloading the data.

Software and Data Setup

---

Clone the repository at esciencecenter-digital-skills/parallel-python-workshop.

Follow software setup instructions there to set up the environment, download the data, and run the unit tests to see if your setup is working.

Discussion

Details

We provide two choices for working environment: uv or conda. If you are on Windows the preferred method is conda. On Linux or MacOS you should be fine with either.

UV

UV is our recommended tool to manage the Python environment. Please follow the UV install instructions if you haven’t already. Then, running from the directory where you cloned this repository, run the following command:

BASH

uv sync

Then, download the New York taxi dataset:

uv run ny-taxi/download.py

Finally, run pytest and see if it completes all tests without errors.

uv run pytest

Conda

From within the cloned repository root directory, run the following commands to create a conda environment from the environment.yml file and activate it

BASH

$ conda env create -f environment.yml
$ conda activate parallel-python

Now, download the New York taxi data set:

BASH

$ python ny-taxi/download.py

Finally, run pytest and see if it completes all tests without errors.

BASH

$ pytest