Summary and Schedule

This is a new lesson built with The Carpentries Workbench.

	Setup Instructions	Download files required for the lesson
Duration: 00h 00m	1. Running commands with Maestro	“How do I run a simple command with Maestro?”
Duration: 01h 00m	2. Running Maestro on the cluster
Duration: 01h 50m	3. MPI applications and Maestro	“How do I run an MPI application via Maestro on the cluster?”
Duration: 02h 40m	4. Placeholders	“How do I make a generic rule?”
Duration: 03h 50m	5. Chaining rules	How do I combine steps into a workflow? How do I make a step that uses the outputs from a previous step?
Duration: 05h 00m	6. Multiple parameters	“How do I specify multiple parameters?” “How do multiple parameters interact?”
Duration: 06h 20m	Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

Tutorial exercises and examples refer to two systems called Ruby and Pascal, which offer identical environments.

If you are working on a different cluster, you will need to install the prerequisites yourself.

Prerequisites

---

• A cluster running Slurm
• A python3 environment including
  • common libraries:
    • sys
    • json
    • matplotlib
    • mpi4py
    • numpy
  • Maestro and related scripts:
    • amdahl
    • maestrowf
    • yamllint
  • the script plot_terse_amdahl_results.py

Setting up your environment

How you set up your environment with the prerequisites will depend on where you are running this tutorial.

If on your own cluster

Some Maestro commands will work locally but others won’t make sense unless you’re connected to a cluster with Slurm installed. Wherever you’re working, you’ll need to install the prerequisite Python packages listed above and download the python plotting script.

The recommended way to install these packages is inside a Python virtual environment. You create one using the venv module and specifying the directory in which your environment will be contained:

BASH

mkdir ~/venvs
python3 -m venv ~/venvs/maestro

This should create the folder at ~/venvs/maestro, containing

BASH

ls ~/venvs/maestro

OUTPUT

bin/  include/  lib/  lib64  pyvenv.cfg

Now you can activate the environment. If your shell is bash,

BASH

source ~/venvs/maestro/bin/activate

Otherwise, use the activate command appropriate for your shell. For example,

BASH

echo $SHELL

OUTPUT

/bin/tcsh

BASH

source ~/venvs/maestro/bin/activate.csh

Your terminal prompt should now have a (maestro) prefix, indicating that the environment was successfully activated. This environment includes pip, a Python package manager. Use it to install the required libraries:

BASH

pip install amdahl maestrowf yamllint

Specifying these three custom libraries should be sufficient: their dependencies will be pulled in by pip automagically!

Finally, copy the plot_terse_amdahl_results.py script into your venv’s bin directory and make it executable:

BASH

cd ~/venvs/maestro/bin
wget https://github.com/xorJane/maestro-workflow-lesson/raw/main/episodes/files/plot_terse_amdahl_results.py
chmod +x plot_terse_amdahl_results.py

If on Ruby/Pascal

If you are working on Ruby, Maestro is installed to a Python virtual environment with binaries in /usr/global/docs/training/janeh/maestro_venv/bin. Instructions for how to use these binaries are contained in the lesson.

You will need the python script at /usr/global/docs/training/janeh/maestro_venv/plot_terse_amdahl_results.py or directly from GitHub.

If you are on LC and trying to set up a virtual environment on your own, follow the instructions for “If on your own cluster”, but add the flag --system-site-packages when creating a virtual environment:

BASH

mkdir ~/venvs
python3 -m venv --system-site-packages ~/venvs/maestro

How to connect over SSH

---

Details

The application you use to connect to your cluster will depend on the type of local machine you’re working on.

Windows

We recommend using Git BASH via git for windows.

MacOS

Use Terminal.app or similar.

Linux

Use Terminal or similar.