Instructor Notes

This is a placeholder file. Please add content here.

Introduction

Intention: Step into the narrative

Set up narrative:

  • Important upcoming conference presentation
  • Time is ticking, the deadline is approaching way too fast
  • The talk is almost done, but, critically, we’re missing a picture for the title slide
  • It should contain three snowmen, and we’ve exhausted our credits for all generative AI models in previous chats with colleagues
  • => Ray tracing a scene to the rescue!
  • Issue: we need to try many different iterations of the scene to find the exact right picture. How can we maximise the number of raytraced snowman images before our conference deadline?
  • Ray tracing is expensive, but luckily we have access to an HPC system

What we’re doing here:

  • Run workflow example for the first time
  • Simple time measurement to get started
  • Introduce different perspectives on efficiency
  • Core-h and correlation to cost in energy/money
  • Either set up the first Slurm job here or in the next episode

Instructor Note

Ask learners: “Why should you care about efficiency?”

After the answers, address the learners with efficient jobs, they can attain/achieve: - shorter turnaround time - lower resource consumption - lower allocation usage - more scientific iterations - lower energy consumption


Instructor Note

Ask learners to run the script with time and estimate what it might be doing before revealing the source code of sum.bash.

Guide learners toward identifying where time is spent: the mathematical calculations themselves or the repeated creation of external processes.


Instructor Note

Ask learners: - How do HPC centers measure resource usage? - How do users know how much of their allocation they have consumed?

Use the upcoming challenge to introduce common HPC accounting concepts such as core-hours, allocations, queues, and resource limits.


Instructor Note

Encourage learners to estimate the ratio between user and real before discussing the meaning of the -np 4 option. Do not explain the result immediately.

The explanation of accumulated CPU time across multiple MPI processes will be introduced in the following episodes.


Slurm Reservation and specific Hardware?

You may need to reserve a set of resources for the course, such that enough resources for the following exercises are available. This is especially important for --exclusive access.

In that case, show how to use --reservation=reservationname to submit jobs.

It may be a good idea to point out the particular hardware of your cluster / partition to emphasize how many cores are available on a single node and when the scaling study goes beyond a single node.


Resource Requirements

Instructor Note

For the next section, the exact memory requirements depend on the cluster configuration (e.g., the MPI backends used). You might have to adapt these numbers for your local cluster to see the out-of-memory behavior.


Todo: Can we demonstrate the callout below?

How can too tight memory limits in Slurm + cgroupsv2 cause cache thrashing?


Instructor Note

At this point you might want to point out to your audience that for certain applications it can be disastrous for performance to set the memory constraint too tightly. The reason is that the memory limit enforced by Slurm does not only affect the resident set size of all the processes in the job allocation, but also the memory used for caching (e.g., file pages). If the allocation runs out of memory for the cache, it will have to evict memory pages to disk, which can cause I/O operations and new memory allocations to block for longer than usual. If the application makes heavy use of this cache (e.g., repeated read and/or write operations on the same file) and the memory pressure in the allocation is high, you can even run into a cache thrashing situation, where the job spends the majority of its time swapping data in and out of system memory and thus slows down to a crawl.


Instructor Note

This error message was generated with OpenMPI. Other MPI implementations might produce different messages.


Instructor Note

At this point you can present some scheduling strategies specific to your cluster. For the sake of time, you have likely reserved some resources for the course participants such that their jobs start instantly. Now would be a good time to show them the harsh reality of HPC scheduling on a contested partition and demonstrate that a major part of using an HPC cluster is waiting for your jobs to start.


Scheduler Tools

Intention: Introduce more basic performance metrics

Narrative:

  • Okay, so first couple of jobs ran, but were they “quick enough”?
  • How many renders could I generate per minute/hour/day according to the current utilization
  • Our cluster uses certain hardware, maybe we didn’t use it as much as we could have?
  • But I couldn’t see all metrics (may be cluster dependent) (Energy, Disk I/O, Network I/O?)

What we’re doing here:

  • What seff and sacct have to offer
  • Introduce simple relation to hardware, what does RSS, CPU, Disk read/write and their utilization mean?
  • Point out what’s missing from a complete picture

Note:

  • seff is an optional SLURM tool. It does not come standard with every SLURM installation. Therefore, make sure beforehand that this tool is available for the students.

Todo: give clear recommendation of what to aim for?

Maybe 80% of job time?


Todo: potential issue?

Running this on our cluster and adding a module load command resulted in 600MB of memory required. My guess is, this is due to cgroups_v2 and Page caches being counted towards the job as well, so loading the modules might spike the resource requirements as well?

Maybe we should play it safe and use a larger value in the following exercise. But we also want to teach not overdoing it, so it’d be good if we can find a useful but generic compromise here


Instructor Note

Note that the information sacct can provide depends on the information that SLURM stores on a given machine. By default this includes Billing, CPU, Energy, Memory, Node, FS/Disk, Pages and VMem. Additional information is available only when SLURM is configured to collect it. These additional trackable resources are listed in AccountingStorageTRES. For I/O fs/lustre is commonly useful, and for the interconnect communication ic/ofed is required. The setting AccountingStorageTRES is found in slurm.conf. Unfortunately there doesn’t seem to be a way to get sacct to print the optional trackable resources.


Todo: extend the following list and examples to include CPU

To reconstruct the CPU utilization reported by seff: - TotalCPU/CPUTime should give the percentage - Could also mention UserCPU and SystemCPU and discuss the difference? Both result in TotalCPU

Maybe remove AveCPUFreq instead, or do we try to teach something specific about it?

Don’t forget to change the example output of all saccts in the following examples/challenges!


Give more insight in the collected sacct metrics

  • AllocCPUS: number of CPU cores we requested for the job
  • MaxRSS = AveRSS: low fluctuation in memory, data is held throughout the whole job
  • MaxPages & AvePages: number of pages loaded into memory
  • MaxDiskRead: Data read from disk by the application, but also to start the application.

Scaling Study

Intention: Introduce/Recollect concept of Speedup and do a simple scaling study

Narrative:

  • We panic, maybe we need more resources to meet the deadline with our title picture!
  • Requesting resources with bigger systems requires a project proposal with an estimate of the resource demand

Todo: Unify MPI and modules across examples

We should address the various ways to start MPI programs at first use and choose a single way. Preparing the software is also cluster-specific (module load ...).

In the mid-/long-term this may call for a site-specific implementation of the course, e.g. via the hpc-intro workbench.


Todo: show, don’t tell

info dump below in this section

Maybe be more specific about which overheads and how we can see them?


When should we stop adding CPU cores?

The learners should realize there is a point of diminishing returns. Where exactly to draw the line is debatable, but in the previous example \(N=32\) seems to be a sweet spot with the best Speedup and an efficiency \(>50\)%.

There are many reasons why a large number of parallel processes may result in worse speedup. Among others, it may be due to I/O overhead, synchronization between threads, or a limited divisibility of the problem domain, where additional processors could not solve additional work.


Performance Overview

Intention: Introduce third party tools for performance reports

Narrative:

  • Scaling study, scheduler tools, project proposal is written and handed in
  • Maybe I can squeeze out more from my current system by trying to understand better how it behaves
  • Another colleague told us about performance measurement tools
  • We are learning more about our application
  • Aha, there IS room to optimize! Compile with vectorization

What we’re doing here:

  • Get a complete picture
  • Introduce additional metrics / definitions, and popular representations of data, e.g. Roofline
  • Relate to hardware on the same level of detail

TODO: Discuss requirements in more detail?

Dependent on the cluster config, some of these may have to be addressed?

cap_perfmon,cap_sys_ptrace,cap_syslog=ep
kernel.perf_event_paranoid

Prepare a reservation!

We’ll have to run a couple of 20 minutes long --exclusive jobs here, so make sure enough resources are available for the exercise!


Know your tool!

We provide alternatives for this episode. Make sure to select the right one in your config.yaml.

ClusterCockpit is a job monitoring systems that can be configured to capture many performance metrics. It is easy to use, but has to be deployed by the cluster administration team.

Be aware of site-specific setups, e.g. limiting access to performance counters, offering non-standard Slurm options during sbatch submission, and how licenses are handled. Not all metrics discussed in this episode may be configured on every cluster — the set of metrics collected and displayed is chosen by the local administration team, so some panels or plots covered here might simply be unavailable, or show different metrics, on your system.


Demonstrating the Job 1 vs. Job 2 comparison

Open Job 1 and Job 2 side by side in two browser tabs, and navigate both to the Roofline plot panel, to demonstrate the comparison live rather than just describing it.

Hovering the cursor over a dot draws guide lines out to the x- and y-axes, letting you read off its operational intensity and achieved performance directly from the axes rather than judging position by eye alone. Hover over a dot in Job 1’s plot, note where its guide lines land, then do the same for a dot at a similar point in the run in Job 2. Comparing the two readings this way gives learners a much clearer feel for exactly how much higher and how much further right Job 2’s dots sit compared to Job 1’s, rather than relying on “it looks a bit higher.”


Intention: Uncover one or two issues in the application

What we’re doing here:

  • Where does our system choke?
  • What’s a bottleneck?
  • How can we identify a bottleneck?
  • “Online” and “after the fact” workflows of performance measurements (trace, accumulated results, attached to the process (live), or after it ran)
  • Point to additional resources of common performance/bottleneck issues, e.g. on hpc-wiki

Maybe something like this already occurred before in 4. Scaling Study, or 5. Performance Overview

Summary could be: - General advice on the workflow - Performance reports may provide an automated summary with recommendations - Performance metrics can be categorized by the underlying hardware, e.g. CPU, memory, I/O, accelerators. - Bottlenecks can appear by metrics being saturated at the physical limits of the hardware or indirectly by other metrics being far from what the physical limits are. - Interpreting bottlenecks is closely related to what the application is supposed to do. - Relative measurements (baseline vs. change) - system is quiescent, fixed CPU freq + affinity, warmups, … - Reproducibility -> link to git course? - Scanning results for smoking guns - Any best practices etc.


Pinning

Intention: Go deeper in performance and hardware relationship

Narrative:

  • We have already seen how to observe resource utilization and identify inefficient use of hardware resources.
  • One way to improve resource utilization is to control where processes and threads are placed.
  • How does the mapping of work to CPUs and memory regions affect performance?

What we’re doing here:

  • Introduce pinning and slurm affinity options
  • Relate to hardware effects
  • A few practical tricks to diagnose and understand poor process placement.

Performance of Accelerators

Intention: Jump onto accelerator with the example application

Narrative:

  • The deadline is creeping up, only few ways to go!
  • Hey, we have a GPU partition! Maybe this will help us speed up the process!

What we’re doing here:

  • What changes?
  • New metrics
  • Transfer to/from accelerator
  • Different options/requirements to scheduler & performance measurement tools

ToDo

Don’t mention FPGAs too much, maybe just a node what accelerators could be, besides GPU. Goal is to keep it simple and accessible, focus on what’s common in most HPC systems these days


ToDo

Explain how to decide where to run something. CPU vs. small GPU vs. high-end GPUs. Touches on transfer overhead etc.


Next Steps

Intention: Provide a roadmap learners could follow

Most important: enable users to translate from example workload to their own code! Guide on how to translate learning goals and key points to their situation. Additionally, provide some info on where and how to dig deeper, if there is interest (application profiling, etc.)

All ideas in this episode may need to be reworked, since they were made with the outlook in mind, not so much to help learners transfer insight

Narrative:

  • Start with picture of beautiful title slide of the talk with the snowman picture
  • Next time we want to tackle the issue way in advance
  • Approach our raytracing application more systematically, such that we can get the title slide done much quicker
  • What could we do to dive deeper in optimizing the raytracer?
  • Where can we go from here?

What we’re doing here:

  • Learning important programming concepts (parallel programming on many levels)
  • Deeper application profiling & tools to use

Instructor Note

Ask learners: - Can a calculation be fast but scientifically useless? - Can a calculation be extremely accurate but unnecessarily expensive?

Guide the discussion toward the trade-off between numerical accuracy and computational cost.

Different scientific applications require different levels of numerical precision. Choosing more precision than necessary may increase runtime, memory usage, and energy consumption without improving scientific results.