Key Points

Introduction

  • Runtime can be measured using tools such as time and date.
  • Repeated process creation can dominate runtime.
  • HPC resource usage is commonly measured in core-hours and GPU-hours.
  • Computational workloads may be compute-bound, memory-bound, or I/O bound.
  • Efficient jobs reduce both resource consumption and energy use.
  • Implementation choices can affect both runtime and numerical accuracy.

Resource Requirements

  • Your cluster might seem to have an enormous amout of computing resources, but these resources are a shared good. You should only use as much as you need.
  • Resource requests are a promise to the scheduler to not use more than a specific amount of resources. If you break your promise to the scheduler and try to use more resources, terrible things will happen.
    • Overstepping memory or time allocations will result in your job being terminated.
    • Oversubscribing CPU cores will at best do nothing and at worst diminish performance.
  • Finding the minimal resource requirements takes a bit of trial and error. Slurm collects a lot of useful metrics to aid you in this.

Scheduler Tools

  • Schedulers provide tools for a high level view on our jobs, e.g. sacct and seff
  • Important basic performance metrics we can gather this way are:
    • CPU Utilization, often as fraction of time where CPU was active/elapsed time of the job
    • Memory utilization, often measured as Resident Set Size (RSS) and number of Pages
  • sacct can also provide metrics about disk I/O and energy consumption
  • Metrics through sacct are accumulated for the whole job runtime and may be too broad for more specific insight

Scaling Study

  • Jobs behave differently with increasing parallel resources and fixed or scaling workloads
  • Scaling studies can help to quantitatively grasp this changing behavior
  • Good working points are defined by configurations where more cores still provide sufficient speedup or improve quality through increasing workloads
  • Amdahl’s law: speedup is limited by the serial fraction of a program
  • Gustafson’s law: more resources for parallel processing still help, if larger workloads can meaningfully contribute to project results

Performance Overview

  • Performance tools measure data as regular samples or by tracing every event
  • The data is either processed and visualized in a timeline or aggregated in a final profile
  • Job performance relates closely to contention points in physical hardware
    • CPU utilization (front-end, ALU, back-end), multithreading, vectorization
    • Memory utilization (capacity, bandwidth, latency)
    • Disk I/O
    • Network I/O

Pinning

  • A Slurm-CPU is a schedulable execution unit visible to the operating system.
  • A process usually equates a Slurm task and involves \(\ge 1\) (Slurm-)CPUs.
  • OpenMP threads are software threads created by an application.
    • It is common to run one OpenMP-thread per CPU.
  • Pinning, or binding, is to keep computation and data close together in order to improve performance through minimized latencies.
    • The CPU- and memory-binding options of srun control resource allocation and placement of Slurm jobs.
    • srun --cpu-bind ... controls where Slurm runs tasks, i.e., where computation runs.
    • srun --mem-bind ... controls where Slurm allocates memory, i.e., where data is allocated.
  • The benefit of CPU and memory placement strongly depends on the application, cluster configuration and hardware.
  • On NUMA systems, CPU- and memory often go together to keep computation and data close to one another.
  • Hints provide guidance to Slurm about the expected characteristics of an application.
    • srun --hint=compute_bound ... may be beneficial for applications that require many CPU resources.
    • srun --hint=memory_bound ... may be beneficial for applications that are limited by memory bandwidth.
  • Memory-intensive applications are often more sensitive to NUMA locality than compute-intensive applications. The stream program is such a case.
  • numactl provides fine-grained control over CPU and memory placement.

Performance of Accelerators

  • Tools to measure GPU/FPGA performance of a job
  • Common symptoms of GPU/FPGA problems

Next Steps

  • There are many profilers, some are language-specific, others are vendor-related, …
  • Simple profile with exclusive resources
  • Repeated measurements for reliability