Key Points

Overview: ML on AWS

  • The cloud offers the current GPU generation on demand, billed per second while a job runs. Campus and national research computing is often cheaper for sustained work, and free to many researchers.
  • Cost and energy efficiency depend on the hardware generation, how busy it is kept, and the data center’s overhead.
  • S3 stores the datasets and results. A small SageMaker notebook instance is the workspace.
  • SageMaker runs training, tuning, processing and inference jobs on instances that exist only for the job.
  • Bedrock gives API access to hosted models, with no GPU instance to manage.

Data Storage: Setting up S3

  • Use S3 for scalable, cost-effective, and flexible storage.

  • EC2 storage is fairly uncommon, but may be suitable for small, temporary datasets; only /home/ec2-user/SageMaker survives a Stop, and nothing survives a Delete.

  • Create the bucket in the same region as your SageMaker notebook, and tag it with the same three keys (Name, Project, Purpose) you will put on every other resource.

  • Track your S3 storage costs, data transfer, and requests to manage expenses.

  • Regularly delete unused data or buckets to avoid ongoing costs.

Notebooks as Controllers

  • Use a minimal SageMaker notebook instance as a controller to manage larger, resource-intensive tasks.
  • The notebook does not do the heavy computation; later episodes use it to launch training, tuning and processing jobs on separate instances.
  • Every resource gets the same three tags; they never propagate, so each new resource needs its own.
  • Stop the notebook instance at the end of class, and delete it when the workshop is over.

Accessing and Managing Data in S3 with SageMaker Notebooks

  • Read S3 objects into memory with get_object; copy one to the instance’s disk with download_file only when a program needs it as a file.
  • Periodically check storage usage and costs to manage S3 budgets.
  • Write outputs back to your bucket under a prefix, and read the bucket’s size from the notebook to see what accumulates.
  • The lesson repository lives at /home/ec2-user/SageMaker/ML_with_AWS_SageMaker/; every later episode adds its scripts/ folder to sys.path and imports helpers.

(Optional) Pushing Your Notebooks to GitHub from SageMaker

  • Your own work belongs in your own repository: create one on GitHub, git init a folder under /home/ec2-user/SageMaker/, and connect the two with git remote add.
  • Authenticate over HTTPS with a scoped, expiring Personal Access Token entered through getpass; nothing is saved on the instance, and the token is revoked when you are done.
  • Version .py copies of notebooks (Jupytext converts both ways) and ignore *.ipynb, caches and data archives in .gitignore.
  • Pull before you push; the first pull into a freshly initialized folder needs --allow-unrelated-histories.

Training Models in SageMaker: Intro

  • Setup: a SageMaker session, the execution role and the bucket name are what every job needs from the notebook.
  • Local vs. managed training: Always test your code locally (on a smaller scale) before scaling things up. This avoids wasting resources on buggy code that doesn’t produce reliable results.
  • Estimators: XGBoost(entry_point=...) is AWS’s container running your script, the pattern every later episode uses with PyTorch and HuggingFace.
  • Training data channels: TrainingInput names the S3 data a job copies onto its instance before the script runs.
  • Estimate the cost before launching (rate × minutes, startup included), then read back BillableTimeInSeconds with job_cost_report() after every job and compare.
  • Long runs: launch with wait=False and carry on, or stop the notebook instance; the job runs and finishes without it, and job_cost_report(job_name, ...) reads its bill later. Set max_run a little above the estimate, and checkpoint to S3 on anything longer than half an hour.

Training Models in SageMaker: PyTorch Example

  • Startup dominates small jobs: Every SageMaker training job waits a minute or so for an instance (not billed), then spends one to three minutes on container pull and data download (billed) before the script runs; for a job whose script finishes in seconds, that overhead is the whole bill.
  • Estimate before you launch: for small models, time a short run and scale it by epochs and rows, plus about three minutes of startup; for large ones, 6 FLOPs per parameter per token, divided by a quarter of the GPU’s peak. helpers.estimate_training_cost does the second; write the number down and compare it to the bill.
  • GPU or CPU is a measurement: the same script on an ml.g4dn.xlarge was slower and dearer than on an ml.m5.large for this network, because the step is too small to keep a GPU busy; the GPU pays when the model and batch are large.
  • More than one machine is the last move: a faster single device first, then more GPUs in one box, then more instances. instance_count=2 on its own runs the whole job twice and bills twice; data parallelism needs distribution= on the estimator and, for your own script, three additions: join the process group, shard the rows, wrap the model in DDP.
  • Parallelism pays when the step’s computation is large next to the gradient exchange: big model, big batch, fast link. helpers.estimate_parallel_speedup gives the speedup and the cost ratio before you rent the devices; on 569 rows of tabular data the measured result was 0.85x the speed at 2.4x the cost.

Hyperparameter Tuning in SageMaker: Neural Network Example

  • Tuner = estimator + ranges + objective: A HyperparameterTuner wraps an ordinary estimator; the script only has to print the objective metric in a format the metric_definitions regex can capture.
  • Start with one job: max_jobs=1, max_parallel_jobs=1 first, to catch script and configuration errors before paying for a search.
  • Parallelism buys time, not quality: Cost is per child training job; max_parallel_jobs changes wall-clock time and, with the Bayesian strategy, how much each job can learn from the ones before it. 2-4 parallel jobs is a good default.
  • Tags propagate from the tuner: tags on the tuner are copied to every child training job.
  • Estimate the whole search before launching it: helpers.estimate_tuning_cost takes the tuner’s own limits and one child job’s billable minutes; startup is paid once per parallel slot, which is why two at a time is cheaper than ten at once.
  • Read the bill from the child jobs: SageMaker bills per second of BillableTimeInSeconds x instance count on each training job; container pull and data download are inside that number and dominate short jobs.
  • Transform the test set with the training scaler: Load the saved scaler and pass it to preprocess_data(test_df, scaler=scaler); never refit on test data.

Fine-tuning a Vision-Language Model with Training Jobs

  • Fine-tuning cost is GPU-hours × hourly rate, and GPU-hours scale with model size × tokens × epochs × attempts. The same recipe runs from under a dollar (3B, LoRA, a few hundred pages) to tens of thousands (70B, full fine-tune, days on multi-GPU nodes). Starting from a model that already knows most of the task is the biggest saving available.
  • Training jobs, not GPU notebooks, are the safe way to run it: they end, max_run guarantees it, a unique job_name prevents a second copy launching by accident, and checkpoints synced to S3 mean a killed job costs an epoch, not the run. Interrupting a notebook cell does not stop a job.
  • Estimate before launching (startup overhead + pages × epochs × seconds per page, at the instance’s rate), then compare to BillableTimeInSeconds.
  • Image resolution (max_pixels) and epochs change cost by multiples; the instance size changes it by 5–45x, which is right when the model does not fit on one GPU and waste when it does.
  • Score the holdout at the end of every epoch: the curve, not a rule of thumb, says how many passes were worth paying for. Measure the gain on the same held-out pages before and after, with the task’s own metric, and a page-level holdout is optimistic when the real test is split by document.

Overview of RAG Workflows on AWS

  • RAG is inference only. Nothing is trained or fine-tuned, so running one does not need a GPU of your own.
  • Bedrock is the default for inference: you pay per token, nothing runs between calls, and there is nothing to forget to shut down.
  • Self-managed GPU compute (Processing jobs, notebook GPUs, inference endpoints) is for a model Bedrock does not offer, or for batch volumes so large that per-token pricing loses; it is not a starting point.
  • Of the GPU options, Processing jobs are the safest, because the instance ends itself; notebook GPUs and endpoints bill by the hour until you stop them.
  • A forgotten GPU instance costs tens of dollars overnight and hundreds to thousands over a month; a full WattBot run on Bedrock costs under a dollar.
  • Tags never propagate. Tag every job at launch and route Bedrock calls through a tagged application inference profile, or the spend cannot be traced.
  • The hands-on episodes start with Bedrock, then the same pipeline on Processing jobs; the notebook-GPU version is optional self-study.

RAG with Bedrock

  • A complete RAG pipeline on AWS needs no GPU. A CPU notebook plus Bedrock-hosted embedding and generation models is enough.
  • Corpus preparation (unpacking the staged PDFs, page extraction, chunking) is cheap CPU work; cache chunks.jsonl on S3 so later episodes and runs can reuse it.
  • Bedrock calls carry no tags; route them through a tagged application inference profile or the spend is anonymous on the bill.
  • converse is the provider-neutral way to call a chat model on Bedrock; its usage block is what you are billed on. DeepSeek V3 is the generator here, and any catalog model is a one-line swap.
  • Bedrock bills per token and leaves nothing running between calls, which is why it is the default route for RAG inference.
  • The model is one tunable component; most accuracy gains come from chunking, retrieval, and post-processing.

RAG with Processing Jobs

  • Bedrock’s catalog is curated and lags releases; when you need a specific open-weight model you rent a GPU and serve it yourself.
  • Of the ways to rent a GPU, a Processing job is the one that cannot be forgotten. It bills while the script runs and ends on its own, with max_runtime_in_seconds as the backstop. Use it for batch work; use Bedrock or an endpoint when you need answers in seconds.
  • Inference needs only the weights. Qwen3-8B is small enough to run unquantized, ~16 GB in bf16 on one 24 GB GPU, with the rest for the prompts’ attention cache; how many prompts decode together is bounded by that remainder. Larger models are quantized to fit.
  • A job’s inputs and outputs are S3 prefixes mounted into the container; the notebook only stages files and reads results, so it stays a cheap CPU instance.
  • The pipeline logic (wattbot_rag.py) ships with the job unchanged from the Bedrock episode; only the two model callables differ.
  • Tags do not propagate from the notebook. Every job gets its own tags=.

(Optional) RAG with a Notebook GPU

  • Why self-host at all: Bedrock’s catalog is curated and lags releases; most open-weight models (including Qwen3.8-27B) are not on it. When the model you need is not offered, you rent a GPU and serve it yourself.
  • Inference uses reduced-precision weights: no gradients means the weights are the memory cost. FP8 puts a 27B model on one 48 GB GPU (ml.g6e.xlarge); bf16 would need 54 GB, an 80 GB GPU or a multi-GPU instance.
  • The meter never stops on its own: ml.g6e.xlarge is $2.61/h, $167 per forgotten weekend, $1,906 per forgotten month, plus $14/month for the storage volume even when stopped. Stop it when you stand up; delete it when the episode is over.
  • Retrieval is a matrix multiply: cosine similarity over the in-memory embedding matrix is enough for a corpus this size; a vector database is the same step for a corpus of millions.
  • Units come from the data: answer_unit is copied from train_QA.csv, never guessed by the model.
  • Two calls per question: one answers and cites; a second, shorter one writes the explanation and names the evidence type.
  • The competition’s format: is_blank marks unanswerable questions and missing fields, as the scorer expects.

(Optional) The Same Jobs from the Terminal

  • The CLI, the SDK and the console are one API. Every call the notebooks made has a CLI command with the same name and fields, and describe-* on a job you ran prints the request the SDK built.
  • The CLI shows you nothing you do not ask for, so its first use is to look and to stop, not to launch.
  • Authenticate with aws sso login and short-lived credentials. Never store an access key.
  • describe-training-job is job_cost_report(). Cost is billable seconds × instance count × the hourly rate, and you do the multiplication.
  • A create-training-job JSON is what estimator.fit() sent, the same script, hyperparameters, channel, cap and tags, with no price printed next to it.
  • Endpoints, EC2 instances and GPU notebooks are not jobs and do not end. The workshop account’s notebook role denies creating them, and your own account should too.

Resource Management and Monitoring

  • Always stop or delete notebook instances when not in use to avoid charges.

  • Stop any training, tuning or Processing job you no longer need.

  • Bedrock application inference profiles are not swept by the instructors’ aws-nuke run; remove them yourself.

  • The refusals you met in this workshop (untagged jobs, wrong instance types, endpoints) are IAM deny rules an administrator attached to the role, not SageMaker defaults; on an account you administer, the instructor pages have the policy to copy.

  • Regularly clean up unused S3 buckets and objects to save on storage costs.

  • Monitor your expenses through the AWS Billing Dashboard and set up alerts.

  • Use tags (set up earlier in the workshop) to track and monitor costs by resource.