Key Points

Overview: ML on AWS

  • The cloud offers the current GPU generation on demand and bills per second while a job runs; on-prem hardware offers full control and a fixed cost but a fixed generation and a queue.
  • Renting is cheaper for bursty research work; owning is cheaper for GPUs that stay busy.
  • An idle rented GPU costs the same as a busy one, so every job in this lesson is launched with a runtime cap.
  • SageMaker runs your script in a preconfigured framework container on an instance that exists only for the job; Bedrock serves AWS-hosted models per token with nothing to start or stop.

Data Storage: Setting up S3

  • Use S3 for scalable, cost-effective, and flexible storage.

  • EC2 storage is fairly uncommon, but may be suitable for small, temporary datasets; only /home/ec2-user/SageMaker survives a Stop, and nothing survives a Delete.

  • Create the bucket in the same region as your SageMaker notebook, and tag it with the same three keys (Name, Project, Purpose) you will put on every other resource.

  • Track your S3 storage costs, data transfer, and requests to manage expenses.

  • Regularly delete unused data or buckets to avoid ongoing costs.

Notebooks as Controllers

  • Use a minimal SageMaker notebook instance as a controller to manage larger, resource-intensive tasks.
  • The notebook does not do the heavy computation; later episodes use it to launch training, tuning and processing jobs on separate instances.
  • Every resource gets the same three tags; they never propagate, so each new resource needs its own.
  • Stop the notebook instance at the end of class, and delete it when the workshop is over.

Accessing and Managing Data in S3 with SageMaker Notebooks

  • Load data from S3 into memory for efficient storage and processing.
  • Periodically check storage usage and costs to manage S3 budgets.
  • Use SageMaker to upload analysis results and maintain an organized workflow.
  • The lesson repository lives at /home/ec2-user/SageMaker/ML_with_AWS_SageMaker/; every later episode adds its scripts/ folder to sys.path and imports helpers.

(Optional) Pushing Your Notebooks to GitHub from SageMaker

  • Your own work belongs in your own repository: create one on GitHub, git init a folder under /home/ec2-user/SageMaker/, and connect the two with git remote add.
  • Authenticate over HTTPS with a scoped, expiring Personal Access Token entered through getpass; nothing is saved on the instance, and the token is revoked when you are done.
  • Version .py copies of notebooks (Jupytext converts both ways) and ignore *.ipynb, caches and data archives in .gitignore.
  • Pull before you push; the first pull into a freshly initialised folder needs --allow-unrelated-histories.

Training Models in SageMaker: Intro

  • Environment initialization: Setting up a SageMaker session, defining roles, and specifying the S3 bucket are essential for managing data and running jobs in SageMaker.
  • Local vs. managed training: Always test your code locally (on a smaller scale) before scaling things up. This avoids wasting resources on buggy code that doesn’t produce reliable results.
  • Estimator classes: SageMaker provides framework-specific Estimator classes (e.g., XGBoost, PyTorch, SKLearn) to streamline training setups, each suited to different model types and workflows.
  • Your script vs. the built-in algorithm: both run in AWS’s XGBoost container. XGBoost(entry_point=...) runs your script in it and is the pattern every later episode uses; Estimator(image_uri=...) with no script runs AWS’s own training code, which needs the data pre-shaped and takes XGBoost’s own parameters.
  • Training data channels: Using TrainingInput ensures SageMaker manages data efficiently.
  • Cost habit: Estimate cost before launching (rate x minutes + ~3 minutes startup), then read back BillableTimeInSeconds with job_cost_report() after every job and compare.
  • Scaling: Upgrade the single instance first; consider multiple instances only when data no longer fits in memory or a run takes more than ~10 hours.

Training Models in SageMaker: PyTorch Example

  • Efficient data handling: The .npz format is optimized for efficient loading, reducing I/O overhead and enabling batch compatibility for PyTorch’s DataLoader.
  • Startup dominates small jobs: Every SageMaker training job waits a minute or so for an instance (not billed), then spends one to three minutes on container pull and data download (billed) before the script runs; for a job whose script finishes in seconds, that overhead is the whole bill.
  • GPU training: While beneficial for larger models or datasets, GPUs may introduce overhead for smaller tasks; selecting the right instance type is critical for cost-efficiency.
  • Data parallelism vs. model parallelism: Data parallelism splits data across instances and synchronizes model weights, suitable for typical neural network tasks. Model parallelism, which divides model layers, is ideal for very large models that exceed memory capacity.
  • instance_count alone is not distributed training: Without distribution= on the estimator and distributed code in the script, instance_count=2 runs the full script on the full data on each instance and bills twice. Data parallelism is three additions to the script (join the process group, shard the rows, wrap the model in DistributedDataParallel) and one argument on the estimator; train_nn_ddp.py has them, and reaches the single-instance run’s loss on half the rows per instance.
  • SageMaker configuration: By adjusting instance counts and types, SageMaker supports scalable training setups. Starting with CPU training and scaling as needed with GPUs or distributed setups allows for performance optimization.
  • Testing locally first: Before deploying large-scale training in SageMaker, test locally with a smaller setup to ensure code correctness and efficient resource usage.
  • Estimate before you launch: for small models, time a smoke test and scale it by epochs and rows, plus about three minutes of startup; for large ones, 6 FLOPs per parameter per example-token, divided by a quarter of the GPU’s peak. helpers.estimate_training_cost does the second; write the number down and compare it to the bill.

Hyperparameter Tuning in SageMaker: Neural Network Example

  • Tuner = estimator + ranges + objective: A HyperparameterTuner wraps an ordinary estimator; the script only has to print the objective metric in a format the metric_definitions regex can capture.
  • Start with one job: max_jobs=1, max_parallel_jobs=1 first, to catch script and configuration errors before paying for a search.
  • Parallelism buys time, not quality: Cost is per child training job; max_parallel_jobs changes wall-clock time and, with the Bayesian strategy, how much each job can learn from the ones before it. 2-4 parallel jobs is a good default.
  • Tags propagate from the tuner: Tags on the tuner are copied to every child training job, the one case in SageMaker where tags are inherited.
  • Read the bill from the child jobs: SageMaker bills per second of BillableTimeInSeconds x instance count on each training job; container pull and data download are inside that number and dominate short jobs.
  • Transform the test set with the training scaler: Load the saved scaler and pass it to preprocess_data(test_df, scaler=scaler); never refit on test data.

Fine-tuning a Vision-Language Model with Training Jobs

  • Fine-tuning cost is GPU-hours × hourly rate, and GPU-hours scale with model size × tokens × epochs × attempts. The same recipe runs from under a dollar (3B, LoRA, a few hundred pages) to tens of thousands (70B, full fine-tune, days on multi-GPU nodes). Starting from a model that already knows most of the task is the biggest saving available.
  • Training jobs, not GPU notebooks, are the safe way to run it: they end, max_run guarantees it, a unique job_name prevents a second copy launching by accident, and checkpoints synced to S3 mean a killed job costs an epoch rather than the run. Interrupting a notebook cell does not stop a job.
  • Estimate before launching (startup overhead + pages × epochs × seconds per page, at the instance’s rate), then compare to BillableTimeInSeconds.
  • Image resolution (max_pixels) and epochs change cost by multiples; the instance size changes it by 5–45x, which is right when the model does not fit on one GPU and waste when it does.
  • Score the holdout at the end of every epoch: the curve, not a rule of thumb, says how many passes were worth paying for. Measure the gain on the same held-out pages before and after, with the task’s own metric, and remember that a page-level holdout is optimistic when the real test is split by document.

Overview of RAG Workflows on AWS

  • RAG is an inference‑only workflow: no training or fine‑tuning required, so you do not need to own a GPU to run one.
  • Amazon Bedrock is the default for RAG inference: you pay per token, nothing runs between calls, and there is nothing to forget to shut down.
  • Self‑managed GPU compute (Processing Jobs, notebook GPUs, inference endpoints) is justified when you need a model Bedrock doesn’t offer or when very large batch volumes make per‑token pricing uncompetitive — not as a starting point.
  • Among the GPU options, Processing Jobs are the safest because the instance terminates itself; notebook GPUs and endpoints bill by the hour until you stop them.
  • A forgotten GPU instance costs tens of dollars overnight and hundreds to thousands over a month; a full WattBot run on Bedrock costs well under a dollar.
  • Tags never propagate: tag every job at launch, and route Bedrock calls through a tagged application inference profile, or the spend is untraceable.
  • The hands‑on episodes start with Bedrock, then show the same pipeline on Processing Jobs; the notebook-GPU version is optional self-study.

RAG with Bedrock

  • A complete RAG pipeline on AWS needs no GPU: a CPU notebook plus Bedrock-hosted embedding and generation models is enough.
  • Corpus preparation (unpacking the staged PDFs, page extraction, chunking) is cheap CPU work; cache chunks.jsonl on S3 so later episodes and runs can reuse it.
  • Bedrock calls carry no tags; route them through a tagged application inference profile or the spend is anonymous on the bill.
  • converse is the provider-neutral way to call a chat model on Bedrock; its usage block is what you are billed on. DeepSeek V3 is the generator here, and any catalog model is a one-line swap.
  • Bedrock bills per token and leaves nothing running between calls, which is why it is the default route for RAG inference.
  • The model is one tunable component; most accuracy gains come from chunking, retrieval, and post-processing.

RAG with Processing Jobs

  • Bedrock’s catalog is curated and lags releases; when you need a specific open-weight model you rent a GPU and serve it yourself.
  • Of the ways to rent a GPU, a Processing job is the one that cannot be forgotten: it bills while the script runs and ends on its own, with max_runtime_in_seconds as the backstop. Use it for batch work; use Bedrock or an endpoint when you need answers in seconds.
  • Inference needs only the weights: Qwen3-8B in bf16 is ~16 GB and fits one 24 GB GPU.
  • A job’s inputs and outputs are S3 prefixes mounted into the container; the notebook only stages files and reads results, so it stays a cheap CPU instance.
  • The pipeline logic (wattbot_rag.py) ships with the job unchanged from the Bedrock episode; only the two model callables differ.
  • Tags do not propagate from the notebook: every job gets its own tags=.

(Optional) RAG with a Notebook GPU

  • Why self-host at all: Bedrock’s catalog is curated and lags releases; most open-weight models (including Qwen3.8-27B) are not on it. When the model you need is not offered, you rent a GPU and serve it yourself.
  • Inference uses reduced-precision weights: no gradients means the weights are the memory cost. FP8 puts a 27B model on one 48 GB GPU (ml.g6e.xlarge); bf16 would need 54 GB and a multi-GPU instance.
  • The meter never stops on its own: ml.g6e.xlarge is $2.61/h, $167 per forgotten weekend, $1,906 per forgotten month, plus $14/month for the storage volume even when stopped. Stop it when you stand up; delete it when the episode is over.
  • Local-first RAG: For teaching (and small corpora), we avoid an external vector database and instead perform cosine similarity search over in-memory embeddings.
  • Ground-truth units: The answer_unit column is always copied directly from train_QA.csv, never guessed by the LLM.
  • Two-stage LLM use: One call focuses on answering and citing; a second, lighter call produces a short explanation tagged with an evidence type.
  • WattBot conventions: We respect the Kaggle competition format, using is_blank for unanswerable questions and for missing fields.
  • Scalability path: The same logic can later be swapped to FAISS/Chroma and larger models, while preserving the interface used here.

(Optional) The Same Jobs from the Terminal

  • The CLI, the SDK and the console are one API; the CLI shows you nothing you do not ask for, so its first use is to look and to stop, not to launch.
  • Authenticate with aws sso login and short-lived credentials; never store an access key.
  • list-*, describe-* and stop-* for training, processing and tuning jobs and notebook instances cover almost every reason to open a terminal.
  • A create-training-job JSON is what estimator.fit() sent: the same tags, the same MaxRuntimeInSeconds, the same instance choice, with no price printed next to it.
  • Endpoints, EC2 instances and GPU notebooks are not jobs and do not end; the workshop account denies creating them, and your own account should too.

Resource Management and Monitoring

  • Always stop or delete notebook instances when not in use to avoid charges.

  • Stop any training, tuning or Processing job you no longer need.

  • Bedrock application inference profiles are not swept by the instructors’ aws-nuke run; remove them yourself.

  • Regularly clean up unused S3 buckets and objects to save on storage costs.

  • Monitor your expenses through the AWS Billing Dashboard and set up alerts.

  • Use tags (set up earlier in the workshop) to track and monitor costs by resource.

  • Following best practices for AWS resource management can significantly reduce costs and improve efficiency.