Hyperparameter Tuning in SageMaker: Neural Network Example

Last updated on 2026-11-02 | Edit this page

Overview

Questions

  • How can we efficiently manage hyperparameter tuning in SageMaker?
  • How can we parallelize tuning jobs to optimize time without increasing costs?

Objectives

  • Set up and run a hyperparameter tuning job in SageMaker.
  • Define ContinuousParameter and CategoricalParameter for targeted tuning.
  • Log and capture objective metrics for evaluating tuning success.
  • Optimize tuning setup to balance cost and efficiency, including parallelization.
  • Estimate a tuning job’s cost from max_jobs, max_parallel_jobs and one child job’s minutes with helpers.estimate_tuning_cost, and read back what it cost from its child training jobs.

Open the notebook from: /ML_with_AWS_SageMaker/notebooks/09_Hyperparameter-tuning.ipynb. Select the conda_pytorch kernel.

Hyperparameter tuning in SageMaker

The PyTorch episode trained one neural network with a learning rate and an epoch count chosen by hand. Those settings change the result, and choosing them by hand means launching a series of experiments and keeping track of their scores yourself. SageMaker’s hyperparameter tuner does that for us. We say which settings may vary, how a good result is measured, and how many training jobs we are willing to pay for.

Each trial is an ordinary SageMaker training job, launched from the controller notebook on its own instance. The tuner changes the hyperparameters between trials and compares the validation metric they report. That makes the search easier to run, but it does not make it free. Ten trials are ten training jobs on the bill, and what keeps a tuning job cheap is the choice of ranges and the number of models trained. We begin with a single trial to check the configuration before paying for a larger search, the same test-first pattern as the training episodes.

The experiment

We reuse the Titanic network and training data from the PyTorch episode and tune two settings, epochs and learning_rate. One is picked from a short list of values, the other sampled from a continuous range. Two settings are enough to see the mechanics without a new dataset or model. In your own work you may tune more.

There are three pieces to connect. An estimator says how each model is trained, a search space says what may change, and a validation metric tells SageMaker which trials did better. The tuner then bounds the search with the total number of trials, max_jobs, and how many run at once, max_parallel_jobs. Afterwards we read the results, add up the billed compute time, and evaluate the chosen model on the test data.

Set up the project folder

Start in the Titanic project folder, as in the previous episodes:

PYTHON 

# The Titanic project folder from the S3 episode, beside the lesson clone: downloads and outputs land here, not in the repository
!mkdir -p /home/ec2-user/SageMaker/titanic
%cd /home/ec2-user/SageMaker/titanic
import sys
sys.path.insert(0, "/home/ec2-user/SageMaker/ML_with_AWS_SageMaker/scripts")
import helpers

Define the training job

We begin with a PyTorch estimator like the one in the previous episode. It names the training script, the instance, the input paths and the metric to record. When it is handed to the tuner, SageMaker uses it as the template for every trial and substitutes new values for the hyperparameters being searched. The training script does not change.

We’ll use our notebook instance name again to label the training jobs launched in this episode

PYTHON 

import boto3

notebook_instance_name = 'SinkOrSwim-DoeJohn' # adjust to your notebook name. we'll use this variable to name the training jobs launched by the tuner.

# the following code just verifies you have the right name for your notebook instance
region = boto3.Session().region_name  # inherited from the notebook instance; do not hard-code a region
sagemaker_client = boto3.client('sagemaker', region_name=region) # Initialize SageMaker client
response = sagemaker_client.describe_notebook_instance(NotebookInstanceName=notebook_instance_name) # Describe the notebook instance

# Display the status and instance type
print(f"Notebook Instance '{notebook_instance_name}' status: {response['NotebookInstanceStatus']}")
local_instance = response['InstanceType']
print(f"Instance Type: {local_instance}")

Cost tracking: As in every training episode, we explicitly tag each training job at launch time. This ensures that compute usage is traceable to a project, a purpose, and a human-readable name, even after the job has completed. Tags on the HyperparameterTuner are copied to every child training job it launches, so we only need to set them once, on the tuner (we also set them on the estimator so a plain .fit() on it is tagged too).

PYTHON 

name = "DoeJohn" # replace with your name
project = "SinkOrSwim"   # your challenge: WattBot2026, BadgerScribe, BadgerCode, BrainDecoding, PoisonedWell or SnapshotWI; SinkOrSwim is the workshop itself
purpose = "Tuning"        # one of: Storage, Notebook, Training, Tuning, Inference

job_tags = [
    {"Key": "Name", "Value": name},
    {"Key": "Project", "Value": project},
    {"Key": "Purpose", "Value": purpose},
]
helpers.check_launch(job_tags)   # the account's rules, checked here first: AWS's own refusal does not say which rule
print(f"Name: {name}, Project: {project}, Purpose: {purpose}")

Next, we’ll configure the baseline estimator that we plan to do hyperparameter search on. Every price in this episode is from the Instances for ML page, us-east-1 on-demand; the tuning job’s cost is that rate times the sum of the child jobs’ billable minutes.

PYTHON 

import sagemaker
# range types for the search: ContinuousParameter (float range), IntegerParameter (whole-number range), CategoricalParameter (explicit list)
from sagemaker.tuner import HyperparameterTuner, IntegerParameter, ContinuousParameter, CategoricalParameter
from sagemaker.pytorch import PyTorch
from sagemaker.inputs import TrainingInput
from sagemaker import get_execution_role

# Initialize role, bucket, and SageMaker session variables
role = get_execution_role()
bucket_name = 'sinkorswim-doejohn'  # replace with your S3 bucket name
session = sagemaker.Session()
s3 = boto3.client('s3')
print("Role:", role)
print("Region:", session.boto_region_name)

# Define instance type/count we'll use for training
instance_type="ml.m5.large"
instance_count=1 # always start with 1; tuning parallelism comes from max_parallel_jobs, not instance_count
price_per_hour = 0.115 # ml.m5.large, us-east-1 on-demand (2026-10-16), from the Instances for ML page; check it for your region

# The runtime cap, in seconds: SageMaker stops the job and releases the instance when it is reached. Set it a little above the estimate;
# the estimate here is three or four minutes, so 20 minutes leaves room for a slow container pull without letting a hung job bill for hours.
max_run = 20*60 # 20 minutes

# Define the PyTorch estimator with entry script and environment details
pytorch_estimator = PyTorch(
    base_job_name=f"{notebook_instance_name}-nn",   # who + which model; the tuner appends its own suffix to each child job
    max_run=max_run, # in seconds; always include (max 48 hours)
    entry_point="/home/ec2-user/SageMaker/ML_with_AWS_SageMaker/scripts/train_nn.py",  # Your script for training
    role=role,
    tags=job_tags,  # tags a plain .fit() on this estimator; the tuner below sets its own
    instance_count=instance_count,
    instance_type=instance_type,
    framework_version="2.3",  # if a job fails with an image error, check with: sagemaker.image_uris.retrieve("pytorch", region, version="2.3", py_version="py311", image_scope="training", instance_type="ml.m5.large")
    py_version="py311",
    # the Regex pulls the number out of each "validation:accuracy = 0.8123" line the script prints; SageMaker records it as the metric
    metric_definitions=[{"Name": "validation:accuracy", "Regex": "validation:accuracy = ([0-9\\.]+)"}],
    hyperparameters={
        "train": "/opt/ml/input/data/train/train_data.npz",  # SageMaker will mount this path
        "val": "/opt/ml/input/data/val/val_data.npz",        # SageMaker will mount this path
        "epochs": 100, # Placeholder initial value. Will be overridden by the tuning values tested
        "learning_rate": 0.001 # Placeholder initial value. Will be overridden by the tuning values tested
    },
    output_path=f"s3://{bucket_name}/tuning/",   # every child job's model.tar.gz lands here, in your bucket
    sagemaker_session=session,
)

Define the search space

In SageMaker, you must explicitly define ranges for any hyperparameters you want to tune. SageMaker supports ContinuousParameter, IntegerParameter and CategoricalParameter types:

  • ContinuousParameter allows SageMaker to dynamically sample numeric values within a specified range, making it ideal for broad, exploratory tuning. The total number of values tested can be controlled through the upcoming max_jobs parameter, which defines how many different combinations SageMaker will evaluate.
  • IntegerParameter does the same for whole-number settings such as a batch size or layer width.
  • CategoricalParameter specifies exact values for SageMaker to test, which is useful when you want the model to only try a specific set of options.

By default, SageMaker uses Bayesian optimization, adjusting future selections based on previous results to efficiently find optimal values. You can also set the strategy to “Random” for uniform sampling across the range, which is effective in larger or more exploratory search spaces. Random search usually needs more jobs to find as good a configuration, and the bill is the number of jobs. Generally, we recommend sticking with the default setting.

PYTHON 

# Hyperparameter tuning ranges
hyperparameter_ranges = {
    "epochs": CategoricalParameter([100, 1000, 10000]),       # exact values to try, nothing in between; adjust as needed
    "learning_rate": ContinuousParameter(0.001, 0.1),  # any float in this range; the Bayesian strategy picks where to sample
}

Configure the tuner

The HyperparameterTuner brings the estimator and the search space together. It also needs an objective. Here it maximizes validation:accuracy, which the training script prints and SageMaker reads out of the log. The strategy chooses which configurations to try, max_jobs caps how many training jobs are launched in total, and max_parallel_jobs says how many run at once. Parallel trials finish sooner, and each one still bills its own instance.

Tuning gains are usually marginal past 20 to 30 configurations, and the search space is where the money goes: a published range for the parameters that matter beats a wide search over all of them.

We first set both limits to one. That single trial checks that the script, the metric extraction, the input files and the permissions all work before we pay for a search.

PYTHON 

# Tuner configuration
tuner = HyperparameterTuner(
    base_tuning_job_name=f"{name}-nn-tune"[:20],   # tuning job names are capped at 32 characters and the SDK appends a 12-character timestamp, so the base gets at most 20; child jobs are named <tuning job>-001-<hash>
    estimator=pytorch_estimator,  # the blueprint: every child training job is this estimator with different hyperparameters
    # same regex as the estimator; the value it captures is what the tuner compares across child jobs
    metric_definitions=[{"Name": "validation:accuracy", "Regex": "validation:accuracy = ([0-9\\.]+)"}],
    objective_metric_name="validation:accuracy",  # Ensure this matches the metric name exactly (the "Name" above)
    objective_type="Maximize",                   # Specify if maximizing or minimizing the metric
    hyperparameter_ranges=hyperparameter_ranges,  # the search space defined above
    strategy="Bayesian",  # Default setting (recommend sticking with this!); can adjust to "Random" for uniform sampling across the range
    max_jobs=1,                # Always start with 1 job for debugging purposes. Adjust based on exploration needs (keep below 30 to be kind to environment). 
    max_parallel_jobs=1,         # Always start with 1 for debugging purposes. Recommended to keep this value <= 4 since SageMaker tests values dynamically.
    tags=job_tags,               # propagated to every training job the tuner launches
)

Check how the objective metric is recorded

A tuner needs a number from every trial. train_nn.py prints the validation accuracy during training, and the estimator’s metric_definitions regex pulls that value out of the job’s log. The name in the log line, in the metric definition and in the tuner’s objective_metric_name have to agree, or SageMaker has nothing to compare the trials on.

Note: It’s best to use an if statement to only print out metrics periodically (e.g., every 100 epochs), so that your print time does not eat up too much of your training time. Printing every epoch is a measurable share of a step this small. On the flipside of this, you don’t want to print metrics so infrequently that you lose resolution in the monitored validation accuracy. Choose a number between 100-1000 epochs or divide your total epoch count by ~25 to yield a reasonable range.

PY 

if (epoch + 1) % 100 == 0 or epoch == epochs - 1:
    print(f"validation:accuracy = {val_accuracy:.4f}", flush=True)  # Log for SageMaker metric tracking

Paired with this, our metric_definitions above uses a regular expression "validation:accuracy = ([0-9\\.]+)" to extract the val_accuracy value from each log line. This regex specifically looks for validation:accuracy =, followed by a floating-point number, which corresponds to the format of our log statement in train_nn.py.

Launch the first trial

Now we give the tuner the training and validation .npz files the PyTorch episode saved to S3. The two TrainingInput channels map those files to the paths train_nn.py expects. tuner.fit() starts the tuning job. SageMaker launches one training job with one configuration from the search space and records the validation metric it reports. With max_jobs=1 this is a test of the setup, not yet a search.

Estimate first, as before every launch. The training episodes estimated one job. A tuning job is many, and helpers.estimate_tuning_cost does the arithmetic from the same numbers the tuner takes. Its first input is the minutes one child job bills after its startup, which the PyTorch episode’s CPU job measured: about 3 billable minutes, 2 of them startup, so about 1 minute of training, evaluation and model upload. The tuner keeps each instance between child jobs, so startup is paid once per parallel slot and every later job on that slot bills only its own minute. Its inputs are the numbers the tuner takes plus one measurement:

  • train_minutes_per_job: the billable minutes one child job spends after its startup, on training, evaluation and the model upload. For a small model, time a short run and scale it, as the PyTorch episode did. For a large one, helpers.estimate_training_cost gives it. Here it is about 1 minute.
  • price_per_hour: the instance’s rate from the Instances for ML page, the same variable the estimator cell above set.
  • max_jobs: how many child jobs the tuner may launch in total, the same max_jobs you give the tuner.
  • max_parallel_jobs: how many run at once, the same max_parallel_jobs. This is the number of instances the tuner keeps up, so it is the number of times startup is paid.
  • startup_minutes: the container pull and data download, 2 by default, from the training episodes’ measured jobs.

It prints the billable minutes with their breakdown, the cost, the wall-clock before the unbilled provisioning, and a max_run for each child job, twice one job’s estimate, to set on the estimator. The whole search, as a dictionary, comes back too.

PYTHON 

helpers.estimate_tuning_cost(
    train_minutes_per_job=1,     # a child job's billable minutes after startup, from the PyTorch episode's CPU job
    price_per_hour=price_per_hour,   # ml.m5.large, set with the estimator above
    max_jobs=1,                  # the tuner's max_jobs: one child job, to test the setup
    max_parallel_jobs=1,         # the tuner's max_parallel_jobs: one instance, so startup is paid once
);

One child job, about 3 billable minutes, about $0.01. If the launch is refused with an AccessDeniedException ending in explicit deny, the account’s checks are the cause: a missing tag, an instance type outside the lesson’s list, or a resource kind the lesson never creates; the first training episode’s callout If the launch is refused says what to look at.

PYTHON 

# Define the input paths
train_input = TrainingInput(f"s3://{bucket_name}/train_data.npz", content_type="application/x-npz")
val_input = TrainingInput(f"s3://{bucket_name}/val_data.npz", content_type="application/x-npz")

import time as t

# Launch the hyperparameter tuning job (blocks until the single job has finished)
start_time = t.time()
tuner.fit({"train": train_input, "val": val_input})
print(f"Hyperparameter tuning job finished in {t.time() - start_time:.1f} seconds (wall-clock, including instance startup)")

Monitor the tuning job

While the above cell runs, we can check on the progress by visiting the SageMaker Console and finding the “Training” tab located on the left panel. Click “Hyperparameter tuning jobs” to view running jobs.

What did that cost?

The money is spent by the training jobs the tuner launches, and that is where SageMaker records billable time. We bring back job_cost_report() from the previous episodes and run it over every child job of the tuner. The list of child jobs comes from tuner.analytics().dataframe() (one row per training job, with its hyperparameters and final objective value).

PYTHON 

def job_cost_report(job_name: str, price_per_hour: float):
    desc = session.sagemaker_client.describe_training_job(TrainingJobName=job_name)
    if "BillableTimeInSeconds" not in desc:   # the job has not finished: no bill yet
        print(f"{job_name}: status {desc['TrainingJobStatus']}; billable time and the model path appear once it is Completed")
        return desc
    n = desc["ResourceConfig"]["InstanceCount"]
    billable = desc["BillableTimeInSeconds"] * n
    wall = (desc["TrainingEndTime"] - desc["CreationTime"]).total_seconds()   # launch request to end, including unbilled provisioning
    print(f"{job_name}: status {desc['TrainingJobStatus']}, wall-clock {wall/60:.1f} min, billable {billable/60:.1f} min on {n} instance(s), cost ${billable/3600*price_per_hour:.3f} at ${price_per_hour}/h")
    print(f"    model: {desc['ModelArtifacts']['S3ModelArtifacts']}")   # the model.tar.gz this job uploaded to output_path
    return desc

def tuning_cost_report(tuner, price_per_hour):
    """Call job_cost_report on every training job the tuner launched and print the total."""
    jobs_df = tuner.analytics().dataframe()  # one row per child training job: its hyperparameters, FinalObjectiveValue and TrainingJobName
    total_billable = 0
    for job_name in jobs_df["TrainingJobName"]:
        desc = job_cost_report(job_name, price_per_hour)
        total_billable += desc["BillableTimeInSeconds"] * desc["ResourceConfig"]["InstanceCount"]
    total_cost = total_billable / 3600 * price_per_hour
    print(f"Tuning job {tuner.latest_tuning_job.name}: {len(jobs_df)} training jobs, {total_billable/60:.1f} billable min, ${total_cost:.3f} total")

tuning_cost_report(tuner, price_per_hour)

Scaling up our approach

If all goes well, we can scale up the experiment with the below code. In this configuration, we’re scaling up the search by allowing SageMaker to test more hyperparameter configurations (max_jobs=10) while setting max_parallel_jobs=2 to manage parallelization efficiently. With two jobs running at once, SageMaker will be able to explore potential improvements more quickly than in a fully sequential setup, while still dynamically selecting promising values as it learns from completed jobs. SageMaker’s Bayesian optimization uses the completed trials to choose the next ones, so it spends fewer jobs on unpromising combinations. Setting max_parallel_jobs higher than 2-4 reduces tuning efficiency, as SageMaker’s ability to learn from completed jobs decreases when too many jobs run simultaneously. Two at a time is a good default.

Estimate first. The same call with the new limits:

PYTHON 

helpers.estimate_tuning_cost(
    train_minutes_per_job=1,     # unchanged: the child jobs are the same job
    price_per_hour=price_per_hour,
    max_jobs=10,                 # ten child jobs
    max_parallel_jobs=2,         # two at a time, so startup is paid twice and the other eight jobs reuse warm instances
);

About 14 billable minutes, $0.03, and about 7 minutes of wall-clock plus provisioning. Startup is paid twice, once per slot, not ten times. Write the numbers down and compare them with the bill below.

PYTHON 

import time as t

# Configuration variables
max_jobs = 10
max_parallel_jobs = 2

# Define the Tuner configuration
tuner = HyperparameterTuner(
    base_tuning_job_name=f"{name}-nn-tune"[:20],   # tuning job names are capped at 32 characters and the SDK appends a 12-character timestamp, so the base gets at most 20; child jobs are named <tuning job>-001-<hash>
    estimator=pytorch_estimator,
    metric_definitions=[{"Name": "validation:accuracy", "Regex": "validation:accuracy = ([0-9\\.]+)"}],
    objective_metric_name="validation:accuracy",  # Ensure this matches the metric name exactly
    objective_type="Maximize",                   # Specify if maximizing or minimizing the metric
    hyperparameter_ranges=hyperparameter_ranges,
    max_jobs=max_jobs,
    max_parallel_jobs=max_parallel_jobs,
    tags=job_tags,
)

# Track start time
start_time = t.time()

# Launch the hyperparameter tuning job
tuner.fit({"train": train_input, "val": val_input})

# Calculate runtime
runtime = t.time() - start_time

# Print confirmation with runtime and configuration details
print(f"Tuning runtime: {runtime:.2f} seconds, Instance Type: {instance_type}, Max Jobs: {max_jobs}, Max Parallel Jobs: {max_parallel_jobs}")

Monitoring tuning

While the above cell runs, we can check on the progress by visiting the SageMaker Console and finding the “Training” tab located on the left panel. Click “Hyperparameter tuning jobs” to view running jobs.

  • Initial Jobs: SageMaker starts by running only max_parallel_jobs (2 in this case) as the initial batch. As each job completes, new jobs from the remaining pool are triggered until max_jobs (10) is reached.
  • Job Completion: Once the first few jobs complete, SageMaker will continue to launch the remaining jobs up to the maximum of 10, but no more than two at a time.

Once it has finished, the cost report for all 10 child jobs:

PYTHON 

tuning_cost_report(tuner, price_per_hour)

Our run finished in 9.3 minutes of wall-clock and billed 14.1 minutes in total, $0.03, the estimate’s 14. Read the per-job lines and the reason the estimate counts startup per slot is visible. The first two child jobs billed about 3 minutes each, like every job so far. The other eight billed about 1 minute each. A tuning job keeps its instances between child jobs, so jobs 3 through 10 ran on the two instances that were already up, with the container pulled and the data downloaded. The startup overhead that dominates a single small job was paid twice, not ten times.

More jobs in parallel

Bayesian optimization chooses each job’s values from the results so far. With max_parallel_jobs well below max_jobs, later jobs learn from earlier ones; with all ten at once, nothing has finished when the values are chosen, and the search is a random one. The All at once challenge below measures what that does to the result, the wall-clock and the bill.

Extracting and evaluating the best model after tuning

Tuning takes about 10 minutes. After SageMaker completes the hyperparameter tuning job, the results, including the trained models for each configuration, are stored in an S3 bucket. The steps to locate and evaluate the best model on test data:

  1. Understanding the folder structure:
    • Each child job uploads its model to the estimator’s output_path, under its own job name: s3://{bucket}/tuning/{job_name}/output/model.tar.gz. Each model is compressed as a .tar.gz file containing the saved model parameters.
  2. Retrieve and load the best model:
    • Using the tuner.best_training_job() method, you can get the name of the best-performing job.
    • From there, retrieve the S3 URI of the best model artifact, download it locally, and extract the files for use.
  3. Prepare test data for final assessment of model generalizability
    • Using the scaler that was fit on the training data.
  4. Evaluate the model on test data:
    • Once extracted, load the saved model weights and evaluate the model on your test dataset to get the final performance metrics.

The code:

View best model details and storage info

We can easily view the best hyperparameters from the tuning procedure…

PYTHON 

# 1. Get the best training job from the completed tuning job
best_job_name = tuner.best_training_job()  # the child job with the highest validation:accuracy
print("Best training job name:", best_job_name)

# 2. Use describe_training_job to retrieve full details, including hyperparameters...
best_job_desc = session.sagemaker_client.describe_training_job(TrainingJobName=best_job_name)
best_hyperparameters = {k: v for k, v in best_job_desc["HyperParameters"].items() if k in hyperparameter_ranges}   # the record also carries the SDK's own sagemaker_* entries
print("Best hyperparameters:", best_hyperparameters)

# ...  and model URI (location on S3)
best_model_s3_uri = best_job_desc['ModelArtifacts']['S3ModelArtifacts']  # the model.tar.gz that job uploaded when it finished
print(f"Best model artifact S3 URI: {best_model_s3_uri}")

Retrieve and load best model

Download the winning job’s model.tar.gz and unpack it; SageMaker packs whatever the script saved to /opt/ml/model into that archive.

PYTHON 

import tarfile

# Download and extract the model artifact: SageMaker packs whatever the script saved to /opt/ml/model into model.tar.gz
local_model_path = "best_model.tar.gz"
model_bucket, model_key = best_model_s3_uri.split('/')[2], '/'.join(best_model_s3_uri.split('/')[3:])  # s3://bucket/key -> bucket, key
s3.download_file(model_bucket, model_key, local_model_path)

# Extract the model files from the tar.gz archive
with tarfile.open(local_model_path, 'r:gz') as tar:
    tar.extractall()
print("Best model downloaded and extracted.")

Prepare test set as test_data.npz

In the PyTorch episode, we converted our train dataset into train/validate subsets, and saved them out as .npz files for efficient processing. We’ll need to preprocess our test data the same way to evaluate it on our model.

Preprocessing lives in a function so that the test set is transformed exactly as the training set was. We’ll import our preprocessing function from train_nn.py, and pass it the scaler that was fit on the training data (saved as scaler.joblib in the previous episode). Calling preprocess_data(test_df) without the scaler would fit a new one on the test set and transform the features differently from what the model saw in training.

PYTHON 

!pip install scikit-learn   # train_nn.py imports it; not in the notebook kernels

Fetch the test CSV and the training-set scaler, transform the test rows the same way the training rows were, and save the arrays.

PYTHON 

import pandas as pd
import numpy as np
import joblib
from train_nn import preprocess_data

# Fetch the test CSV and the training-set scaler from S3 (both uploaded in earlier episodes)
s3.download_file(bucket_name, "titanic_test.csv", "titanic_test.csv")
s3.download_file(bucket_name, "scaler.joblib", "scaler.joblib")
scaler = joblib.load("scaler.joblib")  # the StandardScaler fit on the training data in the PyTorch episode

# Preprocess test data with the same scaler used for training
test_df = pd.read_csv("titanic_test.csv")
X_test, y_test, _ = preprocess_data(test_df, scaler=scaler)  # transform only; passing the scaler stops it refitting on test data

# Save processed data for testing
np.savez('test_data.npz', X_test=X_test, y_test=y_test)

Evaluate the model on test data

Rebuild the network from the script’s class, load the saved weights, and measure accuracy on the test set.

PYTHON 

from train_nn import TitanicNet
from train_nn import calculate_accuracy
import torch

# Load the model (saved as 'nn_model.pth' by train_nn.py)
model = TitanicNet()
model.load_state_dict(torch.load("nn_model.pth", map_location="cpu"))
model.eval()

# Load test data
test_data = np.load("test_data.npz")
X_test = torch.tensor(test_data['X_test'], dtype=torch.float32)
y_test = torch.tensor(test_data['y_test'], dtype=torch.float32).unsqueeze(1)

# Evaluate the model on the test set
with torch.no_grad():
    predictions = model(X_test)
    accuracy = calculate_accuracy(predictions, y_test)
    print(f"Test Accuracy: {accuracy:.4f}")
Challenge

All at once

Rerun the 10-job search with max_parallel_jobs=10 (equal to max_jobs). Before you launch it, run helpers.estimate_tuning_cost with the new limit and predict what changes in (a) total cost, (b) wall-clock time, and (c) the quality of the search. Then run it and check (a) and (b) with tuning_cost_report() and the runtime print.

PYTHON 

tuner_parallel = HyperparameterTuner(
    base_tuning_job_name=f"{name}-nn-tune"[:20],   # tuning job names are capped at 32 characters and the SDK appends a 12-character timestamp, so the base gets at most 20; child jobs are named <tuning job>-001-<hash>
    estimator=pytorch_estimator,
    metric_definitions=[{"Name": "validation:accuracy", "Regex": "validation:accuracy = ([0-9\\.]+)"}],
    objective_metric_name="validation:accuracy",
    objective_type="Maximize",
    hyperparameter_ranges=hyperparameter_ranges,
    max_jobs=10,
    max_parallel_jobs=10,
    tags=job_tags,
)

start_time = t.time()
tuner_parallel.fit({"train": train_input, "val": val_input})
print(f"Tuning runtime: {t.time() - start_time:.2f} seconds with max_parallel_jobs=10")
tuning_cost_report(tuner_parallel, price_per_hour)
print("Best objective, sequential-ish run:", tuner.analytics().dataframe()["FinalObjectiveValue"].max())
print("Best objective, all-parallel run:  ", tuner_parallel.analytics().dataframe()["FinalObjectiveValue"].max())
  • Cost: higher, not the same. The estimator says 30 billable minutes, ten slots’ startup plus ten jobs, against 14 for two slots. Our all-parallel run billed 27.5 minutes ($0.05) against 14.1 minutes ($0.03) for the two-at-a-time run. Ten instances each pull the container and download the data, so the startup overhead is paid ten times; with two at a time it was paid twice and the other eight child jobs reused warm instances. (If your account has a quota on concurrent ml.m5.large training instances, the request may be rejected or throttled, which is the practical limit on this approach.)
  • Wall-clock: roughly one job’s duration. Our run took 4.3 minutes against 9.3, so about twice as fast, not five times: the two-at-a-time run’s later jobs were short because they skipped startup.
  • Search quality: with all 10 launched at once, the Bayesian strategy has no completed results to learn from, so the 10 configurations are effectively a random sample of the space. On a 2-parameter problem this small you will usually see a similar best objective either way; on a larger search space with more jobs, the sequential-ish run tends to find a better configuration for the same number of jobs.
Challenge

Tune the batch size too

Suppose you want to add batch_size to the search as IntegerParameter(16, 256). What has to change in train_nn.py for that to do anything, and what has to change in the notebook?

The tuner can only vary what the script reads. Right now train_nn.py has no --batch_size argument and trains on the full dataset every epoch (one model(X_train) call per epoch), so a batch_size hyperparameter would be passed on the command line, rejected by argparse, and the job would fail. Three changes are needed:

  1. In train_nn.py: add parser.add_argument('--batch_size', type=int, default=64), pass it to train(), and replace the full-batch step with a mini-batch loop, for example by wrapping the tensors in a TensorDataset and iterating over a DataLoader(dataset, batch_size=batch_size, shuffle=True) inside each epoch (with optimizer.zero_grad(), forward, loss, backward() and step() per batch).

PY 

parser.add_argument('--batch_size', type=int, default=64)
...
loader = DataLoader(TensorDataset(X_train, y_train), batch_size=batch_size, shuffle=True)
for epoch in range(epochs):
    model.train()
    for xb, yb in loader:
        optimizer.zero_grad()
        loss = criterion(model(xb), yb)
        loss.backward()
        optimizer.step()

  1. In the estimator: add a placeholder "batch_size": 64 to hyperparameters, so the argument is always passed.

  2. In the ranges: add "batch_size": IntegerParameter(16, 256) to hyperparameter_ranges.

Two side effects: an epoch now takes several optimizer steps instead of one, so the same epochs values train for longer (consider shrinking the epochs choices), and the search space has grown, so 10 jobs will cover it less well; 20-30 would be a reasonable budget, which at these prices is still well under a dollar.

Key Points
  • Tuner = estimator + ranges + objective: A HyperparameterTuner wraps an ordinary estimator; the script only has to print the objective metric in a format the metric_definitions regex can capture.
  • Start with one job: max_jobs=1, max_parallel_jobs=1 first, to catch script and configuration errors before paying for a search.
  • Parallelism buys time, not quality: Cost is per child training job; max_parallel_jobs changes wall-clock time and, with the Bayesian strategy, how much each job can learn from the ones before it. 2-4 parallel jobs is a good default.
  • Tags propagate from the tuner: tags on the tuner are copied to every child training job.
  • Estimate the whole search before launching it: helpers.estimate_tuning_cost takes the tuner’s own limits and one child job’s billable minutes; startup is paid once per parallel slot, which is why two at a time is cheaper than ten at once.
  • Read the bill from the child jobs: SageMaker bills per second of BillableTimeInSeconds x instance count on each training job; container pull and data download are inside that number and dominate short jobs.
  • Transform the test set with the training scaler: Load the saved scaler and pass it to preprocess_data(test_df, scaler=scaler); never refit on test data.