(Optional) Pushing Your Notebooks to GitHub from SageMaker

Last updated on 2026-10-21 | Edit this page

Estimated time: 35 minutes

Overview

Questions

How can I put the code I write on a SageMaker notebook instance under version control on GitHub?
How do I authenticate to GitHub from a notebook instance without leaving a password or key behind?
How can I convert notebooks to .py files and keep .ipynb files out of version control?

Objectives

Create a GitHub repository for your own SageMaker work and connect a folder on the notebook instance to it.
Configure Git on a notebook instance to push over HTTPS with a GitHub Personal Access Token (PAT), entered with getpass.
Convert .ipynb files to .py files for cleaner version control.

Open prefilled .ipynb notebook

Open the notebook from: /ML_with_AWS_SageMaker/notebooks/05_Interacting-with-code-repo.ipynb. Select the conda_python3 kernel.

This episode is optional self-study: it is not needed to run the rest of the lesson, but it is the workflow you will want once you start writing code of your own in SageMaker. It also doubles as a short git refresher. You need a GitHub account for it; nothing else in the lesson does.

Step 0: What we are versioning, and where

So far every notebook you have run came from the lesson repository, which you cloned read-only in the notebooks-as-controllers episode. You cannot push to that repository. This episode sets up the usual situation instead: your own work in your own repository. We will create an empty repository on GitHub, make a matching folder on the notebook instance, copy a notebook into it, and push.

On GitHub, click New repository (the + in the top-right corner). Name it sagemaker-work (any name is fine; the cells below use this one). Leave it Public or make it Private, as you prefer. Tick Add a README file so the repository is not empty, which gives Step 6 a pull with something in it. Click Create repository.
Back in JupyterLab, make the folder on the instance and turn it into a git repository. We put it under /home/ec2-user/SageMaker/, the one directory that survives stopping the instance.

PYTHON

%cd /home/ec2-user/SageMaker/
!mkdir -p sagemaker-work
%cd sagemaker-work
!git init -b main   # start an empty git repository here, with "main" as its first branch

Connect it to the repository you just created, replacing YOUR_GITHUB_USERNAME:

PYTHON

# A remote is a copy of the repository elsewhere; "origin" is the conventional name for the main one (here, GitHub)
!git remote add origin https://github.com/YOUR_GITHUB_USERNAME/sagemaker-work.git
!git remote -v   # list the remotes and their URLs, to check the address

Step 1: A GitHub personal access token (PAT)

A notebook instance is a machine that other people may be able to reach (an administrator, or whoever uses it after you) and that you may delete next month. Neither a saved password nor an SSH key belongs on it. GitHub’s answer is a Personal Access Token: a credential you create with only the permissions you need and an expiry date, use over HTTPS, and revoke in one click. We will enter it with getpass each session so that it never appears in a cell or a saved notebook.

Important Note: A PAT is a password with a scope. Give it the minimum permissions, set an expiry, and never paste it into code.

Generate the token on GitHub

Go to Settings by clicking on your profile picture in the upper-right corner of GitHub.
Click Developer settings at the very bottom of the left sidebar.
Select Personal access tokens, then Tokens (classic).
Click Generate new token (classic).
Give it a descriptive name (e.g., “SageMaker notebook instance”) and an expiry (30 days is plenty for a workshop).
Select the minimum permissions needed:
- For a public repository: only public_repo.
- For a private repository: repo.
Click Generate token and copy it immediately: you cannot see it again once you leave the page.

Caution: Treat your PAT like a password. Store it in a password manager, not in a file on the instance, and revoke it (same page, Delete) when the workshop is over; the cleanup episode’s checklist reminds you.

Enter it with `getpass`

input() and getpass.getpass() prompt you in the notebook; the token is held in a Python variable for this kernel session and is not written into the notebook file.

PYTHON

import getpass   # prompts without echoing what you type, so the token never shows on screen or in the saved notebook

username = input("GitHub Username: ")
token = getpass.getpass("GitHub Personal Access Token (PAT): ")  # kept in this variable for the kernel session only

Note: when you re-run the whole notebook later, this cell prompts again, because nothing is saved.

Step 2: Configure Git settings

Git records an author on every commit. Set yours once; --global stores it in your home directory on the instance, so it persists across restarts of this instance (not across instances).

PYTHON

!git config --global user.name "Your name"             # appears as the author in the commit history
!git config --global user.email your_email@example.edu  # use the email on your GitHub account so commits link to your profile

Step 3: Convert `.ipynb` to `.py`

We want our notebook under version control, but a .ipynb file is JSON with outputs and metadata baked in, which makes diffs unreadable and repositories large. The usual practice is to version a plain .py copy of the code and ignore the notebooks.

Benefits of converting to `.py` before committing

Cleaner version control: readable diffs, reviewable pull requests, sane merges.
Script compatibility: a .py file runs from the command line and imports as a module.
Smaller repository: no outputs, no images, no cell metadata.

Install Jupytext, which converts in both directions.

PYTHON

!pip install jupytext   # converts between .ipynb and .py in both directions

Copy the notebook from the Accessing S3 episode into our repository folder (we are still in /home/ec2-user/SageMaker/sagemaker-work) and convert it there. Copying keeps the lesson clone untouched and puts the .py file where git will see it.

PYTHON

# Copy the notebook into the repository folder ("." is the current directory); the lesson clone stays untouched
!cp /home/ec2-user/SageMaker/ML_with_AWS_SageMaker/notebooks/04_Accessing-S3-via-SageMaker-notebooks.ipynb .
!jupytext --to py 04_Accessing-S3-via-SageMaker-notebooks.ipynb   # writes a .py next to it: code and markdown, no outputs

[jupytext] Reading 04_Accessing-S3-via-SageMaker-notebooks.ipynb in format ipynb
[jupytext] Writing 04_Accessing-S3-via-SageMaker-notebooks.py

With several notebooks, convert them all in a loop:

PYTHON

import subprocess
import os

notebooks = [f for f in os.listdir() if f.endswith('.ipynb')]
for notebook in notebooks:
    output_file = notebook.replace('.ipynb', '.py')
    subprocess.run(["jupytext", "--to", "py", notebook, "--output", output_file])  # the same shell command, run from Python
    print(f"Converted {notebook} to {output_file}")

The same loop is helpers.convert_files() in the lesson’s scripts/helpers.py. It converts every .ipynb in the current directory, so run it from inside the folder you mean to convert:

PYTHON

import sys
# Put the lesson's scripts/ folder on Python's import path, as in the Accessing S3 episode
sys.path.insert(0, "/home/ec2-user/SageMaker/ML_with_AWS_SageMaker/scripts")
import helpers
helpers.convert_files(direction="notebook_to_python")

Step 4: Add and commit the `.py` file

Check the status of the repository. Make sure you are in the repository folder first.

PYTHON

%cd /home/ec2-user/SageMaker/sagemaker-work
!git status   # shows the untracked files (the .ipynb and the .py); nothing is staged yet

Add and commit. We add the .py by name rather than git add ., so the .ipynb copy is never tracked (Step 5 ignores it for good).

PYTHON

!git add 04_Accessing-S3-via-SageMaker-notebooks.py   # stage: mark this file to go into the next commit
!git commit -m "Add Accessing-S3 notebook as a script"   # a commit is a snapshot of the staged files; the message says what changed, specifically

Check the status again.

PYTHON

!git status   # the .py is committed; only the untracked .ipynb remains, which Step 5 ignores

Step 5: Ignore notebooks and clutter with `.gitignore`

A .gitignore file lists patterns git should never track. Ignoring *.ipynb across the repository means a stray git add . cannot commit a notebook by accident.

Confirm you are in the repository folder.

PYTHON

!pwd

Create the file (hidden in the file browser because it starts with a dot; ls -a shows it) and add the notebook pattern.

PYTHON

with open(".gitignore", "a") as gitignore:   # "a" appends, so re-running the cell keeps earlier rules
    gitignore.write("# Jupyter notebooks: we version the .py copies instead\n*.ipynb\n")  # one pattern per line; * matches any name
!cat .gitignore

While we are here, ignore the usual clutter too: caches, temporary files, and the NumPy archives the training episodes write next to notebooks.

PYTHON

with open(".gitignore", "a") as gitignore:
    gitignore.write("\n# Cache and temp files\n__pycache__/\n*.tmp\n*.log\n")
!cat .gitignore

Add and commit .gitignore so the rule travels with the repository.

PYTHON

!git add .gitignore   # .gitignore is itself tracked, so everyone who clones gets the same rules
!git commit -m "Ignore notebooks, caches and temp files"

Step 6: Pull, then push

Before pushing, pull: the remote already has the README that GitHub created, and in a shared repository it may have your collaborators’ commits. Pulling first means your push is a simple addition rather than a rejected one.

Which pull strategy

Git has to decide how to combine your local commits with remote ones. The three options:

Merge (pull.rebase false): the remote changes come in as a merge commit. Simplest, shows where you integrated. The default and the right choice for most collaborative work.
Rebase (pull.rebase true): your commits are replayed on top of the remote ones, giving a linear history. Cleaner, but conflicts have to be resolved commit by commit.
Fast-forward only (pull.ff only): refuses to pull unless nothing diverged. Useful as a safety check, not as a daily setting.

We use merge. Because our folder and the GitHub repository started separately (one from git init, one from Create repository), git also needs --allow-unrelated-histories for this very first pull; never again after that.

PYTHON

!git config pull.rebase false   # pull strategy: merge (remote commits come in through a merge commit)
# pull = fetch origin's main and merge it into ours; the flag is needed only this once, as the two histories share no commit
!git pull origin main --allow-unrelated-histories --no-edit

If git reports a merge conflict, each conflicting file has to be resolved one way or the other and then staged and committed. You will not get one here (the README and your script are different files), but this is what it looks like:

PYTHON

# Example only.
# Keep YOUR version of a file:
# !git checkout --ours 04_Accessing-S3-via-SageMaker-notebooks.py
# Keep the REMOTE version of a file:
# !git checkout --theirs README.md
# Then stage and commit the resolution:
# !git add 04_Accessing-S3-via-SageMaker-notebooks.py README.md
# !git commit -m "Resolve merge conflicts"

Push with the PAT

The token goes into the URL for this one command, built from the variables getpass filled. It is not stored by git and does not appear in the notebook.

PYTHON

github_url = f"github.com/{username}/sagemaker-work.git"
# IPython substitutes the Python variables in {braces}; user:token@ in an HTTPS URL authenticates this one push and is not stored
!git push https://{username}:{token}@{github_url} main

Open https://github.com/YOUR_GITHUB_USERNAME/sagemaker-work in a browser: the README, the .py file and .gitignore should all be there, with your two commits in the history.

Step 7: Pulling `.py` files and converting back to notebooks

Suppose a collaborator edits the .py file on GitHub, or you edit it from another machine. Pull as usual:

PYTHON

!git config pull.rebase false
!git pull origin main   # a routine pull: no extra flag now that the histories are joined

Then convert the script back to a notebook with Jupytext. The .ipynb it writes is ignored by git (Step 5), so the round trip leaves the repository clean.

PYTHON

# --to notebook converts the other way; the .ipynb it writes matches the *.ipynb rule in .gitignore
!jupytext --to notebook 04_Accessing-S3-via-SageMaker-notebooks.py --output 04_Accessing-S3-via-SageMaker-notebooks.ipynb

To convert every .py in the folder back to notebooks, helpers.convert_files(direction="python_to_notebook") does the loop; remember it converts every .py in the current directory, so keep real modules out of that folder.

Challenge

Challenge: ignore the big files too

The training episodes save NumPy arrays (.npz) next to your notebooks. Those are binary and can be large, which makes them a poor fit for git. Add *.npz to your repository’s .gitignore, commit it, push with your PAT, and verify the change on GitHub.

Show me the solution

Still inside /home/ec2-user/SageMaker/sagemaker-work:

PYTHON

with open(".gitignore", "a") as gitignore:
    gitignore.write("\n# NumPy archives saved by the training episodes\n*.npz\n")
!cat .gitignore
!git add .gitignore
!git commit -m "Ignore .npz arrays"
!git push https://{username}:{token}@github.com/{username}/sagemaker-work.git main

Then open https://github.com/YOUR_GITHUB_USERNAME/sagemaker-work/blob/main/.gitignore in a browser: the *.npz line should be there, and the commit list should show “Ignore .npz arrays” as the latest commit. If the push is rejected with “authentication failed”, the PAT has expired or lacks the public_repo (or repo) scope; generate a new one and re-run the getpass cell.

Key Points

Your own work belongs in your own repository: create one on GitHub, git init a folder under /home/ec2-user/SageMaker/, and connect the two with git remote add.
Authenticate over HTTPS with a scoped, expiring Personal Access Token entered through getpass; nothing is saved on the instance, and the token is revoked when you are done.
Version .py copies of notebooks (Jupytext converts both ways) and ignore *.ipynb, caches and data archives in .gitignore.
Pull before you push; the first pull into a freshly initialised folder needs --allow-unrelated-histories.

(Optional) Pushing Your Notebooks to GitHub from SageMaker

Overview

Questions

Objectives

Open prefilled .ipynb notebook

Step 0: What we are versioning, and where

PYTHON

PYTHON

Step 1: A GitHub personal access token (PAT)

Generate the token on GitHub

Enter it with getpass

PYTHON

Step 2: Configure Git settings

PYTHON

Step 3: Convert .ipynb to .py

Benefits of converting to .py before committing

PYTHON

PYTHON

PYTHON

PYTHON

Step 4: Add and commit the .py file

PYTHON

PYTHON

PYTHON

Step 5: Ignore notebooks and clutter with .gitignore

PYTHON

PYTHON

PYTHON

PYTHON

Step 6: Pull, then push

Which pull strategy

PYTHON

PYTHON

Push with the PAT

PYTHON

Step 7: Pulling .py files and converting back to notebooks

PYTHON

PYTHON

Challenge: ignore the big files too

Show me the solution

PYTHON

Enter it with `getpass`

Step 3: Convert `.ipynb` to `.py`

Benefits of converting to `.py` before committing

Step 4: Add and commit the `.py` file

Step 5: Ignore notebooks and clutter with `.gitignore`

Step 7: Pulling `.py` files and converting back to notebooks