This lesson is in the early stages of development (Alpha version)

Reproducible Computational Environments using Containers

Introducing Containers

Overview

Teaching: 20 min
Exercises: 0 min
Questions
  • What are containers, and why might they be useful to me?

Objectives
  • Show how software depending on other software leads to configuration management problems.

  • Identify the problems that software installation problems can pose for research.

  • Give two examples of how containers can solve software configuration problems.

Disclaimers

  1. Docker is complex software used for many different purposes. We are unlikely to give examples that suit all of your potential ideal use-cases, but would be delighted to at least open up discussion of what those use-cases might be.

  2. Containers are a topic that requires significant amounts of technical background to understand in detail. Most of the time containers, particularly as wrapped up by Docker, do not require you to have a deep technical understanding of container technology, but when things go wrong, the diagnostic messages may turn opaque rather quickly.

Scientific Software Challenges

What’s Your Experience?

Take a minute to think about challenges that you have experienced in using scientific software (or software in general!) for your research. Then, share with your neighbors and try to come up with a list of common gripes or challenges.

You may have come up with some of the following:

Etc.

A lot of these characteristics boil down to one fact: the main program you want to use likely depends on many, many, different other programs (including the operating system!), creating a very complex, and often fragile system. One change or missing piece may stop the whole thing from working or break something that was already running. It’s no surprise that this situation is sometimes informally termed “dependency hell”.

Software and Science

Again, take a minute to think about how the software challenges we’ve discussed could impact (or have impacted!) the quality of your work. Share your thoughts with your neighbors. What can go wrong if our software doesnt work?

Unsurprisingly, software installation and configuration challenges can have negative consequences for research:

Thankfully there are ways to get underneath (a lot of) this mess: containers to the rescue! Containers provide a way to package up software dependencies and access to resources such as files and communications networks in a uniform manner.

What is a Container?

Docker is a tool that allows you to build what are called “containers.” It’s not the only tool that can create containers, but is the one we’ve chosen for this workshop. But what is a container?

To understand containers, let’s first talk briefly about your computer.

Your computer has some standard pieces that allow it to work - often what’s called the hardware. One of these pieces is the CPU or processor; another is the amount of memory or RAM that your computer can use to store information temporarily while running programs; another is the hard drive, which can store information over the long-term. All these pieces work together to do the “computing” of a computer, but we don’t see them, because they’re hidden away.

Instead, what we see is our desktop, program windows, different folders, and files. These all live in what’s called the file system. Everything on your computer - programs, pictures, documents - lives somewhere in the file system. One way to think of the file system is the layer of stuff that can be activated to use use the CPU, memory and hard drive of your computer.

NOW, imagine you wanted to have a second computer. You don’t want to buy a whole new computer because it’s too expensive. What if, instead, you could have another filesystem that you could store and access from your main computer, but that is self-contained?

A container system (like Docker) is a special program on your computer that does this. The term “container” can be usefully considered with reference to shipping containers. Before shipping containers were developed, packing and unpacking cargo ships was time consuming, and error prone, with high potential for different clients’ goods to become mixed up. Software containers standardise the packaging of a complete software system: you can drop a container into a computer with the container software installed (also called a container host), and it should “just work”.

Virtualization

Containers are an example of what’s called virtualization – having a second “virtual” computer running and accessible from a main or host computer. Another example of virtualization are virtual machines or VMs. A virtual machine typically contains a whole copy of an operating system in addition to its own file system and has to get booted up in the same way a computer would. A container is considered a lightweight version of a virtual machine; underneath, the container is using the Linux kernel and simply has some flavor of Linux + the file system inside.

One final term: if the container is an alternative file system layer that you can access and run from your computer, the container image is like a template for that container. The container image has all the needed information to start up a running copy of the container. A running container tends to be transient and can be started and shut down. The image is more long-lived, as a source file for the container. You could think of the container image like a cookie cutter – it can be used to create multiple copies of the same shape (or container) and is relatively unchanging, where cookies come and go. If you want a different type of container (cookie) you need a different image (cookie cutter).

Putting the Pieces Together

Think back to some of the challenges we described at the beginning. The many layers of scientific software installations make it hard to install and re-install scientific software – which ultimately, hinders reliability and reproducibility.

But now, think about what a container is - a self-contained, complete, separate computer file system. What if you put your scientific software tools into a container?

This solves several of our problems:

The rest of this workshop will show you how to download and run pre-existing containers on your own computer, and how to create and share your own containers.

Key Points

  • Almost all software depends on other software components to function, but these components have independent evolutionary paths.

  • Projects involving many software components can rapidly run into a combinatoric explosion in the number of software version configurations available, yet only a subset of possible configurations actually works as desired.

  • Containers collect software components together and can help avoid software dependency problems.

  • Virtualisation is an old technology that container technology makes more practical.

  • Docker is just one software platform that can create containers and the resources they use.


Introducing the Docker command line

Overview

Teaching: 10 min
Exercises: 0 min
Questions
  • How do I interact with Docker?

Objectives
  • Explain how to check that Docker is installed and is ready to use.

  • Demonstrate some initial Docker command line interactions.

Docker command line

Start the Docker application that you installed in working through the setup instructions for this session. Note that this might not be necessary if your laptop is running Linux.

The Docker application will usually provide a way for you to log in using the application’s menu (macOS) or systray icon (Windows). This will require you to use your Docker Hub username and your password.

Determining your Docker Hub username

If you no longer recall your Docker Hub username, e.g., because you have been logging into the Docker Hub using your email address, you can find out what it is through the steps:

  • Open http://hub.docker.com/ in a web browser window
  • Sign-in using your email and password (don’t tell us what it is)
  • In the top-right of the screen you will see your username. Mine’s dme26.

Now open a shell window, and run the following command in your shell to check that Docker is installed. I have appended the output that I see on my Mac, but the specific version is unlikely to matter much: it certainly does not have to precisely match mine.

$ docker --version
Docker version 18.09.1, build 4c52b90

Ensure that your command line docker commands are able to reach the Docker Hub by running the following command:

$ docker login
Authenticating with existing credentials...
Login Succeeded

(I wasn’t prompted for authentication details, if you are, then you need to use your Docker Hub username and password.)

The Login Succeeded message means that your docker command line tool is ready to access the Docker Hub. We will return to discussion of the Docker Hub soon…

Key Points

  • A toolbar icon indicates that Docker is ready to use containers.

  • You will typically interact with Docker using the command line.


Exploring and Running Containers

Overview

Teaching: 20 min
Exercises: 10 min
Questions
  • How do I interact with a Docker container on my computer?

Objectives
  • Use the correct command to see which Docker images are on your computer.

  • Download new Docker images.

  • Demonstrate how to start an instance of a container from an image.

  • Describe at least two ways to run commands inside a running Docker container.

Reminder of terminology: images and containers

Recall that a container “image” is the template from which particular instances of containers will be created.

Let’s explore our first Docker container. The Docker team provides a simple container image online called hello-world. We’ll start with that one.

Downloading Docker images

The docker image command is used to list and modify Docker images. You can find out what container images you have on your computer by using the following command (“ls” is short for “list”):

$ docker image ls

If you’ve just installed Docker, you won’t see any images listed.

To get a copy of the hello-world Docker image from the internet, run this command:

$ docker pull hello-world

You should see output like this:

Using default tag: latest
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:f9dfddf63636d84ef479d645ab5885156ae030f611a56f3a7ac7f2fdd86d7e4e
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

DockerHub

Where did the hello-world image come from? It came from the DockerHub website, which is a place to share Docker images with other people. More on that in a later episode.

Exercise: Check on Your Images

What command would you use to see if the hello-world Docker image had downloaded successfully and was on your computer? Give it a try before checking the solution.

Solution

To see if the hello-world image is now on your computer, run:

$ docker image ls

Note that the downloaded hello-world image is not in the folder where you are in the terminal! (Run ls by itself to check.) The image is not a file like our normal programs and files; Docker stores it in a specific location that isn’t commonly accessed, so it’s necessary to use the special docker image command to see what Docker images you have on your computer.

Running the hello-world container

To create and run containers from named Docker images you use the docker run command. Try the following docker run invocation. Note that it does not matter what your current working directory is.

$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

What just happened? When we use the docker run command, Docker does three things:

1. Starts a Running Container 2. Performs Default Action 3. Shuts Down the Container
starts a running container, based on the image. Think of this as the “alive” or”inflated” version of the container – it’s actually doing something If the container has a default action set, it will perform that default action. This could be as simple as printing a message (as above) or running a whole analysis pipeline! Once the default action is complete, the container stops running (or exits). The image is still there, but nothing is actively running.

The hello-world container is set up to run an action by default - namely to print this message.

Using docker run to get the image

We could have skipped the docker pull step; if you use the docker run command and you don’t already have a copy of the Docker image, Docker will automatically pull the image first and then run it.

Running a container with a chosen command

But what if we wanted to do something different with the container? The output just gave us a suggestion of what to do – let’s use a different Docker image to explore what else we can do with the docker run command. The suggestion above is to use ubuntu, but we’re going to run a different type of Linux, alpine instead because it’s quicker to download.

Run the Alpine Docker container

Try downloading and running the alpine Docker container. You can do it in two steps, or one. What are they?

What happened when you ran the Alpine Docker container?

$ docker run alpine

Probably nothing! That’s because this particular container is designed for you to provide commands yourself. Try running this instead:

$ docker run alpine cat /proc/version

You should see the output of the cat /proc/version command, which prints out the version of Linux that this container is using.

Hello World, Part 2

Can you run the container and make it print a “hello world” message?

Give it a try before checking the solution.

Solution

Use the same command as above, but with the echo command to print a message.

$ docker run alpine echo 'Hello World'

So here, we see another option – we can provide commands at the end of the docker run command and they will execute inside the running container.

Running containers interactively

In all the examples above, Docker has started the container, run a command, and then immediately shut down the container. But what if we wanted to keep the container running so we could log into it and test drive more commands? The way to do this is by adding the interactive flag -it to the docker run command and by providing a shell (usually bash or sh) as our command.

$ docker run -it alpine sh

Technically…

Technically, the interactive flag is just -i, the extra -t (combined as -it above) is an option that allows you to connect to a shell like bash. But since usually you want to have a command line when run interactively, it always makes sense to use the two together.

Your prompt should change significantly to look like this:

/ #

That’s because you’re now inside the running container! Try these commands:

All of these are being run from inside the running container, so you’ll get information about the container itself, instead of your computer. To finish using the container, just type exit.

/ # exit

Practice Makes Perfect

Can you find out the version of Linux installed on the busybox container? Can you find the busybox program? What does it do? (Hint: passing --help to almost any command will give you more information.)

Solution 1 - Interactive

Run the busybox container interactively – you can use docker pull first, or just run it with this command:

$ docker run -it busybox sh

Then try, running these commands

/# cat /proc/version
/# busybox --help

Exit when you’re done.

/# exit

Solution 2 - Run commands

Run the busybox container, first with a command to read out the Linux version:

$ docker run busybox cat /proc/version

Then run the container again with a command to print out the busybox help:

$ docker run busybox busybox --help

Conclusion

So far, we’ve seen how to download Docker images, use them to run commands inside running containers, and even how to explore a running container from the inside. Next, we’ll take a closer look at all the different kinds of Docker images that are out there.

Key Points

  • The docker pull command downloads Docker images from the internet.

  • The docker image command lists Docker images that are (now) on your computer.

  • The docker run command creates running containers from images and can run commands inside them.

  • When using the docker run command, a container can run a default action (if it has one), a user specified action, or a shell to be used interactively.


Finding Containers on the Docker Hub

Overview

Teaching: 10 min
Exercises: 10 min
Questions
  • What is the Docker Hub, and why is it useful?

Objectives
  • Explain how the Docker Hub augments Docker use.

  • Explore the Docker Hub webpage for a popular Docker image.

  • Find the list of tags for a particular Docker image.

  • Identify the three components of a container’s identifier.

In the previous episode, we ran a few different containers: hello-world, alpine, and maybe busybox. Where did these containers come from? The Docker Hub!

Introducing the Docker Hub

The Docker Hub is an online repository of container images, a vast number of which are publicly available. A large number of the images are curated by the developers of the software that they package. Also, many commonly used pieces of software that have been containerised into images are specifically endorsed, which means that you can trust the containers to have been checked for functionality, stability, and that they don’t contain malware.

Docker can be used without connecting to the Docker Hub

Note that while the Docker Hub is well integrated into Docker functionality, the Docker Hub is certainly not required for all types of use of Docker containers. For example, some organisations may run container infrastructure that is entirely disconnected from the Internet.

Exploring an Example Docker Hub Page

As an example of a Docker Hub page, let’s explore the page for the python language. The most basic form of containerised python is in the “python” image (which is endorsed by the Docker team). Open your web browser to https://hub.docker.com/_/python to see what is on a typical Docker hub software page.

The top-left provides information about the name, short description, popularity (i.e., over a million downloads in the case of this image), and endorsements.

The top-right provides the command to pull this image to your computer.

The main body of the page contains many used headings, such as:

At least in my experience, the “Examples of how to use the image” section of most images’ pages will provide examples that are likely to adequately cover your intended use of the image.

Exploring Image Versions

A single Docker Hub page can have many different versions of container images, based on the version of the software inside. These versions are indicated by “tags”. When referring to the specific version of a container by its tag, you use a colon, :, like this:

CONTAINERNAME:TAG

So if I wanted to download the python container, with Python 3.8, I would use this name:

$ docker pull python:3.8

But if I wanted to download a Python 3.6 container, I would use this name:

$ docker pull python:3.6

The default tag (which is used if you don’t specify one) is called latest.

So far, we’ve only seen containers that are maintained by the Docker team. However, it’s equally common to use containers that have been produced by individual owners or organizations. Containers that you create and upload to Docker Hub would fall into this category, as would the containers maintained by organizations like ContinuumIO (the folks who develop the Anaconda Python environment) or community groups like rocker, a group that builds community R containers.

The name for these group- or individually-managed containers have this format:

OWNER/CONTAINERNAME:TAG

Repositories

The technical name for the contents of a Docker Hub page is a “repository.” The tag indicates the specific version of the container image that you’d like to use from a particular repository. So a slightly more accurate version of the above example is:

OWNER/REPOSITORY:TAG

What’s in a name?

How would I download the Docker container produced by the rocker group that has version 3.6.1 of R and the tidyverse installed?

Solution

First, search for rocker in Docker Hub. Then look for their tidyverse image. You can look at the list of tags, or just guess that the tag is 3.6.1. Altogether, that means that the name of the container we want to download is:

$ docker pull rocker/tidyverse:3.6.1

Many Different Containers

There are many different containers on Docker Hub. This is where the real advantage of using containers shows up – each container represents a complete software installation that you can use and access without any extra work!

The easiest way to find containers is to search on Docker Hub, but sometimes software pages have a link to their containers from their home page.

What container is right for you?

Find a Docker container that’s relevant to you. If you’re unsuccessful in your search, or don’t know what to look for, you can use the R or Python containers we’ve already seen.

Once you find a container, use the skills from the previous episode to download the image and explore it.

Key Points

  • The Docker Hub is an online repository of container images.

  • Many Docker Hub images are public, and may be officially endorsed.

  • Each Docker Hub page about an image provides structured information and subheadings

  • Most Docker Hub pages about images contain sections that provide examples of how to use those images.

  • Many Docker Hub images have multiple versions, indicated by tags.

  • The naming convention for Docker containers is: OWNER/CONTAINER:TAG


Cleaning Up Containers

Overview

Teaching: 10 min
Exercises: 0 min
Questions
  • How do I interact with a Docker container on my computer?

Objectives
  • Explain how to list running and completed containers.

Removing images

The images and their corresponding containers can start to take up a lot of disk space if you don’t clean them up occasionally, so it’s a good idea to periodically remove container images that you won’t be using anymore.

In order to remove a specific image, you need to find out details about the image, specifically, the “image ID”. For example say my laptop contained the following image.

$ docker image ls
REPOSITORY       TAG         IMAGE ID       CREATED          SIZE
hello-world      latest      fce289e99eb9   15 months ago    1.84kB

You can remove the image with a docker image rm command that includes the image ID, such as:

$ docker image rm fce289e99eb9

or use the image name, like so:

$ docker image rm hello-world

However, you may see this output:

Error response from daemon: conflict: unable to remove repository reference "hello-world" (must force) - container e7d3b76b00f4 is using its referenced image fce289e99eb9

This happens when Docker hasn’t cleaned up some of the times when a container has been actually run. So before removing the container image, we need to be able to see what containers are currently running, or have been run recently, and how to remove these.

What containers are running?

Working with containers, we are going to shift to a new docker command: docker container. Similar to docker image, we can list running containers by typing:

$ docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Notice that this command didn’t return any containers because our containers all exited and thus stopped running after they completed their work.

docker ps

The command docker ps serves the same purpose as docker container ls, and comes from the Unix shell command ps which describes running processes.

What containers have run recently?

There is also a way to list running containers, and those that have completed recently, which is to add the --all/-a flag to the docker container ls command as shown below.

$ docker container ls --all
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
9c698655416a        hello-world         "/hello"            2 minutes ago       Exited (0) 2 minutes ago                       zen_dubinsky
6dd822cf6ca9        hello-world         "/hello"            3 minutes ago       Exited (0) 3 minutes ago                       eager_engelbart

Keeping it clean

You might be surprised at the number of containers Docker is still keeping track of. One way to prevent this from happening is to add the --rm flag to docker run. This will completely wipe out the record of the run container when it exits. If you need a reference to the running container for any reason, don’t use this flag.

How do I remove an exited container?

To delete an exited container you can run the following command, inserting the CONTAINER ID for the container you wish to remove. It will repeat the CONTAINER ID back to you, if successful.

$ docker container rm 9c698655416a
9c698655416a

If you want to remove all exited containers at once you can use the docker containers prune command. Be careful with this command. If you have containers you may want to reconnect to, you should not use this command. It will ask you if to confirm you want to remove these containers, see output below. If successfull it will print the full CONTAINER ID back to you.

$ docker container prune
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
9c698655416a848278d16bb1352b97e72b7ea85884bff8f106877afe0210acfc
6dd822cf6ca92f3040eaecbd26ad2af63595f30bb7e7a20eacf4554f6ccc9b2b

Removing images, for real this time

Now that we’ve removed any potentially running or stopped containers, we can try again to delete the hello-world image.

$ docker image rm hello-world
Untagged: hello-world:latest
Untagged: hello-world@sha256:5f179596a7335398b805f036f7e8561b6f0e32cd30a32f5e19d17a3cda6cc33d
Deleted: sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e
Deleted: sha256:af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3

The reason that there are a few lines of output, is that a given image may have been formed by merging multiple underlying layers. Any layers that are used by multiple Docker images will only be stored once. Now the result of docker image ls should no longer include the hello-world image.

Key Points

  • The docker container command lists containers that have been created.


Creating your own container images

Overview

Teaching: 20 min
Exercises: 15 min
Questions
  • How can I make my own Docker images?

Objectives
  • Explain the purpose of a Dockerfile and show some simple examples.

  • Demonstrate how to build a Docker image from a Dockerfile.

  • Compare the steps of creating a container interactively versus a Dockerfile.

  • Create an installation strategy for a container

  • Demonstrate how to upload (‘push’) your container images to the Docker Hub.

  • Describe the significance of the Docker Hub naming scheme.

There are lots of reasons why you might want to create your own Docker image.

Interactive installation

Before creating a reproducible installation, let’s experiment with installing software inside a container. Start the alpine container from before, interactively:

$ docker run -it alpine sh

Because this is a basic container, there’s a lot of things not installed – for example, python.

/# python
sh: python: not found

Inside the container, we can run commands to install Python. The Alpine version of Linux has a installation tool called apk that we can use to install Python.

/# apk add --update python py-pip python-dev

We can test our installation by running a Python command:

/# python --version

Once Python is installed, we can add Python packages using the pip package installer:

/# pip install cython

Exercise: Searching for Help

Can you find instructions for installing R on Alpine Linux? Do they work?

Solution

A quick search should hopefully show that the way to install R on Alpine Linux is:

/# apk add R

Once we exit, these changes are not saved to a new container by default. There is a command that will “snapshot” our changes, but building containers this way is not very reproducible. Instead, we’re going to take what we’ve learned from this interactive installation and create our container from a reproducible recipe, known as a Dockerfile.

If you haven’t already, exit out of the interactively running container.

/# exit

Put installation instructions in a Dockerfile

A Dockerfile is a plain text file with keywords and commands that can be used to create a new container image.

From your shell, go to the folder you downloaded at the start of the lesson and print out the Dockerfile inside:

$ cd ~/Desktop/docker-intro/basic
$ cat Dockerfile
FROM <EXISTING IMAGE>
RUN <INSTALL CMDS FROM SHELL>
RUN <INSTALL CMDS FROM SHELL>
CMD <CMD TO RUN BY DEFUALT>

Let’s break this file down:

Exercise: Take a Guess

Do you have any ideas about what we should use to fill in the sample Dockerfile to replicate the installation we did above?

Solution:

Based on our experience above, edit the Dockerfile (in your text editor of choice) to look like this:

FROM alpine
RUN apk add --update python py-pip python-dev
RUN pip install cython
CMD cat /proc/version && python --version

The recipe provided by this Dockerfile will use Alpine Linux as the base container, add Python and the Cython library, and set a default print command.

Create a new Docker image

So far, we just have a file. We want Docker to take this file, run the install commands inside, and then save the resulting container as a new container image. To do this we will use the docker build command.

We have to provide docker build with two pieces of information:

$ docker build -t USERNAME/CONTAINERNAME .

The -t option names the container; the final dot indicates that the Dockerfile is in our current directory.

For example, if my user name was alice and I wanted to call my image alpine-python, I would use this command:

$ docker build -t alice/alpine-python .

Exercise: Review!

  1. Think back to earlier. What command can you run to check if your image was created successfully? (Hint: what command shows the images on your computer?)

  2. We didn’t specify a tag for our image name. What did Docker automatically use?

  3. What command will run the container you’ve created? What should happen by default if you run the container? Can you make it do something different, like print “hello world”?

Solution

  1. To see your new image, run docker image ls. You should see the name of your new image under the “REPOSITORY” heading.

  2. In the output of docker image ls, you can see that Docker has automatically used the latest tag for our new image.

  3. We want to use docker run to run the container.

$ docker run alice/alpine-python

should run the container and print out our default message, including the version of Linux and Python.

$ docker run alice/alpine-python echo "Hello World"

will run the container and print out “Hello world” instead.

While it may not look like you have achieved much, you have already effected the combination of a lightweight Linux operating system with your specification to run a given command that can operate reliably on macOS, Microsoft Windows, Linux and on the cloud!

Boring but important notes about installation

There are a lot of choices when it comes to installing software - sometimes too many! Here are some things to consider when creating your own container:

In general, a good strategy for installing software is:

TODO: Exercises

Have a set of “choose your own adventure” software installation examples

Share your new container on Docker Hub

Images that you release publicly can be stored on the Docker Hub for free. If you name your image as described above, with your Docker Hub username, all you need to do is run the opposite of docker pulldocker push.

$ docker push alice/alpine-python

Make sure to substitute the full name of your container!

In a web browser, open https://hub.docker.com, and on your user page you should now see your container listed, for anyone to use or build on.

Logging In

Technically, you have to be logged into Docker on your computer for this to work. Usually it happens by default, but if docker push doesn’t work for you, run docker login first, enter your Docker Hub username and password, and then try docker push again.

What’s in a name? (again)

You don’t have to name your containers using the USERNAME/CONTAINER:TAG naming> scheme. On your own computer, you can call containers whatever you want and refer to them by the names you choose. It’s only when you want to share a container that it needs the correct naming format.

You can rename images using the docker tag command. For example, imagine someone named Alice has been working on a workflow container and called it workflow-test on her own computer. She now wants to share it in her alice Docker Hub account with the name workflow-complete and a tag of v1. Her docker tag command would look like this:

$ docker tag workflow-test alice/workflow-complete:v1

She could then push the re-named container to Docker Hub, using docker push alice/workflow-complete:v1

Key Points

  • Dockerfiles specify what is within Docker images.

  • The docker build command is used to build an image from a Dockerfile

  • You can share your Docker images through the Docker Hub so that others can create Docker containers from your images.


Creating More Complex Container Images

Overview

Teaching: 30 min
Exercises: 30 min
Questions
  • How can I make more complex container images?

Objectives
  • Explain how you can include files within Docker images when you build them.

  • Explain how you can access files on the Docker host from your Docker containers.

In order to create and use your own containers, you may need more information than our previous example. You may want to use files from outside the container, copy those files into the container, and just generally learn a little bit about software installation. This episode will cover these. Note that the examples will get gradually more and more complex - most day-to-day use of containers can be accomplished using the first 1-2 sections on this page.

Using scripts and files from outside the container

In your shell, change to the sum folder in the docker-intro folder and look at the files inside.

$ cd ~/Desktop/docker-intro/sum
$ ls

This folder has both a Dockerfile and a python script called sum.py. Let’s say we wanted to try running the script using our recently created alpine-python container.

Running containers

What command would we use to run python from the alpine-python container?

If we try running the container and Python script, what happens?

$ docker run alice/alpine-python python sum.py
python: can't open file 'sum.py': [Errno 2] No such file or directory

No such file or directory

What does the error message mean? Why might the Python inside the container not be able to find or open our script?

The problem here is that the container and its file system is separate from our host computer’s file system. When the container runs, it can’t see anything outside itself, including any of the files on our computer. In order to use Python (inside the container) and our script (outside the container, on our computer), we need to create a link between the directory on our computer and the container.

This link is called a “mount” and is what happens automatically when a USB drive or other external hard drive gets connected to a computer - you can see the contents appear as if they were on your computer.

We can create a mount between our computer and the running container by using an additional option to docker run. We’ll also use the variable $PWD which will substitute in our current working directory. The option will look like this

-v $PWD:/temp

What this means is – link my current directory with the container, and inside the container, name the directory /temp

Let’s try running the command now:

$ docker run -v $PWD:/temp alice/alpine-python python sum.py

But we get the same error!

python: can't open file 'sum.py': [Errno 2] No such file or directory

This final piece is a bit tricky – we really have to remember to put ourselves inside the container. Where is the sum.py file? It’s in the directory that’s been mapped to /temp – so we need to include that in the path to the script. This command should give us what we need:

$ docker run -v $PWD:/temp alice/alpine-python python /temp/sum.py

Note that if we create any files in the /temp directory while the container is running, these files will appear on our host filesystem in the original directory and will stay there even when the container stops.

Exercise: Explore the script

What happens if you use the docker run command above and put numbers after the script name?

Solution

This script comes from the Python Wiki > > and is set to add all numbers that are passed to it as arguments.

Exercise: Checking the options

Our Docker command has gotten much longer! Can you go through each piece of the Docker command above the explain what it does? How would you characterize the key components of a Docker command?

Solution

Here’s a breakdown of each piece of the command above

  • docker run: use Docker to run a container
  • -v $PWD:/temp: connect my current working directory ($PWD) as a folder inside the container called /temp
  • alice/alpine-python: name of the container to run
  • python /temp/sum.py: what commands to run in the container

More generally, every Docker command will have the form: `docker [action] [docker options] [docker image] [command to run inside]

Exercise: Interactive jobs

Try using the directory mount option but run the container interactively. Can you find the folder that’s connected to your computer? What’s inside?

Solution

The docker command to run the container interactively is:

$ docker run -v $PWD:/temp -it alice/alpine-python sh

Once inside, you should be able to navigate to the /temp folder and see that’s contents are the same as the files on your computer:

/# cd /temp
/# ls

Mounting a folder can be very useful when you want to run the software inside your container on many different input files. In other situations, you may want to save or archive an authoritative version of your data by adding it to the container permanently. That’s what we will cover next.

Including personal scripts and data in a container

Our next project will be to add our own files to a container - something you might want to do if you’re sharing a finished analysis or just want to have an archived copy of your entire analysis including the data. Let’s assume that we’ve finished with our sum.py script and want to add it to the container itself.

In your shell, you should still be in the sum folder in the docker-intro folder.

$ pwd
$ /Users/yourname/Desktop/docker-intro/sum

Take a look at the Dockerfile. It looks similar to the one we used before, but it has an additional line with the COPY keyword.

COPY sum.py /home

This line will cause Docker to copy the file from your computer into the container’s file system. Let’s build the container like before, but give it a different name:

$ docker build -t alice/alpine-sum .

Exercise: Did it work?

Can you remember how to run a container interactively? Try that with this one. Once inside, try running the Python script.

Solution

You can start the container interactively like so:

$ docker run -it alice/alpine-sum sh

You should be able to run the python command inside the container like this:

/# python /home/sum.py

This COPY keyword can be used to place your own scripts or own data into a container that you want to publish or use as a record. Note that it’s not necessarily a good idea to put your scripts inside the container if you’re constantly changing or editing them. Then, referencing the scripts from outside the container is a good idea, as we did in the previous section. You also want to think carefully about size – if you run docker image ls you’ll see the size of each image all the way on the right of the screen. The bigger your image becomes, the harder it will be to easily download.

Copying alternatives

Another trick for getting your own files into a container is by using the RUN keyword and downloading the files from the internet. For example, if your code is in a GitHub repository, you could include this statement in your Dockerfile to download the latest version every time you build the container:

RUN git clone https://github.com/alice/mycode

Similarly, the wget command can be used to download any file publicly available on the internet:

RUN wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/2.10.0/ncbi-blast-2.10.0+-x64-linux.tar.gz

More fancy Dockerfile options (optional, for presentation or as exercises)

We can expand on the example above to make our container even more “automatic”. Here are some ideas:

FROM alpine

COPY sum.py /home
RUN apk add --update python py-pip python-dev

# Run the sum.py script as the default command
CMD python /home/sum.py
# OR
# CMD ["python", "/home/sum.py"]

Build and test it:

$ docker build -t alpine-sum:v1 .
$ docker run alpine-sum:v1
FROM alpine

COPY sum.py /home
RUN apk add --update python py-pip python-dev

# Run the sum.py script as the default command and
# allow people to enter arguments for it
ENTRYPOINT ["python", "/home/sum.py"]

Build and test it:

$ docker build -t alpine-sum:v2 .
$ docker run alpine-sum:v2 1 2 3 4
FROM alpine

COPY sum.py /home
# set script permissions
RUN chmod +x /home/sum.py
# add /home folder to the PATH
ENV PATH /home:$PATH

RUN apk add --update python py-pip python-dev

Build and test it:

$ docker build -t alpine-sum:v3 .
$ docker run alpine-sum:v3 sum.py 1 2 3 4

Key Points

  • You can include files from your Docker host into your Docker images by using the COPY instruction in your Dockerfile.

  • Docker allows containers to read and write files from the Docker host.


Creating containers in the cloud

Overview

Teaching: 20 min
Exercises: 0 min
Questions
  • How can I create Docker containers in the cloud?

Objectives
  • Demonstrate how to effect creation of a container from the Docker image in the cloud.

  • Gain an initial experience of the container functionality provided by the Bitbucket repository storage service.

Containers can be created on cloud computing platforms

There are lots of ways containers can be created on cloud computing platforms (a.k.a., “in the cloud”). Most commercial cloud providers now offer a container hosting service that will connect to the Docker Hub in order to fetch the container images that you specify, and charge for the time and resources that the containers use. The container hosting will usually be significantly cheaper than full virtualisation services that might be on offer.

Note also that most cloud providers will give you sign-up credit that you can use for free after you first create your account.

For this lesson, though, we instead use an excellent software project repository platform, Bitbucket, that allows users a monthly quota of minutes for which containers of your choice can be run. Bitbucket allows containers to be created in response to the modification of files within your software project.

There are many excellent sites for storing public software repositories

  • Note that Bitbucket, GitHub and GitLab all achieve similar functions.
  • Bitbucket offers container-based features that are easier to get at than the equivalent functions in GitHub, although GitHub will soon catch up when they release their GitHub Actions functionality publicly.

Running a container in the cloud, using your Bitbucket account

Because the ability to use the git version management tool is not a prerequisite of this session, we will use Bitbucket in an atypical manner. However we should still be able to clearly see Bitbucket’s cloud servers running a container of your choice, under your control.

Edit your repository’s bitbucket-pipelines.yml file through the web

You should be looking at a web-based text editor that is headed “bitbucket-pipelines.yml”. The one that I see has 13 lines, all of which are numbered on the left of the text editor.

Bitbucket Pipelines allow you to specify software tools to run, for example, in response to files being changed in your Bitbucket projects. The Bitbucket servers run your software tools within Docker containers, and thus Bitbucket Pipelines can specify Docker images to fetch from the Docker Hub.

Change your repository’s bitbucket-pipelines.yml file to be similar to the following example, but note that you need to replace my Docker Hub user ID (dme26) with yours. Also, ensure that your indentation steps in line-by-line, the language being used (YAML) gives significance to the indentation of the lines.

image: dme26/my-container

pipelines:
  default:
    - step:
        script:
          - /bin/cat /root/my_message

Click the “commit file” button. After you commit your bitbucket-pipelines.yml file, the Bitbucket Pipeline will download the Docker image you specified from the Docker Hub, and display the progress of the computations it runs.

When using the docker run command (as you have done previously), the container takes some default actions after being created, which are specified in your Dockerfile (e.g., the CMD line). Bitbucket Pipelines disable these default actions, instead using the commands listed under the “script:” section in your bitbucket-pipelines.yml. Note that hyphens at the same indentation level are treated as an itemised list. There is only one item in our script: list, namely the command /bin/cat /root/my_message.

If the pipeline runs successfully, a green heading containing a tick icon will be shown near the top of the page. On the right-hand-side of the page, you should see the following headings:

Click on the /bin/cat /root/my_message heading, and you should see that your custom message was shown.

While it is difficult to argue that this container achieves important computational work, you have, nonetheless, demonstrated that Docker images that you create can be run on the cloud. Moreover, many cloud organisations that are willing to create containers from your images will offer generous allowances to you to do so, even if you only have a free account.

Key Points

  • You can create Docker containers on cloud computing resources just using a web browser.

  • Bitbucket is an online repository storage service that can create Docker containers to perform computation in response to files changing in your repositories.


Containers used in generating this lesson

Overview

Teaching: 20 min
Exercises: 0 min
Questions
  • How can containers be useful to me for building websites?

Objectives
  • Demonstrate how to construct a website using containers to transform a specification into a fully-presented website.

The website for this lesson is generated mechanically, based on a set of files that specify the configuration of the site, its presentation template, and the content to go on this page. This is far more manageable than editing each webpage of the lesson separately, for example, if the page header needs to change, this change can be made in one place, and all the pages regenerated. The alternative would be needing to edit each page to repeat the change: this is not productive or suitable work for humans to do!

In your shell window, in your container-playground create a new directory copy-of-docker-intro and cd into it. We will later be expanding a ZIP file into this directory later.

Now open a web browser window and:

  1. Navigate to the GitHub repository that contains the files for this session, at https://github.com/dme26/docker-introduction/;
  2. Click the green “Clone or download” button on the right-hand side of the page;
  3. Click “Download ZIP”.
  4. The downloaded ZIP file should contain one directory named docker-introduction-gh-pages.
  5. Move the docker-introduction-gh-pages folder into the copy-of-docker-intro folder you created above.

There are many ways to work with ZIP files

Note that the last two steps can be achieved using a Mac or Windows graphical user interface. There are also ways to effect expanding the ZIP archive on the command line, for example, on my Mac I can achieve the effect of those last two steps through running the command unzip ~/Downloads/docker-introduction-gh-pages.zip.

In your shell window, if you cd into the docker-introduction-gh-pages folder and list the files, you should see something similar to what I see:

$ cd docker-introduction-gh-pages
$ ls
AUTHORS			_episodes		code
CITATION		_episodes_rmd		data
CODE_OF_CONDUCT.md	_extras			fig
CONTRIBUTING.md		_includes		files
LICENSE.md		_layouts		index.md
Makefile		aio.md			reference.md
README.md		assets			setup.md
_config.yml		bin

You can now request that a container is created that will compile the files in this set into the lesson website, and will run a simple webserver to allow you to view your version of the website locally. Note that this command will be long and fiddly to type, so you probably want to copy-and-paste it into your shell window. This command will continue to (re-)generate and serve up your version of the lesson website, so you will not get your shell prompt back until you type control+c. This will stop the webserver, since it cleans away the container.

If you happen to have the make tool already installed…

We are taking an atypical approach in using the command that follows, since you are not required to have set up the make tool on your computer. However you may want to see whether you happen to have make installed anyway, by typing make docker-serve instead of the command below. At worst, this will fail and you can use the command shown below.

For macOS, Linux and PowerShell:

$ docker run --rm -it -v ${PWD}:/srv/jekyll -p 127.0.0.1:4000:4000 jekyll/jekyll:3.7.3 make serve

For cmd.exe shells on Microsoft Windows:

> docker run --rm -it -v "%CD%":/srv/jekyll -p 127.0.0.1:4000:4000 jekyll/jekyll:3.7.3 make serve

When I ran the macOS command, the output was as follows:

Unable to find image 'jekyll/jekyll:3.7.3' locally
3.7.3: Pulling from jekyll/jekyll
ff3a5c916c92: Pull complete 
8e2da6035957: Pull complete 
42e99ed6de92: Pull complete 
70c638bbd0d9: Pull complete 
8f8df9937b34: Pull complete 
Digest: sha256:2b907c5f836ee66d6dde39aa021eebadcadd59dffab693ceecb73be7cfa2808b
Status: Downloaded newer image for jekyll/jekyll:3.7.3
jekyll serve
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-musl]
Configuration file: /srv/jekyll/_config.yml
            Source: /srv/jekyll
       Destination: /srv/jekyll/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
                    done in 2.647 seconds.
 Auto-regeneration: enabled for '/srv/jekyll'
    Server address: http://0.0.0.0:4000
  Server running... press ctrl-c to stop.
[2019-02-07 15:37:35] ERROR `/assets/favicons/favicon-96x96.png' not found.
[2019-02-07 15:37:35] ERROR `/assets/favicons/favicon-196x196.png' not found.
[2019-02-07 15:37:35] ERROR `/assets/favicons/favicon-16x16.png' not found.
[2019-02-07 15:37:35] ERROR `/assets/favicons/favicon-128.png' not found.
[2019-02-07 15:37:35] ERROR `/assets/favicons/favicon-32x32.png' not found.

In the preceding output, you see Docker downloading the image for Jekyll, which is a tool for building websites from specification files such as those used for this lesson. The line jekyll serve indicates a command that runs within the Docker container instance. The output below that is from the Jekyll tool itself, highlighting that the website has been built, and indicating that there is a server running.

Open a web browser window and visit the address http://localhost:4000/. You should see a site that looks very similar to that at https://dme26.github.io/docker-introduction/.

Using a new shell window, or using your laptop’s GUI, locate the file index.md within the docker-introduction-gh-pages directory, and open it in your preferred editor program.

Near the top of this file you should see the description starting “This session aims to introduce the use of Docker containers with the goal of using them to effect reproducible computational environments.” Make a change to this message, and save the file.

If you reload your web browser, the change that you just made should be visible. This is because the Jekyll container saw that you changed the index.md file, and regenerated the website.

You can stop the Jekyll container by clicking in its terminal window and typing control+c.

You have now achieved using a reproducible computational environment to reproduce a lesson about reproducible computing environments.

Key Points

  • The generation of this lesson website can be effected using a container.