This lesson is being piloted (Beta version)
If you teach this lesson, please tell the authors and provide feedback by opening an issue in the source repository

Intermediate Research Software Development: Setup

You will need the following software installed and working correctly on your system to be able to follow the course.

Common Issues & Tips

If you are having issues installing or running some of the tools below, check a list of common issues other course participants encountered and some useful tips for using the tools and working through the material.

Command Line Tool

You will need a command line tool (shell/console) in order to run Python scripts and version control your code with Git.

To test your command line tool, start it up and type:

$ date

If your command line program is working - it should return the current date and time similar to:

Wed 21 Apr 2021 11:38:19 BST

Git Version Control Tool

Git is a program that can be accessed from your command line tool.

To test your Git installation, start your command line tool and type:

$ git help

If your Git installation is working you should see something like:

usage: git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           [--config-env=<name>=<envvar>] <command> [<args>]

These are common Git commands used in various situations:

start a working area (see also: git help tutorial)
   clone     Clone a repository into a new directory
   init      Create an empty Git repository or reinitialize an existing one

work on the current change (see also: git help everyday)
   add       Add file contents to the index
   mv        Move or rename a file, a directory, or a symlink
   restore   Restore working tree files
   rm        Remove files from the working tree and from the index

examine the history and state (see also: git help revisions)
   bisect    Use binary search to find the commit that introduced a bug
   diff      Show changes between commits, commit and working tree, etc
   grep      Print lines matching a pattern
   log       Show commit logs
   show      Show various types of objects
   status    Show the working tree status

grow, mark and tweak your common history
   branch    List, create, or delete branches
   commit    Record changes to the repository
   merge     Join two or more development histories together
   rebase    Reapply commits on top of another base tip
   reset     Reset current HEAD to the specified state
   switch    Switch branches
   tag       Create, list, delete or verify a tag object signed with GPG

collaborate (see also: git help workflows)
   fetch     Download objects and refs from another repository
   pull      Fetch from and integrate with another repository or a local branch
   push      Update remote refs along with associated objects

'git help -a' and 'git help -g' list available subcommands and some
concept guides. See 'git help <command>' or 'git help <concept>'
to read about a specific subcommand or concept.
See 'git help git' for an overview of the system.

When you use Git on a machine for the first time, you need to configure a few things:

This can be done from the command line as follows:

$ git config --global user.name "Your Name"
$ git config --global user.email "name@example.com"
$ git config --global core.editor "nano -w"

GitHub Account

GitHub is a free, online host for Git repositories that you will use during the course to store your code in so you will need to open a free GitHub account unless you do not already have one.

Secure Access To GitHub Using Git From Command Line

In order to access GitHub using Git from your machine securely, you need to set up a way of authenticating yourself with GitHub through Git. The recommended way to do that for this course is to set up SSH authentication - a method of authentication that is more secure than sending passwords over HTTPS and which requires a pair of keys - one public that you upload to your GitHub account, and one private that remains on your machine.

GitHub provides full documentation and guides on how to:

A short summary of the commands you need to perform is shown below.

To generate an SSH key pair, you will need to run the ssh-keygen command from your command line tool/GitBash and provide your identity for the key pair (e.g. the email address you used to register with GitHub) via the -C parameter as shown below. Note that the ssh-keygen command can be run with different parameters - e.g. to select a specific public key algorithm and key length; if you do not use them ssh-keygen will generate an RSA key pair for you by default. GitHub now recommends that you use a newer cryptographic standard (such as EdDSA variant algorithm Ed25519), so please be sure to specify it using the -t flag as shown below. It will also prompt you to answer a few questions - e.g. where to save the keys on your machine and a passphrase to use to protect your private key. Pressing ‘Enter’ on these prompts will get ssh-keygen to use the default key location (within .ssh folder in your home directory) and set the passphrase to empty.

$ ssh-keygen -t ed25519 -C "your-github-email@example.com"

Generating public/private ed25519 key pair.
Enter file in which to save the key (/Users/<YOUR_USERNAME>/.ssh/id_ed25519): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/<YOUR_USERNAME>/.ssh/id_ed25519
Your public key has been saved in /Users/<YOUR_USERNAME>/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:qjhN/iO42nnYmlpink2UTzaJpP8084yx6L2iQkVKdHk your-github-email@example.com
The key's randomart image is:
+--[ED25519 256]--+
|.. ..            |
| ..o A           |
|. o..            |
| .o.o .          |
| ..+ =  B        |
| .o = ..         |
|o..X *.          |
|++B=@.X          |
|+*XOoOo+         |
+----[SHA256]-----+

Next, you need to copy your public key (not your private key - this is important!) over to your GitHub account. The ssh-keygen command above will let you know where your public key is saved (the file should have the extension “.pub”), and you can get its contents (e.g. on a Mac OS system) as follows:

$ cat /Users/<YOUR_USERNAME>/.ssh/id_ed25519.pub

ssh-ed25519 AABAC3NzaC1lZDI1NTE5AAAAICWGVRsl/pZsxx85QHLwSgJWyfMB1L8RCkEvYNkP4mZC your-github-email@example.com

Copy the line of output that starts with “ssh-ed25519” and ends with your email address (it may start with a different algorithm name based on which one you used to generate the key pair and it may have gone over multiple lines if your command line window is not wide enough).

Finally, go to your GitHub Settings -> SSH and GPG keys -> Add New page to add a new SSH public key. Give your key a memorable name (e.g. the name of the computer you are working on that contains the private key counterpart), paste the public key from your clipboard into the box labelled “Key” (making sure it does not contain any line breaks), then click the “Add SSH key” button.

Now, we can check that the SSH connection is working:

$ ssh -T git@github.com

What About Passwords?

While using passwords over HTTPS for authentication is easier to setup and will allow you read access to your repository on GitHub from your machine, it alone is not sufficient any more to allow you to send changes or write to your remote repository on GitHub. This is because, on 13 August 2021, GitHub has strengthened security requirements for all authenticated Git operations. This means you would need to use a personal access token instead of your password for added security each time you need to authenticate yourself to GitHub from the command line (e.g. when you want to push your local changes to your code repository on GitHub). While using SSH key pair for authentication may seem complex, once set up, it is actually more convenient than keeping track of/caching your access token.

Python 3 Distribution

To download the latest Python 3 distribution for your operating system, please head to Python.org.

If you are on Linux, it is likely that the system Python 3 already installed will satisfy the requirements of this course (the material has been tested using the standard Python distribution version 3.11 but any supported version should work).

The course uses venv for virtual environment management and pip for package management. The material has not been extensively tested with other Python distributions and package managers, but most sections are expected to work with some modifications. For example, package installation and virtual environments would need to be managed differently, but Python script invocations should remain the same regardless of the Python distribution used.

We recommend using the latest Python version but any supported version should work. Specifically, we recommend upgrading from Python 2.7 wherever possible; continuing to use it will likely result in difficulty finding supported dependencies or syntax errors.

You can test your Python installation from the command line with:

$ python3 --version # on Mac/Linux
$ python --version # on Windows — Windows installation comes with a python.exe file rather than a python3.exe file

If you are using Windows and invoking python command causes your Git Bash terminal to hang with no error message or output, you may need to create an alias for the python executable python.exe, as explained in the troubleshooting section.

If all is well with your installation, you should see something like:

Python 3.11.4

To make sure you are using the standard Python distribution and not some other distribution you may have on your system, type the following in your shell:

 $ python3 # python on Windows

This should enter you into a Python console and you should see something like:

Python 3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information. 
>>>

Press CONTROL-D or type exit() to exit the Python console.

venv and pip

If you are using a Python 3 distribution from Python.org, venv and pip will be automatically installed for you. If not, please make sure you have these two tools (that correspond to your Python distribution) installed on your machine.

PyCharm IDE

We use JetBrains’s PyCharm Python Integrated Development Environment for the course. PyCharm can be downloaded from the JetBrains website. The Community edition is fine, though if you are developing software for the purpose of academic research you may be eligible for a free license for the Professional edition which contains extra features.