Manual Pages

Overview

Teaching: 10 min
Exercises: 0 min

Questions

• How to use man pages?

Objectives

• Use man to display the manual page for a given command.

• Explain how to read the synopsis of a given command while using man.

• Search for specific options or flags in the manual page for a given command.

We can get help for any Unix command with the man (short for manual) command. For example, here is the command to look up information on cp:

$ man cp

The output displayed is referred to as the “man page”.

Most man pages contain much more information than can fit in one terminal screen.
To help facilitate reading, the man command tries to use a “pager” to move and search through the information screenfull by screenfull. The most common pager is called less. Detailed information is available using man less. less is typically the default pager for Unix systems and other tools may use it for output paging as well.

When less displays a colon ‘:’, we can press the space bar to get the next page, the letter ‘h’ to get help, or the letter ‘q’ to quit.

man’s output is typically complete but concise, as it is designed to be used as a reference rather than a tutorial. Most man pages are divided into sections:

• NAME: gives the name of the command and a brief description
• SYNOPSIS: how to run the command, including optional and mandatory parameters. (We will explain the syntax later.)
• DESCRIPTION: a fuller description than the synopsis, including a description of all the options to the command. This section may also include example usage or details about how the command works.
• EXAMPLES: self-explanatory.
• SEE ALSO: list other commands that we might find useful or other sources of information that might help us.

Other sections we might see include AUTHOR, REPORTING BUGS, COPYRIGHT, HISTORY, (known) BUGS, and COMPATIBILITY.

How to Read the Synopsis

Here is the is synopsis for the cp command on Ubuntu Linux:

SYNOPSIS
   cp [OPTION]... [-T] SOURCE DEST
   cp [OPTION]... SOURCE... DIRECTORY
   cp [OPTION]... -t DIRECTORY SOURCE...

This tells the reader that there are three ways to use the command. Let’s look at the first usage:

cp [OPTION]... [-T] SOURCE DEST

[OPTION] means the cp command can be followed by one or more optional flags. We can tell they’re optional because of the square brackets, and we can tell that one or more are welcome because of the ellipsis (…). For example, the fact that [-T] is in square brackets, but after the ellipsis, means that it’s optional, but if it’s used, it must come after all the other options.

SOURCE refers to the source file or directory, and DEST to the destination file or directory. Their precise meanings are explained at the top of the DESCRIPTION section.

The other two usage examples can be read in similar ways. Note that to use the last one, the -t option is mandatory (because it isn’t shown in square brackets).

The DESCRIPTION section starts with a few paragraphs explaining the command and its use, then expands on the possible options one by one:

     The following options are available:

-a    Same as -pPR options. Preserves structure and attributes of
           files but not directory structure.

     -f    If the destination file cannot be opened, remove it and create
           a new file, without prompting for confirmation regardless of
           its permissions.  (The -f option overrides any previous -n
           option.)

           The target file is not unlinked before the copy.  Thus, any
           existing access rights will be retained.

      ...  ...

Finding Help on Specific Options

If we want to skip ahead to the option you’re interested in, we can search for it using the slash key ‘/’. (This isn’t part of the man command: it’s a feature of less.) For example, to find out about -t, we can type /-t and press return. After that, we can use the ‘n’ key to navigate to the next match until we find the detailed information we need:

-t, --target-directory=DIRECTORY
     copy all SOURCE arguments into DIRECTORY

This means that this option has the short form -t and the long form --target-directory and that it takes an argument. Its meaning is to copy all the SOURCE arguments into DIRECTORY. Thus, we can give the destination explicitly instead of relying on having to place the directory at the end.

Limitations of Man Pages

Man pages can be useful for a quick confirmation of how to run a command, but they are not famous for being readable. If you can’t find what you need in the man page— or you can’t understand what you’ve found— try entering “unix command copy file” into your favorite search engine: it will often produce more helpful results.

You May Also Enjoy…

The explainshell.com site does a great job of breaking complex Unix commands into parts and explaining what each does. Sadly, it doesn’t work in reverse…

Key Points

• man command displays the manual page for a given command.

• [OPTION]... means the given command can be followed by one or more optional flags.

• Flags specified after ellipsis are still optional but must come after all other flags.

• While inside the manual page,use / followed by your pattern to do interactive searching.

Working Remotely

Overview

Teaching: 10 min
Exercises: 0 min

Questions

• How do I use ‘ssh’ and ‘scp’ ?

Objectives

• Learn what SSH is

• Learn what an SSH key is

• Generate your own SSH key pair

• Learn how to use your SSH key

• Learn how to work remotely using ssh and scp

• Add your SSH key to an remote server

Let’s take a closer look at what happens when we use the shell on a desktop or laptop computer. The first step is to log in so that the operating system knows who we are and what we’re allowed to do. We do this by typing our username and password; the operating system checks those values against its records, and if they match, runs a shell for us.

As we type commands, the 1’s and 0’s that represent the characters we’re typing are sent from the keyboard to the shell. The shell displays those characters on the screen to represent what we type, and then, if what we typed was a command, the shell executes it and displays its output (if any).

What if we want to run some commands on another machine, such as the server in the basement that manages our database of experimental results? To do this, we have to first log in to that machine. We call this a remote login.

In order for us to be able to login, the remote computer must be running a remote login server and we will run a client program that can talk to that server. The client program passes our login credentials to the remote login server and, if we are allowed to login, that server then runs a shell for us on the remote computer.

Once our local client is connected to the remote server, everything we type into the client is passed on, by the server, to the shell running on the remote computer. That remote shell runs those commands on our behalf, just as a local shell would, then sends back output, via the server, to our client, for our computer to display.

SSH History

Back in the day, when everyone trusted each other and knew every chip in their computer by its first name, people didn’t encrypt anything except the most sensitive information when sending it over a network and the two programs used for running a shell (usually back then, the Bourne Shell, sh) on, or copying files to, a remote machine were named rsh and rcp, respectively. Think (r)emote sh and cp

However, anyone could watch the unencrypted network traffic, which meant that villains could steal usernames and passwords, and use them for all manner of nefarious purposes.

The SSH protocol was invented to prevent this (or at least slow it down). It uses several sophisticated, and heavily tested, encryption protocols to ensure that outsiders can’t see what’s in the messages going back and forth between different computers.

The remote login server which accepts connections from client programs is known as the SSH daemon, or sshd.

The client program we use to login remotely is the secure shell, or ssh, think (s)ecure sh.

The ssh login client has a companion program called scp, think (s)ecure cp, which allows us to copy files to or from a remote computer using the same kind of encrypted connection.

A remote login using ssh

To make a remote login, we issue the command ssh username@computer which tries to make a connection to the SSH daemon running on the remote computer we have specified.

After we log in, we can use the remote shell to use the remote computer’s files and directories.

Typing exit or Control-D terminates the remote shell, and the local client program, and returns us to our previous shell.

In the example below, the remote machine’s command prompt is moon> instead of just $. To make it clearer which machine is doing what, we’ll indent the commands sent to the remote machine and their output.

$ pwd

/users/vlad

$ ssh vlad@moon.euphoric.edu
Password: ********

    moon> hostname

    moon

    moon> pwd

    /home/vlad

    moon> ls -F

    bin/     cheese.txt   dark_side/   rocks.cfg

    moon> exit

$ pwd

/users/vlad

Copying files to, and from a remote machine using scp

To copy a file, we specify the source and destination paths, either of which may include computer names. If we leave out a computer name, scp assumes we mean the machine we’re running on. For example, this command copies our latest results to the backup server in the basement, printing out its progress as it does so:

$ scp results.dat vlad@backupserver:backups/results-2011-11-11.dat
Password: ********

results.dat              100%  9  1.0 MB/s 00:00

Note the colon :, seperating the hostname of the server and the pathname of the file we are copying to. It is this character that informs scp that the source or target of the copy is on the remote machine and the reason it is needed can be explained as follows:

In the same way that the default directory into which we are placed when running a shell on a remote machine is our home directory on that machine, the default target, for a remote copy, is also the home directory.

This means that

$ scp results.dat vlad@backupserver:

would copy results.dat into our home directory on backupserver, however, if we did not have the colon to inform scp of the remote machine, we would still have a valid commmad

$ scp results.dat vlad@backupserver

but now we have merely created a file called vlad@backupserver on our local machine, as we would have done with cp.

$ cp results.dat vlad@backupserver

Copying a whole directory betwen remote machines uses the same syntax as the cp command: we just use the -r option to signal that we want copying to be recursively. For example, this command copies all of our results from the backup server to our laptop:

$ scp -r vlad@backupserver:backups ./backups
Password: ********

results-2011-09-18.dat              100%  7  1.0 MB/s 00:00
results-2011-10-04.dat              100%  9  1.0 MB/s 00:00
results-2011-10-28.dat              100%  8  1.0 MB/s 00:00
results-2011-11-11.dat              100%  9  1.0 MB/s 00:00

Running commands on a remote machine using ssh

Here’s one more thing the ssh client program can do for us. Suppose we want to check whether we have already created the file backups/results-2011-11-12.dat on the backup server. Instead of logging in and then typing ls, we could do this:

$ ssh vlad@backupserver "ls results*"
Password: ********

results-2011-09-18.dat  results-2011-10-28.dat
results-2011-10-04.dat  results-2011-11-11.dat

Here, ssh takes the argument after our remote username and passes them to the shell on the remote computer. (We have to put quotes around it to make it look like a single argument.) Since those arguments are a legal command, the remote shell runs ls results for us and sends the output back to our local shell for display.

SSH Keys

Typing our password over and over again is annoying, especially if the commands we want to run remotely are in a loop. To remove the need to do this, we can create an SSH key to tell the remote machine that it should always trust us.

SSH keys come in pairs, a public key that gets shared with services like GitHub, and a private key that is stored only on your computer. If the keys match, you’re granted access.

The cryptography behind SSH keys ensures that no one can reverse engineer your private key from the public one.

The first step in using SSH authorization is to generate your own key pair.

You might already have an SSH key pair on your machine. You can check to see if one exists by moving to your .ssh directory and listing the contents.

$ cd ~/.ssh
$ ls

If you see id_rsa.pub, you already have a key pair and don’t need to create a new one.

If you don’t see id_rsa.pub, use the following command to generate a new key pair. Make sure to replace your@email.com with your own email address.

$ ssh-keygen -t rsa -C "your@email.com"

When asked where to save the new key, hit enter to accept the default location.

Generating public/private rsa key pair.
Enter file in which to save the key (/Users/username/.ssh/id_rsa):

You will then be asked to provide an optional passphrase. This can be used to make your key even more secure, but if what you want is avoiding type your password every time you can skip it by hitting enter twice.

Enter passphrase (empty for no passphrase):
Enter same passphrase again:

When the key generation is complete, you should see the following confirmation:

Your identification has been saved in /Users/username/.ssh/id_rsa.
Your public key has been saved in /Users/username/.ssh/id_rsa.pub.
The key fingerprint is:
01:0f:f4:3b:ca:85:d6:17:a1:7d:f0:68:9d:f0:a2:db your@email.com
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|        . E +    |
|       . o = .   |
|      . S =   o  |
|       o.O . o   |
|       o .+ .    |
|      . o+..     |
|       .+=o      |
+-----------------+

The random art image is an alternate way to match keys but we won’t be needing this.

Now you need to place a copy of your public key ony any servers you would like to use SSH to connect to, instead of logging in with a username and passwd.

Display the contents of your new public key file with cat:

$ cat ~/.ssh/id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA879BJGYlPTLIuc9/R5MYiN4yc/YiCLcdBpSdzgK9Dt0Bkfe3rSz5cPm4wmehdE7GkVFXrBJ2YHqPLuM1yx1AUxIebpwlIl9f/aUHOts9eVnVh4NztPy0iSU/Sv0b2ODQQvcy2vYcujlorscl8JjAgfWsO3W4iGEe6QwBpVomcME8IU35v5VbylM9ORQa6wvZMVrPECBvwItTY8cPWH3MGZiK/74eHbSLKA4PY3gM4GHI450Nie16yggEg2aTQfWA1rry9JYWEoHS9pJ1dnLqZU3k/8OWgqJrilwSoC5rGjgp93iu0H8T6+mEHGRQe84Nk1y5lESSWIbn6P636Bl3uQ== your@email.com

Copy the contents of the output.

Login to the remote server with your username and password.

$ ssh vlad@moon.euphoric.edu
Password: ********

Paste the content that you copy at the end of ~/.ssh/authorized_keys.

    moon> nano ~/.ssh/authorized_keys

After append the content, logout of the remote machine and try login again. If you setup your SSH key correctly you won’t need to type your password.

    moon> exit

$ ssh vlad@moon.euphoric.edu

SSH Files and Directories

The example of copying our public key to a remote machine, so that it can then be used when we next SSH into that remote machine, assumed that we already had a directory ~/.ssh/.

Whilst a remote server may support the use of SSH to login, your home directory there may not contain a .ssh directory by default.

We have already seen that we can use SSH to run commands on remote machines, so we can ensure that everything is set up as required before we place the copy of our public key on a remote machine.

Walking through this process allows us to highlight some of the typical requirements of the SSH protocol itself, as documented in the man-page for the ssh command.

Firstly, we check that we have a .ssh/ directory on another remote machine, comet

$ ssh vlad@comet "ls -ld ~/.ssh"
Password: ********

    ls: cannot access /home/vlad/.ssh: No such file or directory

Oh dear! We should create the directory; and check that it’s there (Note: two commands, seperated by a semicolon)

$ ssh vlad@comet "mkdir ~/.ssh; ls -ld ~/.ssh"
Password: ********

    drwxr-xr-x 2 vlad vlad 512 Jan 01 09:09 /home/vlad/.ssh

Now we have a dot-SSH directory, into which to place SSH-related files but we can see that the default permissions allow anyone to inspect the files within that directory.

For a protocol that is supposed to be secure, this is not considered a good thing and so the recommended permissions are read/write/execute for the user, and not accessible by others.

Let’s alter the permissions on the directory

$ ssh vlad@comet "chmod 700 ~/.ssh; ls -ld ~/.ssh"
Password: ********

    drwx------ 2 vlad vlad 512 Jan 01 09:09 /home/vlad/.ssh

That’s looks much better.

In the above example, it was suggested that we paste the content of our public key at the end of ~/.ssh/authorized_keys, however as we didn’t have a ~/.ssh/ on this remote machine, we can simply copy our public key over as the initial ~/.ssh/authorized_keys, and of course, we will use scp to do this, even though we don’t yet have passwordless SSH access set up.

$ scp ~/.ssh/id_rsa.pub vlad@comet:.ssh/authorized_keys
Password: ********

Note that the default target for the scp command on a remote machine is the home directory, so we have not needed to use the shorthand ~/.ssh/ or even the full path /home/vlad/.ssh/ to our home directory there.

Checking the permissions of the file we have just created on the remote machine, also serves to indicate that we no longer need to use our password, because we now have what’s needed to use SSH without it.

$ ssh vlad@comet "ls -l ~/.ssh"

    -rw-r--r-- 2 vlad vlad 512 Jan 01 09:11 /home/vlad/.ssh/authorized_keys

Whilst the authorized keys file is not considered to be highly sensitive, (after all, it contains public keys), we alter the permissions to match the man page’s recommendations

$ ssh vlad@comet "chmod go-r ~/.ssh/authorized_keys; ls -l ~/.ssh"

    -rw------- 2 vlad vlad 512 Jan 01 09:11 /home/vlad/.ssh/authorized_keys

Key Points

• SSH is a secure alternative to username/password authorization

• SSH keys are generated in public/private pairs. Your public key can be shared with others. The private keys stays on your machine only.

• The ‘ssh’ and ‘scp’ utilities are secure alternatives to logging into, and copying files to/from remote machine

Permissions

Overview

Teaching: 10 min
Exercises: 0 min

Questions

• Understanding file/directory permissions

Objectives

• What are file/directory permissions?

• How to view permissions?

• How to change permissions?

• File/directory permissions in Windows

Unix controls who can read, modify, and run files using permissions. We’ll discuss how Windows handles permissions at the end of the section: the concepts are similar, but the rules are different.

Let’s start with Nelle. She has a unique user name, nnemo, and a user ID, 1404.

Why Integer IDs?

Why integers for IDs? Again, the answer goes back to the early 1970s. Character strings like alan.turing are of varying length, and comparing one to another takes many instructions. Integers, on the other hand, use a fairly small amount of storage (typically four characters), and can be compared with a single instruction. To make operations fast and simple, programmers often keep track of things internally using integers, then use a lookup table of some kind to translate those integers into user-friendly text for presentation. Of course, programmers being programmers, they will often skip the user-friendly string part and just use the integers, in the same way that someone working in a lab might talk about Experiment 28 instead of “the chronotypical alpha-response trials on anacondas”.

Users can belong to any number of groups, each of which has a unique group name and numeric group ID. The list of who’s in what group is usually stored in the file /etc/group. (If you’re in front of a Unix machine right now, try running cat /etc/group to look at that file.)

Now let’s look at files and directories. Every file and directory on a Unix computer belongs to one owner and one group. Along with each file’s content, the operating system stores the numeric IDs of the user and group that own it.

The user-and-group model means that for each file every user on the system falls into one of three categories: the owner of the file, someone in the file’s group, and everyone else.

For each of these three categories, the computer keeps track of whether people in that category can read the file, write to the file, or execute the file (i.e., run it if it is a program).

For example, if a file had the following set of permissions:

it would mean that:

• the file’s owner can read and write it, but not run it;
• other people in the file’s group can read it, but not modify it or run it; and
• everybody else can do nothing with it at all.

Let’s look at this model in action. If we cd into the labs directory and run ls -F, it puts a * at the end of setup’s name. This is its way of telling us that setup is executable, i.e., that it’s (probably) something the computer can run.

$ cd labs
$ ls -F

safety.txt    setup*     waiver.txt

Necessary But Not Sufficient

The fact that something is marked as executable doesn’t actually mean it contains a program of some kind. We could easily mark this HTML file as executable using the commands that are introduced below. Depending on the operating system we’re using, trying to “run” it will either fail (because it doesn’t contain instructions the computer recognizes) or cause the operating system to open the file with whatever application usually handles it (such as a web browser).

Now let’s run the command ls -l:

$ ls -l

-rw-rw-r-- 1 vlad bio  1158  2010-07-11 08:22 safety.txt
-rwxr-xr-x 1 vlad bio 31988  2010-07-23 20:04 setup
-rw-rw-r-- 1 vlad bio  2312  2010-07-11 08:23 waiver.txt

The -l flag tells ls to give us a long-form listing. It’s a lot of information, so let’s go through the columns in turn.

On the right side, we have the files’ names. Next to them, moving left, are the times and dates they were last modified. Backup systems and other tools use this information in a variety of ways, but you can use it to tell when you (or anyone else with permission) last changed a file.

Next to the modification time is the file’s size in bytes and the names of the user and group that owns it (in this case, vlad and bio respectively). We’ll skip over the second column for now (the one showing 1 for each file) because it’s the first column that we care about most. This shows the file’s permissions, i.e., who can read, write, or execute it.

Let’s have a closer look at one of those permission strings: -rwxr-xr-x. The first character tells us what type of thing this is: ‘-‘ means it’s a regular file, while ‘d’ means it’s a directory, and other characters mean more esoteric things.

The next three characters tell us what permissions the file’s owner has. Here, the owner can read, write, and execute the file: rwx. The middle triplet shows us the group’s permissions. If the permission is turned off, we see a dash, so r-x means “read and execute, but not write”. The final triplet shows us what everyone who isn’t the file’s owner, or in the file’s group, can do. In this case, it’s ‘r-x’ again, so everyone on the system can look at the file’s contents and run it.

To change permissions, we use the chmod command (whose name stands for “change mode”). Here’s a long-form listing showing the permissions on the final grades in the course Vlad is teaching:

$ ls -l final.grd

-rwxrwxrwx 1 vlad bio  4215  2010-08-29 22:30 final.grd

Whoops: everyone in the world can read it—and what’s worse, modify it! (They could also try to run the grades file as a program, which would almost certainly not work.)

The command to change the owner’s permissions to rw- is:

$ chmod u=rw final.grd

The ‘u’ signals that we’re changing the privileges of the user (i.e., the file’s owner), and rw is the new set of permissions. A quick ls -l shows us that it worked, because the owner’s permissions are now set to read and write:

$ ls -l final.grd

-rw-rwxrwx 1 vlad bio  4215  2010-08-30 08:19 final.grd

Let’s run chmod again to give the group read-only permission:

$ chmod g=r final.grd
$ ls -l final.grd

-rw-r--rw- 1 vlad bio  4215  2010-08-30 08:19 final.grd

And finally, let’s give “all” (everyone on the system who isn’t the file’s owner or in its group) no permissions at all:

$ chmod a= final.grd
$ ls -l final.grd

-rw-r----- 1 vlad bio  4215  2010-08-30 08:20 final.grd

Here, the ‘a’ signals that we’re changing permissions for “all”, and since there’s nothing on the right of the “=”, “all”’s new permissions are empty.

We can search by permissions, too. Here, for example, we can use -type f -perm -u=x to find files that the user can execute:

$ find . -type f -perm -u=x

./tools/format
./tools/stats

Before we go any further, let’s run ls -a -l to get a long-form listing that includes directory entries that are normally hidden:

$ ls -a -l

drwxr-xr-x 1 vlad bio     0  2010-08-14 09:55 .
drwxr-xr-x 1 vlad bio  8192  2010-08-27 23:11 ..
-rw-rw-r-- 1 vlad bio  1158  2010-07-11 08:22 safety.txt
-rwxr-xr-x 1 vlad bio 31988  2010-07-23 20:04 setup
-rw-rw-r-- 1 vlad bio  2312  2010-07-11 08:23 waiver.txt

The permissions for . and .. (this directory and its parent) start with a ‘d’. But look at the rest of their permissions: the ‘x’ means that “execute” is turned on. What does that mean? A directory isn’t a program—how can we “run” it?

In fact, ‘x’ means something different for directories. It gives someone the right to traverse the directory, but not to look at its contents. The distinction is subtle, so let’s have a look at an example. Vlad’s home directory has three subdirectories called venus, mars, and pluto:

Each of these has a subdirectory in turn called notes, and those sub-subdirectories contain various files. If a user’s permissions on venus are ‘r-x’, then if she tries to see the contents of venus and venus/notes using ls, the computer lets her see both. If her permissions on mars are just ‘r–’, then she is allowed to read the contents of both mars and mars/notes. But if her permissions on pluto are only ‘–x’, she cannot see what’s in the pluto directory: ls pluto will tell her she doesn’t have permission to view its contents. If she tries to look in pluto/notes, though, the computer will let her do that. She’s allowed to go through pluto, but not to look at what’s there. This trick gives people a way to make some of their directories visible to the world as a whole without opening up everything else.

What about Windows?

Those are the basics of permissions on Unix. As we said at the outset, though, things work differently on Windows. There, permissions are defined by access control lists, or ACLs. An ACL is a list of pairs, each of which combines a “who” with a “what”. For example, you could give the Mummy permission to append data to a file without giving him permission to read or delete it, and give Frankenstein permission to delete a file without being able to see what it contains.

This is more flexible that the Unix model, but it’s also more complex to administer and understand on small systems. (If you have a large computer system, nothing is easy to administer or understand.) Some modern variants of Unix support ACLs as well as the older read-write-execute permissions, but hardly anyone uses them.

Challenge

If ls -l myfile.php returns the following details:

-rwxr-xr-- 1 caro zoo  2312  2014-10-25 18:30 myfile.php

Which of the following statements is true?

1. caro (the owner) can read, write, and execute myfile.php
2. caro (the owner) cannot write to myfile.php
3. members of caro (a group) can read, write, and execute myfile.php
4. members of zoo (a group) cannot execute myfile.php

Key Points

• Correct permissions are critical for the security of a system.

• File permissions describe who and what can read, write, modify, and access a file.

• Use ls -l to view the permissions for a specific file.

• Use chmod to change permissions on a file or directory.

Directory structure

Overview

Teaching: 5 min
Exercises: 0 min

Questions

• Understanding the concept of Unix directory structure

Objectives

• FIXME

All Unix files are integrated in a single directory structure. The file-system is arranged in a structure like an inverted tree. The top of this tree is the root and is written as a slash ‘/’.

The tree command

FIXME

Key Points

• FIXME

Job control

Overview

Teaching: 5 min
Exercises: 0 min

Questions

• How do keep track of the process running on my machine?

• Can I run more than one program/script from within a shell?

Objectives

• Learn how to use ps to get information about the state of processes

• Learn how to control, ie., “stop/pause/background/foreground” processes

The shell-novice lesson explained how we run programs or scripts from the shell’s command line.

We’ll now take a look at how to control programs once they’re running. This is called job control, and while it’s less important today than it was back in the Dark Ages, it is coming back into its own as more people begin to leverage the power of computer networks.

When we talk about controlling programs, what we really mean is controlling processes. As we said earlier, a process is just a program that’s in memory and executing. Some of the processes on your computer are yours: they’re running programs you explicitly asked for, like your web browser. Many others belong to the operating system that manages your computer for you, or, if you’re on a shared machine, to other users.

The ps command

You can use the ps command to list processes, just as you use ls to list files and directories.

Behaviour of the ps command

The ps command has a swathe of option flags that control its behaviour and, what’s more, the sets of flags and default behaviour vary across different platforms.

A bare invocation of ps only shows you basic information about your, active processes.

After that, this is a command that it is worth reading the ‘man page’ for.

$ ps

  PID TTY          TIME CMD
12767 pts/0    00:00:00 bash
15283 pts/0    00:00:00 ps

At the time you ran the ps command, you had two active processes, your (bash) shell and the (ps) command you had invoked in it.

Chances are that you were aware of that information, without needing to run a command to tell you it, so let’s try and put some flesh on that bare bones information.

$ ps -f

UID        PID  PPID  C STIME TTY          TIME CMD
vlad     12396 25397  0 14:28 pts/0    00:00:00 ps -f
vlad     25397 25396  0 12:49 pts/0    00:01:39 bash

In case you haven’t had time to do a man ps yet, be aware that the -f flag doesn’t stand for “flesh on the bones” but for “Do full-format listing”, although even then, there are “fuller” versions of the ps output.

But what are we being told here?

Every process has a unique process id (PID). Remember, this is a property of the process, not of the program that process is executing: if you are running three instances of your browser at once, each will have its own process ID.

The third column in this listing, PPID, shows the ID of each process’s parent. Every process on a computer is spawned by another, which is its parent (except, of course, for the bootstrap process that runs automatically when the computer starts up).

Clearly, the ps -f that was run is a child process of the (bash) shell it was invoked in.

Column 1 shows the username of the user the processes are being run by. This is the username the computer uses when checking permissions: each process is allowed to access exactly the same things as the user running it, no more, no less.

Column 5, STIME, shows when the process started running, whilst Column 7, TIME, shows you how much time process has used, whilst Column 8, CMD, shows what program the process is executing.

Column 6, TTY, shows the ID of the terminal this process is running in. Once upon a time, this really would have been a terminal connected to a central timeshared computer. It isn’t as important these days, except that if a process is a system service, such as a network monitor, ps will display a question mark for its terminal, since it doesn’t actually have one.

The fourth column, C, is an indication of the perCentage of processor utilization.

Your version of ps may show more or fewer columns, or may show them in a different order, but the same information is generally available everywhere, and the column headers are generally consistent.

Stopping, pausing, resuming, and backgrounding, processes

The shell provides several commands for stopping, pausing, and resuming processes. To see them in action, let’s run our analyze program on our latest data files. After a few minutes go by, we realize that this is going to take a while to finish. Being impatient, we kill the process by typing Control-C. This stops the currently-executing program right away. Any results it had calculated, but not written to disk, are lost.

$ ./analyze results*.dat

...a few minutes pass...
^C

Let’s run that same command again, with an ampersand & at the end of the line to tell the shell we want it to run in the background:

$ ./analyze results*.dat &

When we do this, the shell launches the program as before. Instead of leaving our keyboard and screen connected to the program’s standard input and output, though, the shell hangs onto them. This means the shell can give us a fresh command prompt, and start running other commands, right away. Here, for example, we’re putting some parameters for the next run of the program in a file:

$ cat > params.txt
density: 22.0
viscosity: 0.75
^D

(Remember, \^D is the shell’s way of showing Control-D, which means “end of input”.) Now let’s run the jobs command, which tells us what processes are currently running in the background:

$ jobs

[1] ./analyze results01.dat results02.dat results03.dat

Since we’re about to go and get coffee, we might as well use the foreground command, fg, to bring our background job into the foreground:

$ fg

...a few minutes pass...

When analyze finishes running, the shell gives us a fresh prompt as usual. If we had several jobs running in the background, we could control which one we brought to the foreground using fg %1, fg %2, and so on. The IDs are not the process IDs. Instead, they are the job IDs displayed by the jobs command.

The shell gives us one more tool for job control: if a process is already running in the foreground, Control-Z will pause it and return control to the shell. We can then use fg to resume it in the foreground, or bg to resume it as a background job. For example, let’s run analyze again, and then type Control-Z. The shell immediately tells us that our program has been stopped, and gives us its job number:

$ ./analyze results01.dat
^Z

[1]  Stopped   ./analyze results01.dat

If we type bg %1, the shell starts the process running again, but in the background. We can check that it’s running using jobs, and kill it while it’s still in the background using kill and the job number. This has the same effect as bringing it to the foreground and then typing Control-C:

$ bg %1

$ jobs

[1] ./analyze results01.dat

$ kill %1

Job control was important when users only had one terminal window at a time. It’s less important now: if we want to run another program, it’s easy enough to open another window and run it there. However, these ideas and tools are making a comeback, as they’re often the easiest way to run and control programs on remote computers elsewhere on the network. This lesson’s ssh episode has more to say about that.

Key Points

• When we talk of ‘job control’, we really mean ‘process control’

• A running process can be stopped, paused, and/or made to run in the background

• A process can be started so as to immediately run in the background

• Paused or backgrounded processes can be brought back into the foreground

• Process information can be inspected with ps

Aliases and bash customization

Overview

Teaching: 10 minutes min
Exercises: 0 min

Questions

• How do I customize my bash environment?

Objectives

• Create aliases.

• Add customizations to the .bashrc and .bash_profile files.

• Change the prompt in a bash environment.

Bash allows us to customize our environments to fill our own particular needs.

Aliases

Sometimes we need to use long commands that have to be typed over and over again. Fortunately, the alias command allows us to create shortcuts for these long commands.

As an example, let’s create aliases for going up one, two, or three directories.

alias up='cd ..'
alias upup='cd ../..'
alias upupup='cd ../../..'

Let’s try these commands out.

cd /usr/local/bin
upup
pwd

/usr

We can also remove a shortcut with unalias.

unalias upupup

If we create one of these aliases in a bash session, they will only last until the end of that session. Fortunately, bash allows us to specify customizations that will work whenever we begin a new bash session.

Bash customization files

Bash environments can be customized by adding commands to the .bashrc, .bash_profile, and .bash_logout files in our home directory. The .bashrc file is executed whenever entering interactive non-login shells whereas .bash_profile is executed for login shells. If the .bash_logout file exists, then it will be run after exiting a shell session.

Let’s add the above commands to our .bashrc file. Be careful to append to .bashrc, with >>. for concatenate, rather than one > which would overwrite.

echo "alias up='cd ..'" >> ~/.bashrc
tail -n 1 ~/.bashrc

alias up='cd ..'

We can execute the commands in .bashrc using source, so this creates the alias up which we can then use in directory /usr/local/bin:

source ~/.bashrc
cd /usr/local/bin
up
pwd

/usr/local

Having to add customizations to two files can be cumbersome. It we would like to always use the customizations in our .bashrc file, then we can add the following lines to our .bash_profile file.

if [ -f $HOME/.bashrc ]; then
        source $HOME/.bashrc
fi

Customizing your prompt

We can also customize our bash prompt by setting the PS1 system variable. To set our prompt to be $ , then we can run the command

export PS1="$ "

To set the prompt to $ for all bash sessions, add this line to the end of .bashrc.

Further bash prompt customizations are possible. To have our prompt be username@hostname[directory]: , we would set

export PS1="\u@\h[\W]: "

where \u represents username, \h represents hostname, and \W represents the current directory.

Key Points

• Aliases are used to create shortcuts or abbreviations

• The .bashrc and .bash_profile files allow us to customize our bash environment.

• The PS1 system variable can be changed to customize your bash prompt.

Shell Variables

Overview

Teaching: 10 min
Exercises: 0 min

Questions

• How are variables set and accessed in the Unix shell?

Objectives

• Understand how variables are implemented in the shell

• Explain how the shell uses the PATH variable to search for executables

• Read the value of an existing variable

• Create new variables and change their values

The shell is just a program, and like other programs, it has variables. Those variables control its execution, so by changing their values you can change how the shell and other programs behave.

Let’s start by running the command set and looking at some of the variables in a typical shell session:

$ set

COMPUTERNAME=TURING
HOME=/home/vlad
HOMEDRIVE=C:
HOSTNAME=TURING
HOSTTYPE=i686
NUMBER_OF_PROCESSORS=4
OS=Windows_NT
PATH=/Users/vlad/bin:/usr/local/git/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
PWD=/home/vlad
UID=1000
USERNAME=vlad
...

As you can see, there are quite a few—in fact, four or five times more than what’s shown here. And yes, using set to show things might seem a little strange, even for Unix, but if you don’t give it any arguments, it might as well show you things you could set.

Every variable has a name. By convention, variables that are always present are given upper-case names. All shell variables’ values are strings, even those (like UID) that look like numbers. It’s up to programs to convert these strings to other types when necessary. For example, if a program wanted to find out how many processors the computer had, it would convert the value of the NUMBER_OF_PROCESSORS variable from a string to an integer.

Similarly, some variables (like PATH) store lists of values. In this case, the convention is to use a colon ‘:’ as a separator. If a program wants the individual elements of such a list, it’s the program’s responsibility to split the variable’s string value into pieces.

The PATH Variable

Let’s have a closer look at that PATH variable. Its value defines the shell’s search path, i.e., the list of directories that the shell looks in for runnable programs when you type in a program name without specifying what directory it is in.

For example, when we type a command like analyze, the shell needs to decide whether to run ./analyze or /bin/analyze. The rule it uses is simple: the shell checks each directory in the PATH variable in turn, looking for a program with the requested name in that directory. As soon as it finds a match, it stops searching and runs the program.

To show how this works, here are the components of PATH listed one per line:

/Users/vlad/bin
/usr/local/git/bin
/usr/bin
/bin
/usr/sbin
/sbin
/usr/local/bin

On our computer, there are actually three programs called analyze in three different directories: /bin/analyze, /usr/local/bin/analyze, and /users/vlad/analyze. Since the shell searches the directories in the order they’re listed in PATH, it finds /bin/analyze first and runs that. Notice that it will never find the program /users/vlad/analyze unless we type in the full path to the program, since the directory /users/vlad isn’t in PATH.

Showing the Value of a Variable

Let’s show the value of the variable HOME:

$ echo HOME

HOME

That just prints “HOME”, which isn’t what we wanted (though it is what we actually asked for). Let’s try this instead:

$ echo $HOME

/home/vlad

The dollar sign tells the shell that we want the value of the variable rather than its name. This works just like wildcards: the shell does the replacement before running the program we’ve asked for. Thanks to this expansion, what we actually run is echo /home/vlad, which displays the right thing.

Creating and Changing Variables

Creating a variable is easy—we just assign a value to a name using “=”:

$ SECRET_IDENTITY=Dracula
$ echo $SECRET_IDENTITY

Dracula

To change the value, just assign a new one:

$ SECRET_IDENTITY=Camilla
$ echo $SECRET_IDENTITY

Camilla

If we want to set some variables automatically every time we run a shell, we can put commands to do this in a file called .bashrc in our home directory. (The ‘.’ character at the front prevents ls from listing this file unless we specifically ask it to using -a: we normally don’t want to worry about it. The “rc” at the end is an abbreviation for “run control”, which meant something really important decades ago, and is now just a convention everyone follows without understanding why.)

For example, here are two lines in /home/vlad/.bashrc:

export SECRET_IDENTITY=Dracula
export TEMP_DIR=/tmp
export BACKUP_DIR=$TEMP_DIR/backup

These three lines create the variables SECRET_IDENTITY, TEMP_DIR, and BACKUP_DIR, and export them so that any programs the shell runs can see them as well. Notice that BACKUP_DIR’s definition relies on the value of TEMP_DIR, so that if we change where we put temporary files, our backups will be relocated automatically.

While we’re here, it’s also common to use the alias command to create shortcuts for things we frequently type. For example, we can define the alias backup to run /bin/zback with a specific set of arguments:

alias backup=/bin/zback -v --nostir -R 20000 $HOME $BACKUP_DIR

As you can see, aliases can save us a lot of typing, and hence a lot of typing mistakes. You can find interesting suggestions for other aliases and other bash tricks by searching for “sample bashrc” in your favorite search engine.

Key Points

• Shell variables are by default treated as strings

• The PATH variable defines the shell’s search path

• Variables are assigned using “=” and recalled using the variable’s name prefixed by “$”

The Unix Shell

Overview

Teaching: min
Exercises: min

Questions

Objectives

Learning Objectives {.objectives}

• Understand the need for flexibility regarding arguments
• Generate the values of the arguments on the fly using command substitution
• Understand the difference between pipes/redirection, and the command substitution operator

Introduction

In the Loops topic we saw how to improve productivity by letting the computer do the repetitive work. Often, this involves doing the same thing to a whole set of files, e.g.:

$ cd data/pdb
$ mkdir sorted
$ for file in *cyclo*.pdb; do
>     sort $file > sorted/sorted-$file
> done

In this example, the shell generates for us the list of things to loop over, using the wildcard mechanism we saw in the Pipes and Filters topic. This results in the *cyclo*.pdb being replaced with cyclobutane.pdb cyclohexanol.pdb cyclopropane.pdb ethylcyclohexane.pdb before the loop starts.

Another example is a so-called parameter sweep, where you run the same program a number of times with different arguments. Here is a fictitious example:

$ for cutoff in 0.001 0.01 0.05; do
>   run_classifier.sh --input ALL-data.txt --pvalue $cutoff --output results-$cutoff.txt
> done

In the second example, the things to loop over: "0.001 0.01 0.05" are spelled out by you.

Looping over the words in a string {.callout}

In the previous example you can make your code neater and self-documenting by putting the cutoff values in a separate string:

$ cutoffs="0.001 0.01 0.05"
$ for cutoff in $cutoffs; do
  run_classifier.sh --input ALL-data.txt --pvalue $cutoff --output results-$cutoff.txt
done

This works because, just as with the filename wildcards, $cutoffs is replaced with 0.001 0.01 0.05 before the loop starts.

However, you don’t always know in advance what you have to loop over. It could well be that it is not a simple file name pattern (in which case you can use wildcards), or that it is not a small, known set of values (in which case you can write them out explicitly as was done in the second example). It would therefore be nice if you could loop over filenames or over words contained in a file. Suppose that file cohort2010.txt contains the filenames over which to iterate, then it would be nice to able to say something like:

# (imaginary syntax)
$ for file in [INSERT THE CONTENTS OF cohort2010.txt HERE]
> do
>    run_classifier.sh --input $file --pvalue -0.05 --output $file.results
> done

Command substitution

This would be more general, more flexible and more tractable than relying on the wildcard mechanism. What we need, therefore, is a mechanism that actually replaces everytying beween [ and ] with the desired names of input files, just before the loop starts. Thankfully, this mechanism exists, and it is called the command substitution operator (previously written using the backtick operator). It looks much like the previous snippet:

# (actual syntax)
$ for file in $(cat cohort2010.txt)
> do
>    run_classifier.sh --input $file --pvalue -0.05 --output $file.results
> done

It works simply as follows: everything between the $( and the ) is executed as a Unix command, and the command’s standard output replaces everything from $( up to and including ), just before the loop starts. For convenience, newlines in the command’s output are replaced with simple spaces.

Backtick operator

In legacy code, you may see the same construct but with a different syntax. It starts and ends with backticks, ` (not to be confused with the single quote ' !). The backticks work exactly the same as the command substitution done by $( and ). However its use is discouraged as backticks cannot be nested.

Example

OK. Recall from the Pipes and Filters topic that cat prints the contents of its argument (a filename) to standard output. So, if the contents of file cohort2010.txt look like

patient1033130.txt 
patient1048338.txt 
patient7448262.txt 
.
.
.
patient1820757.txt

then the construct

$ for file in $(cat cohort2010.txt)
> do
>     ...
> done

will be expanded to

$ for file in patient1033130.txt patient1048338.txt patient7448262.txt ... patient1820757.txt
> do
>     ...
> done

(notice the convenience of newlines having been replaced with simple spaces).

This example uses $(cat somefilename) to supply arguments to the for variable in ... do ... done-construct, but any output from any command, or even pipeline, can also be used. For example, if cohort2010.txt contains a few thousand patients but you just want to try the first two for a test run, you can use the head command to just get the first few lines of its argument, like so:

$ for file in $(cat cohort2010.txt | head -n 2)
> do
>     ...
> done

which will expand to

$ for file in patient1033130.txt patient1048338.txt
> do
>     ...
> done

simply because cat cohort2010.txt | head -n 2 produces patient1033130.txt patient1048338.txt after the command substitution.

Everything between the $( and ) is executed verbatim by the shell, so also the -n 2 argument to the head command works as expected.

Important

Recall from the Loops and the Shell Scripts topics that Unix uses whitespace to separate command, options (flags) and parameters / arguments. For the same reason it is essential that the command (or pipeline) inside the backticks produces clean output: single word output works best within single commands and whitespace- or newline-separated words works best for lists over which to iterate in loops.

Generating filenames based on a timestamp {.challenge}

It can be useful to create the filename ‘on the fly’. For instance, if some program called qualitycontrol is run periodically (or unpredictably) it may be necessary to supply the time stamp as an argument to keep all the output files apart, along the following lines:

qualitycontrol --inputdir /data/incoming/  --output qcresults-[INSERT TIMESTAMP HERE].txt

Getting [INSERT TIMESTAMP HERE] to work is a job for the command subsitution operator. The Unix command you need here is the date command, which provides you with the current date and time (try it).

In the current form, its output is less useful for generating filenames because it contains whitespace (which, as we know from now, should preferably be avoided in filenames). You can tweak date’s format in great detail, for instance to get rid of whitespace:

$ date +"%Y-%m-%d_%T"

(Try it).

Write the command that will copy a file of your choice to a new file whose name contains the time stamp. Test it by executing the command a few times, waiting a few seconds between invocations (use the arrow-up key to avoid having to retype the command)

Juggling filename extensions {.challenge}

When running an analysis program with a certain input file, it is often required that the output has the same name as the input, but with a different filename extension, e.g.

$ run_classifier.sh --input patient1048338.txt --pvalue -0.05 --output patient1048338.results

A good trick here is to use the Unix basename command. It takes a string (typically a filename), and strips off the given extension (if it is part of the input string). Example:

$ basename patient1048338.txt    .txt

gives

patient1048338

Write a loop that uses the command substitution operator and the basename command to sort each of the *.pdb files into a corresponding *.sorted file. That is, make the loop do the following:

$ sort ammonia.pdb > ammonia.sorted

but for each of the .pdb-files.

Closing remarks

The command subsitution operator provides us with a powerful new piece of ‘plumbing’ that allows us to connect “small pieces, loosely together” to keep with the Unix philosophy. It is remotely similar to the | operator in the sense that it connects two programs. But there is also a clear difference: | connects the standard output of one command to the standard input of another command, where as $(command) is substituted ‘in-place’ into the the shell script, and always provides parameters, options, arguments to other commands.

Key Points

AWK

Overview

Teaching: min
Exercises: min

Questions

Objectives

Learning Objectives {.objectives}

• Explain why AWK is useful and when it is better than pipes.
• Show a basic usage similar to the command “cut”
• Introduce the filed separator parameter.
• Use regulax expressions to perform different instructions.
• Introduce BEGIN and END keywords.
• Use the if then structure to change behaviour for the same matching regex.
• Introduce the array data structure.
• Use the for loop to cycle through an array

Prerequisite: shell and any programming language

If we need to count the number of lines in a file, we can use the previously showed command for word counting wc

$ wc -l example.txt

As you probably remember, -l is an option that asks for the number of lines only.

However, wc counts the number of newlines in the file, if the last line does not contain a carriage return (i.e. there is no emptyline at the end of the file), the result is going be the actual number of lines minus one.

A workaround is to use Awk. Awk is command line program that takes as input a set of instructions and one or more files. The instructions are executed on each line of the input file(s).

The instructions are enclosed in single quotes or they can be read from a file.

Example:

$ awk '{print $0}' example.txt

This command has the same output of “cat”: it prints each line from the example.txt file.

The structure of the instruction is the following:

• curly braces surround the set of instructions
• print is the instruction that sends its arguments to the terminal
• $0 is a variable, it means “the content of the current line”

As you can see, the file contains a table.

Awk automatically splits the processed line by looking at spaces: in our case it has knowledge of the different columns in the table.

Each column value for the current line is stored into a variable: $1 for the first column, $2 for the second and so on.

So, if we like to print only the second column from the table, we execute

$ awk '{print $2}' example.txt

We can also print more than one value, or add text to the printed line:

$ awk '{print "chr",$2,$4}' example.txt

The comma puts a space between the printed values. Strings of text should be enclosed in double quotes. In this case we are printing the text “chr”, the second and the fourth column for each row in the table.

So, $0 is the whole line, $1 the first field, $2 the second and so on. What if we want to print the last column, but we don’t know its number? Maybe it is a huge table, or maybe different lines have a different number of columns.

Awk helps us thanks to the variable NF. NF stores the number of fields (our columns) in the row. Let’s see for our table:

$ awk '{print NF}' example.txt

We can see that some lines contain 6 fields while others contain 7 of them. Since NF is the number of the last field, $NF contains its value.

$ awk '{print "This line has",NF,"columns. The last one contains",$NF}' example.txt

Field separator {.callout}

Out there we have different file formats: our data may be comma separated (csv), tab separated (tsv), by semicolon or by any other character.

To specify the field separator, we should provide it at command line like:

$ awk -F "," '{print $2}' example2.txt

In this case , we are printing the second field in each line, using comma as separator. Please notice that the character space is now part of the field value, since it is no longer the separator.

Matching lines {.callout}

Maybe we would like to perform different instruction on different lines.

Awk allows you to specify a matching pattern, like the command grep does.

Let’s look at the file content

$ awk '{print $0}' example.pdb

It seems an abriged PDB file. If we would like to print only lines starting with the word “ATOM”, we type:

awk '/^ATOM/ {print $0}' example.pdb

In this case, we specify the pattern before the instructions: only lines starting with the text “ATOM”. As you remember, ^ means “at the beginning of the line”.

We can specify more that one pattern:

awk '/^ATOM/ {print $7,$8,$9} /^HEADER/ {print $NF}' example.pdb

In this case, we are printing the spatial coordinates of each atom.

Key Points