This lesson is in the early stages of development (Alpha version)

# Running Tests with pytest

## Overview

Teaching: 10 min
Exercises: 0 min
Questions
• How do I automate my tests?

Objectives
• Understand how to run a test suite using the pytest framework

• Understand how to read the output of a pytest test suite

We created a suite of tests for our mean function, but it was annoying to run them one at a time. It would be a lot better if there were some way to run them all at once, just reporting which tests fail and which succeed.

Thankfully, that exists. Recall our tests:

``````from mean import *

def test_ints():
num_list = [1,2,3,4,5]
obs = mean(num_list)
exp = 3
assert obs == exp

def test_zero():
num_list=[0,2,4,6]
obs = mean(num_list)
exp = 3
assert obs == exp

def test_double():
# This one will fail in Python 2
num_list=[1,2,3,4]
obs = mean(num_list)
exp = 2.5
assert obs == exp

def test_long():
big = 100000000
obs = mean(range(1,big))
exp = big/2.0
assert obs == exp

def test_complex():
# given that complex numbers are an unordered field
# the arithmetic mean of complex numbers is meaningless
num_list = [2 + 3j, 3 + 4j, -32 - 2j]
obs = mean(num_list)
exp = NotImplemented
assert obs == exp
``````

Once these tests are written in a file called `test_mean.py`, the command `pytest` can be run on the terminal or command line from the directory containing the tests (note that you’ll have to use `py.test` for older versions of the `pytest` package):

``````\$ pytest
``````
``````collected 5 items

test_mean.py ....F

================================== FAILURES ===================================
________________________________ test_complex _________________________________

def test_complex():
# given that complex numbers are an unordered field
# the arithmetic mean of complex numbers is meaningless
num_list = [2 + 3j, 3 + 4j, -32 - 2j]
obs = mean(num_list)
exp = NotImplemented
>       assert obs == exp
E       assert (-9+1.6666666666666667j) == NotImplemented

test_mean.py:34: AssertionError
===================== 1 failed, 4 passed in 2.71 seconds ======================
``````

In the above case, the pytest package ‘sniffed-out’ the tests in the directory and ran them together to produce a report of the sum of the files and functions matching the regular expression `[Tt]est[-_]*`.

The major benefit a testing framework provides is exactly that, a utility to find and run the tests automatically. With pytest, this is the command-line tool called `pytest`. When `pytest` is run, it will search all directories below where it was called, find all of the Python files in these directories whose names start or end with `test`, import them, and run all of the functions and classes whose names start with `test` or `Test`. This automatic registration of test code saves tons of human time and allows us to focus on what is important: writing more tests.

When you run `pytest`, it will print a dot (`.`) on the screen for every test that passes, an `F` for every test that fails or where there was an unexpected error. In rarer situations you may also see an `s` indicating a skipped tests (because the test is not applicable on your system) or a `x` for a known failure (because the developers could not fix it promptly). After the dots, pytest will print summary information.

Without changing the tests, alter the mean.py file from the previous section until it passes. When it passes, `pytest` will produce results like the following:

``````\$ pytest
``````
``````collected 5 items

test_mean.py .....

========================== 5 passed in 2.68 seconds ===========================
``````

## Show what tests are executed

Using `pytest -v` will result in `pytest` listing which tests are executed and whether they pass or not:

``````\$ py.test
``````
``````collected 5 items

test_mean.py .....

test_mean.py::test_ints PASSED
test_mean.py::test_zero PASSED
test_mean.py::test_double PASSED
test_mean.py::test_long PASSED
test_mean.py::test_complex PASSED

========================== 5 passed in 2.57 seconds ===========================
``````

As we write more code, we would write more tests, and pytest would produce more dots. Each passing test is a small, satisfying reward for having written quality scientific software. Now that you know how to write tests, let’s go into what can go wrong.

## Key Points

• The `pytest` command collects and runs tests starting with `Test` or `test_`.

• `.` means the test passed

• `F` means the test failed or erred

• `x` is a known failure

• `s` is a purposefully skipped test