Pytest Module

Indices and tables

• Index
• Module Index
• Search Page

Introduction

Testing is the practice of exhaustively verifying that your code operates correctly. This is broken down into:

• Unit testing: verifying the individual methods or functions
• Integration testing: verifying higher level functionality involving multiple functions or methods

At the most basic level, this would just involve setting some input conditions, running some code, and using Python’s built-in assert statement to check some output against an expected result. So why bother using a test framework?

Python test frameworks such as PyTest provide a robust feature set that makes them much more user friendly than simple asserts. This includes:

1. Built-in test discovery
2. More detailed information on failures
3. Test fixtures
4. Parametrized tests

This section will cover testing with PyTest. In order to use PyTest, you must install it:

$ pip3 install pytest

Running Tests

One of the advantages of PyTest is it’s ability to automatically discover tests. This saves the user from having to manually maintain a list of tests for any given test suite.

When invoked from the command line, PyTest uses the following default rules for test discovery:

1. Starting from (and including) the current directory, recurse into all subdirectories (directories can be filtered using the norecursedirs flag).
2. In each directory, look for files named test_*.py or *_test.py and import them by their test package name.
3. In each file,
   1. look for functions or methods prefixed with test_ that are not part of any class , or,
   2. look for functions or methods prefixed with test_ inside a class prefixed with Test.

Alternatively, tests can be run using command line arguments to filter by keyword, run specific test files by name, or run specific test methods. Some examples of PyTest commands to run tests could include:

$ python3 -m pytest -k filenamekeyword                               # tests matching keyword
$ python3 -m pytest tests/utils/test_sample.py                       # single test file
$ python3 -m pytest tests/utils/test_sample.py::test_answer_correct  # single test method

Assertion Introspection

As we saw with plain asserts, it can be difficult to diagnose the issue using the AssertionError that is raised when an assert condition is false. Even with a descriptive error message, all we know is where the assertion occured but not much about how it occured.

Enter: PyTest’s deep assertion analysis.

import sys
import pytest

@pytest.mark.parametrize('num', [0, -1])
def test_foo(num):
    assert 0 <= num

if __name__ == "__main__":
    test_foo(int(sys.argv[1]))

Running the test as a plain python execution:

$ python test_foo.py 0
$ python test_foo.py -1
Traceback (most recent call last):
  File "test_foo.py", line 9, in <module>
    test_foo(int(sys.argv[1]))
  File "test_foo.py", line 6, in test_foo
    assert 0 <= num
AssertionError

Running the test using PyTest:

$ pytest -v test_foo.py
=============================================================================================== test session starts ===============================================================================================
platform linux2 -- Python 2.7.12, pytest-3.4.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kbutler/del.me, inifile:
collected 2 items

test_foo.py::test_foo[0] PASSED    [ 50%]
test_foo.py::test_foo[-1] FAILED   [100%]

==================================================================================================== FAILURES =====================================================================================================
__________________________________________________________________________________________________   test_foo[-1]   ___________________________________________________________________________________________________

num = -1

    @pytest.mark.parametrize("num", [0, -1])
        def test_foo(num):
        >       assert 0 <= num
        E       assert 0 <= -1

        test_foo.py:6: AssertionError
        =======================================================================================           1 failed, 1 passed in 0.02 seconds           ========================================================================================

Test Structure

Tests are structured according to the following general format:

def test_name():
    # Arrange (setup/initialization)
    # Act (invoke the code)
    # Assert (check the outputs/system state; standard form: assert expected == actual)

Arrange, Act, Assert

Assume we have a function called calculate_hypotenuse that takes in two values (i.e. the legs of the right-angle triangle) and returns the length of the hypotenuse:

import math
def calculate_hypotenuse(a, b):
    return math.sqrt(a**2 + b**2)

In order to write tests for this, we first have to come up with some sample data sets to test. We choose two scenarios, h² = 3² +4² and h² = 4² + 4² and we know the results should be 5 and 5.657 (rounded), respectively. We write the following tests to check these cases:

def test_calculate_hypotenuse_3_4():
    result = calculate_hypotenuse(3, 4)
    assert 5 == result

def test_calculate_hypotenuse_4_4():
    result = calculate_hypotenuse(4, 4)
    assert 5.657 == result

Run the tests (the -v flag specifies a verbose test result output):

$ pytest -v
======================================================= test session starts =======================================================
test_sample.py::test_calculate_hypotenuse_3_4 PASSED    [ 50%]
test_sample.py::test_calculate_hypotenuse_4_4 FAILED    [100%]

As you can see, one of our tests failed. PyTest gives us a very detailed error message indicating where the failure occurred:

__________________________________________ test_normal_4_4 __________________________________________
def test_calculate_hypotenuse_4_4():
>
E
result = calculate_hypotenuse(4, 4)
assert 5.657 == result
assert 5.657 == 5.656854249492381

Expected Value Tolerance

The above failure occurs because the expected result does not match the actual result with sufficient precision.

We can use the pytest function pytest.approx to specify the desired level of precision:

def test_calculate_hypotenuse_4_4():
    result = calculate_hypotenuse(4, 4)
    assert 5.657 == pytest.approx(result, abs=0.001)

Here, the test will pass if the result of the calculation falls between 5.656 and 5.658, as the tolerance was specified to be the absolute value of 0.001.

Parametrized Tests

You will notice that the above tests involve quite a bit of repetition; you call the function with two values, capture the return, and compare the return value to expected value within a certain level of precision.

Whenever tests involve repetition, it is a good opportunity to make use of a parametrized test. This type of tests allows multiple data sets to be applied to the same set of test steps.Test parametrization is done using the @pytest.mark.parametrize decorator, where the first argument is a string denoting a comma separated list of variables to pass to the function. The second argument is a list that contains one or more tuples of identical size as the number of arguments in the list.

@pytest.mark.parametrize("var_a,var_b,var_c", [
    (var_a_1, var_b_1, var_c_1),
    (var_a_2, var_b_2, var_c_2),
])
def test_mytest(var_a, var_b, var_c):
    pass

In our example, we can rewrite the two tests as follows:

@pytest.mark.parametrize("a,b,expected", [
    (3,4,5),
    (4,4,5.657),
])
def test_calculate_hypotenuse(a, b, expected):
    result = calculate_hypotenuse(a, b)
    assert expected == pytest.approx(result, abs=0.001)

Now, when we run this test, we see two results are produced:

test_sample.py::test_calculate_hypotenuse[3-4-5] PASSED      [ 50%]
test_sample.py::test_calculate_hypotenuse[4-4-5.657] PASSED  [100%]

You can see that the tests get the name of the generic test, suffixed with some information about the data set. This is not usually very helpful, as one of the characteristics of good tests is a succinct but descriptive name that tells the reader immediately what the test does. You can supply the ids argument to @pytest.mark.parametrize to supply a descriptive name for each test:

@pytest.mark.parametrize("a,b,expected", [
    (3,4,5),
    (4,4,5.657),
], ids=[
    'A=3, B=4, H=5; no decimals',
    'A=4, B=4, H=5.657; decimal result with 1e-03 precision'
])
def test_calculate_hypotenuse(a, b, expected):
    result = calculate_hypotenuse(a, b)
    assert expected == pytest.approx(result, abs=0.001)

Now when we run the tests, you can see it is much easier to grasp the test just from the name presented:

test_sample.py::test_calculate_hypotenuse[A=3, B=4, H=5; no decimals] PASSED                              [ 50%]
test_sample.py::test_calculate_hypotenuse[A=4, B=4, H=5.657; decimal result with 1e-03 precision] PASSED  [100%]

Skipped Tests

If there is a desire to skip certain tests in the test suite, you can mark them using the @pytest.mark.skip decorator:

deftest_run_this_test():
    pass
@pytest.mark.skip
def test_dont_run_this_test():
    pass

Running these tests would result in the following:

test_sample.py::test_run_this_test PASSED        [ 50%]
test_sample.py::test_dont_run_this_test SKIPPED  [100%]

As you can see, the test still appears in the output report, but is marked as being skipped.

An alternative to always skipping certain tests is to skip them if certain conditions are met. For example, maybe certain features of a test are only compatible with certain versions of python. In this case, you could choose to skip based on the version of python using @pytest.mark.skipif(condition, reason).

@pytest.mark.skipif(sys.version_info < (3,0),reason="requires python3.0")
def test_that_requires_python_3():
    pass

When running tests, the -ra option will produce extra test summary information, which includes the skip reasons.

$ python2 -m pytest -vra
test_sample.py::test_that_requires_python_3 SKIPPED   [100%]
============================================= short test summary info ==============================================
SKIP [1] test_sample.py:11: requires python3.0

$ python3 -m pytest -vra
test_sample.py::test_that_requires_python_3 PASSED [100%]

An alternative to skipping the test entirely is to run it, but not fail the suite if the test itself fails. This is done using @pytest.mark.xfail

@pytest.mark.xfail
def test_that_sometimes_fails():
    assert random.randint(1,100) < 50

In the output report, this will produce the following if the test fails:

test_sample.py::test_that_sometimes_fails[100%] XPASS  [100%]

test_sample.py::test_that_sometimes_fails[100%] xfail  [100%]

As you can see, this still allows us to run the test and see results, but does not fail the overall test results if it happens to fail.

Verifying Exception Throwing

Suppose one of our tests is to check whether the function throws an exception under certain circumstances.

def calculate_hypotenuse(a, b):
    if not a or not b:
        raise ValueError("Supplied lengths cannot be zero.")
    return math.sqrt(a**2 + b**2)

We can use the pytest.raises context manager to check for this:

@pytest.mark.parametrize("a,b", [
    (0, 1),
    (1, 0),
])
def test_zero_value_exception(a, b):
    with pytest.raises(ValueError):
        calculate_hypotenuse(a, b)

Running the test results in the following output:

test_sample.py::test_zero_value_exception[0-1] PASSED   [ 50%]
test_sample.py::test_zero_value_exception[1-0] PASSED   [100%]

Test Fixtures

Using Test Fixtures

If tests repeatedly have the same setup steps in the Arrange portion of the test, it is a good opportunity to create a test fixture. Think of a test fixture like a little helper function that can be re-used in multiple tests. This saves a tester from having to repeat the same code over and over in each test. To use the fixture, pass it’s name as an argument to the test in question.

@pytest.fixture()
def fixture_name():
    print ("\nSetting up")
    return obj # provide the fixture value

def test_that_uses_fixture(fixture_name):
    print ("\nRunning a test")

Running this with the -s flag so we can see print statements, we observe:

$ pytest -vs
=============================================== test session starts ================================================
test_fixtures.py::test_that_uses_fixture
Setting up
Running a test
PASSED

Often, if there is setup, there is some sort of cleanup afterwards (sometimes called teardown). In order for our test to use this, we use a yield instead of a return.

@pytest.fixture()
def fixture_name():
    print ("\nSetting up")
    yield obj  # provide the fixture value
    print ("\nCleaning up")

def test_that_uses_fixture(fixture_name):
    print ("\nRunning a test")

After running the test, we see the following output:

$ pytest -vs test_fixtures.py
=============================================== test session starts ================================================
test_fixtures.py::test_that_uses_fixture
Setting up
Running a test
PASSED
Cleaning up

Test Fixture Scope

Test fixtures have the concept of scope; this is equivalent to their lifetime. A fixture with a function scope (default) only lives for the duration of the test that uses it. Other scope options include:

• class: fixture lives for the duration of all tests inside a test class
• module: fixture lives for the duration of all tests in a single test module
• session: fixture lives for the duration of the entire PyTest run, which can span multiple files

Taking our previous example, adding a second test, and changing the fixture scope to module would result in the following:

@pytest.fixture(scope='module')
def fixture_name():
    print ("\nSetting up")
    yield obj  # provide the fixture value
    print ("\nCleaning up")

def test_first_test(fixture_name):
    print ("\nRunning test 1")

def test_second_test(fixture_name):
    print ("\nRunning test 2")

Running these tests would result in the following output:

$ pytest -vs test_fixtures.py
=============================================== test session starts ================================================
test_fixtures.py::test_first_test
Setting up
Running test 1
PASSED
test_fixtures.py::test_second_test
Running test 2
PASSED
Cleaning up

As you can see, despite both tests including the fixture, it was only called once for the entire scope that we specified; in this case, once for the entire module.

Fixtures (a real example)

Imagine you were unit testing a function which writes to a file. This function does not open or close the file.

def write_message_to_file(file, message):
    return file.write(message)

In order to test this, each test would need to open a new file for writing, pass this file into the code under test, and be responsible for closing the file afterwards. That’s a lot of repetition!

def test_one():
    f = open("/tmp/test.txt", "a+")
    length = write_message_to_file(f, "test1")
    assert 5 == length
    f.close()

def test_two():
    f = open("/tmp/test.txt", "a+")
    length = write_message_to_file(f, "mytest2")
    assert 7 == length
    f.close()

This is where fixtures can save us a lot of time. The fixture can take care of opening the file (the setup) and closing (the teardown).

@pytest.fixture()
def file_output():
    f = open("/tmp/test.txt", "a+")
    yield f
    f.close()

def test_one(file_output):
    length = write_message_to_file(file_output, "test1")
    assert 5 == length

def test_two(file_output):
    length = write_message_to_file(file_output, "mytest2")
    assert 7 == length