Pytest Module
=============

.. toctree::
   :maxdepth: 1


Indices and tables
------------------

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`


Introduction
------------

Testing is the practice of exhaustively verifying that your code operates correctly. This is broken down into:

* Unit testing: verifying the individual methods or functions

* Integration testing: verifying higher level functionality involving multiple functions or methods

At the most basic level, this would just involve setting some input conditions, running some code, and using Python's built-in :py:keyword:`assert` statement to check some output against an expected result. So why bother using a test framework?

Python test frameworks such as PyTest provide a robust feature set that makes them much more user friendly than simple asserts. This includes:

1. Built-in test discovery
2. More detailed information on failures
3. Test fixtures
4. Parametrized tests

This section will cover testing with PyTest. In order to use PyTest, you must install it:

.. code-block:: bash

   $ pip3 install pytest


.. _section_heading-Running_Tests:

Running Tests
-------------
One of the advantages of PyTest is it's ability to automatically discover tests. This saves the user from having to manually maintain a list of tests for any given test suite.

When invoked from the command line, PyTest uses the following default rules for test discovery:

1. Starting from (and including) the current directory, recurse into all subdirectories (directories can be filtered using the *norecursedirs* flag).

2. In each directory, look for files named ``test_*.py`` or ``*_test.py`` and :py:keyword:`import` them by their test package name.

3. In each file,

   a. look for functions or methods prefixed with ``test_`` that are not part of any class , **or**,

   b. look for functions or methods prefixed with ``test_`` inside a class prefixed with ``Test``.

Alternatively, tests can be run using command line arguments to filter by keyword, run specific test files by name, or run specific test methods. Some examples of PyTest commands to run tests could include:

.. code-block:: bash

   $ python3 -m pytest -k filenamekeyword                               # tests matching keyword
   $ python3 -m pytest tests/utils/test_sample.py                       # single test file
   $ python3 -m pytest tests/utils/test_sample.py::test_answer_correct  # single test method

.. _section_heading-Assertion_Introspection:

Assertion Introspection
-----------------------

As we saw with plain asserts, it can be difficult to diagnose the issue using
the :py:exc:`AssertionError` that is raised when an assert condition is
false. Even with a descriptive error message, all we know is `where` the
assertion occured but not much about `how` it occured.

Enter: PyTest's deep assertion analysis.

.. code-block:: python

   import sys
   import pytest

   @pytest.mark.parametrize('num', [0, -1])
   def test_foo(num):
       assert 0 <= num

   if __name__ == "__main__":
       test_foo(int(sys.argv[1]))

Running the test as a plain python execution:

.. code-block:: bash

   $ python test_foo.py 0
   $ python test_foo.py -1
   Traceback (most recent call last):
     File "test_foo.py", line 9, in <module>
       test_foo(int(sys.argv[1]))
     File "test_foo.py", line 6, in test_foo
       assert 0 <= num
   AssertionError

Running the test using PyTest:

.. code-block:: bash

   $ pytest -v test_foo.py
   =============================================================================================== test session starts ===============================================================================================
   platform linux2 -- Python 2.7.12, pytest-3.4.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
   cachedir: .pytest_cache
   rootdir: /home/kbutler/del.me, inifile:
   collected 2 items

   test_foo.py::test_foo[0] PASSED    [ 50%]
   test_foo.py::test_foo[-1] FAILED   [100%]

   ==================================================================================================== FAILURES =====================================================================================================
   __________________________________________________________________________________________________   test_foo[-1]   ___________________________________________________________________________________________________

   num = -1

       @pytest.mark.parametrize("num", [0, -1])
           def test_foo(num):
           >       assert 0 <= num
           E       assert 0 <= -1

           test_foo.py:6: AssertionError
           =======================================================================================           1 failed, 1 passed in 0.02 seconds           ========================================================================================

.. _section_heading-Test_Structure:

Test Structure
--------------
Tests are structured according to the following general format:

.. code-block:: python

   def test_name():
       # Arrange (setup/initialization)
       # Act (invoke the code)
       # Assert (check the outputs/system state; standard form: assert expected == actual)


.. _section_heading-Arrange_Act_Assert:

Arrange, Act, Assert
--------------------

Assume we have a function called ``calculate_hypotenuse`` that takes in two values (i.e. the legs of the right-angle triangle) and returns the length of the hypotenuse:

.. code-block:: python

   import math
   def calculate_hypotenuse(a, b):
       return math.sqrt(a**2 + b**2)

In order to write tests for this, we first have to come up with some sample data sets to test. We choose two scenarios, h\ :sup:`2` = 3\ :sup:`2` +4\ :sup:`2` and h\ :sup:`2` = 4\ :sup:`2` + 4\ :sup:`2` and we know the results should be 5 and 5.657 (rounded), respectively. We write the following tests to check these cases:

.. code-block:: python

   def test_calculate_hypotenuse_3_4():
       result = calculate_hypotenuse(3, 4)
       assert 5 == result

   def test_calculate_hypotenuse_4_4():
       result = calculate_hypotenuse(4, 4)
       assert 5.657 == result

Run the tests (the -v flag specifies a verbose test result output):

.. code-block:: bash

   $ pytest -v
   ======================================================= test session starts =======================================================
   test_sample.py::test_calculate_hypotenuse_3_4 PASSED    [ 50%]
   test_sample.py::test_calculate_hypotenuse_4_4 FAILED    [100%]


As you can see, one of our tests failed. PyTest gives us a very detailed error message indicating where the failure occurred:

.. code-block:: text

   __________________________________________ test_normal_4_4 __________________________________________
   def test_calculate_hypotenuse_4_4():
   >
   E
   result = calculate_hypotenuse(4, 4)
   assert 5.657 == result
   assert 5.657 == 5.656854249492381


.. _section_heading-Expected_Value_Tolerance:

Expected Value Tolerance
------------------------
The above failure occurs because the expected result does not match the actual result with sufficient precision.

We can use the :py:mod:`pytest` function :py:mod:`pytest.approx` to specify the desired level of precision:

.. code-block:: python

   def test_calculate_hypotenuse_4_4():
       result = calculate_hypotenuse(4, 4)
       assert 5.657 == pytest.approx(result, abs=0.001)

Here, the test will pass if the result of the calculation falls between 5.656 and 5.658, as the tolerance was specified to be the absolute value of 0.001.



.. _section_heading-Parametrized_Tests:

Parametrized Tests
------------------

You will notice that the above tests involve quite a bit of repetition; you call the function with two values, capture the return, and compare the return value to expected value within a certain level of precision.

Whenever tests involve repetition, it is a good opportunity to make use of a parametrized test. This type of tests allows multiple data sets to be applied to the same set of test steps.Test parametrization is done using the ``@pytest.mark.parametrize`` decorator, where the first argument is a string denoting a comma separated list of variables to pass to the function. The second argument is a list that contains one or more tuples of identical size as the number of arguments in the list.

.. code-block:: python

   @pytest.mark.parametrize("var_a,var_b,var_c", [
       (var_a_1, var_b_1, var_c_1),
       (var_a_2, var_b_2, var_c_2),
   ])
   def test_mytest(var_a, var_b, var_c):
       pass

In our example, we can rewrite the two tests as follows:

.. code-block:: python

   @pytest.mark.parametrize("a,b,expected", [
       (3,4,5),
       (4,4,5.657),
   ])
   def test_calculate_hypotenuse(a, b, expected):
       result = calculate_hypotenuse(a, b)
       assert expected == pytest.approx(result, abs=0.001)

Now, when we run this test, we see two results are produced:

.. code-block:: text

   test_sample.py::test_calculate_hypotenuse[3-4-5] PASSED      [ 50%]
   test_sample.py::test_calculate_hypotenuse[4-4-5.657] PASSED  [100%]

You can see that the tests get the name of the generic test, suffixed with some information about the data set. This is not usually very helpful, as one of the characteristics of good tests is a succinct but descriptive name that tells the reader immediately what the test does. You can supply the ids argument to @pytest.mark.parametrize to supply a descriptive name for each test:

.. code-block:: python

   @pytest.mark.parametrize("a,b,expected", [
       (3,4,5),
       (4,4,5.657),
   ], ids=[
       'A=3, B=4, H=5; no decimals',
       'A=4, B=4, H=5.657; decimal result with 1e-03 precision'
   ])
   def test_calculate_hypotenuse(a, b, expected):
       result = calculate_hypotenuse(a, b)
       assert expected == pytest.approx(result, abs=0.001)

Now when we run the tests, you can see it is much easier to grasp the test just from the name presented:

.. code-block:: text

   test_sample.py::test_calculate_hypotenuse[A=3, B=4, H=5; no decimals] PASSED                              [ 50%]
   test_sample.py::test_calculate_hypotenuse[A=4, B=4, H=5.657; decimal result with 1e-03 precision] PASSED  [100%]


.. _section_heading-Skipped_Tests:

Skipped Tests
-------------
If there is a desire to skip certain tests in the test suite, you can mark them using the ``@pytest.mark.skip`` decorator:

.. code-block:: python

   deftest_run_this_test():
       pass
   @pytest.mark.skip
   def test_dont_run_this_test():
       pass

Running these tests would result in the following:

.. code-block:: text

   test_sample.py::test_run_this_test PASSED        [ 50%]
   test_sample.py::test_dont_run_this_test SKIPPED  [100%]

As you can see, the test still appears in the output report, but is marked as being skipped.

An alternative to always skipping certain tests is to skip them if certain conditions are met. For example, maybe certain features of a test are only compatible with certain versions of python. In this case, you could choose to skip based on the version of python using ``@pytest.mark.skipif(condition, reason)``.

.. code-block:: python

   @pytest.mark.skipif(sys.version_info < (3,0),reason="requires python3.0")
   def test_that_requires_python_3():
       pass

When running tests, the ``-ra`` option will produce extra test summary information, which includes the skip reasons.


.. code-block:: bash

   $ python2 -m pytest -vra
   test_sample.py::test_that_requires_python_3 SKIPPED   [100%]
   ============================================= short test summary info ==============================================
   SKIP [1] test_sample.py:11: requires python3.0


.. code-block:: bash

   $ python3 -m pytest -vra
   test_sample.py::test_that_requires_python_3 PASSED [100%]


An alternative to skipping the test entirely is to run it, but not fail the suite if the test itself fails. This is done using ``@pytest.mark.xfail``

.. code-block:: python

   @pytest.mark.xfail
   def test_that_sometimes_fails():
       assert random.randint(1,100) < 50

In the output report, this will produce the following if the test fails:

.. code-block:: text

   test_sample.py::test_that_sometimes_fails[100%] XPASS  [100%]

.. code-block:: text

   test_sample.py::test_that_sometimes_fails[100%] xfail  [100%]

As you can see, this still allows us to run the test and see results, but does not fail the overall test results if it happens to fail.

.. _section_heading-Verifying_Exception_Throwing:

Verifying Exception Throwing
----------------------------
Suppose one of our tests is to check whether the function throws an exception under certain circumstances.

.. code-block:: python

   def calculate_hypotenuse(a, b):
       if not a or not b:
           raise ValueError("Supplied lengths cannot be zero.")
       return math.sqrt(a**2 + b**2)


We can use the ``pytest.raises`` context manager to check for this:

.. code-block:: python

   @pytest.mark.parametrize("a,b", [
       (0, 1),
       (1, 0),
   ])
   def test_zero_value_exception(a, b):
       with pytest.raises(ValueError):
           calculate_hypotenuse(a, b)

   Running the test results in the following output:

.. code-block:: text

   test_sample.py::test_zero_value_exception[0-1] PASSED   [ 50%]
   test_sample.py::test_zero_value_exception[1-0] PASSED   [100%]


.. _section_heading-Test_Fixtures:

Test Fixtures
-------------

.. _section_heading-Using_Test_Fixtures:

Using Test Fixtures
^^^^^^^^^^^^^^^^^^^
If tests repeatedly have the same setup steps in the **Arrange** portion of the test, it is a good opportunity to create a test fixture. Think of a test fixture like a little helper function that can be re-used in multiple tests. This saves a tester from having to repeat the same code over and over in each test. To use the fixture, pass it's name as an argument to the test in question.

.. code-block:: python

   @pytest.fixture()
   def fixture_name():
       print ("\nSetting up")
       return obj # provide the fixture value

   def test_that_uses_fixture(fixture_name):
       print ("\nRunning a test")

Running this with the ``-s`` flag so we can see print statements, we observe:

.. code-block:: bash

   $ pytest -vs
   =============================================== test session starts ================================================
   test_fixtures.py::test_that_uses_fixture
   Setting up
   Running a test
   PASSED

Often, if there is setup, there is some sort of cleanup afterwards (sometimes called *teardown*). In order for our test to use this, we use a :py:keyword:`yield` instead of a :py:keyword:`return`.

.. code-block:: python

   @pytest.fixture()
   def fixture_name():
       print ("\nSetting up")
       yield obj  # provide the fixture value
       print ("\nCleaning up")

   def test_that_uses_fixture(fixture_name):
       print ("\nRunning a test")

After running the test, we see the following output:

.. code-block:: bash

   $ pytest -vs test_fixtures.py
   =============================================== test session starts ================================================
   test_fixtures.py::test_that_uses_fixture
   Setting up
   Running a test
   PASSED
   Cleaning up


.. _section_heading-Test_Fixture_Scope:

Test Fixture Scope
^^^^^^^^^^^^^^^^^^
Test fixtures have the concept of scope; this is equivalent to their lifetime. A fixture with a function scope (default) only lives for the duration of the test that uses it. Other scope options include:

* class: fixture lives for the duration of all tests inside a test class
* module: fixture lives for the duration of all tests in a single test module
* session: fixture lives for the duration of the entire PyTest run, which can span multiple files

Taking our previous example, adding a second test, and changing the fixture scope to module would result in the following:

.. code-block:: python

   @pytest.fixture(scope='module')
   def fixture_name():
       print ("\nSetting up")
       yield obj  # provide the fixture value
       print ("\nCleaning up")

   def test_first_test(fixture_name):
       print ("\nRunning test 1")

   def test_second_test(fixture_name):
       print ("\nRunning test 2")

Running these tests would result in the following output:

.. code-block:: bash

   $ pytest -vs test_fixtures.py
   =============================================== test session starts ================================================
   test_fixtures.py::test_first_test
   Setting up
   Running test 1
   PASSED
   test_fixtures.py::test_second_test
   Running test 2
   PASSED
   Cleaning up

As you can see, despite both tests including the fixture, it was only called once for the entire scope that we specified; in this case, once for the entire module.


.. _section_heading-Fixed_A_Real_Example:

Fixtures (a real example)
^^^^^^^^^^^^^^^^^^^^^^^^^

Imagine you were unit testing a function which writes to a file. This function does not open or close the file.

.. code-block:: python

   def write_message_to_file(file, message):
       return file.write(message)

In order to test this, each test would need to open a new file for writing, pass this file into the code under test, and be responsible for closing the file afterwards. That's a lot of repetition!

.. code-block:: python

   def test_one():
       f = open("/tmp/test.txt", "a+")
       length = write_message_to_file(f, "test1")
       assert 5 == length
       f.close()

   def test_two():
       f = open("/tmp/test.txt", "a+")
       length = write_message_to_file(f, "mytest2")
       assert 7 == length
       f.close()

This is where fixtures can save us a lot of time. The fixture can take care of opening the file (the setup) and closing (the teardown).

.. code-block:: python

   @pytest.fixture()
   def file_output():
       f = open("/tmp/test.txt", "a+")
       yield f
       f.close()

   def test_one(file_output):
       length = write_message_to_file(file_output, "test1")
       assert 5 == length

   def test_two(file_output):
       length = write_message_to_file(file_output, "mytest2")
       assert 7 == length