Testing#

Estimated reading time: 35 minutes

codecov

Here we provide some high-level guidelines for contributors on how-to test geovista with pytest, along with a summary of the automated quality assurance infrastructure.

Continuous Integration#

Reference:

  • .github/workflows

  • pyproject.toml

  • requirements

  • tests

The following testing workflows are available:

Table 20: Testing Workflows#

Workflow

Description

ci-test

The ci-tests.yml Action executes both the Image Tests and Unit Tests for each SPEC 0 supported distribution of python.

A codecov report is generated only for the latest supported distribution of python.

The tests floors job runs the same tests, without the image tests, in the locked floors environment, which solves each direct dependency at its floor on the oldest supported distribution of python (floors spec §4.1). Its check is required on main, so the pull request that breaks a floor is the one that raises it (floors spec §2 decision 11). To run it locally:

$ pixi run --frozen -e floors tests-unit "not image"

When it fails on a bot’s pull request, raise the floor where the fix can merge: on the branch of a dependabot pull request, as for a ceiling, and in a pull request merged ahead of the weekly lock pull request, whose branch ci-locks.yml rewrites each week.

Also see the testing Pixi Workflow tests-unit task.

ci-docs

The ci-tests-docs.yml Action executes both the documentation image tests and the Browser Tests for the latest SPEC 0 supported distribution of python.

Also see the documentation Pixi Workflow tests-docs task, and the testing Pixi Workflow tests-docs-browser task.

ci-lock

The ci-tests-lock.yml cron based Action regularly schedules the execution of both the Image Tests and Unit Tests for the latest SPEC 0 supported distribution of python.

Prior to running the tests pixi performs an ephemeral refresh of the pixi.lock file based on the pyproject.toml manifest and the latest package versions available from the conda and PyPI ecosystems.

The purpose of this Action is to preemptively test against the latest resolved python variant of the geovista-py3xx environment to uncover and highlight any issues prior to refreshing the pixi.lock file with the scheduled ci-locks.yml Action. See the packaging Continuous Integration section for further details.

Any subsequent test failures will result in a bespoke Issue being raised for contributors to investigate referencing the Action run-id of the failed job.

ci-pypi

The ci-tests-pypi.yml cron based Action regularly schedules the execution of both the Image Tests and Unit Tests for the latest SPEC 0 supported distribution of python.

Prior to running the tests pip installs the latest package dependencies from the PyPI ecosystem. See the packaging Python Package Index section for further details.

The purpose of this Action is to preemptively test against the latest resolved PyPI environment to uncover and highlight any issues prior to the scheduled dependabot.yml orchestrated refresh of our PyPI package dependencies.

Any subsequent test failures will result in a bespoke Issue being raised for contributors to investigate referencing the Action run-id of the failed job.

ci-spec

The ci-spec-status.yml Action checks the work that each design specification status cites, by asking about each pull request and issue it references. See Specifications.

On a pull-request that changes a specification, it checks that a landed roadmap row cites merged pull requests on its date, and that a Resolved or Abandoned item cites real work. A status may cite the pull-request itself while it is still open.

Each night it also reports a status the work has moved past, such as an in progress row whose pull request has merged. A finding raises a bespoke Issue for contributors to investigate.

Also see the testing Pixi Workflow check-spec-status task.

Image Tests#

Reference:

  • pyproject.toml

  • src/geovista/examples

  • tests/plotting

This section outlines how to create and run our image tests, and also highlights the key infrastructure that underpins it.

Markers#

Reference:

  • pyproject.toml

The following pytest markers are configured for image unit tests:

Table 21: Image Unit Test Markers#

Marker

Description

image

Generic marker to be used on all image unit tests.

example

Specific marker only to be used on Examples gallery unit tests.

The following marker expressions can be used for finer-grained control of unit test selection for execution:

Table 22: Marker Expressions#

Marker Expression

Description

pytest -m image

Execute all image unit tests.

pytest -m "not image"

Execute all unit tests except the image unit tests.

pytest -m example

Execute only the gallery unit tests.

pytest -m "not example"

Execute all unit tests except the gallery unit tests.

pytest -m "image and not example"

Execute all image unit tests that are not gallery unit tests.

Fixtures#

Reference:

  • tests/conftest.py

All image unit tests require to use the following pytest fixtures:

Table 23: Image Unit Test Fixtures#

Fixture

Description

plot_nodeid

This fixture generates the unique dotted nodeid to identify the unit test relative to the tests root directory e.g., geodesic.test_BBox.test_outline is the nodeid of the test_outline unit test defined within the test_BBox.py file located within the tests/geodesic directory.

The fixture also ensures that the baseline image for the unit test is available in the image cache. See Caching for further details.

verify_image_cache

The pytest-pyvista pytest plugin is used to perform the image comparison between a unit test baseline image (expected) against its pyvista rendered image (actual).

This fixture requires the unit test to call the show() method as a hook so that it can initiate the image comparison.

Template#

Reference:

  • pyproject.toml

  • tests/conftest.py

Image unit tests should adopt the following usage pattern of Markers and Fixtures e.g.,

Listing 1: Image Unit Test Template#
 1import pytest
 2
 3import geovista as gv
 4
 5
 6@pytest.mark.image
 7def test(plot_nodeid, verify_image_cache):
 8    """Single line doc-string."""
 9    verify_image_cache.test_name = plot_nodeid
10    p = gv.GeoPlotter()
11    ...
12    p.show()

Note that:

  • line: 6 - The @pytest.mark.image marker must always decorate the unit test

  • line: 7 - Both the plot_nodeid and verify_image_cache fixures are used

  • line: 9 - The verify_image_cache fixure is configured with the name of the unit test baseline image

  • line: 12 - The show() method of the plotter is called to render the unit test scene and then invoke image comparison via the pytest-pyvista plugin configured by its verify_image_cache fixture. Be aware that the show() method must only be invoked once within the unit test. However the test itself may be decorated with the @pytest.mark.parametrize marker to execute it multiple times.

Caching#

Reference:

  • pyproject.toml

  • src/geovista/cache/__init__.py

  • src/geovista/config.py

  • src/geovista/tests/plotting/__init__.py

  • src/geovista/tests/plotting/unit_image_cache

The pytest-pyvista plugin performs the image comparison between a unit test baseline image (expected) and rendered image (actual) via its verify_image_cache fixture.

Unit test baseline images are automatically fetched and cached by the pooch CACHE manager on request by the plot_nodeid fixture.

The CACHE manager fetches assets for a specific DATA_VERSION release tag from the geovista-data repository and stores them according to the resource configuration or the GEOVISTA_CACHEDIR environment variable.

These cached assets are made available to the verify_image_cache fixture via the tests/plotting/unit_image_cache symbolic-link, as configured within the [tool.pytest.ini_options] table of the pyproject.toml manifest.

The tests/plotting/unit_image_cache symbolic-link to the CACHE manager image cache is automatically verified and managed by the tests.plotting module.

Rendering#

Reference:

  • pyproject.toml

  • tests/plotting/test_examples.py

CI renders the image tests with llvmpipe, Mesa’s software rasterizer, and the baseline images match its output. The pixi environment carries the same Mesa build. A hardware GPU, though, draws some scenes differently enough to exceed the image tolerances, so generate and compare images with llvmpipe wherever you can.

To see which renderer VTK will use e.g.,

$ pixi run python -c "import pyvista as pv; print(pv.GPUInfo().renderer)"

If it names a hardware GPU, set the GALLIUM_DRIVER environment variable to select llvmpipe, and run the command again to confirm it e.g.,

$ GALLIUM_DRIVER=llvmpipe pixi run python -c "import pyvista as pv; print(pv.GPUInfo().renderer)"
llvmpipe (LLVM 23.1.3, 256 bits)

The variable only takes effect where the OpenGL driver in use is Mesa’s. Where it has no effect, rely on CI to compare images. If the command fails instead, VTK has nothing to render with, so run under xvfb-run -a, as CI does.

A few gallery examples have image tolerances relaxed in CI only. These are the thresholds in tests/plotting/test_examples.py, applied when the CI environment variable is true. Set both variables to reproduce a CI run e.g.,

$ CI=true GALLIUM_DRIVER=llvmpipe pixi run tests-unit "image"

The same applies to the documentation image tests of the tests-docs task, see Pixi Workflow.

Generation#

Reference:

  • pyproject.toml

  • src/geovista/cache/__init__.py

  • src/geovista/cache/registry.txt

Unit test rendered image (actual) generation is managed through the pytest-pyvista plugin.

For example, to serizalize all unit test rendered images to file within the test_images directory e.g.,

$ pytest -m image --generated_image_dir test_images

To serialize both the baseline image (expected) and rendered image (actual) of all unit test failures to file within the test_images_failed directory e.g.,

$ pytest -m image --failed_image_dir test_images_failed

Typically, we recommend the following:

  • Use the --generated_image_dir option to capture the rendered image (actual) of any new or existing unit tests

  • Use the --failed_image_dir option to capture and investigate any failed image unit tests

To register a new baseline image for a unit test:

  1. Generate the baseline image with llvmpipe (see Rendering), using the --generated_image_dir option to the pytest-pyvista plugin

  2. Add the new baseline image to the assets/tests/unit directory of the geovista-data repository via a pull-request

  3. Once the pull-request has been reviewed and merged, a new geovista-data release will be tagged automatically

  4. Update the DATA_VERSION within geovista.cache to the new geovista-data release version

  5. Finally, update the registry.txt with the unit test baseline image name (relative to the assets directory) along with the hexadecimal SHA256SUM of the baseline image file contents

The pooch CACHE manager will automatically download and cache the new baseline image from the geovista-data repository when re-running the unit test. Alternatively, download all the new assets for the DATA_VERSION using the CLI e.g.,

$ geovista download --all --decompress

Browser Tests#

Reference:

  • pyproject.toml

  • docs

  • tests/docs

The browser unit tests drive a headless chromium browser over a built documentation site using playwright, in order to cover the theme chrome that is assembled in the browser and is therefore beyond the reach of any other unit test i.e., the primary and secondary sidebar toggles, the dialogs that they open, the Examples gallery carousel, the Tooltips, which are attached to a page only once it has loaded, and the colours of the section cards, which the theme switches between light and dark mode.

The tests/docs directory also carries unit tests of the sphinx configuration itself, which require no browser. Some build a throwaway documentation set in-process, such as those covering the Intersphinx fallback behaviour; others read the built site directly, such as those covering which links are given a tooltip. They share the skip-or-require guard described below, and the whole directory is selected for execution instead of the browser marker, so that all of them are covered by the same Action.

The tests/docs/test_readingtime_coverage.py gate is the one module there with nothing to skip on. It reads the documentation sources as text, needing neither a browser nor a build nor sphinx itself, and holds every published page to the banner rule. The pages it governs are derived from docs/src rather than listed, so a page is covered from the day it lands.

Prerequisites#

Reference:

  • pyproject.toml

  • requirements/pypi-optional-test.txt

  • tests/docs/conftest.py

The browser unit tests require the following:

Table 24: Browser Unit Test Prerequisites#

Prerequisite

Description

playwright

The python bindings that drive the browser, which are sourced from PyPI rather than conda-forge, as the conda-forge package of the same name provides the node command line interface but no python bindings.

Declared in the [tool.pixi.feature.test.pypi-dependencies] table of the pyproject.toml manifest, and in requirements/pypi-optional-test.txt.

chromium

The browser itself, which playwright manages separately from its python bindings. Install it once with the tests-docs-browser-install pixi task.

A documentation build

The tests are performed against the docs/_build/html site, which is opened over file://: no web server is required. See Building for further details.

Each of the above is guarded by a pytest fixture that skips rather than fails when its prerequisite is unavailable, so a plain pytest run remains unaffected for contributors who have neither playwright nor a documentation build.

That is the wrong default for ci-tests-docs.yml, which installs every prerequisite and builds the gallery deliberately, and so would report success having silently stopped covering anything. The --browser-strict option requires each prerequisite instead e.g.,

$ pytest tests/docs --browser-strict

Markers#

Reference:

  • pyproject.toml

The following pytest marker is configured for browser unit tests:

Table 25: Browser Unit Test Markers#

Marker

Description

browser

Generic marker to be used on all documentation theme chrome unit tests.

The marker is applied to an entire test module with a pytestmark global e.g.,

Listing 2: Browser Unit Test Marker#
1import pytest
2
3pytestmark = pytest.mark.browser

A module that mixes browser and non-browser unit tests, such as tests/docs/test_tooltips.py, decorates each browser unit test with @pytest.mark.browser instead, so that the marker expressions below still select the remainder.

The following marker expressions can be used for finer-grained control of unit test selection for execution:

Table 26: Browser Marker Expressions#

Marker Expression

Description

pytest -m browser

Execute all browser unit tests.

pytest -m "not browser"

Execute all unit tests except the browser unit tests.

Fixtures#

Reference:

  • tests/docs/conftest.py

The following pytest fixtures are available to browser unit tests:

Table 27: Browser Unit Test Fixtures#

Fixture

Description

browser

A launched headless chromium browser, shared for the duration of the test session and closed on teardown. Under pytest-xdist each worker is a session of its own, and so launches its own browser.

goto

A factory that opens a built page at a given viewport width, defaulting to index.html e.g., page = goto(390).

The factory waits for the theme chrome to settle before handing the page back, and closes every page that it opened on teardown.

html_root

The root directory of the documentation build, docs/_build/html.

api_page

The path of a built API reference page, relative to html_root.

Such a page carries a secondary On this page sidebar with enough entries to be worth revealing, which the landing page does not.

require

A guard for a prerequisite that only a unit test itself can detect, such as the gallery carousel, which skips or, under --browser-strict, fails e.g.,

require("no gallery carousel in this build").

Helpers#

Reference:

  • tests/docs/_theme.py

The tests/docs/_theme.py module provides the helpers that browser unit tests use to interrogate the theme chrome, such as click_visible to activate a toggle, dialog to report the state of a sidebar dialog, and sidebar_offset to distinguish a sidebar that collapses in place from one that opens as a dialog.

Both themes render a duplicate of each toggle and hide one of them, so “the button” is ambiguous: the helpers consistently act upon the one that a reader can actually see.

Pixi Workflow#

Reference:

  • pyproject.toml

Testing may be performed using the convenience of pixi run tasks, which do not require to be executed from within geovista root directory e.g.,

$ pixi run <pixi-task>
Table 28: Pixi Tasks#

Pixi Task

Description

download

Download and cache offline assets.

This task calls the geovista download command. Provide optional argument all, clean, doc-images, dry-run, images, list, natural-earth, operational, pantry, rasters, unit-images or verify. Defaults to all e.g.,

$ pixi run download rasters

check-spec-status

Check the work each design specification status cites against GitHub. See Specifications e.g.,

$ pixi run check-spec-status

Provide the number of a pull request to check as that pull request, which a status may cite while it is still open, or drift to also report a status the work has moved past e.g.,

$ pixi run check-spec-status drift

check-spec-renumbering

Check the specification citations against a renumbered section. See Specification Renumbering e.g.,

$ pixi run check-spec-renumbering

Provide fix to rewrite the citations of moved sections in place, or audit to compare every citation with its specification when it was written e.g.,

$ pixi run check-spec-renumbering fix

benchmarks

Perform the benchmarks under CodSpeed, without memray. See Benchmarks e.g.,

$ pixi run benchmarks

benchmarks-memory

Perform the benchmarks under memray, holding each to its memory ceiling e.g.,

$ pixi run benchmarks-memory

tests-clean

Purge both the documentation and unit test image caches, along with any images generated from previous test sessions e.g.,

$ pixi run tests-clean

tests-unit

Perform the unit tests.

This task calls the pytest command. Defaults to executing all unit tests discoverable from the geovista root directory.

Accepts a valid pytest marker expression as an optional argument. Refer to the [tool.pytest.ini_options] table entry in the pyproject.toml manifest for configured markers e.g.,

$ pixi run tests-unit "not image"

Runs the unit tests in parallel, one worker per CPU core (see Parallel Testing). Accepts the number of workers as an optional second argument, where 0 runs them serially in a single process e.g.,

$ pixi run tests-unit "not image" 0

Note that the tests-clean task is called prior to running this task.

tests-docs-browser

Perform the browser unit tests.

This task calls the pytest command for the tests/docs directory.

Accepts an optional documentation build target as an argument, which is built first. Defaults to html-noplot e.g.,

$ pixi run tests-docs-browser html-gallery

A non-empty second argument additionally passes the --browser-strict option, which requires rather than skips each prerequisite e.g.,

$ pixi run tests-docs-browser html-gallery strict

tests-docs-browser-install

Install the chromium browser required by the Browser Tests e.g.,

$ pixi run tests-docs-browser-install

This is a one-off, as playwright manages the browser separately from its python bindings.

verification-report

Report each design specification’s decisions, roadmap and open items, and where each decision is cited. See Decisions and Verification e.g.,

$ pixi run verification-report

Provide a specification’s prefix, such as heights, to report that specification alone. It is never a check: it exits 0 whatever it finds.

Unit Tests#

Reference:

  • pyproject.toml

  • tests

The unit tests are located within the tests root directory and are organised into sub-directories, typically one for each geovista top-level module or sub-package.

The exceptions to this rule are tests/plotting, which contains the Image Tests, and tests/docs, which contains the Browser Tests.

To execute all unit tests:

$ pytest

Finer-grained control of unit test can be achieved by using our pytest image and browser markers.

Parallel Testing#

Reference:

  • .github/workflows

  • pyproject.toml

  • src/geovista/cache/__init__.py

The tests-unit task and continuous integration both spread the unit tests over several worker processes with the pytest-xdist plugin, which is several times faster than a serial run on a multi-core machine:

$ pytest --numprocesses auto --dist worksteal

Plain pytest still runs serially, which suits debugging with --pdb or stopping at the first failure with -x. To run the task serially, pass 0 as its second argument.

The tests-docs-browser task runs the Browser Tests in parallel too, and takes the number of workers as its third argument.

Each worker is a separate process with its own module state, and takes the next free test as it finishes one. A unit test must pass whatever order the tests run in, and must not rely on another test having run first in the same process.

Benchmarks#

Reference:

  • .github/workflows/ci-benchmarks.yml

  • benchmarks

  • pyproject.toml

The benchmarks measure the speed and the memory of geovista’s computation, and live in the benchmarks root directory, apart from the unit tests, which never collect them. The benchmarks specification, bench spec §2, records why each choice was made.

The ci-benchmarks.yml workflow runs them on every pull request and every push to main, in two jobs:

  • codspeed benchmarks runs them under CodSpeed in simulation mode, which counts the CPU instructions each one runs under Valgrind. A count does not move with the load on a shared runner, so CodSpeed compares each pull request with the result stored for its base commit, and comments on the pull request with what changed.

  • memory ceilings runs them under memray, through the pytest-memray plugin, and fails a benchmark whose peak memory passes its ceiling.

Neither job is a required check. A regression is reported on the pull request that causes it, including one caused by a change to pixi.lock, and the maintainer decides whether it merges (bench spec §2 decision 4).

CodSpeed reports in two places on a pull request. Its comment lists each benchmark that got faster or slower than on the base commit, by how much, and links each to its page on CodSpeed, where a flame graph shows which calls the difference lies in. Its check, CodSpeed Performance Analysis, fails when a benchmark slows past CodSpeed’s threshold. A failing check never blocks the merge. Look into the cause, and if the slowdown is the price of the change, as it was in #2912, say so in the pull request, so the maintainer can weigh it. A new benchmark is listed as new, and is compared from the next commit of main on.

To run them locally:

$ pixi run benchmarks
$ pixi run benchmarks-memory

Without Valgrind, the benchmarks task measures wall time, which is enough to compare two branches on one machine. Both tasks run OpenMP on a single thread, as the workflow does. OpenBLAS picks its kernel for the CPU, and the buffers it sets up on the first matrix product depend on that kernel, so benchmarks/conftest.py makes that product before any benchmark runs. A ceiling measured on one machine then holds on any runner.

Adding a Benchmark#

A benchmark is a test in benchmarks/test_<module>.py, one module for each geovista module it measures. It hands the call it measures to the benchmark fixture, and carries its memory ceiling:

@pytest.mark.limit_memory("22 MB")
def test_line(benchmark, line):
    """Build a geodesic line through 2,000 vertices."""
    result = benchmark(geodesic.line, *line)
    assert result.n_points > line[0].size

Build its input in a fixture of benchmarks/conftest.py, outside the measured call, and from a seeded generator where it is random. Download nothing, and render nothing.

The ceiling is the benchmark’s measured peak plus 20%, rounded up to a whole MiB (bench spec §2 decision 5). To measure it, give the benchmark a generous placeholder, such as limit_memory("1 GB"), and run:

$ pixi run benchmarks-memory

Read the benchmark’s peak from the “📦 Total memory allocated” line under its “Allocation results for” heading in the report, add 20%, and round up. pytest-memray reads MB as 1024² bytes, a MiB. A benchmark with no ceiling is refused when --memray is given, naming it. A change that rightly raises a peak raises its ceiling in the same pull request, where the review sees it.

The task passes --trace-python-allocators, so memray counts the objects Python allocates. Without it, memray sees the arenas Python’s own allocator takes from the system a mebibyte at a time, and a benchmark’s peak then depends on the arenas the benchmarks before it left free: one run alone could breach a ceiling it keeps in the full run.