Testing#
Estimated reading time: 35 minutes
Here we provide some high-level guidelines for contributors on how-to test
geovista with pytest, along with a summary of the automated quality assurance
infrastructure.
Note
The unit tests are not bundled within the geovista package distributed
on conda-forge or PyPI.
Continuous Integration#
Reference:
.github/workflowspyproject.tomlrequirementstests
Note
We follow Scientific Python Ecosystem Coordination (SPEC)
recommendation for Minimum Supported Dependencies (SPEC 0), and
typically support the latest 2-3 distributions of python.
The following testing workflows are available:
Workflow |
Description |
|---|---|
The ci-tests.yml Action executes both the Image Tests
and Unit Tests for each SPEC 0 supported distribution of A codecov report is generated only for the latest supported distribution of The $ pixi run --frozen -e floors tests-unit "not image"
When it fails on a bot’s pull request, raise the floor where the fix can merge: on the branch of a dependabot pull request, as for a ceiling, and in a pull request merged ahead of the weekly lock pull request, whose branch ci-locks.yml rewrites each week. Also see the testing Pixi Workflow tests-unit task. Note Failed Image Tests are captured via the |
|
The ci-tests-docs.yml Action executes both the documentation image tests and the
Browser Tests for the latest SPEC 0 supported distribution of
Also see the documentation Pixi Workflow tests-docs task, and the testing Pixi Workflow tests-docs-browser task. Note The Browser Tests require a built documentation site, so this job
builds the html-gallery target: the cheapest build that renders the gallery, and therefore
the only one carrying the carousel that the browser tests cover. It is run with the When this job fails, it uploads the output of |
|
The ci-tests-lock.yml Prior to running the tests The purpose of this Action is to preemptively test against the latest resolved Any subsequent test failures will result in a bespoke Issue being raised for contributors to
investigate referencing the Action |
|
The ci-tests-pypi.yml Prior to running the tests The purpose of this Action is to preemptively test against the latest resolved Any subsequent test failures will result in a bespoke Issue being raised for contributors to
investigate referencing the Action Attention pyvista performs a |
|
The ci-spec-status.yml Action checks the work that each design specification status cites, by asking about each pull request and issue it references. See Specifications. On a Each night it also reports a status the work has moved past, such as an Also see the testing Pixi Workflow check-spec-status task. |
Image Tests#
Reference:
pyproject.tomlsrc/geovista/examplestests/plotting
This section outlines how to create and run our image tests, and also highlights the key infrastructure that underpins it.
Markers#
Reference:
pyproject.toml
The following pytest markers are configured for image unit tests:
Marker |
Description |
|---|---|
image |
Generic marker to be used on all image unit tests. |
example |
Specific marker only to be used on Examples gallery unit tests. |
See also
For further details see the markers array entry in the [tool.pytest.ini_options]
table of the pyproject.toml manifest.
The following marker expressions can be used for finer-grained control of unit test selection for execution:
Marker Expression |
Description |
|
Execute all image unit tests. |
|
Execute all unit tests except the image unit tests. |
|
Execute only the gallery unit tests. |
|
Execute all unit tests except the gallery unit tests. |
|
Execute all image unit tests that are not gallery unit tests. |
See also
Refer to the Pixi Workflow tests-unit task.
Attention
pytest must be executed from within the geovista root directory.
Important
Mark every unit test that draws with image. That includes any test that
calls show, screenshot, render, plot or an export_*
method, directly or through a helper or fixture. Rendering needs a display
or a GPU, and crashes without one, so pytest -m "not image" is only safe
on such a machine if the marker is never missing. Every test marked
example must also be marked image.
The tests/test_image_marker.py unit test reads the source of every
test module and fails on a test that breaks either rule.
Fixtures#
Reference:
tests/conftest.py
All image unit tests require to use the following pytest fixtures:
Fixture |
Description |
|
This fixture generates the unique dotted The fixture also ensures that the baseline image for the unit test is available in the image cache. See Caching for further details. |
|
The pytest-pyvista This fixture requires the unit test to call the
See also For further details see |
Template#
Reference:
pyproject.tomltests/conftest.py
Image unit tests should adopt the following usage pattern of Markers and Fixtures e.g.,
1import pytest
2
3import geovista as gv
4
5
6@pytest.mark.image
7def test(plot_nodeid, verify_image_cache):
8 """Single line doc-string."""
9 verify_image_cache.test_name = plot_nodeid
10 p = gv.GeoPlotter()
11 ...
12 p.show()
Note that:
line: 6- The@pytest.mark.imagemarker must always decorate the unit testline: 7- Both theplot_nodeidandverify_image_cachefixures are usedline: 9- Theverify_image_cachefixure is configured with the name of the unit test baseline imageline: 12- Theshow()method of the plotter is called to render the unit test scene and then invoke image comparison via thepytest-pyvistaplugin configured by itsverify_image_cachefixture. Be aware that theshow()method must only be invoked once within the unit test. However the test itself may be decorated with the@pytest.mark.parametrizemarker to execute it multiple times.
Caching#
Reference:
pyproject.tomlsrc/geovista/cache/__init__.pysrc/geovista/config.pysrc/geovista/tests/plotting/__init__.pysrc/geovista/tests/plotting/unit_image_cache
The pytest-pyvista plugin performs the image comparison between a
unit test baseline image (expected) and rendered image (actual) via its
verify_image_cache fixture.
Unit test baseline images are automatically fetched and cached by the pooch
CACHE manager on request by the plot_nodeid fixture.
See also
The geovista download command may be used to cache offline assets.
The CACHE manager fetches assets for a specific
DATA_VERSION release tag from the geovista-data repository
and stores them according to the resource configuration
or the GEOVISTA_CACHEDIR
environment variable.
These cached assets are made available to the verify_image_cache fixture via
the tests/plotting/unit_image_cache symbolic-link, as configured
within the [tool.pytest.ini_options] table of the pyproject.toml manifest.
The tests/plotting/unit_image_cache symbolic-link to the CACHE
manager image cache is automatically verified and managed by the tests.plotting module.
Rendering#
Reference:
pyproject.tomltests/plotting/test_examples.py
CI renders the image tests with llvmpipe, Mesa’s software rasterizer, and the
baseline images match its output. The pixi environment carries the same Mesa
build. A hardware GPU, though, draws some scenes differently enough to exceed the
image tolerances, so generate and compare images with llvmpipe wherever you can.
To see which renderer VTK will use e.g.,
$ pixi run python -c "import pyvista as pv; print(pv.GPUInfo().renderer)"
If it names a hardware GPU, set the
GALLIUM_DRIVER environment variable to
select llvmpipe, and run the command again to confirm it e.g.,
$ GALLIUM_DRIVER=llvmpipe pixi run python -c "import pyvista as pv; print(pv.GPUInfo().renderer)"
llvmpipe (LLVM 23.1.3, 256 bits)
The variable only takes effect where the OpenGL driver in use is Mesa’s. Where it
has no effect, rely on CI to compare images. If the command fails instead, VTK has
nothing to render with, so run under xvfb-run -a, as CI does.
A few gallery examples have image tolerances relaxed in CI only. These are the
thresholds in tests/plotting/test_examples.py, applied when the
CI environment variable is true. Set both variables to reproduce a
CI run e.g.,
$ CI=true GALLIUM_DRIVER=llvmpipe pixi run tests-unit "image"
The same applies to the documentation image tests of the tests-docs task, see Pixi Workflow.
Generation#
Reference:
pyproject.tomlsrc/geovista/cache/__init__.pysrc/geovista/cache/registry.txt
Unit test rendered image (actual) generation is managed through the
pytest-pyvista plugin.
For example, to serizalize all unit test rendered images to file within the
test_images directory e.g.,
$ pytest -m image --generated_image_dir test_images
To serialize both the baseline image (expected) and rendered image (actual) of
all unit test failures to file within the test_images_failed directory e.g.,
$ pytest -m image --failed_image_dir test_images_failed
Attention
pytest must be executed from within the geovista root directory.
Typically, we recommend the following:
Use the
--generated_image_diroption to capture the rendered image (actual) of any new or existing unit testsUse the
--failed_image_diroption to capture and investigate any failed image unit tests
See also
Refer to pytest-pyvista for further CLI and configuration options.
Also see the tests-unit pixi task in the
following Pixi Workflow section, as a
convenience.
To register a new baseline image for a unit test:
Generate the baseline image with
llvmpipe(see Rendering), using the--generated_image_diroption to thepytest-pyvistapluginAdd the new baseline image to the
assets/tests/unitdirectory of the geovista-data repository via apull-requestOnce the
pull-requesthas been reviewed and merged, a newgeovista-datarelease will be tagged automaticallyUpdate the DATA_VERSION within
geovista.cacheto the newgeovista-datarelease versionFinally, update the
registry.txtwith the unit test baseline image name (relative to theassetsdirectory) along with the hexadecimalSHA256SUMof the baseline image file contents
The pooch CACHE manager will automatically download
and cache the new baseline image from the geovista-data repository
when re-running the unit test. Alternatively, download all the new assets for
the DATA_VERSION using the CLI e.g.,
$ geovista download --all --decompress
Browser Tests#
Reference:
pyproject.tomldocstests/docs
The browser unit tests drive a headless chromium browser over a built
documentation site using playwright, in order to cover the theme chrome that is
assembled in the browser and is therefore beyond the reach of any other unit test
i.e., the primary and secondary sidebar toggles, the dialogs that they open, the
Examples gallery carousel, the
Tooltips, which are
attached to a page only once it has loaded, and the colours of the section cards,
which the theme switches between light and dark mode.
Note
The browser unit tests deliberately assert on what a reader can see and do, rather than on the markup of any particular theme release, so that they survive a theme bump and report what actually broke.
The tests/docs directory also carries unit tests of the sphinx
configuration itself, which require no browser. Some build a throwaway
documentation set in-process, such as those covering the
Intersphinx fallback
behaviour; others read the built site directly, such as those covering which
links are given a
tooltip.
They share the skip-or-require guard described below, and the whole directory is
selected for execution instead of the browser marker, so that all of
them are covered by the same Action.
The tests/docs/test_readingtime_coverage.py gate is the one module there
with nothing to skip on. It reads the documentation sources as text, needing
neither a browser nor a build nor sphinx itself, and holds every published
page to the
banner rule.
The pages it governs are derived from docs/src rather than listed, so a
page is covered from the day it lands.
Prerequisites#
Reference:
pyproject.tomlrequirements/pypi-optional-test.txttests/docs/conftest.py
The browser unit tests require the following:
Prerequisite |
Description |
|---|---|
|
The Declared in the |
|
The browser itself, which |
A documentation build |
The tests are performed against the |
Each of the above is guarded by a pytest fixture that skips
rather than fails when its prerequisite is unavailable, so a plain pytest run
remains unaffected for contributors who have neither playwright nor a
documentation build.
That is the wrong default for ci-tests-docs.yml, which installs every
prerequisite and builds the gallery deliberately, and so would report success
having silently stopped covering anything. The --browser-strict option
requires each prerequisite instead e.g.,
$ pytest tests/docs --browser-strict
Note
pytest honours the pytest_addoption hook only within the test root
tests/conftest.py, which is therefore where the --browser-strict
option is registered, not alongside the browser unit tests that
consume it.
See also
Refer to the Pixi Workflow
tests-docs-browser-install and tests-docs-browser tasks,
which satisfy the chromium and documentation build prerequisites
respectively.
Attention
The default html-noplot documentation build target sets
plot_gallery = False, which leaves the Examples gallery
carousel with nothing to render, so the carousel unit tests skip, or fail
under --browser-strict. Only the html-gallery target, or another
plotting target, covers them, and rendering the gallery thumbnails requires a
display.
Exactly like the Image Tests, the carousel is therefore only meaningfully exercised by the ci-tests-docs.yml Action. See Continuous Integration for further details.
Markers#
Reference:
pyproject.toml
The following pytest marker is configured for browser unit tests:
Marker |
Description |
|---|---|
browser |
Generic marker to be used on all documentation theme chrome unit tests. |
The marker is applied to an entire test module with a pytestmark global
e.g.,
1import pytest
2
3pytestmark = pytest.mark.browser
A module that mixes browser and non-browser unit tests, such as
tests/docs/test_tooltips.py, decorates each browser unit test with
@pytest.mark.browser instead, so that the marker expressions below
still select the remainder.
The following marker expressions can be used for finer-grained control of unit test selection for execution:
Marker Expression |
Description |
|---|---|
|
Execute all browser unit tests. |
|
Execute all unit tests except the browser unit tests. |
See also
For further details see the markers array entry in the
[tool.pytest.ini_options] table of the pyproject.toml manifest.
Attention
pytest must be executed from within the geovista root directory.
Fixtures#
Reference:
tests/docs/conftest.py
The following pytest fixtures are available to browser unit tests:
Fixture |
Description |
|---|---|
|
A launched headless |
|
A factory that opens a built page at a given viewport width, defaulting
to The factory waits for the theme chrome to settle before handing the page back, and closes every page that it opened on teardown. Attention Both themes inject their toggles from |
|
The root directory of the documentation build, |
|
The path of a built API reference page, relative to Such a page carries a secondary On this page sidebar with enough entries to be worth revealing, which the landing page does not. |
|
A guard for a prerequisite that only a unit test itself can detect,
such as the gallery carousel, which skips or, under
|
Helpers#
Reference:
tests/docs/_theme.py
The tests/docs/_theme.py module provides the helpers that browser unit tests
use to interrogate the theme chrome, such as click_visible to activate a
toggle, dialog to report the state of a sidebar dialog, and
sidebar_offset to distinguish a sidebar that collapses in place from one
that opens as a dialog.
Both themes render a duplicate of each toggle and hide one of them, so “the button” is ambiguous: the helpers consistently act upon the one that a reader can actually see.
Attention
The sphinx-design sd-stretched-link class covers its card through
an ::after overlay, so the bounding rectangle of the anchor itself is
no more than its text. Click coverage must therefore be verified by hit-testing
with document.elementFromPoint, and not by comparing rectangles.
A carousel also deliberately hangs its cards past its own clipping edge, so only those cards lying wholly inside the clipping rectangle can be hit-tested. Assert that at least one card qualifies, otherwise the unit test passes vacuously.
Pixi Workflow#
Reference:
pyproject.toml
Pixi Environment
The testing pixi tasks are available in the test,
geovista and py3xx variant of those pixi environments
e.g., geovista-py313, and in the floors environment. See Packaging
for further details.
Testing may be performed using the convenience of
pixi run tasks, which
do not require to be executed from within geovista root directory
e.g.,
$ pixi run <pixi-task>
Pixi Task |
Description |
|---|---|
download |
Download and cache offline assets. This task calls the geovista download
command. Provide optional argument $ pixi run download rasters
|
check-spec-status |
Check the work each design specification status cites against GitHub. See Specifications e.g., $ pixi run check-spec-status
Provide the number of a pull request to check as that pull
request, which a status may cite while it is still open, or
$ pixi run check-spec-status drift
|
check-spec-renumbering |
Check the specification citations against a renumbered section. See Specification Renumbering e.g., $ pixi run check-spec-renumbering
Provide $ pixi run check-spec-renumbering fix
|
benchmarks |
Perform the benchmarks under CodSpeed, without memray. See Benchmarks e.g., $ pixi run benchmarks
|
benchmarks-memory |
Perform the benchmarks under memray, holding each to its memory ceiling e.g., $ pixi run benchmarks-memory
|
tests-clean |
Purge both the documentation and unit test image caches, along with any images generated from previous test sessions e.g., $ pixi run tests-clean
|
tests-unit |
Perform the unit tests. This task calls the Accepts a valid $ pixi run tests-unit "not image"
Runs the unit tests in parallel, one worker per CPU core (see
Parallel Testing). Accepts
the number of workers as an optional second argument, where
$ pixi run tests-unit "not image" 0
Note that the tests-clean task is called prior to running this task. Note Failed Image Tests are
automatically captured via the Additionally all generated images are captured via the
|
tests-docs-browser |
Perform the browser unit tests. This task calls the Accepts an optional documentation build target as
an argument, which is built first. Defaults to
$ pixi run tests-docs-browser html-gallery
A non-empty second argument additionally
passes the $ pixi run tests-docs-browser html-gallery strict
Attention The default |
tests-docs-browser-install |
Install the $ pixi run tests-docs-browser-install
This is a one-off, as |
verification-report |
Report each design specification’s decisions, roadmap and open items, and where each decision is cited. See Decisions and Verification e.g., $ pixi run verification-report
Provide a specification’s prefix, such as |
Tip
Apply the --frozen option to avoid pixi checking and updating the pixi.lock file.
E.g., To execute all non-image based unit tests:
$ pixi run --frozen tests-unit "not image"
Unit Tests#
Reference:
pyproject.tomltests
The unit tests are located within the tests root directory and are
organised into sub-directories, typically one for each geovista top-level
module or sub-package.
The exceptions to this rule are tests/plotting, which contains the
Image Tests, and tests/docs, which
contains the Browser Tests.
To execute all unit tests:
$ pytest
See also
Refer to the Pixi Workflow tests-unit task.
Attention
pytest must be executed from within the geovista root directory.
Finer-grained control of unit test can be achieved by using our pytest
image and
browser markers.
Parallel Testing#
Reference:
.github/workflowspyproject.tomlsrc/geovista/cache/__init__.py
The tests-unit task and continuous integration both spread the unit tests over several worker processes with the pytest-xdist plugin, which is several times faster than a serial run on a multi-core machine:
$ pytest --numprocesses auto --dist worksteal
Plain pytest still runs serially, which suits debugging with --pdb or
stopping at the first failure with -x. To run the task serially, pass 0
as its second argument.
The tests-docs-browser task runs the Browser Tests in parallel too, and takes the number of workers as its third argument.
Each worker is a separate process with its own module state, and takes the next free test as it finishes one. A unit test must pass whatever order the tests run in, and must not rely on another test having run first in the same process.
Attention
All workers share the CACHE. When it starts empty,
for example after a DATA_VERSION bump, several
workers may request the same asset at once. geovista decompresses each
asset into a temporary file and renames it into place, so a worker never
reads a partly written file. Workers may still download the same asset more
than once; run the download task first to avoid that.
Benchmarks#
Reference:
.github/workflows/ci-benchmarks.ymlbenchmarkspyproject.toml
The benchmarks measure the speed and the memory of geovista’s computation,
and live in the benchmarks root directory, apart from the unit tests,
which never collect them. The benchmarks specification, bench spec §2, records
why each choice was made.
The ci-benchmarks.yml workflow runs them on every pull request and every push
to main, in two jobs:
codspeed benchmarksruns them under CodSpeed in simulation mode, which counts the CPU instructions each one runs under Valgrind. A count does not move with the load on a shared runner, so CodSpeed compares each pull request with the result stored for its base commit, and comments on the pull request with what changed.memory ceilingsruns them under memray, through the pytest-memray plugin, and fails a benchmark whose peak memory passes its ceiling.
Neither job is a required check. A regression is reported on the pull request
that causes it, including one caused by a change to pixi.lock, and the
maintainer decides whether it merges (bench spec §2 decision 4).
CodSpeed reports in two places on a pull request. Its comment lists each
benchmark that got faster or slower than on the base commit, by how much, and
links each to its page on CodSpeed, where a flame graph shows which calls the
difference lies in. Its check, CodSpeed Performance Analysis, fails when a
benchmark slows past CodSpeed’s threshold. A failing check never blocks the
merge. Look into the cause, and if the slowdown is the price of the change, as
it was in #2912, say so in the pull request, so the maintainer can weigh
it. A new benchmark is listed as new, and is compared from the next commit of
main on.
To run them locally:
$ pixi run benchmarks
$ pixi run benchmarks-memory
Without Valgrind, the benchmarks task measures wall time, which is
enough to compare two branches on one machine. Both tasks run OpenMP on a single
thread, as the workflow does. OpenBLAS picks its kernel for the CPU, and the
buffers it sets up on the first matrix product depend on that kernel, so
benchmarks/conftest.py makes that product before any benchmark runs. A
ceiling measured on one machine then holds on any runner.
Adding a Benchmark#
A benchmark is a test in benchmarks/test_<module>.py, one module for
each geovista module it measures. It hands the call it measures to the
benchmark fixture, and carries its memory ceiling:
@pytest.mark.limit_memory("22 MB")
def test_line(benchmark, line):
"""Build a geodesic line through 2,000 vertices."""
result = benchmark(geodesic.line, *line)
assert result.n_points > line[0].size
Build its input in a fixture of benchmarks/conftest.py, outside the
measured call, and from a seeded generator where it is random. Download nothing,
and render nothing.
The ceiling is the benchmark’s measured peak plus 20%, rounded up to a whole
MiB (bench spec §2 decision 5). To measure it, give the benchmark a generous
placeholder, such as limit_memory("1 GB"), and run:
$ pixi run benchmarks-memory
Read the benchmark’s peak from the “📦 Total memory allocated” line under its
“Allocation results for” heading in the report, add 20%, and round up.
pytest-memray reads MB as 1024² bytes, a MiB. A benchmark with no ceiling
is refused when --memray is given, naming it. A change that rightly raises a
peak raises its ceiling in the same pull request, where the review sees it.
The task passes --trace-python-allocators, so memray counts the objects
Python allocates. Without it, memray sees the arenas Python’s own allocator
takes from the system a mebibyte at a time, and a benchmark’s peak then depends
on the arenas the benchmarks before it left free: one run alone could breach a
ceiling it keeps in the full run.