Packaging#

Estimated reading time: 20 minutes

Pixi SPEC0

Pixi

Reference:

  • pyproject.toml

Package management is orchestrated and performed by pixi. It needs pixi 0.84.2 or later, as requires-pixi in the [tool.pixi.workspace] table declares, since an older one misreads the pixi.lock file. An older pixi stops with an error naming that version, and pixi self-update installs a newer one.

Our pixi environments, features, and tasks are defined within the pyproject.toml manifest.

pixi offers fast, reproducible, cross-platform environment management that enables us to resolve and provision robust, consistent environments blended with packages from both conda and PyPI ecosystems.

We offer several similar collections of pixi environments for each supported distribution of python. Each environment within a collection belongs to the same solve-group i.e., environments within the same solve-group have their dependencies resolved at the same time, which means that all those environments share the exact same dependencies but may also include additional dependencies compatible within the solve-group.

Pixi Solve-Groups#

Reference:

  • pyproject.toml

Our pixi environments are organized into collections by solve-group. See the [tool.pixi.environments] table defined in the pyproject.toml manifest.

The primary solve-group is called default and will always contain the latest supported distribution of python. This is denoted by the {py} feature which represents the latest distribution of python as recommended by SPEC 0.

Table 14: Pixi default Solve-Group#

Environment

Features

Description

default

{default}, {py}

This environment contains the core dependencies of geovista along with the latest supported distribution of python. See the [tool.pixi.dependencies] table in the pyproject.toml manifest.

devs

{default}, {devs}, {py}

As per the default environment plus additional development dependencies.

docs

{default}, {devs}, {docs}, {py}

As per the devs environment plus additional documentation dependencies.

geovista

{default}, {devs}, {docs}, {geovista}, {py}, {test}

This environment is the union of all environments in the default solve-group.

test

{default}, {devs}, {py}, {test}

As per the devs environment plus additional test dependencies.

Given the above default solve-group we have the following inheritance relationship between its environments:

        ---
config:
   theme: base
   themeVariables:
      lineColor: "#F8B229"
      nodeBorder: "#2569E9"
      primaryColor: "#DCE7FC"
      secondaryColor: "#F4DDFF"
---
graph LR
   default(["`**default**`"])
   devs(["`**devs**`"])
   docs(["`**docs**`"])
   test(["`**test**`"])
   geovista(["`**geovista**`"])
   default -. "`*core*`" .-> devs
   devs ---> docs
   devs ---> test
   docs ---> geovista
   test ---> geovista
    

Fig. 7: Environment Hierarchy for default Solve-Group#

Several secondary solve-groups are available, each of which are named after the distribution of python that they support e.g., py313.

The number of different python distribution solve-groups on offer at any given time is dictated by SPEC 0.

The generic py3xx solve-group is used here as a convenience to represent each of the different python distribution solve-groups available, all of which are identical in structure and content apart from the specific distribution of python that they support.

Also note that each py3xx solve-group will always contain the similarly named {py3xx} feature e.g., the py313 solve-group will contain the {py313} feature which in turn defines the python=3.13 dependency to be included in all py313 environments.

Table 15: Pixi py3xx Solve-Groups#

Environment

Feature

Description

devs-py3xx

{default}, {devs}, {py3xx}

As per the py3xx environment plus additional development dependencies.

docs-py3xx

{default}, {devs}, {docs} , {py3xx}

As per the devs-py3xx environment plus additional documentation dependencies.

geovista-py3xx

{default}, {devs}, {docs}, {geovista}, {py3xx}, {test}

This environment is the union of all environments in the same py3xx solve-group.

py3xx

{default}, {py3xx}

This environment contains the core dependencies of geovista along with python distribution py3xx e.g., py313.

test-py3xx

{default}, {devs}, {py3xx}, {test}

As per the devs-py3xx environment plus additional test dependencies.

Given the above py3xx solve-group we have the following inheritance relationship between its environments:

        ---
config:
   theme: base
   themeVariables:
      lineColor: "#F8B229"
      nodeBorder: "#2569E9"
      primaryColor: "#DCE7FC"
      secondaryColor: "#F4DDFF"
---
graph LR
   py3xx(["`**py3xx**`"])
   devs(["`**devs-py3xx**`"])
   docs(["`**docs-py3xx**`"])
   test(["`**test-py3xx**`"])
   geovista(["`**geovista-py3xx**`"])
   py3xx --> |"`*core*`"| devs
   devs --> docs
   devs --> test
   docs --> geovista
   test --> geovista
    

Fig. 8: Environment Hierarchy for py3xx Solve-Group#

One environment belongs to no solve-group at all. The floors environment resolves each direct dependency at the lowest release its floor admits, and each transitive dependency at its newest, against the oldest supported distribution of python. The tests floors check of the Continuous Integration runs the suite in it, which tests the minimum versions of the pixi dependencies that the suite needs (floors spec §4.1).

Table 16: Pixi Floors Environment#

Environment

Features

Description

floors

{default}, {devs}, {py3xx}, {test}

As per the test-py3xx environment for the oldest {py3xx} feature, with each direct dependency solved at its floor.

It declares solve-strategy = "lowest-direct" and no solve-group, since a group shared with environments solved at their newest would solve it with them. It is locked in pixi.lock like any other. Raising a floor above the version locked makes pixi lock re-solve it, but lowering one does not, since the version locked still satisfies the floor, so follow a lowered floor with pixi update --environment floors <package>. The weekly pixi update of the ci-locks.yml workflow moves only its transitive dependencies. Where two floors cannot be resolved together, the solve lifts one above its floor rather than failing, and the lock shows the version it was lifted to.

Pixi Features#

Reference:

  • pyproject.toml

A pixi environment is defined by combining one or more features. For further details see this pixi tutorial on how to create and use features in a multi-environment scenario.

Our features are defined under the [tool.pixi.feature] tables in the pyproject.toml manifest.

Each named feature table e.g., [tool.pixi.feature.devs], may contain various fields, such as dependencies, pypi-dependencies, pypi-options, system-requirements, activation, platforms, channels, channel-priority, target and tasks.

Table 17: Pixi Features#

Feature

TOML Table

Description

{default}

[tool.pixi.dependencies]

This feature is used to define the core dependencies and tasks of geovista. Note that the {default} feature is automatically included in all environments by pixi.

{devs}

[tool.pixi.feature.devs]

This feature is used to define the development dependencies and pypi-dependencies. Note that an editable install of geovista is performed by the {devs} feature.

{docs}

[tool.pixi.feature.docs]

This feature is used to define the documentation dependencies, pypi-dependencies and tasks.

{geovista}

[tool.pixi.feature.geovista]

This feature is only used to define tasks.

{py3xx}

[tools.pixi.feature.py3xx]

This feature is used to explicitly define the distribution of python supported e.g., py313. Note that the dependencies of this feature additionally includes the pip package. The number of different python distributions supported is governed by SPEC 0.

{test}

[tool.pixi.feature.test]

This feature is used to define the test dependencies, pypi-dependencies and tasks.

Pixi Tasks#

Reference:

  • pyproject.toml

A pixi task is a custom cross-platform workflow command that is defined as part of a feature within an environment.

All our tasks are defined within the pyproject.toml manifest.

Tasks expose a convenient and easy to use entry-point to commands that allow streamlined and automated custom workflows.

The following tasks are defined for each of our features:

Table 18: Pixi Tasks#

Feature

Task

Description

{default}

download

Download and cache offline assets.

This task calls the geovista download command. Provide optional argument all, clean, doc-images, dry-run, images, list, natural-earth, operational, pantry, rasters, unit-images or verify. Defaults to all e.g.,

$ pixi run download operational

{docs}

clean

Purge all sphinx-autoapi, sphinx-gallery sphinx-tags, carousel, and other sphinx-build artifacts e.g.,

$ pixi run clean

This task is an alias for the make clean command.

clean-all

Perform both the clean and clean-cache tasks e.g.,

$ pixi run clean-all

This task is an alias for the make clean-all command.

clean-cache

Purge the myst-nb Jupyter cache. See myst-nb configuration for further details e.g.,

$ pixi run clean-cache

This task is an alias for the make clean-cache command.

doctest

Execute sphinx.ext.doctest to test code snippets within the documentation i.e.,

$ pixi run doctest

Note that the clean task is called prior to running this task.

This task is an alias for the make doctest command.

fetch-inventories

Refresh the vendored intersphinx fallback inventories e.g.,

$ pixi run fetch-inventories

Each inventory is the local fallback consulted only when its remote counterpart cannot be reached. See Intersphinx.

make

Build the documentation.

Provide optional argument html, html-docstring, html-docstring-inline, html-gallery, html-inline or html-noplot. Defaults to html-noplot e.g.,

$ pixi run make html-gallery

Note that the clean task is called prior to running this task.

This task is an alias for the make command.

serve-html

Build the documentation and start a local HTTP server on port 11000 to view the rendered documentation. This is necessary in order to support interactive scenes.

Note that the clean and make tasks are called prior to running this task.

Defaults to passing html-noplot to the make task. Override this behaviour by providing an alternative optional argument as per the make task e.g.,

$ pixi run serve-html html

This task is an alias for the make serve-html command.

{geovista}

tests-docs

Perform documentation image tests of pyvista-plot directive static scenes e.g.,

$ pixi run tests-docs

This task calls pytest --doc_mode to perform the documentation image tests using the pytest-pyvista plugin. Refer to the [tool.pytest.ini_options] table entry in the pyproject.toml manifest for default configuration options.

Note that the tests-clean, download doc-images and make html-docstring-inline tasks are called prior to running this task.

This task is only available in the geovista and geovista-py3xx environments.

tests-docs-browser

Perform browser tests of the documentation theme chrome e.g.,

$ pixi run tests-docs-browser

This task calls pytest tests/docs to drive a headless chromium browser over the built documentation using playwright.

Provide an optional documentation build target argument, which is built first. Defaults to html-noplot e.g.,

$ pixi run tests-docs-browser html-gallery

A non-empty second argument passes --browser-strict, which fails rather than skips a test whose prerequisite is missing. The tests run in parallel, one worker per CPU core, each launching its own browser over the one build. The optional third argument is the number of workers, where 0 runs them serially in a single process e.g.,

$ pixi run tests-docs-browser html-noplot "" 0

Note that the make task is called prior to running this task.

This task is only available in the geovista and geovista-py3xx environments.

tests-docs-browser-install

Install the chromium browser required by the tests-docs-browser task e.g.,

$ pixi run tests-docs-browser-install

This is a one-off, as playwright manages the browser separately from its python bindings.

This task is only available in the geovista and geovista-py3xx environments.

{test}

check-spec-status

Check the work each design specification status cites against GitHub e.g.,

$ pixi run check-spec-status

Provide the number of a pull request to check as that pull request, which a status may cite while it is still open, or drift to also report a status the work has moved past. See Specifications.

check-spec-renumbering

Check the specification citations against a renumbered section e.g.,

$ pixi run check-spec-renumbering

Provide fix to rewrite the citations of moved sections in place, or audit to compare every citation with its specification when it was written. See Specification Renumbering.

benchmarks

Perform the benchmarks under CodSpeed, without memray e.g.,

$ pixi run benchmarks

Run locally, it measures wall time. In CI, CodSpeed counts instructions under Valgrind instead. See Benchmarks.

benchmarks-memory

Perform the benchmarks under memray, holding each to its memory ceiling e.g.,

$ pixi run benchmarks-memory

See Benchmarks.

tests-clean

Purge both the documentation and unit test image caches, along with any images generated from previous test sessions e.g.,

$ pixi run tests-clean

tests-unit

Perform the unit tests.

This task calls the pytest command. Defaults to executing all unit tests discoverable from the geovista root directory.

Accepts a valid pytest marker expression as an optional argument. Refer to the [tool.pytest.ini_options] table entry in the pyproject.toml manifest for configured markers e.g.,

$ pixi run tests-unit "not image"

Runs the unit tests in parallel, one worker per CPU core (see Parallel Testing). Accepts the number of workers as an optional second argument, where 0 runs them serially in a single process e.g.,

$ pixi run tests-unit "not image" 0

Note that the tests-clean task is called prior to running this task.

verification-report

Report each design specification’s decisions, roadmap and open items, and where each decision is cited e.g.,

$ pixi run verification-report

Provide a specification’s prefix, such as heights, to report that specification alone. The report is a view of the ledger, never a check: it exits 0 whatever it finds. See Decisions and Verification.

Continuous Integration#

Reference:

  • .github/workflows

The following packaging workflows are available:

Table 19: Packaging Workflows#

Workflow

Description

ci-locks

The ci-locks.yml cron based GHA workflow regularly schedules pixi to refresh the pixi.lock file based on the pyproject.toml manifest and the latest package versions available from the conda and PyPI ecosystems.

A failed run of the schedule opens an issue, or comments on the open one, quoting the pixi error that names the environment it could not solve, since no environment’s lock moves until it is fixed.

Only the latest python variant of the geovista-py3xx environment e.g., geovista-py314, will be exported to a conda explicit specification file (.txt) and also converted to an equivalent explicit environment YAML file (.yml). These resources are available in the requirements/locks directory.

Additionally, the geovista-py3xx environment will be exported to the conda environment YAML file requirements/geovista.yml containing only the top-level dependencies of that environment, as defined by its features.

ci-wheels

The ci-wheels.yml GHA workflow builds, tests and publishes the source distribution (sdist) and binary wheel of geovista.

We have adopted PyPI Trusted Publishing with OpenID Connect (OIDC) for secure deployments of assets.

Note that the sdist and wheel assets are automatically published to Test PyPI for each pull-request merged to the main branch, and PyPI for each tag release.

Python Package Index#

Reference:

  • pyproject.toml

  • requirements

PyPI package dependencies are configured in the pyproject.toml manifest under the [tool.setuptools.dynamic] and [tool.setuptools.dynamic.optional-dependencies] table entries.

Version management of our PyPI dependencies is orchestrated by Dependabot. Refer to the package-ecosystem: "pip" key entry in the .github/dependabot.yml file for further details.

The Dependabot service is regularly scheduled to check the versions of our dependencies and automatically increment them using maximum capping pins via a pull-request in response to the latest available package updates within the PyPI ecosystem.

Every requirement also carries a floor, as in numpy >=2.2.6,<2.6, equal to the floor of its pixi declaration, which the Dependency Floors check holds (floors spec §4.4). A floor moves on both sides in one commit, followed by pixi lock. Lowering one also needs pixi update --environment floors <package>, since pixi lock keeps the version already locked.