Packaging#
Estimated reading time: 20 minutes
Note
We follow Scientific Python Ecosystem Coordination (SPEC)
recommendation for Minimum Supported Dependencies (SPEC 0), and
typically support the latest 2-3 distributions of python.
Reference:
pyproject.toml
Package management is orchestrated and performed by pixi. It needs pixi
0.84.2 or later, as requires-pixi in the [tool.pixi.workspace]
table declares, since an older one misreads the pixi.lock file. An older
pixi stops with an error naming that version, and pixi self-update
installs a newer one.
Our pixi environments, features,
and tasks are defined within the pyproject.toml manifest.
pixi offers fast, reproducible, cross-platform environment management that
enables us to resolve and provision robust, consistent environments
blended with packages from both conda and PyPI ecosystems.
We offer several similar collections of pixi environments for each
supported distribution of python. Each environment within a collection belongs
to the same solve-group i.e., environments within the same solve-group
have their dependencies resolved at the same time, which means that all
those environments share the exact same dependencies but may also include
additional dependencies compatible within the solve-group.
Tip
We’ve adopted the following format convention to help clarify each type of
pixi component:
environment-name
{feature-name}task-namesolve-group-name
Pixi Solve-Groups#
Reference:
pyproject.toml
Our pixi environments are organized into collections by solve-group.
See the [tool.pixi.environments] table defined in the pyproject.toml
manifest.
The primary solve-group is called default and will always
contain the latest supported distribution of python. This is denoted by
the {py} feature which represents the latest distribution of python
as recommended by SPEC 0.
Environment |
Features |
Description |
|---|---|---|
default |
|
This environment contains the core
dependencies of |
devs |
|
As per the default environment plus additional development dependencies. |
docs |
|
As per the devs environment plus additional documentation dependencies. |
geovista |
|
This environment is the union of all environments in the default solve-group. |
test |
|
As per the devs environment plus additional test dependencies. |
See also
The pixi info command provides key information about the system, environments,
solve-groups, dependencies and tasks in the workspace.
Given the above default solve-group we have the following inheritance relationship between its environments:
---
config:
theme: base
themeVariables:
lineColor: "#F8B229"
nodeBorder: "#2569E9"
primaryColor: "#DCE7FC"
secondaryColor: "#F4DDFF"
---
graph LR
default(["`**default**`"])
devs(["`**devs**`"])
docs(["`**docs**`"])
test(["`**test**`"])
geovista(["`**geovista**`"])
default -. "`*core*`" .-> devs
devs ---> docs
devs ---> test
docs ---> geovista
test ---> geovista
Fig. 7: Environment Hierarchy for default Solve-Group#
Note
As its name suggests, the default environment is used
by default when no explicit environment name is specified to pixi.
e.g., the pixi shell command starts a shell with the default
environment activated, whereas the pixi shell -e devs command will
activate the devs environment instead.
Several secondary solve-groups are available, each of which are named
after the distribution of python that they support e.g., py313.
The number of different python distribution solve-groups on offer at any
given time is dictated by SPEC 0.
The generic py3xx solve-group is used here as a convenience to
represent each of the different python distribution solve-groups available,
all of which are identical in structure and content apart from the specific
distribution of python that they support.
Also note that each py3xx solve-group will always contain the
similarly named {py3xx} feature e.g., the py313 solve-group
will contain the {py313} feature which in turn defines the python=3.13
dependency to be included in all py313 environments.
Environment |
Feature |
Description |
|---|---|---|
devs-py3xx |
|
As per the py3xx environment plus additional development dependencies. |
docs-py3xx |
|
As per the devs-py3xx environment plus additional documentation dependencies. |
geovista-py3xx |
|
This environment is the union of all environments in the same py3xx solve-group. |
py3xx |
|
This environment contains the core
dependencies of |
test-py3xx |
|
As per the devs-py3xx environment plus additional test dependencies. |
Attention
The generic py3xx solve-group does not exist, neither does
the generic {py3xx} feature.
Given the above py3xx solve-group we have the following inheritance relationship between its environments:
---
config:
theme: base
themeVariables:
lineColor: "#F8B229"
nodeBorder: "#2569E9"
primaryColor: "#DCE7FC"
secondaryColor: "#F4DDFF"
---
graph LR
py3xx(["`**py3xx**`"])
devs(["`**devs-py3xx**`"])
docs(["`**docs-py3xx**`"])
test(["`**test-py3xx**`"])
geovista(["`**geovista-py3xx**`"])
py3xx --> |"`*core*`"| devs
devs --> docs
devs --> test
docs --> geovista
test --> geovista
Fig. 8: Environment Hierarchy for py3xx Solve-Group#
One environment belongs to no solve-group at all. The floors
environment resolves each direct dependency at the lowest release its
floor admits, and each transitive dependency at its newest, against the
oldest supported distribution of python. The tests floors check of
the Continuous Integration runs the suite in
it, which tests the minimum versions of the pixi dependencies that the
suite needs (floors spec §4.1).
Environment |
Features |
Description |
|---|---|---|
floors |
|
As per the test-py3xx environment
for the oldest |
It declares solve-strategy = "lowest-direct" and no solve-group, since a
group shared with environments solved at their newest would solve it with them.
It is locked in pixi.lock like any other. Raising a floor above the
version locked makes pixi lock re-solve it, but lowering one does not, since
the version locked still satisfies the floor, so follow a lowered floor with
pixi update --environment floors <package>. The weekly pixi update
of the ci-locks.yml workflow moves only its transitive dependencies. Where two
floors cannot be resolved together, the solve lifts one above its floor rather
than failing, and the lock shows the version it was lifted to.
Pixi Features#
Reference:
pyproject.toml
A pixi environment is defined by combining one or
more features. For further details see this pixi tutorial
on how to create and use features in a multi-environment scenario.
Our features are defined under the [tool.pixi.feature] tables in the
pyproject.toml manifest.
Each named feature table e.g., [tool.pixi.feature.devs], may contain
various fields, such as dependencies, pypi-dependencies,
pypi-options, system-requirements, activation, platforms,
channels, channel-priority, target and tasks.
Feature |
TOML Table |
Description |
|---|---|---|
|
|
This feature is used to define the core
|
|
|
This feature is used to define the development
|
|
|
This feature is used to define the
documentation |
|
|
This feature is only used to define |
|
|
This feature is used to explicitly define the
distribution of |
|
|
This feature is used to define the test
|
Pixi Tasks#
Reference:
pyproject.toml
A pixi task is a custom cross-platform workflow command
that is defined as part of a feature within an
environment.
All our tasks are defined within the pyproject.toml manifest.
Tasks expose a convenient and easy to use entry-point to commands that allow streamlined and automated custom workflows.
The following tasks are defined for each of our features:
Feature |
Task |
Description |
|---|---|---|
|
|
Download and cache offline assets. This task calls the geovista download command. Provide optional
argument $ pixi run download operational
|
|
|
Purge all sphinx-autoapi, sphinx-gallery sphinx-tags, carousel, and other sphinx-build artifacts e.g., $ pixi run clean
This task is an alias for the |
|
Perform both the $ pixi run clean-all
This task is an alias for the |
|
|
Purge the myst-nb Jupyter cache. See myst-nb configuration for further details e.g., $ pixi run clean-cache
This task is an alias for the |
|
|
Execute sphinx.ext.doctest to test code snippets within the documentation i.e., $ pixi run doctest
Note that the This task is an alias for the |
|
|
Refresh the vendored intersphinx fallback inventories e.g., $ pixi run fetch-inventories
Each inventory is the local fallback consulted only when its remote counterpart cannot be reached. See Intersphinx. |
|
|
Build the documentation. Provide optional argument $ pixi run make html-gallery
Note that the This task is an alias for the |
|
|
Build the documentation and start a local Note that the Defaults to passing $ pixi run serve-html html
This task is an alias for the |
|
|
|
Perform documentation image tests of $ pixi run tests-docs
This task calls Note that the This task is only available in the geovista and geovista-py3xx environments. Note Failed image tests are captured via the |
|
Perform browser tests of the documentation theme chrome e.g., $ pixi run tests-docs-browser
This task calls Provide an optional documentation build target argument, which is built
first. Defaults to $ pixi run tests-docs-browser html-gallery
A non-empty second argument passes $ pixi run tests-docs-browser html-noplot "" 0
Note that the This task is only available in the geovista and geovista-py3xx environments. Attention The default |
|
|
Install the $ pixi run tests-docs-browser-install
This is a one-off, as playwright manages the browser separately from its
This task is only available in the geovista and geovista-py3xx environments. |
|
|
|
Check the work each design specification status cites against GitHub e.g., $ pixi run check-spec-status
Provide the number of a pull request to check as that pull request, which a status
may cite while it is still open, or |
|
Check the specification citations against a renumbered section e.g., $ pixi run check-spec-renumbering
Provide |
|
|
Perform the benchmarks under CodSpeed, without memray e.g., $ pixi run benchmarks
Run locally, it measures wall time. In CI, CodSpeed counts instructions under Valgrind instead. See Benchmarks. |
|
|
Perform the benchmarks under memray, holding each to its memory ceiling e.g., $ pixi run benchmarks-memory
See Benchmarks. |
|
|
Purge both the documentation and unit test image caches, along with any images generated from previous test sessions e.g., $ pixi run tests-clean
|
|
|
Perform the unit tests. This task calls the Accepts a valid $ pixi run tests-unit "not image"
Runs the unit tests in parallel, one worker per CPU core (see
Parallel Testing). Accepts the number of workers
as an optional second argument, where $ pixi run tests-unit "not image" 0
Note that the Note Failed Image Tests are captured via the
|
|
|
Report each design specification’s decisions, roadmap and open items, and where each decision is cited e.g., $ pixi run verification-report
Provide a specification’s prefix, such as |
See also
The pixi task list command describes each task available within the workspace.
Whereas the pixi task list --summary command enumerates the tasks available per environment.
Continuous Integration#
Reference:
.github/workflows
The following packaging workflows are available:
Workflow |
Description |
|---|---|
The ci-locks.yml A failed run of the schedule opens an issue, or comments on the open one, quoting
the Only the latest Additionally, the geovista-py3xx environment will be exported
to the |
|
The ci-wheels.yml GHA workflow builds, tests and publishes the
source distribution ( We have adopted PyPI Trusted Publishing with OpenID Connect (OIDC) for secure deployments of assets. Note that the |
Python Package Index#
Reference:
pyproject.tomlrequirements
PyPI package dependencies are configured in the pyproject.toml manifest
under the [tool.setuptools.dynamic] and
[tool.setuptools.dynamic.optional-dependencies] table entries.
Version management of our PyPI dependencies is orchestrated by
Dependabot. Refer to the package-ecosystem: "pip" key entry in the
.github/dependabot.yml file for further details.
The Dependabot service is regularly scheduled to check the
versions of our dependencies and automatically increment them using maximum
capping pins via a pull-request in response to the
latest available package updates within the PyPI ecosystem.
Every requirement also carries a floor, as in numpy >=2.2.6,<2.6, equal to the
floor of its pixi declaration, which the
Dependency Floors check holds
(floors spec §4.4). A floor moves on both sides in one commit, followed by
pixi lock. Lowering one also needs pixi update --environment floors
<package>, since pixi lock keeps the version already locked.
Attention
Updates to the PyPI maximum capping pins may require to be manually
reflected in the pyproject.toml manifest for similar pixi managed
conda dependencies.
Track native Dependabot support for pixi in
dependabot/dependabot-core issue#2227 🤞
Attention
A conda-forge package of the same name is not always the python one.
A tool shipping both a node and a python distribution may be packaged on
conda-forge as the node command line interface alone, with no python
bindings. The conda package then installs cleanly, and the subsequent
import raises a ModuleNotFoundError. playwright, required by
the Browser Tests, is the case in point.
Such a dependency belongs in the pypi-dependencies table of its
feature, along with the appropriate
requirements/pypi-optional-*.txt file.