We’d like this to be a community driven project, so all kinds of input are welcome!
There are numerous way you could contribute:
Report bugs by submitting issues
Request features by submitting issues
Write examples and improve documentation
Contribute code: bug fixes, new features
This document is loosely based on the Contributing to xarray guide. It’s worth reading, it covers many of the subjects below in greater detail.
You can report bugs on the Issues pages. Please include a self-contained Python snippet that reproduces the problem. In the majority of cases, a Minimal, Complete, and Verifiable Example (MCVE) or Minimum Working Example (MWE) is the best way to communicate the problem and provide insight. Have a look at this stackoverflow article for an in-depth description.
We use Git for version control. Git is excellent software, but it might take some time to wrap your head around it. There are many excellent resources online. Have a look at the extensive manual online, a shorter handbook, searchable GitHub help, a cheatsheet, or try this interactive tutorial.
We use black for automatic code formatting. Like Black, we are
uncompromising about formatting. Continuous Integration will fail if
black . from within the repository root folder would make
any formatting changes.
Integration black into your workflow is easy. Find the instructions here. If you’re using VisualStudioCode (which we heartily recommend), consider enabling the Format On Save option – it’ll save a lot of hassle.
If you add functionality or fix a bug, always add a test. For a new feature, you’re testing anyway to see if it works… you might as well clean it up and include it in the test suite! In case of a bug, it means our test coverage is insufficient. Apart from fixing the bug, also include a test that addresses the bug so it can’t happen again in the future.
pytest to do automated testing. You can run the test suite
locally by simply calling
pytest in the project directory.
pytest will pick up on all tests and run them automatically. Check
the pytest documentation, and have a look at the test suite to figure
out how it works.
Automation with tox#
This package uses tox to automate code formatting, linting, and testing. tox builds an isolated environment for these steps. Because imod has a large number of dependencies, we rely on the tox-conda plugin to build these isolated environments with mamba.
Running the steps below will result in the creation of three conda
environments in the
.tox directory. The format and linting environments
are comparatively small (140 MB), but the build environment requires a full
installation of the imod package and all its dependencies (>2.5 GB).
To format the code with isort and black, run:
tox -e format
To lint the code with isort, black, and flake8:
tox -e lint
To run the tests and build the documentation, run:
tox -e build
This will run all the tests with pytest, and builds the documentation with
sphinx. Building the documentation is performed in the same step and
environment as running the tests as building the documentation includes running
all the examples in the
.examples directory, which requires a full
installation of the imod package and all its dependencies. Separating the
testing and documentation building requires building two large identical
Create a branch, and send a merge or pull request. Your code doesn’t have to be perfect! We’ll have a look, and we will probably suggest some modifications or ask for some clarifications.
How to release a new version#
To follow these steps, you need to be one of the maintainers for imod on both PyPI and conda-forge.
Update the Changelog.
Tag in Gitlab UI. Old tags are here. Old releases are here. To make a tag show up under releases, fill in the release notes in the UI. Since we keep changes in the Changelog only, just put
See https://deltares.gitlab.io/imod/imod-python/api/changelog.htmlin both the
Release notesbox. The tag name should be
vx.y.z, where x, y and z are version numbers according to Semantic Versioning.
git fetch --tagsand
git pull, verify you are on the commit you want to release, and that it is clean.
imod.egg-infofolders if present.
Create a source distribution under
python -m build --sdist
Upload the files from step 5 and 6 to PyPI with
twine upload dist/*
For conda-forge, a PR will be created automatically. If the requirements are up to date in meta.yaml then you can merge it. Otherwise you have to edit them and push this to the bot’s branch.
All our examples are available as:
As a rendered HTML gallery online
As downloadable Python scripts or Jupyter notebooks
As the original Python scripts in the
./examplesdirectory, which can be browsed directly on the online repository.
We use Sphinx-Gallery to render the Python files as HTML. We could also use Jupyter notebooks as they are nicely rendered and executable by a user. However, Sphinx-Gallery has a number of advantages over Jupyter notebooks:
To render Jupyter notebooks online, cell output has to be stored in the notebooks. This is fine for text output, but images are stored as (inline) binary blobs. These result in large commits bloating the Git repository. Tools such as nbstripout will remove the cell outputs, but this comes at the obvious cost of not having the rendered notebooks available online.
Not everybody likes Jupyter notebooks and Jupyter notebooks require special software to run. Python scripts can be run with just a Python interpreter. Furthermore, Sphinx-Gallery also provides Jupyter notebooks: from the Python scripts it will automatically generate them.
Sphinx-Gallery uses reStructured Text (rST) rather than Markdown. rST syntax is somewhat less straightforward than Markdown, but it also provides additional features such as easily linking to the API (including other projects, via intersphinx).
For Sphinx-Gallery, rST is embedded as module docstrings at the start of a
scripts and as comments in between the executable Python code. We use
as the block splitter rather than 79
#’s, as the former is recognized by
editors such as Spyder and VSCode, while the latter is not. The former also
introduces less visual noise into the examples when reading it as an unrendered
Note that documentation that includes a large portion of executable code such as the User Guide has been written as Python scripts with embedded rST as well, rather than via the use of IPython Sphinx Directives.
Building documentation and examples#
docs directory, run:
Sphinx will build the documentation in a few steps. This is generally useful, as it means only part of the documentation needs to be rebuilt after some changes. However, to start afresh, run:
This will get rid of all files generated by Sphinx.
Building the documentation is also part of the
tox -e build step , see:
Automation with tox.
Debugging Continuous Integration#
Continuous Integration runs on an image with a specific operating system, and Python installation. Due to system idiosyncrasies, CI failing might not reproduce locally. If an issue requires more than trial-and-error changes, Docker may be the easiest way to debug.
On windows, install Docker: https://docs.docker.com/docker-for-windows/install/
Pull the CI image (at the time of writing), and run it interactively:
docker pull condaforge/miniforge3:latest docker run -it condaforge/miniforge3
This should land you in the docker image. Next, we reproduce the CI setup steps. Some changes are required, such as installing git and cloning the repository, which happens automatically within CI.
conda install mamba tox apt-get update -q -y apt-get install -y build-essential conda install git cd /usr/src git clone https://gitlab.com/deltares/imod/imod-python.git cd imod-python conda env create -f imod-environment.yml source activate imod pip install -e . curl -O -L https://gitlab.com/deltares/imod/imod-python/uploads/a8ed27675150689c6acd425239531a5e/mf6.gz gunzip mf6.gz chmod +x mf6 mv mf6 /opt/conda/envs/imod/bin
At this point, everything should be ready to run the tests on the Docker image.