zenv

Python Environment Manager for HPC and Development Systems

zenv is a command-line tool written in Zig that manages Python virtual environments, primarily designed for High-Performance Computing (HPC) environments and development systems.

Features

Core Functionality

The tool provides several key features:

1. Environment Management: Create, activate, and manage Python virtual environments with configurations stored in zenv.json
2. Registry: Track environments globally so they can be activated from any directory
3. System Targeting: Configure environments for specific machines or clusters
4. Dependency Management: Install Python packages with awareness of what's already provided by system modules
5. Add packages on the fly: zenv add installs a package into an environment and records it in the project's manifest (requirements.txt / pyproject.toml / zenv.json)
6. Module Integration: Load HPC modules required for environment setup

Installation

Homebrew (macOS / Linux)

brew install anoopkcn/zenv/zenv

Upgrade later with brew upgrade zenv.

Use install script

curl -fsSL https://raw.githubusercontent.com/anoopkcn/zenv/HEAD/install.sh | bash

Run the same command to update already insalled zenv version

Alternative methods of installation

Manual download of stable release

Change the <tag> according to the latest version

# Replace <tag> with last stable release version:
curl -LO "https://github.com/anoopkcn/zenv/releases/download/<tag>/zenv-x86_64-linux-musl-small.tar.gz"

# Extract the 'zenv' executable and move it somewhere in your 'PATH'
tar -xvf zenv-x86_64-linux-musl-small.tar.gz
mv zenv ~/.local/bin/

Build from Source

# Clone the repository
git clone https://github.com/anoopkcn/zenv.git

# Build the project
cd zenv
zig build

# Optional: Move executable to ~/.local/bin (assumes ~/.local/bin is in PATH)
mv zig-out/bin/zenv ~/.local/bin/
# OR
# Optional: Add to your PATH
export PATH="$PATH:path/to/zig-out/bin"

Check release for specific versions. Supported OS: Linux(aarch64, x86_64), MacOS(aarch64, x86_64). Windows support is not planned

Usage

Initialize and setup an environment

zenv init [name] [description]
# This creates a `zenv.json` configuration file in your project directory
# You can then modify this json file according to your needs and run:
zenv setup <name>
# This will also register the environment to global ZENV_DIR/registry.json

A minimal example of a zenv.json file:

{
  "base_dir": "zenv",
  "test": {
    "target_machines": ["*"],
    "description": "Basic environment JURECA and any machine",
    "modules": ["Stages/2025", "StdEnv", "Python"],
    "dependency_file": "requirements.txt"
  }
}

Check the Configuration Reference for full list of key-values. The optional dependency_file can be requirements.txt OR pyproject.toml file. If you run zenv init then it will be automatically populated according to what is found in the project.

Listing environments

List all environments registered for the current machine:

zenv list # for listing envs configured for current computer
# OR
zenv list --all # for listing all available envs in the registry

Example output:

- test
  id      : c3c494547b40f070b4c080f95c707622d84fe749
  target  : jureca, juwels, *
  project : /path/to/project/
  venv    : /path/to/project/zenv/test
  desc    : Test python environment

Found 1 environment(s) for the current machine ('jrlogin01.jureca').

Activating environments

Activate an environment by name or ID

Example:

# Activate by name
source $(zenv activate test)

# Activate by full ID
source $(zenv activate c3c494547b40f070b4c080f95c707622d84fe749)

# Activate by partial ID (first 7+ characters)
source $(zenv activate c3c4945)

Direct run without activation

One could run commands and tools without explicit activation of Environments using zenv run command

zenv run <name|id> <command>

For example to run a script using python from the environment:

zenv run test python my_test_file.py

Or to run a server:

zenv run test jupyter notebook
# OR
zenv run test mkdocs serve

OR make environment avilable for your editor without activaton:

# for vim or neovim
zenv run test vim

# for vs code
# zenv run test code -n .

Adding packages to an environment

zenv add installs a package into an existing environment and records it in the project's manifest, so the dependency persists and is reinstalled on the next setup/rebuild. The package is installed first; the manifest is only updated if the install succeeds.

zenv add <name|id|.> <package> [--dev] [--zenv] [--uv]

The <package> may include a version specifier (e.g. flask>=2.0). Re-adding a package that is already recorded replaces its spec, so you can use zenv add to update a pin.

# install requests and record it in the manifest
zenv add test requests

# install with a version constraint
zenv add test "numpy>=1.26,<2.0"

# add a development dependency
zenv add test pytest --dev

Where the package is recorded depends on the environment's dependency_file and the flags:

Environment uses	normal dependency	--dev (development dependency)
requirements.txt	appended to requirements.txt	dev_dependencies in zenv.json
pyproject.toml	[project].dependencies	[dependency-groups].dev (PEP 735)
no dependency_file	dependencies in zenv.json	dev_dependencies in zenv.json

Flags:

• --dev — record the package as a development dependency (see the table above).
• --zenv — record the package only in zenv.json (under dependencies, or dev_dependencies with --dev), leaving requirements.txt / pyproject.toml untouched. The package is still installed into the environment.
• --uv — use uv pip install instead of python -m pip install.

Note: --dev is interpreted per-command — it means editable install of the current project for zenv setup, but development dependency for zenv add.

The environment must already be built (zenv setup <name>); zenv add does not create it.

Registering and Deregistering Environments

A metadata information about the environments are stored at ZENV_DIR/registry.json. by default ZENV_DIR is $HOME/.zenv. But one can set ZENV_DIR environment variable as any directory with write permission.

Register an environment in the global registry:

This is done automatically when you run zenv setup <name>

zenv register <name>

Remove an environment from the registry:

zenv deregister <name>     # Remove by name or ID

Python Management

The default priority of the Python is as follows:

1. Module-provided Python (if HPC modules are loaded)
2. Explicitly configured 'fallback_python' from zenv.json (if not null)
3. zenv-managed pinned Python
4. System python3
5. System python

If you would like to use zenv-managed default Python for the environment, run:

# Install a python version if not done already
zenv python install <version>

# Pin a specic python version
zenv python pin <version>

# usethe pinned version
zenv setup <name> --python

Configuration Reference

One can have multiple environment configurations in the same zenv.json file and it supports the following structure:

{
  "base_dir": "<base dirrectory where all envs listed in the config will be stored>",
  "<name>": {
    "target_machines": ["<machine identifier accepts wild card *>"],
    "fallback_python": "<path to python executable OR null>",
    "description": "<optional description OR null>",
    "modules": ["<module1>", "<module2>"],
    "modules_file": "<path to modules file OR null>"
    "module_cache": true,
    "dependency_file": "<optional path to requirements_txt OR pyproject_toml OR null>",
    "dependencies": ["<package name with or without version>"],
    "dev_dependencies": ["<development dependency, typically added via 'zenv add --dev'>"],
    "setup": {
      "commands": ["<list of shell commands which is run during setup>"],
      "script": "<path to a shell script which is run during setup>"
    },
    "activate": {
      "commands": ["<list of shell commands which is run during activation>"],
      "script": "<path to a shell script which will be run during activation>"
    }
  },
  "<another_name>": {
    "target_machines": ["<anothor machine identifier>"]
  }
}

One can run zenv validate to validate the config file. If there are errors in the JSON it will try to inform the context of the error.

In the configuration target_machines is required key(If you want, you can disable the validation check using --no-host), all other entries are optional. Top-level base_dir can be an absolute path or relative one(relative to the zenv.json file), if not provided it will create a directory called .zenv at the project root. One can use wildcards to target specific systems, to mantch any machine use * or any ("target_machines": ["*"]). The lookup location of the dependency_file is the same directory as zenv.json.

The dependencies and dev_dependencies arrays list packages that zenv setup installs into the environment in addition to anything from dependency_file. Both are usually populated for you by zenv add (see Adding packages to an environment) rather than edited by hand, but you can also add entries directly.

The machine identity that target_machines is matched against is taken from $SYSTEMNAME when it is set (falling back to $HOSTNAME/$HOST, then the hostname command). On HPC systems every node of a cluster exports the same $SYSTEMNAME while per-node hostnames differ (e.g. jrlogin01 vs jrc0042), so keying off $SYSTEMNAME lets a single entry like "target_machines": ["jureca"] match on login and compute nodes alike. This is the same cluster identity used for the module cache (below). Off HPC, where $SYSTEMNAME is unset, matching uses the hostname as before.

Host-aware selection (shared filesystems). When several machines share a filesystem (e.g. HPC login/compute nodes), you can register one environment per machine — distinct names like env_machine1, env_machine2, each with its own target_machines — and give them all the same alias (e.g. zenv alias create dev env_machine1, then ... dev env_machine2). Resolving that alias, or the . current-directory shortcut, then auto-selects the environment whose target_machines matches the machine you are on, so the same command (zenv run dev, zenv activate ., zenv cd .) works on every node. If the current host matches none of the candidates — or more than one (e.g. two share a */any target) — resolution reports an ambiguity listing each candidate and its target machines; use the exact environment name or id to disambiguate. A unique alias (one holder) still resolves regardless of host.

For custom scripts, you can use activate_hook and setup_hook to specify paths to shell scripts that will be copied to the environment's directory and executed during activation or setup. These scripts allow for more complex customization than inline commands. The scripts are copied to the environment directory, making the environment portable and independent of the original script location.

The key-val "modules_file": "path/to/file.txt" can be specified in an environment to load module names from an external file. The file can contain module names separated by spaces, tabs, commas, or newlines. When specified, any modules listed in the "modules" array are ignored.

Module caching

On HPC systems, module load (Lmod) can be slow, and the generated activate.sh runs it on every activation. With "module_cache": true (the default), zenv setup captures the environment that module load produces and writes it to the environment directory (.zenv_module_cache.sh + .zenv_module_cache.stamp). Subsequent activations replay that captured environment instead of invoking Lmod, which is significantly faster.

The cache is used only when it is trustworthy; otherwise activation transparently falls back to a real module --force purge + module load:

• It is keyed to the cluster via $SYSTEMNAME (falling back to hostname), so it is reused across all nodes of a system but never replayed on a different one.
• Path-style variables (PATH, LD_LIBRARY_PATH, …) are replayed as prepends, so your live login environment is preserved.
• If any module fails to load during setup, no cache is written.

When this auto-rebuild fires, zenv prints a short notice on stderr (zenv: rebuilding environment '<name>' (...; auto-setup)... followed by a completion line) so you know a rebuild happened and why. It is on stderr deliberately: commands like zenv activate / zenv cd put their shell-eval payload on stdout (source $(zenv activate <name>)), and stderr does not corrupt that capture while still being visible in your terminal. The notice appears only when a rebuild actually runs — a routine activation with no changes stays silent. Set ZENV_NO_AUTO_SETUP=1 to disable the auto-rebuild (and its notice) entirely.

Reuse on rebuild. zenv re-runs zenv setup automatically when it detects that zenv.json or a referenced dependency/modules file changed. To avoid paying the Lmod cost on every such rebuild, the stamp also records a signature of the module set (modules_sig). When a rebuild leaves the module set unchanged — e.g. you only edited requirements.txt — setup skips module --force purge + module load and instead re-sources the existing cache to establish the build environment, provided the cache is still trustworthy for this cluster (same $SYSTEMNAME, not untrusted). When the module set itself changes (you edit modules/modules_file), setup re-runs Lmod and re-captures the cache. zenv setup --force always re-captures.

The cache is not auto-invalidated when the system's modules change underneath you. If a module load is updated by site maintenance, re-run zenv setup (or zenv setup --force) to refresh the cache. Set "module_cache": false to always run module load at activation. Environments without modules are unaffected.

Help

zenv help

Output:

Usage: zenv <command> [name|id] [options]

Manages Python virtual environments based on zenv.json configuration.

Commands:
  init [name] [desc]         Initializes a new 'zenv.json' in the current directory.
                             Creates a 'test' environment if 'name' is not provided.
                             Use to start defining your environments[z].

  setup <name>               Creates and configures the virtual environment for '<name>'.
                             Builds the environment in '<base_dir>/<name>' as per 'zenv.json'.
                             This is the primary command to build an environment.

  activate <name|id|.>       Outputs the activation script path for an environment.
                             To use: source $(zenv activate <name|id|.>)

  run <name|id|.> <command>  Executes a <command> within the specified isolated environment.
                             Does NOT require manual activation of the environment.

  add <name|id|.> <package>  Installs <package> into the environment and records it in the
                             project's manifest (requirements.txt / pyproject.toml / zenv.json).
                             <package> may include a version (e.g. 'flask>=2.0').
                             Options: --dev (dev dependency), --zenv (record only in
                             zenv.json), --uv (use uv instead of pip).

  cd <name|id|.>             Outputs the project directory path for an environment.
                             To use: cd $(zenv cd <name|id|.>)

  list                       Lists registered environments accessible on this machine.

  list --all                 Lists all registered environments.

  register <name>            Adds the environment '<name>' (from current 'zenv.json') to the
                             global registry[a], making it accessible from any location.

  deregister <name|id|.>     Removes an environment from the global registry.
                             The virtual environment files are NOT deleted.

  rename <old|id> <new>      Renames an environment from 'old' to 'new'.
                             Updates the registry, renames the virtual environment directory,
                             updates generated scripts, and updates any associated Jupyter kernels.
                             Preserves all configuration and metadata.

  rm <name|id>               De-registers the environment AND permanently deletes its
                             virtual environment directory from the filesystem.

  validate [config]          Validates the configuration file. If no arguent provided it
                             will validate the 'zenv.json' file in the current directory.
                             Reports errors with line numbers and field names if found.

  log <name|id|.>            Displays the setup log file for the specified environment.

  alias <subcommand>         Manages environment aliases for easier access:
    create <alias> <env>     Creates an alias for an environment. The same alias may
                             be given to several environments that target different
                             machines (see "host-aware selection" below).
    remove <alias>           Removes an existing alias.
    list                     Lists all defined aliases.
    show <alias>             Shows the environment(s) the alias resolves to, with
                             each one's target machines.

  jupyter <subcommand>       Manages Jupyter kernels for environments:
    create <env_name>        Creates a Jupyter kernel for the specified environment.
    remove <env_name>        Removes the Jupyter kernel for the specified environment.
    list                     Lists all zenv-managed Jupyter kernels.
    check                    Checks if Jupyter is installed and available.

  python <subcommand>        (Experimantal feature) Manages Python installations:
    install <version>        Downloads and installs a specific Python version for zenv.
    pin <version>            Sets <version> as the pinned Python for zenv to prioritize.
    list                     Shows Python versions installed and managed by zenv.

  version, -v, --version     Prints the installed zenv version.

  help, --help               Shows this help message.

Options for 'zenv setup <name>':
  --init                     Creates and populates 'zenv.json' file before 'zenv setup'.
                             Convenient for creating and setting up in one step.

  --dev                      Installs the current directory's project in editable mode.
                             Equivalent to 'pip install --editable .' command.

  --uv                       Uses 'uv' instead of 'pip' for package operations.
                             Ensure 'uv' is installed and accessible.

  --no-host                  Bypasses hostname validation during setup.
                             Equivalent to "target_machines": ["*"] in zenv.json.
                             Use if an environment should be set up regardless of the machine.

  --python                   Use the zenv-pinned Python for creating environment.
                             Ignores the default Python priority[b] list.

  --force                    Forces reinstallation of all dependencies.
                             Useful if dependencies from loaded modules cause conflicts.

  --no-cache                 Disables the package cache when installing dependencies.
                             Ensures fresh package downloads for each installation.

  --jupyter                  Creates a Jupyter kernel for the environment after setup.
                             Equivalent to running 'zenv jupyter create <name>' after setup.

Options for 'zenv add <name> <package>':
  --dev                      Records the package as a development dependency.
                             pyproject.toml -> [dependency-groups].dev; otherwise zenv.json.
                             (Note: --dev means editable install for 'setup', dev dependency
                             for 'add' — its meaning is per-command.)

  --zenv                     Records the package only in zenv.json, leaving requirements.txt
                             and pyproject.toml untouched. The package is still installed.

  --uv                       Uses 'uv pip install' instead of 'python -m pip install'.

[z] Configuration (zenv.json):
  The 'zenv.json' file is a JSON formatted file that defines your environments.
  Each top-level key is an environment name. "base_dir": "path/to/venvs" is a special
  top-level key specifying the storage location for virtual environments.
  Paths can be absolute (e.g., /path/to/venvs) or relative to the 'zenv.json' location.

[a] Registry (ZENV_DIR/registry.json):
  A global JSON file (path in ZENV_DIR environment variable, typically $HOME/.zenv)
  that tracks registered environments. This allows 'zenv' commands to manage
  these environments from any directory. Environments are added via 'zenv setup'
  or 'zenv register'.

[b] Python Priority List (for 'zenv setup' without '--python' flag):
  zenv attempts to find a Python interpreter in the following order:
  1. HPC module-provided Python (if HPC environment modules are loaded).
  2. Path explicitly specified by the 'fallback_python' key in zenv.json.
  3. zenv-pinned Python (set via 'zenv python use <version>').
  4. System Python.
  Use 'zenv setup <name> --python' to use only the pinned version.

[.] Dot Notation:
  Use '.' as the environment identifier to automatically select an environment
  from the current directory. This works when you're in a directory containing
  a zenv.json file and have registered environments for that directory.

Issues

If you encounter any bugs open an Issue. To use the debug logging feature, users can set the ZENV_DEBUG environment variable:

Example:

ZENV_DEBUG=1 zenv setup <name>

License

MIT License