Back to Pixi

Commands as lists so you can also add documentation in between.

docs/workspace/advanced_tasks.md

0.67.225.4 KB
Original Source

When building a package, you often have to do more than just run the code. Steps like formatting, linting, compiling, testing, benchmarking, etc. are often part of a workspace. With Pixi tasks, this should become much easier to do.

Here are some quick examples

toml
[tasks]
# Commands as lists so you can also add documentation in between.
configure = { cmd = [
    "cmake",
    # Use the cross-platform Ninja generator
    "-G",
    "Ninja",
    # The source is in the root directory
    "-S",
    ".",
    # We wanna build in the .build directory
    "-B",
    ".build",
] }

# Add task descriptions, to be surfaced when the tasks are listed
say-hello = { cmd = ["echo", "hello world"], description = "Greet the world." }

# Depend on other tasks
build = { cmd = ["ninja", "-C", ".build"], depends-on = ["configure"] }

# Using environment variables
run = "python main.py $PIXI_PROJECT_ROOT"
set = "export VAR=hello && echo $VAR"

# Cross platform file operations
copy = "cp pixi.toml pixi_backup.toml"
clean = "rm pixi_backup.toml"
move = "mv pixi.toml backup.toml"

# Setting a default environment for the task
test = { cmd = "pytest", default-environment = "test" }

Depends on

Just like packages can depend on other packages, our tasks can depend on other tasks. This allows for complete pipelines to be run with a single command.

An obvious example is compiling before running an application.

Checkout our cpp_sdl example for a running example. In that package we have some tasks that depend on each other, so we can assure that when you run pixi run start everything is set up as expected.

fish
pixi task add configure "cmake -G Ninja -S . -B .build"
pixi task add build "ninja -C .build" --depends-on configure
pixi task add start ".build/bin/sdl_example" --depends-on build

Results in the following lines added to the pixi.toml

toml
[tasks]
# Configures CMake
configure = "cmake -G Ninja -S . -B .build"
# Build the executable but make sure CMake is configured first.
build = { cmd = "ninja -C .build", depends-on = ["configure"] }
# Start the built executable
start = { cmd = ".build/bin/sdl_example", depends-on = ["build"] }

The tasks will be executed after each other:

  • First configure because it has no dependencies.
  • Then build as it only depends on configure.
  • Then start as all its dependencies are run.

If one of the commands fails (exit with non-zero code.) it will stop and the next one will not be started.

With this logic, you can also create aliases as you don't have to specify any command in a task.

shell
pixi task add fmt ruff
pixi task add lint pylint
toml
--8<-- "docs/source_files/pixi_tomls/pixi_task_alias.toml:not-all"

!!! tip "Hiding Tasks" Tasks can be hidden from user facing commands by naming them with an _ prefix.

Shorthand Syntax

Pixi supports a shorthand syntax for defining tasks that only depend on other tasks. Instead of using the more verbose depends-on field, you can define a task directly as an array of dependencies.

Executing:

pixi task alias style fmt lint

results in the following pixi.toml:

toml
--8<-- "docs/source_files/pixi_tomls/pixi_task_alias.toml:all"

Now you can run both tools with one command.

shell
pixi run style

Environment specification for task dependencies

You can specify the environment to use for a dependent task:

toml
--8<-- "docs/source_files/pixi_tomls/tasks_depends_on.toml:tasks"

This allows you to run tasks in different environments as part of a single pipeline. When you run the main task, Pixi ensures each dependent task uses its specified environment:

shell
pixi run test-all

The environment specified for a task dependency takes precedence over the environment specified via the CLI --environment flag. This means even if you run pixi run test-all --environment py312, the first dependency will still run in the py311 environment as specified in the TOML file.

In the example above, the test-all task runs the test task in both Python 3.11 and 3.12 environments, allowing you to verify compatibility across different Python versions with a single command.

Working directory

Pixi tasks support the definition of a working directory.

cwd stands for Current Working Directory. The directory is relative to the Pixi workspace root, where the pixi.toml file is located.

By default, tasks are executed from the Pixi workspace root. To change this, use the --cwd flag. For example, consider a Pixi workspace structured as follows:

shell
├── pixi.toml
└── scripts
    └── bar.py

To add a task that runs the bar.py file from the scripts directory, use:

shell
pixi task add bar "python bar.py" --cwd scripts

This will add the following line to manifest file:

toml
[tasks]
bar = { cmd = "python bar.py", cwd = "scripts" }

Default environment

You can set the default Pixi environment used by a task using the default-environment field:

toml
--8<-- "docs/source_files/pixi_tomls/task_default_environment.toml:default-environment"

The default environment can be overridden as usual with the --environment argument:

shell
pixi run -e other_environment test

Task Arguments

Tasks can accept arguments that can be referenced in the command. This provides more flexibility and reusability for your tasks.

Why Use Task Arguments?

Task arguments make your tasks more versatile and maintainable:

  • Reusability: Create generic tasks that can work with different inputs rather than duplicating tasks for each specific case
  • Flexibility: Change behavior at runtime without modifying your pixi.toml file
  • Clarity: Make your task intentions clear by explicitly defining what values can be customized
  • Validation: Define required arguments to ensure tasks are called correctly
  • Default values: Set sensible defaults while allowing overrides when needed

For example, instead of creating separate build tasks for development and production modes, you can create a single parameterized task that handles both cases.

Arguments can be:

  • Required: must be provided when running the task
  • Optional: can have default values that are used when not explicitly provided
  • Constrained: can be restricted to a set of allowed values using choices

Defining Task Arguments

Define arguments in your task using the args field:

toml
--8<-- "docs/source_files/pixi_tomls/task_arguments.toml:project_tasks"

!!! note "Argument naming restrictions" Argument names cannot contain dashes (-) due to them being seen as a minus sign in MiniJinja. Use underscores (_) or camelCase instead.

Using Task Arguments

When running a task, provide arguments in the order they are defined:

shell
# Required argument
pixi run greet John
✨ Pixi task (greet in default): echo Hello, John!

# Default values are used when omitted
pixi run build
✨ Pixi task (build in default): echo Building my-app in development mode

# Override default values
pixi run build my-project production
✨ Pixi task (build in default): echo Building my-project in production mode

# Mixed argument types
pixi run deploy auth-service
✨ Pixi task (deploy in default): echo Deploying auth-service to staging
pixi run deploy auth-service production
✨ Pixi task (deploy in default): echo Deploying auth-service to production

Passing Extra Arguments with --

When a task defines typed args, all command-line values are matched against those definitions. If you need to pass additional flags or arguments directly to the underlying command on top of the typed args, use -- as a separator. Everything after -- is forwarded verbatim to the command, regardless of the task's args definition.

toml
[tasks.test]
cmd = "pytest {{ target }} -v"
args = [{ arg = "target", default = "tests/unit" }]
shell
# Without --, extra flags would cause an error:
# × task 'test' received more arguments than expected
#   hint: use `--` to separate task arguments from extra passthrough arguments
pixi run test tests/integration --tb=short --maxfail=5

# With --, the typed arg is filled and the rest is forwarded:
pixi run test tests/integration -- --tb=short --maxfail=5
✨ Pixi task (test in default): pytest tests/integration -v --tb=short --maxfail=5

You can also use -- when relying on a default argument value:

shell
# "target" uses its default, "--maxfail=5" is forwarded to the command
pixi run test -- --maxfail=5
✨ Pixi task (test in default): pytest tests/unit -v --maxfail=5

!!! note "Tasks without typed args" For tasks that do not define args, -- is passed through to the underlying command unchanged. This preserves its meaning for programs that use -- themselves (e.g. git log -- somefile).

Restricting Values with Choices

You can restrict the allowed values of an argument using the choices field. If a value is provided that is not in the list, Pixi will report an error instead of running the task.

toml
--8<-- "docs/source_files/pixi_tomls/task_arguments.toml:project_tasks_choices"
shell
# Providing a valid choice
pixi run compile debug
✨ Pixi task (compile): echo 'Compiling in debug mode'
Compiling in debug mode

# Default value must also be one of the choices
pixi run test
✨ Pixi task (test): echo 'Running unit tests'
Running unit tests

# Providing an invalid value results in an error
pixi run compile fast
× got 'fast' for argument 'mode' of task 'compile', choose from: debug, release

Passing Arguments to Dependent Tasks

You can pass arguments to tasks that are dependencies of other tasks:

toml
--8<-- "docs/source_files/pixi_tomls/task_arguments_dependent.toml:project_tasks"

When executing a dependent task, the arguments are passed to the dependency:

shell
pixi run install-release
✨ Pixi task (install in default): echo Installing with manifest /path/to/manifest and flag --debug

pixi run deploy
✨ Pixi task (install in default): echo Installing with manifest /custom/path and flag --verbose
✨ Pixi task (deploy in default): echo Deploying

When a dependent task doesn't specify all arguments, the default values are used for the missing ones:

toml
--8<-- "docs/source_files/pixi_tomls/task_arguments_partial.toml:project_tasks"
shell
pixi run partial-override
✨ Pixi task (base-task in default): echo Base task with override1 and default2

For a dependent task to accept arguments to pass to the dependency, you can use the same syntax as passing arguments to the command:

toml
--8<-- "docs/source_files/pixi_tomls/task_arguments_partial.toml:project_tasks_with_arg"
shell
pixi run partial-override-with-arg
✨ Pixi task (base-task in default): echo Base task with override1 and new-default2
pixi run partial-override-with-arg cli-arg
✨ Pixi task (base-task in default): echo Base task with override1 and cli-arg

MiniJinja Templating for Task Arguments

Task commands and env variables defined in the manifest support MiniJinja templating syntax for accessing and formatting argument values. This provides powerful flexibility when constructing commands.

!!! note "Templating and ad-hoc CLI commands" MiniJinja templating is only applied to tasks defined in your manifest file (pixi.toml / pyproject.toml). Ad-hoc commands passed directly to pixi run on the command line are not templated by default, so commands like pixi run echo '{{ hello }}' are passed through as-is. Use the --templated flag to opt in to template rendering for CLI commands:

```shell
pixi run --templated echo '{{ pixi.platform }}'
```

Basic syntax for using an argument in your command:

toml
--8<-- "docs/source_files/pixi_tomls/task_minijinja_simple.toml:tasks"

You can also use filters to transform argument values:

toml
--8<-- "docs/source_files/pixi_workspaces/minijinja/task_args/pixi.toml:tasks"

Pixi Variables

In addition to task arguments, Pixi automatically provides a pixi object in the MiniJinja context with system variables:

VariableDescriptionExample Value
pixi.platformThe platform name for the environment in which the task will runlinux-64, osx-arm64, win-64
pixi.environment.nameThe name of the current environment (when available)default, prod, test
pixi.manifest_pathAbsolute path to the manifest file/path/to/project/pixi.toml
pixi.versionThe version of pixi being used0.59.0
pixi.init_cwdThe current working directory when pixi was invoked/path/to/cwd
pixi.is_winBoolean flag indicating if the platform is Windowstrue or false
pixi.is_unixBoolean flag indicating if the platform is Unix-liketrue or false
pixi.is_linuxBoolean flag indicating if the platform is Linuxtrue or false
pixi.is_osxBoolean flag indicating if the platform is macOStrue or false

These variables are particularly useful for creating platform-specific or environment-aware tasks:

toml
[tasks]
# Platform-specific commands
build = { cmd = "cargo build --target {{ pixi.platform }}", args = [] }
download-binary = { cmd = "curl -O https://example.com/binary-{{ pixi.platform }}.tar.gz", args = [] }

# Conditional execution based on platform
install = { cmd = "{% if pixi.is_win %}install.bat{% else %}./install.sh{% endif %}", args = [] }

# Environment-aware tasks
deploy = { cmd = "deploy.sh --env {{ pixi.environment.name }}", args = [] }

# Using manifest path
validate = { cmd = "validator --manifest {{ pixi.manifest_path }}", args = [] }

# Using init_cwd in inputs/outputs for caching
mkdir_test = { cmd = "mkdir -p {{ pixi.init_cwd }}/test", outputs = ["{{ pixi.init_cwd }}/test"] }

The pixi variables can also be combined with task arguments:

toml
[tasks]
deploy = {
    cmd = "deploy.sh --platform {{ pixi.platform }} --env {{ environment }} --version {{ pixi.version }}",
    args = [{ arg = "environment", default = "staging" }]
}

When running tasks with typed arguments, the platform will automatically reflect the best platform for the environment where the task executes.

For more information about available filters and template syntax, see the MiniJinja documentation.

Task Names

A task name follows these rules:

  • No spaces are allowed in the name.
  • Must be unique within the table.
  • _ at the start of the name will hide the task from the pixi task list command.

Hiding tasks can be useful if your workspace defines many tasks but your users only need to use a subset of them.

toml
--8<-- "docs/source_files/pixi_tomls/task_visibility.toml:project_tasks"

Caching

When you specify inputs and/or outputs to a task, Pixi will reuse the result of the task.

For the cache, Pixi checks that the following are true:

  • No package in the environment has changed.
  • The selected inputs and outputs are the same as the last time the task was run. We compute fingerprints of all the files selected by the globs and compare them to the last time the task was run.
  • The command is the same as the last time the task was run.

If all of these conditions are met, Pixi will not run the task again and instead use the existing result.

Inputs and outputs can be specified as globs, which will be expanded to all matching files. You can also use MiniJinja templates in your inputs and outputs fields to parameterize the paths, making tasks more reusable:

toml
--8<-- "docs/source_files/pixi_tomls/tasks_minijinja_inputs_outputs.toml:tasks"

When using template variables in inputs/outputs, Pixi expands the templates using the provided arguments or environment variables, and uses the resolved paths for caching decisions. This allows you to create generic tasks that can handle different files without duplicating task configurations:

shell
# First run processes the file and caches the result
pixi run process-file data1

# Second run with the same argument uses the cached result
pixi run process-file data1  # [cache hit]

# Run with a different argument processes a different file
pixi run process-file data2

Note: if you want to debug the globs you can use the --verbose flag to see which files are selected.

shell
# shows info logs of all files that were selected by the globs
pixi run -v start

Environment Variables

You can set environment variables directly for a task, as well as by other means. See the environment variable priority documentation for full details of ways to set environment variables, and how those ways interact with each other.

Notes on environment variables in tasks:

  • Values set via tasks.<name>.env are interpreted by deno_task_shell when the task runs. Shell-style expansions like env = { VAR = "$FOO" } therefore work the same on all operating systems.
  • Templating is allowed in env variables, when you have something like { cmd="pytest", env={ BACKEND="{{ backend }}" }, args=[{arg="backend", default="numpy"}] }. The arg {{ backend }} value is interpreted by the backend values passed in.

!!! warning

In older versions of Pixi, this priority was not well-defined, and there are a number of known
deviations from the current priority which exist in some older versions:

- `activation.scripts` used to take priority over `activation.env`
- activation scripts of dependencies used to take priority over `activation.env`
- outside environment variables used to override variables set in `task.env`

If you previously relied on a certain priority which no longer applies, you may need to change your
task definitions.

For the specific case of overriding `task.env` with outside environment variables, this behavior can
now be recreated using [task arguments](#task-arguments). For example, if you were previously using
a setup like:

```toml title="pixi.toml"
[tasks]
echo = { cmd = "echo $ARGUMENT", env = { ARGUMENT = "hello" } }
```

```shell
ARGUMENT=world pixi run echo
✨ Pixi task (echo in default): echo $ARGUMENT
world
```

you can now recreate this behavior like:

```toml title="pixi.toml"
[tasks]
echo = { cmd = "echo {{ ARGUMENT }}", args = [{"arg" = "ARGUMENT", "default" = "hello" }] }
```

```shell
pixi run echo world
✨ Pixi task (echo): echo world
world
```

Clean environment

You can make sure the environment of a task is "Pixi only". Here Pixi will only include the minimal required environment variables for your platform to run the command in. The environment will contain all variables set by the conda environment like "CONDA_PREFIX". It will however include some default values from the shell, like: "DISPLAY", "LC_ALL", "LC_TIME", "LC_NUMERIC", "LC_MEASUREMENT", "SHELL", "USER", "USERNAME", "LOGNAME", "HOME", "HOSTNAME","TMPDIR", "XPC_SERVICE_NAME", "XPC_FLAGS"

toml
[tasks]
clean_command = { cmd = "python run_in_isolated_env.py", clean-env = true }

This setting can also be set from the command line with pixi run --clean-env TASK_NAME.

!!! warning "clean-env not supported on Windows" On Windows it's hard to create a "clean environment" as conda-forge doesn't ship Windows compilers and Windows needs a lot of base variables. Making this feature not worthy of implementing as the amount of edge cases will make it unusable.

Our task runner: deno_task_shell

To support the different OS's (Windows, OSX and Linux), Pixi integrates a shell that can run on all of them. This is deno_task_shell. The task shell is a limited implementation of a bourne-shell interface. Task command lines and the values of tasks.<name>.env are parsed and expanded by this shell.

Built-in commands

Next to running actual executable like ./myprogram, cmake or python the shell has some built-in commands.

  • cp: Copies files.
  • mv: Moves files.
  • rm: Remove files or directories. Ex: rm -rf [FILE]... - Commonly used to recursively delete files or directories.
  • mkdir: Makes directories. Ex. mkdir -p DIRECTORY... - Commonly used to make a directory and all its parents with no error if it exists.
  • pwd: Prints the name of the current/working directory.
  • sleep: Delays for a specified amount of time. Ex. sleep 1 to sleep for 1 second, sleep 0.5 to sleep for half a second, or sleep 1m to sleep a minute
  • echo: Displays a line of text.
  • cat: Concatenates files and outputs them on stdout. When no arguments are provided, it reads and outputs stdin.
  • exit: Causes the shell to exit.
  • unset: Unsets environment variables.
  • xargs: Builds arguments from stdin and executes a command.

Syntax

  • Boolean list: use && or || to separate two commands.
    • &&: if the command before && succeeds continue with the next command.
    • ||: if the command before || fails continue with the next command.
  • Sequential lists: use ; to run two commands without checking if the first command failed or succeeded.
  • Environment variables:
    • Set env variable using: export ENV_VAR=value
    • Use env variable using: $ENV_VAR
    • unset env variable using unset ENV_VAR
  • Shell variables: Shell variables are similar to environment variables, but won't be exported to spawned commands.
    • Set them: VAR=value
    • use them: VAR=value && echo $VAR
  • Pipelines: Use the stdout output of a command into the stdin a following command
    • |: echo Hello | python receiving_app.py
    • |&: use this to also get the stderr as input.
  • Command substitution: $() to use the output of a command as input for another command.
    • python main.py $(git rev-parse HEAD)
  • Negate exit code: ! before any command will negate the exit code from 1 to 0 or visa-versa.
  • Redirects: > to redirect the stdout to a file.
    • echo hello > file.txt will put hello in file.txt and overwrite existing text.
    • python main.py 2> file.txt will put the stderr output in file.txt.
    • python main.py &> file.txt will put the stderr and stdout in file.txt.
    • echo hello >> file.txt will append hello to the existing file.txt.
  • Glob expansion: * to expand all options.
    • echo *.py will echo all filenames that end with .py
    • echo **/*.py will echo all filenames that end with .py in this directory and all descendant directories.
    • echo data[0-9].csv will echo all filenames that have a single number after data and before .csv

More info in deno_task_shell documentation.

Best Practices

Task can be very handy and may even replace simple Makefiles. There are a few principles that can make tasks more useful.

Use a description

It is a good habit to add a description to your task:

toml
[tasks]
echo = { cmd = "echo Hello Pixi user", description = "Friendly greeting to a Pixi user" }
build = { cmd = "build", description = "Build everything" }
test = { cmd = "test", description = "Run all tests" }

Now, the command pixi task list will not only list all task names but also the their descriptions.

shell
pixi task list
Tasks that can run on this machine:
-----------------------------------
build, echo, test
Task   Description
build  Build everything
echo   Friendly greeting to a Pixi user
test   Run all tests

This list can be very helpful to quickly find the right tasks.

Use different TOML syntax for long commands

Tasks can get quite long:

toml
echo-arg = { cmd = "echo {{ ARGUMENT }}", args = [{"arg" = "ARGUMENT", "default" = "hello"}], description = "Display the given argument" }

While it is possible to add line breaks inside the task in your pixi.toml to make the task more readable:

toml
echo-arg = {
    cmd = "echo {{ ARGUMENT }}",
    args = [{"arg" = "ARGUMENT", "default" = "hello"}],
    description = "Display the given argument"
}

it will not work in pyproject.toml because it supports only TOML 1.0, that doesn't allow line breaks inside the task specification. Other tools can't parse the pyproject.toml anymore. So if you use an editable dependency:

toml
[tool.pixi.pypi-dependencies]
myproject = { path = ".", editable = true }

Pixi will use your build system and reports an error:

shell
Error:   × Failed to update PyPI packages for environment 'default'
  ├─▶ Failed to prepare distributions
  ├─▶ Failed to build `proj1 @ file:///.../proj1`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `hatchling.build.build_editable` failed (exit status: 1)

There is an alternative syntax, adding the task name to the table header:

toml
[tool.pixi.tasks.echo-arg]
cmd = "echo {{ ARGUMENT }}"
args = [{"arg" = "ARGUMENT", "default" = "hello"}]
description = "Display the given argument"

This is valid TOML 1.0 syntax and makes more complex task better readable.