Managing Credentials in Pipenv

This guide covers best practices for securely managing credentials and authentication in Pipenv, including environment variables, private repositories, and security considerations.

Credentials in Package Sources

When working with private package repositories or authenticated services, you need to securely manage credentials in your Pipenv workflow.

Environment Variable Expansion

Pipenv automatically expands environment variables in your Pipfile, providing a secure way to inject credentials without storing them in version control.

Basic Environment Variable Syntax

You can use environment variables in your Pipfile using the following syntax:

${VARIABLE_NAME} - Standard syntax
$VARIABLE_NAME - Alternative syntax
%VARIABLE_NAME% - Windows-specific syntax (also supported on all platforms)

Example: Private Repository Authentication

toml

[[source]]
url = "https://${USERNAME}:${PASSWORD}@mypypi.example.com/simple"
verify_ssl = true
name = "private-pypi"

When Pipenv reads this Pipfile, it will replace ${USERNAME} and ${PASSWORD} with the values of the corresponding environment variables.

Setting Environment Variables

Before running Pipenv commands, set the required environment variables:

Linux/macOS:

bash

$ export USERNAME="your-username"
$ export PASSWORD="your-password"
$ pipenv install

Windows (Command Prompt):

cmd

> set USERNAME=your-username
> set PASSWORD=your-password
> pipenv install

Windows (PowerShell):

powershell

> $env:USERNAME="your-username"
> $env:PASSWORD="your-password"
> pipenv install

Security Considerations

Pipenv hashes your Pipfile before expanding environment variables and substitutes them again when installing from the lock file. This means:

Your credentials are never stored in the lock file
You don't need to commit any secrets to version control
Different developers can use different credentials with the same Pipfile

{warning}

While environment variables provide better security than hardcoded credentials, they are still accessible to any process running with the same user permissions. For highly sensitive credentials, consider using a dedicated secrets management solution.

URL Encoding for Special Characters

If your credentials contain special characters, they must be URL-encoded according to RFC 3986.

For example, if your password is p@ssw0rd!, it should be encoded as p%40ssw0rd%21:

toml

[[source]]
url = "https://${USERNAME}:${PASSWORD_ENCODED}@mypypi.example.com/simple"
verify_ssl = true
name = "private-pypi"

You can generate URL-encoded strings using Python:

python

import urllib.parse
print(urllib.parse.quote("p@ssw0rd!", safe=""))
# Output: p%40ssw0rd%21

Credentials in Package Requirements

Environment variables can also be used in package requirement specifiers, but with some limitations.

VCS Repository Authentication

For version control system (VCS) repositories that require authentication:

toml

[packages]
requests = {git = "git://${USERNAME}:${PASSWORD}@private.git.com/psf/requests.git", ref = "2.22.0"}

{note}

For VCS repositories, only the `${VAR_NAME}` syntax is supported. Neither `$VAR_NAME` nor `%VAR_NAME%` will work in this context.

Runtime vs. Install-time Expansion

It's important to understand that environment variables are expanded at runtime, not when the Pipfile or Pipfile.lock is created. This means:

The entries in Pipfile and Pipfile.lock remain untouched
You need to have the environment variables set every time you run Pipenv commands
Different environments (development, CI/CD, production) can use different credentials

Keyring Integration

Pipenv supports keyring integration for authentication to private registries. With pip 23.1+, there are two approaches to configure keyring authentication.

Modern Approach: System-Wide Keyring (pip >= 23.1)

Starting with pip 23.1, you can install keyring system-wide (outside of your project virtualenv) and configure pip to use it via the subprocess provider. This is the recommended approach as it doesn't require installing keyring in every project.

Setup Steps

Install keyring globally (using pip, pipx, or your system package manager):

bash

# Using pip (user install)
$ pip install --user keyring keyrings.google-artifactregistry-auth

# Or using pipx (recommended for CLI tools)
$ pipx install keyring
$ pipx inject keyring keyrings.google-artifactregistry-auth

Ensure keyring is on your PATH:

bash

$ which keyring
/home/user/.local/bin/keyring

Ensure virtualenv seeds pip >= 23.1 (one-time setup):
bash
```
$ virtualenv --upgrade-embed-wheels
```
Configure the keyring provider using one of these methods:

Option A: Environment variable (recommended for Pipenv):
bash
```
$ export PIPENV_KEYRING_PROVIDER=subprocess
```
Option B: pip configuration:
bash
```
# Global configuration
$ pip config set --global keyring-provider subprocess

# Or user-level configuration
$ pip config set --user keyring-provider subprocess
```
!!! note Setting PIPENV_KEYRING_PROVIDER ensures that keyring credentials are used during both dependency resolution and installation, even though Pipenv disables interactive pip input by default. This is particularly important for credential managers like Windows Credential Manager.
Configure your index URL with the appropriate username:
- For Google Artifact Registry: use oauth2accesstoken as username
- For Azure Artifacts: use VssSessionToken as username
toml
```
# Pipfile
[[source]]
url = "https://[email protected]/my-project/python/simple/"
verify_ssl = true
name = "private-gcp"
```

Legacy Approach: Keyring in Project Virtualenv

Alternatively, you can install keyring directly in your project's virtual environment. This approach works with all pip versions but requires installing keyring in every project.

Google Cloud Artifact Registry

Google Cloud Artifact Registry supports authentication via keyring:

Install the required packages:

bash

$ pipenv run pip install keyring keyrings.google-artifactregistry-auth

Configure your Pipfile:

toml

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://europe-python.pkg.dev/my-project/python/simple"
verify_ssl = true
name = "private-gcp"

[packages]
flask = "*"
private-test-package = {version = "*", index = "private-gcp"}

If the keyring might ask for user input, you may need to disable input enforcement:
toml
```
[pipenv]
disable_pip_input = false
```
Alternatively, if your keyring provider does not require user interaction (e.g. Windows Credential Manager), you can set PIPENV_KEYRING_PROVIDER=subprocess instead.

Azure Artifact Registry

For Azure Artifact Registry:

Install the required packages:

bash

$ pipenv run pip install keyring artifacts-keyring

Configure your Pipfile similar to the Google Cloud example above, using VssSessionToken as the username in your index URL.

Windows Credential Manager

On Windows, pip can use credentials stored in Windows Credential Manager via the keyring library. To use these credentials with Pipenv:

Install keyring globally:
powershell
```
> pip install --user keyring
```
Set the keyring provider:
powershell
```
> $env:PIPENV_KEYRING_PROVIDER="subprocess"
```
Or set it permanently via system environment variables.
Store your credentials in Windows Credential Manager (via Control Panel or cmdkey).

Configure your Pipfile with the private index URL (with or without the username):

toml

[[source]]
url = "https://[email protected]/simple"
verify_ssl = true
name = "private"

AWS CodeArtifact

For AWS CodeArtifact:

Install the required packages:

bash

$ pipenv run pip install keyring keyrings.codeartifact

Configure your Pipfile with the appropriate AWS CodeArtifact URL.

Best Practices for Credential Management

Use Environment Variables

Always use environment variables instead of hardcoding credentials:

toml

# Good
url = "https://${USERNAME}:${PASSWORD}@private-repo.example.com/simple"

# Bad
url = "https://actual-username:[email protected]/simple"

Store Credentials in .env Files

For local development, store credentials in .env files that are not committed to version control:

# .env
USERNAME=your-username
PASSWORD=your-password

Add .env to your .gitignore:

# .gitignore
.env

Pipenv will automatically load variables from .env files when you run commands.

Provide Templates

Include a template .env.example file in your repository:

# .env.example
USERNAME=
PASSWORD=

This helps other developers understand which environment variables they need to set.

Use Different Credentials for Different Environments

Maintain separate credential sets for different environments:

# .env.development
USERNAME=dev-username
PASSWORD=dev-password

# .env.production
USERNAME=prod-username
PASSWORD=prod-password

Use them with:

bash

$ PIPENV_DOTENV_LOCATION=.env.development pipenv install

Rotate Credentials Regularly

Regularly rotate your credentials to minimize the impact of potential leaks:

Generate new credentials
Update your environment variables or .env files
Revoke the old credentials

Consider Credential Managers

For team environments, consider using dedicated credential management solutions:

These can be integrated into your workflow to provide secure, centralized credential management.

CI/CD Integration

GitHub Actions

For GitHub Actions, use secrets to store credentials:

yaml

name: Python Package

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.10'
    - name: Install dependencies
      env:
        USERNAME: ${{ secrets.PYPI_USERNAME }}
        PASSWORD: ${{ secrets.PYPI_PASSWORD }}
      run: |
        pip install pipenv
        pipenv install --dev

GitLab CI

For GitLab CI, use CI/CD variables:

yaml

stages:
  - test

test:
  stage: test
  image: python:3.10
  variables:
    USERNAME: ${PYPI_USERNAME}
    PASSWORD: ${PYPI_PASSWORD}
  script:
    - pip install pipenv
    - pipenv install --dev
    - pipenv run pytest

Jenkins

For Jenkins, use credentials binding:

groovy

pipeline {
    agent {
        docker {
            image 'python:3.10'
        }
    }
    stages {
        stage('Build') {
            steps {
                withCredentials([
                    string(credentialsId: 'pypi-username', variable: 'USERNAME'),
                    string(credentialsId: 'pypi-password', variable: 'PASSWORD')
                ]) {
                    sh 'pip install pipenv'
                    sh 'pipenv install --dev'
                    sh 'pipenv run pytest'
                }
            }
        }
    }
}

Troubleshooting

Common Issues

Environment Variables Not Being Expanded

If environment variables aren't being expanded:

Verify the variables are set correctly:
bash
```
$ echo $USERNAME
$ echo $PASSWORD
```
Check the syntax in your Pipfile:
- Use ${VARIABLE_NAME} for the most reliable expansion
- Ensure there are no typos in variable names
Try setting the variables directly in the command:
bash
```
$ USERNAME=user PASSWORD=pass pipenv install
```

Authentication Failures

If you're experiencing authentication failures:

Verify your credentials work outside of Pipenv:

bash

$ curl -u "${USERNAME}:${PASSWORD}" https://private-repo.example.com/simple

Check for special characters that might need URL encoding
Ensure your credentials have the necessary permissions

Keychain Integration Issues

If keychain integration isn't working:

Verify the keychain packages are installed:
bash
```
$ pipenv run pip list | grep keyring
```
Check if the keychain is properly configured for your system
Try disabling pip input enforcement:
toml
```
[pipenv]
disable_pip_input = false
```

Security Considerations

Credential Leakage Risks

Be aware of these common credential leakage risks:

Command history: Credentials passed directly in commands are stored in shell history
Process listing: Environment variables set on the command line may be visible in process listings
Log files: Debug logs might include expanded environment variables
Core dumps: Application crashes might include memory containing credentials

Mitigating Risks

To mitigate these risks:

Use .env files instead of setting variables on the command line
Limit access to environments where credentials are used
Use temporary credentials or tokens when possible
Implement the principle of least privilege for all credentials
Monitor for unauthorized access to your repositories

Conclusion

Properly managing credentials in Pipenv is essential for security and maintainability. By using environment variables, .env files, and following best practices, you can securely authenticate to private repositories and services without compromising sensitive information.

Remember that credential management is an ongoing process that requires regular review and updates to maintain security. Always follow the principle of least privilege and rotate credentials regularly to minimize potential security risks.