ContextQMD
Back to Gitlabhq

Create a new Python projects

doc/development/python_guide/create_project.md

18.11.27.3 KB
Original Source

When creating a new Python repository, some guidelines help keep our code standardized.

Development & testing

  • pytest: Primary testing framework for writing and running tests.
  • pytest-cov: Test coverage reporting plugin for pytest.
  • black: Opinionated code formatter that ensures consistent code style.
  • flake8: Linter for style enforcement.
  • pylint: Comprehensive linter for error detection and quality enforcement.
  • mypy: Static type checker.
  • isort: Utility to sort imports.

Package manager & build system

  • poetry: Modern packaging and dependency management.

Common utilities

  • typer: Library for building CLI applications.
  • python-dotenv: Environment variable management.
  • pydantic: Data validation and settings management using Python type annotations.
  • fastapi: Modern, high-performance web framework for building APIs.
  • structlog: Structured logging library.
  • httpx: Asynchronous and performant HTTP client.
  • rich: Terminal formatting library for rich text.
  • sqlmodel: Intuitive and robust ORM.
  • tqdm: Fast, extensible progress bar for CLI.

Depending on the type of project, for example, API service, CLI application or library, the folder structure can be varied. The following structure is for a standard CLI application.

plaintext
project_name/
├── .gitlab/                     # GitLab-specific configuration
│   ├── issue_templates/         # Issue templates
│   └── merge_request_templates/ # MR templates
├── .gitlab-ci.yml               # CI/CD configuration
├── project_name/                # Main package directory
│   ├── __init__.py              # Package initialization
│   ├── cli.py                   # Command-line interface entry points
│   ├── config.py                # Configuration handling
│   └── core/                    # Core functionality
│       └── __init__.py
├── tests/                       # Test directory
│   ├── __init__.py
│   ├── conftest.py              # pytest fixtures and configuration
│   └── test_*.py                # Test modules
├── docs/                        # Documentation
├── scripts/                     # Utility scripts
├── README.md                    # Project overview
├── CONTRIBUTING.md              # Contribution guidelines
├── LICENSE                      # License information
├── pyproject.toml               # Project metadata and dependencies (Poetry)

Linter configuration

We should consolidate configurations into pyproject.toml as much as possible.

pyproject.toml

toml
[tool.black]
line-length = 120

[tool.isort]
profile = "black"

[tool.mypy]
python_version = 3.12
ignore_missing_imports = true

[tool.pylint.main]
jobs = 0
load-plugins = [
  # custom plugins
]

[tool.pylint.messages_control]
enable = [
  # custom plugins
]

[tool.pylint.reports]
score = "no"

setup.cfg

ini
[flake8]
extend-ignore = E203,E501
extend-exclude = **/__init__.py,.venv,tests
indent-size = 4
max-line-length = 120

Example Makefile

makefile
# Excerpt from project Makefile showing common targets

# lint
.PHONY: install-lint-deps
install-lint-deps:
    @echo "Installing lint dependencies..."
    @poetry install --only lint

.PHONY: format
format: black isort

.PHONY: black
black: install-lint-deps
    @echo "Running black format..."
    @poetry run black ${CI_PROJECT_DIR}

.PHONY: isort
isort: install-lint-deps
    @echo "Running isort format..."
    @poetry run isort ${CI_PROJECT_DIR}

.PHONY: lint
lint: flake8 check-black check-isort check-pylint check-mypy

.PHONY: flake8
flake8: install-lint-deps
    @echo "Running flake8..."
    @poetry run flake8 ${CI_PROJECT_DIR}

.PHONY: check-black
check-black: install-lint-deps
    @echo "Running black check..."
    @poetry run black --check ${CI_PROJECT_DIR}

.PHONY: check-isort
check-isort: install-lint-deps
    @echo "Running isort check..."
    @poetry run isort --check-only ${CI_PROJECT_DIR}

.PHONY: check-pylint
check-pylint: install-lint-deps install-test-deps
    @echo "Running pylint check..."
    @poetry run pylint ${CI_PROJECT_DIR}

.PHONY: check-mypy
check-mypy: install-lint-deps
    @echo "Running mypy check..."
    @poetry run mypy ${CI_PROJECT_DIR}

# test
.PHONY: test
test: install-test-deps
    @echo "Running tests..."
    @poetry run pytest

.PHONY: test-coverage
test-coverage: install-test-deps
    @echo "Running tests with coverage..."
    @poetry run pytest --cov=duo_workflow_service --cov=lints --cov-report term --cov-report html

Example GitLab CI Configuration

yaml
# Excerpt from .gitlab-ci.yml showing linting and testing jobs

image: python:3.13

stages:
  - lint
  - test

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  POETRY_CACHE_DIR: "$CI_PROJECT_DIR/.cache/poetry"
  POETRY_VERSION: "2.1.2"

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - $PIP_CACHE_DIR
    - $POETRY_CACHE_DIR
    - .venv/

# Base template for Python jobs
.poetry:
  before_script:
    - pip install poetry==${POETRY_VERSION}
    - poetry config virtualenvs.in-project true
    - poetry add --dev black isort flake8 pylint mypy pytest pytest-cov

# Linting jobs
black:
  extends: .poetry
  stage: lint
  script:
    - poetry run black --check ${CI_PROJECT_DIR}

isort:
  extends: .poetry
  stage: lint
  script:
    - poetry run isort --check-only ${CI_PROJECT_DIR}

flake8:
  extends: .poetry
  stage: lint
  script:
    - poetry run flake8 ${CI_PROJECT_DIR}

pylint:
  extends: .poetry
  stage: lint
  script:
    - poetry run pylint ${CI_PROJECT_DIR}

mypy:
  extends: .poetry
  stage: lint
  script:
    - poetry run mypy ${CI_PROJECT_DIR}

# Testing jobs
test:
  extends: .poetry
  stage: test
  script:
    - poetry run pytest --cov=duo_workflow_service --cov-report=term --cov-report=xml:coverage.xml --junitxml=junit.xml
  coverage: '/TOTAL.+?(\d+\%)/'
  artifacts:
    when: always
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

Adding reviewer roulette

We recommend reviewer roulette to distribute review workload across reviewers and maintainers. A pool of Python Reviewers is available for small Python projects and can be configured following these steps.

To create a pool of reviewers specific to a project:

  1. Follow the GitLab Dangerfiles instructions to add the configuration to your project.
  2. Implement the Danger Reviewer component in your GitLab CI pipeline to automatically trigger the roulette.