docs/dev/initiative-g-pure-python-design.md
Status: Phases 1 + 2 shipped (Phase 2 with CI bench gate skipped);
phases 3–4 awaiting maintainer sign-off.
Phase 1 (the standalone PEP 691 client + parsed-manifest cache +
parallel fetcher) is on main as of T17 (see §11 below for the
per-criterion sign-off). Phase 2 (the prefetch bridge that wires the
parallel fetcher into do_lock behind the opt-in
[pipenv] prefetch_index_manifests setting) shipped at T22 without
the originally-planned CI bench measurement gate (T21) — see the
Phase-2 sign-off note in §11a for the explicit scoping decision and the
honest "claim is theoretical, not measured" caveat. Phases 3–4 remain
at design stage pending the bench-data sign-off described in §11.
Companion documents:
initiative-f-backends-design.md —
established the Backend protocol and pluggable-backend registry
this initiative slots into. Initiative G is a new backend
implementation, not a new architecture.initiative-f-typed-design.md —
the typed ResolverRequest / ResolverResponse envelope every
backend speaks.modernization-plan.md — overall
modernization framing.Initiative G is a multi-phase effort to replace the parts of pipenv's
resolution path that depend on pip's _internal APIs with an in-tree,
pure-Python implementation modeled on uv's architecture (PEP 691
simple-API client, parallel fetch, parsed-manifest cache).
It implements a new backend under the Initiative F framework
(pipenv/resolver/backends/pure_python.py). pip remains the default
backend during the migration. The pure-Python backend ships first as
opt-in via [pipenv] resolver_backend = "pure-python" (or
--backend pure-python); promotion to default and eventual removal of
the pip backend is gated on real-CI parity over a release window.
The two goals, in priority order:
pipenv/patched/pip/_internal/... consumers;
each of phases 3 and 4 of the 2026-05 modernization initiative was
driven by chasing that churn. A pure-Python client whose only
external contract is PEP 691 / PEP 503 is stable against pip's
internal refactors.maintenance/code-cleanup-phase5-perf-2026-06
commit history) measured the warm-relock ceiling at ~19 s for a
100-package Pipfile. ~9 s of that is urllib3 streaming/decoding
inside pip's sequential per-package index revalidation. Pip's
architecture (Cache-Control: max-age=0 forced revalidation, raw
response cache, sequential find_all_candidates) is the bound; no
amount of pipenv-side optimisation breaks it. uv resolves the
same fixture in ~1-2 s by replacing those decisions, not by being
Rust.This document covers what to build, in what order, against which
contract. Phase 1 ships the standalone client; phase 2 wires it as a
cache-prime layer behind pip; phase 3 replaces the
PackageFinder.find_all_candidates call entirely; phase 4 (optional)
replaces pip's resolvelib Provider with one that consumes our typed
Candidates directly.
The pipenv/patched/pip/_vendor/ and pipenv/patched/pip/_internal/
trees contain ~MB of vendored pip code that we patch and re-distribute.
The patches exist because pip's resolution path uses APIs that pip
explicitly marks as internal:
pip._internal.commands.install.InstallCommand — used as a config
parser proxy; pip docs forbid this use.pip._internal.index.package_finder.PackageFinder — pipenv drives
it directly via Resolver.get_resolver().pip._internal.network.session.PipSession —
re-used as our HTTP client.pip._internal.req.req_install.InstallRequirement — pipenv
constructs these directly.pip._internal.models.link.Link — used as the lockfile-entry
source-of-truth.pip._internal.req.constructors.parse_req_from_line — pipenv's
parse path.pip._internal.resolution.resolvelib.resolver.Resolver — the
internal driver.Counting the May 2026 modernization commits, ~40 % of the merged
work was reactive churn to keep these consumers compiling after
upstream pip refactors. Phase 3's typed-resolver schema reduced this
to a single seam (the ResolverResponse envelope), but the seam is
still pip-shaped on the inside.
Profile of a representative 100-package pipenv lock (warm cache),
in-process resolver, no spinner:
| component (cumulative time) | wall | notes |
|---|---|---|
Resolver.resolve_for_pipenv (parent dispatch) | 12.5 s | full wall |
find_best_candidate / find_all_candidates | 9.7 s | inside the resolver |
process_project_url | 8.5 s | per-package PyPI fetch |
| urllib3 streaming + decode | 9.4 s | mostly inside the fetch |
Link.from_json | 9.6 s (cum) | per-link URL construction |
_ensure_quoted_url | 8.9 s (cum) | called from Link.from_json |
evaluate_link self time | 0.24 s | per-link compatibility check |
The 8.5 s in process_project_url is ~50 sequential HTTP round-trips
(PyPI simple-API GETs with Cache-Control: max-age=0 — pip
deliberately revalidates every read; see _get_simple_response in
pipenv/patched/pip/_internal/index/collector.py:108). On the
benchmark CI (lock-warm = 19.25 s) the ratio is similar.
The May 2026 phase-5 work hit a ceiling around 7-11 % improvement because every remaining cut runs into one of:
Link.from_json running urlsplit/urlunsplit on every
candidate (cumulative wins are mostly I/O, not CPU).None of these are pipenv-side problems. The only durable wins require changing the decisions, not the implementation details.
uv ships ~10× faster on the same workload. The Rust isn't the win; the architecture is. Decomposed:
| decision | pip | uv |
|---|---|---|
| transport | HTTP/1.1, up to ~10 parallel connections via urllib3 pool | HTTP/2, multiplexed over one TCP connection |
| simple-API parse | Link.from_json per candidate, urlsplit/urlunsplit per URL | one parse per package, store typed Candidate |
| cache format | raw HTTP responses in CacheControl/SafeFileCache | parsed candidate manifests on disk |
| revalidation | Cache-Control: max-age=0 forces conditional GET on every read | trust cache within session; revalidate on --refresh |
| fetch order | sequential per-package inside find_all_candidates | all top-level packages' candidates fetched concurrently before resolution |
| resolver | resolvelib (PubGrub-shaped, backtracking) | pubgrub (same family, different impl) |
Three of those (transport, simple-API parse, cache format) are language-agnostic and replicable in pure Python. The other three (revalidation policy, fetch ordering, resolver) flow from there.
Initiative G targets the first three first — the transport + parse + cache rewrite is where the architectural lift comes from. Replacing resolvelib is explicitly out of scope for this initiative.
application/vnd.pypi.simple.v1+json)
client with PEP 503 HTML fallback.Candidate dataclass and a parsed-manifest on-disk cache
with explicit TTL, invalidation, and schema versioning.Backend (pipenv/resolver/backends/pure_python.py)
under the Initiative F framework.resolvelib, just with our own Provider and our own
candidate source.pipenv install's wheel
download + install path stays on pip install for now; only the
resolution side moves.prepare_metadata_for_build_wheel.pipenv/patched/pip/_vendor/ entirely. Pip remains
vendored for the install side and as a fallback backend. ┌──────────────────────────────────────────┐
│ pipenv parent process │
│ │
│ do_lock / do_install │
│ │ │
│ ▼ │
│ ResolverRequest (Initiative F typed) │
│ │ │
└──────┼─────────────────────────────────────┘
│
┌──────────────┴─────────────┐
│ │
pip backend pure-python backend ← NEW (Initiative G)
(existing) │
│ ┌──────────┴──────────┐
│ │ │
│ PEP691Client ParsedManifestCache
│ (HTTP/1.1 -> 2) (~/.cache/pipenv/manifests/)
│ │ ▲
│ └─────────┬───────────┘
│ │
│ pure_python.Provider
│ (resolvelib.Provider impl
│ over our Candidate)
│ │
│ ▼
└──────► resolvelib.Resolver ◄───────────┘
│
▼
ResolverResponse (typed)
Key points:
ResolverRequest / ResolverResponse typed envelope is the
unchanged contract between pipenv's parent process and any
backend. Backend selection is the Initiative F existing
mechanism.pip._internal.* symbols. Its only pip dependency is
resolvelib (vendored under pipenv/patched/pip/_vendor/resolvelib/),
which is upstream-stable and not pip-internal.PEP691Client and ParsedManifestCache are reusable across
backends — a future "uv-backend" delegating to the uv binary could
use the same cache, and a future replacement for pip's installation
side could share the manifest data.PEP691Client (phase 1)Module: pipenv/resolver/pep691.py
Responsibility: fetch a single package's simple-API page from a
single index URL. Returns a parsed list of Candidates. Does not
cache. Does not retry beyond HTTP-level transient errors.
Interface (proposed):
class PEP691Client:
def __init__(
self,
*,
session: httpx.Client | urllib3.PoolManager,
# ↑ Pluggable for testability; concrete choice in §6.
netrc: NetrcLookup | None = None,
cert: tuple[str, str] | None = None,
) -> None: ...
def fetch(
self,
index_url: str,
package_name: str,
*,
if_none_match: str | None = None, # ETag for conditional GET
) -> SimplePageResponse: ...
@dataclass(frozen=True)
class SimplePageResponse:
candidates: tuple[Candidate, ...]
etag: str | None
last_modified: str | None
raw_meta: dict # api-version, etc., for forward-compat
status: Literal["fresh", "not-modified", "missing"]
@dataclass(frozen=True)
class Candidate:
name: str # canonical (PEP 503)
version: str # PEP 440
url: str # ABSOLUTE; no further quoting needed
filename: str # for tag inspection
hashes: frozenset[Hash] # (algo, value)
requires_python: str | None
yanked: bool
yanked_reason: str | None
upload_time: datetime | None
is_wheel: bool # derived from filename
# Wheel-only fields; None for sdists
wheel_tags: frozenset[Tag] | None
Critical design notes:
url is stored already-absolute and already-quoted. Done once at
parse time, not per-evaluation. This is the replacement for pip's
_ensure_quoted_url-per-link cost.wheel_tags is computed at parse time from the filename via
pipenv/vendor/packaging.tags.parse_tag (vendored packaging, not
patched-pip), so compatibility checks become a frozenset
intersection instead of pip's Wheel.supported(tags) call.Hash is a (algo: str, value: str) tuple, frozen and hashable so
hash-set comparisons are fast.Accept: application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html; q=0.1, text/html; q=0.01)
and parse HTML as a fallback. HTML parsing uses
html.parser.HTMLParser (stdlib).Cache-Control: max-age=0. Freshness is
controlled by the cache layer (§5.2), not by forcing
revalidation on every read.Tests (tests/unit/test_pep691_client.py):
Candidate set.Candidate set.status="not-modified".status="missing".yanked=True, reason preserved.manylinux, musllinux,
macosx, win_amd64, any, abi3.hashes and HTML
data- attributes.ParsedManifestCache (phase 1)Module: pipenv/resolver/manifest_cache.py
Responsibility: persist parsed Candidate tuples to disk, keyed
by (index_url, package_name). Replace pip's CacheControl + JSON
re-parse hot path.
Interface:
class ParsedManifestCache:
def __init__(self, root: Path, schema_version: int = 1) -> None: ...
def get(
self, index_url: str, package_name: str
) -> CachedManifest | None: ...
def put(self, index_url: str, package_name: str,
candidates: Sequence[Candidate],
etag: str | None,
ttl_seconds: int) -> None: ...
def invalidate(self, index_url: str, package_name: str) -> None: ...
@dataclass(frozen=True)
class CachedManifest:
candidates: tuple[Candidate, ...]
etag: str | None
cached_at: datetime
expires_at: datetime
Disk format:
~/.cache/pipenv/manifests-v{schema_version}/.<sha256(index_url)>/<canonical_package_name>.msgpack
(or json — see §10 Q1).CachedManifest including version,
schema_version, cached_at, expires_at, etag, candidates.tempfile.NamedTemporaryFile + os.replace.Freshness policy (versus pip's max-age=0 revalidate-every-time):
If-None-Match: <etag>); on 304,
extend expires_at by TTL; on 200, replace.--clear invalidates the whole cache root.pipenv update --refresh-index (new flag) bypasses TTL for the
current resolve.This is a behaviour change from pip's hard-coded
revalidate-every-read. The honest tradeoff: twine upload && pipenv install race window grows from "~10 minutes" (pip's PyPI Cache-Control
window) to "TTL seconds", but the more important property — fresh
resolves pick up matching new releases — is unchanged because the user
running pipenv update always gets a fresh fetch.
Module: pipenv/resolver/fetcher.py
Responsibility: given a list of (index_url, package_name) pairs
and a ParsedManifestCache, populate the cache concurrently. Returns
a dict[package_name, CachedManifest | FetchError].
Concurrency model:
concurrent.futures.ThreadPoolExecutor with up to 16
workers (matches the urllib3 connection-pool ceiling we measured —
beyond 16 we hit "Connection pool is full, discarding connection"
warnings).asyncio + httpx[http2] for
multiplexed fetches. Spec'd behind a feature flag in phase 3 design.Error handling:
FetchError(kind="missing"), the
package may still resolve from another index.FetchError(kind="transient"),
the resolver retries the affected lookup on demand.pure_python.Provider — resolvelib integration (phase 3)Module: pipenv/resolver/backends/pure_python_provider.py
Responsibility: implement the resolvelib.AbstractProvider
interface (identify, get_preference, find_matches,
is_satisfied_by, get_dependencies) over our Candidate types,
backed by PEP691Client + ParsedManifestCache.
Critical methods:
find_matches(identifier, requirements, incompatibilities) →
returns candidates from our cache, filtered to those matching
every requirement and not matching any incompatibility.
This is the hottest path; correctness here drives the whole
resolution.get_dependencies(candidate) → for wheel candidates, fetch the
wheel's METADATA from the index (using PEP 658 metadata files
where the index advertises them; downloading the wheel head
bytes for those that don't). For sdists, fall back to pip's
metadata extraction (sdist build is out of scope for Initiative G).get_preference(...) → match pip's preference ordering exactly
so lockfiles produced by pure_python backend ≡ lockfiles produced
by pip backend for the same input. This is the parity criterion.pure_python Backend (phase 3)Module: pipenv/resolver/backends/pure_python.py
Responsibility: implement Initiative F's Backend protocol.
Translates ResolverRequest → resolvelib resolve via our
Provider, translates the resolved graph back into the typed
ResolverResponse.
Pseudo-code shape:
class PurePythonBackend:
name = "pure-python"
def resolve(self, request: ResolverRequest) -> ResolverResponse:
client = PEP691Client(session=...)
cache = ParsedManifestCache(root=...)
fetcher = ParallelFetcher(client, cache, max_workers=16)
# Pre-fetch top-level packages.
top_level = [(src.url, name) for src in request.sources
for name in request.packages.specs]
fetcher.populate(top_level)
provider = PurePythonProvider(client, cache, request)
resolver = resolvelib.Resolver(provider, ...)
result = resolver.resolve(...)
return _result_to_response(result, request)
Options (ranked):
urllib3 directly (phase 1 ships here).
pipenv/vendor/ transitively.httpx[http2] (phase 3 — deferred).
Phase 1 chooses urllib3 because it ships zero new dependency risk and the architectural lift (parsed-manifest cache, parallel fetch, single parse-per-package) is most of the win. HTTP/2 is a phase 3 multiplier on an already-faster baseline.
json stdlib. PEP 691 responses are well under 1 MB
typically; no streaming-parser need.html.parser.HTMLParser stdlib. PEP 503 anchor parsing is
trivial; no lxml dependency.pipenv.vendor.packaging is the version / specifier / marker /
tag library. Already vendored, not pip-internal. Initiative G
uses it directly.Phase 1's only requirement is the stdlib + already-vendored
pipenv.vendor.packaging + already-vendored
pipenv.patched.pip._vendor.urllib3. Adding httpx is a phase 3
decision; the design doc for that phase lays out the vendoring
posture explicitly.
The pluggable-backend registry from Initiative F is the integration point. No work needed here; just confirming the seam exists.
Deliverable: PEP691Client, Candidate, ParsedManifestCache,
ParallelFetcher with unit tests. Not wired into do_lock.
Acceptance: feed the client a list of 100 packages, get back
parsed Candidate tuples that match pip's Link.from_json output
candidate-for-candidate (name, version, url, hashes,
requires_python, yanked) on a fixture index.
Scope estimate: ~1 week of focused work.
Deliverable: a new optional code path in
pipenv/routines/lock.py that, when enabled by a Pipfile setting,
runs ParallelFetcher.populate(top_level_packages) before the
resolver subprocess fires. pip's resolver still does the actual
resolution; we just warm its cache more intelligently.
This is a smaller cousin of the "parallel prefetch" experiment from phase-5 that didn't pay off. The difference: we cache the parsed form, so even on dev boxes with warm pip cache we save the per-link parse cost.
Acceptance: on CI lock-warm bench, ≥10 % wall-time reduction versus phase-5 baseline. No regression on tests.
Scope estimate: ~3-5 days after phase 1 lands.
pure_python.Provider)Deliverable: the pure_python backend selectable via
[pipenv] resolver_backend = "pure-python". Bypasses
PackageFinder.find_all_candidates entirely. Reads only from
our ParsedManifestCache (warmed by ParallelFetcher).
Drives resolvelib with our Provider.
Acceptance (parity criterion):
pure-python are byte-identical (modulo
field ordering) to lockfiles produced by pip for the same
Pipfile, across:
tests/integration/test_lock.py).Scope estimate: 4-6 weeks of focused work.
After phase 3 ships and a release cycle of opt-in users report no
regressions, flip the default in a major version bump. The pip
backend remains available as [pipenv] resolver_backend = "pip"
for the next major thereafter, then is removed.
Not gated by this design doc. Phase 4 gets its own sign-off based on phase-3 production data.
_meta.hash, same per-package
shape. Initiative F's typed schema is the only contract.[pipenv] settings: resolver_backend is the only new field
in phase 3. Defaults to "pip".--backend is the optional
selector (already specified by Initiative F).tests/unit/test_pep691_client.py — protocol-level fetch tests
against synthetic responses, no real network.tests/unit/test_candidate.py — Candidate construction,
hashing, equality, wheel-tag derivation.tests/unit/test_manifest_cache.py — TTL, etag round-trip,
atomic write, schema versioning, concurrent read/write safety.tests/unit/test_parallel_fetcher.py — pool sizing, error
handling, FetchError propagation.tests/integration/test_pure_python_prefetch.py — exercise the
prefetch-only mode against a real PyPI fixture (or tests/pytest-pypi/
local index). Verify lockfile parity with pip-only mode.tests/integration/test_backend_parity.py — same Pipfile resolved
with both backends, lockfiles compared field-by-field. Run on
the 100-package bench fixture plus a curated 10-real-project list.tests/integration/test_pure_python_backend.py — full integration
smoke; lock + install + run for the bench fixture.The existing tests/integration/test_lock.py etc. continue to run
under the pip backend during phases 1-3. No changes required.
A separate CI matrix entry runs the same suite under the
pure-python backend during phase 3. Failures block phase-3 sign-off.
Q1: Cache file format — msgpack or JSON?
JSON is human-readable for debugging; msgpack is ~3× smaller and ~5× faster to deserialise. For a per-package manifest at 5-20 KB this matters at scale (100 packages × 10 KB = 1 MB read on warm relock; msgpack would be 200-300 KB).
Recommendation: JSON for phase 1 (debug-friendliness wins while the format is in flux); revisit in phase 3 once the format stabilises.
Q2: Should Candidate carry the wheel-METADATA contents
inline?
PEP 658 provides per-file core-metadata URLs / content-hashes.
We could fetch + cache the METADATA at the same time as the manifest
and avoid the second round of fetches during get_dependencies.
Recommendation: defer to phase 3 — the bandwidth and storage cost isn't trivial and we need real-data measurement before committing the cache format to it.
Q3: Default TTL for parsed manifests?
Tradeoffs: longer TTL = faster relocks but staler "did a new
release land" detection. pip's Cache-Control: max-age=0 is
the conservative extreme (every read revalidates); 600 s matches
PyPI's CDN; 1 hour is uv's default.
Recommendation: 600 s for phase 1. Reconsider after measuring real usage patterns.
Q4: Phase 3 — resolvelib Provider parity with pip's?
pip's provider.py has dozens of small behaviours (preference
ordering, prerelease handling, yanked-pinned override, etc.).
Phase 3 acceptance requires matching them exactly. How aggressively
do we audit pip's Provider for behaviours to replicate?
Recommendation: phase 3 produces a "parity matrix" doc listing every behaviour we replicate and every divergence (with justification). Sign-off requires no unjustified divergence.
Q5: Vendoring strategy for the new modules?
The Initiative G code lives in pipenv/resolver/ (not
pipenv/patched/... or pipenv/vendor/...). This is consistent
with the existing typed-resolver code from Initiative F.
Recommendation: confirmed — new code is first-party, not vendored.
Q6: Auth / netrc / keyring parity?
pip's PipSession integrates with keyring providers and netrc.
Our PEP691Client needs to do the same so private indexes keep
working. How much of pip's auth handling do we replicate?
Recommendation: phase 1 replicates netrc + basic-auth-in-URL +
PIP_CLIENT_CERT. Keyring is phase 3 (less common, deferrable).
Shipped at T17 on branch maintenance/code-cleanup-phase5-perf-2026-06.
PEP691Client.fetch returns the same candidate set as pip's
Link.from_json for a fixture of 20 packages spanning JSON
and HTML simple-API formats. (T10 parity fixture green.)pip._internal.* imports anywhere in the new code.
(Enforced ongoing by the no-pip-internal-in-resolver
pre-commit hook added in T17 — scoped to ^pipenv/resolver/,
pattern-anchored on actual import statements rather than raw
substring so docstring mentions don't false-positive.)packaging are imported by the phase-1 surface.)pipenv lock --clear / pipenv install --clear invalidate the
parsed-manifest cache root alongside pip's HTTP cache. (T17 wired
_clear_parsed_manifest_cache at the top of do_lock.)--cov-fail-under=90 on the resolver-module
suite) and pre-commit pip._internal gate ship together so the
acceptance criteria above are enforced continuously, not just at
merge time.Shipped at T22 on branch maintenance/code-cleanup-phase5-perf-2026-06.
The CI bench measurement gate (T21) is deferred — see the Phase-2
sign-off note in §11a for the scoping rationale.
tests/integration/test_prefetch_manifest.py verify lockfile
parity, best-effort failure handling, --clear short-circuiting,
and no URL leakage at any verbosity level.)Phase 2 ships without the CI bench measurement gate (T21). The maintainer scoped T21 out during execution review (no appetite for a multi-run statistically-guarded CI bench step at this time). The Phase-2 perf claim is therefore theoretical rather than measured against the current CI baseline:
--clear
short-circuiting, and no URL leakage at any verbosity level).false) specifically because
the phase-5 parallel-prefetch experiment was empirically
net-harmful on warm-cache dev machines. Users with cold-cache
workflows (typical CI without persisted pip cache) are the target
beneficiaries.If future maintainer interest produces real benchmark data, the
Phase-2 acceptance criteria should be revisited and the bench gate
re-spec'd. Until then, this is shipped as a low-risk opt-in feature
gated on user opt-in via [pipenv] prefetch_index_manifests = true.
[pipenv] resolver_backend = "pure-python" documented as supported.Explicitly not part of Initiative G:
resolvelib with another resolver (out of scope
entirely — resolvelib is upstream-stable, not pip-internal,
and not the bottleneck).