vscode-rollout — A/B loader for the SDK extension rollout

The VS Code Marketplace has no staged rollouts: publishing a version updates every user. This package lets us ship the SDK-based extension (main's apps/vscode, "next") to a percentage of users while everyone else keeps running the legacy extension (the legacy-extension branch), inside a single published VSIX.

How it works

The published VSIX contains a ~40 KB loader as its entrypoint and two complete, independently built extension bundles:

extension.js          ← loader (this package)
package.json          ← UNION of both bundles' manifests (generated, see below)
assets/, walkthrough/ ← manifest-referenced resources (VSIX-root-relative)
next/                 ← SDK extension  (dist/, webview-ui/build/, assets/)
legacy/               ← legacy extension (dist/, webview-ui/build/, assets/, codicons)

Per window, the loader:

1. Reads the cached cohort assignment from its own globalState keys — synchronously, never from the network.
2. Sets the cline.sdkBundle context key (gates cohort-specific menu items / palette entries in the union manifest).
3. require()s exactly one bundle and calls its activate() with a Proxy-wrapped ExtensionContext whose extensionUri / extensionPath / asAbsolutePath point into that bundle's subdirectory — so each bundle resolves its own webview build and assets without knowing it was relocated. Storage properties pass through untouched: both bundles share the same ~/.cline/data + VS Code storage they used as standalone extensions.
4. After the selected bundle activates, evaluates the PostHog flags in the background and caches the assignment for the next window. Flag changes never flip a live window. A crash fallback skips this refresh so it cannot overwrite the legacy pin.

If the next bundle throws during activation, the loader disposes whatever it half-registered, pins this VSIX version back to legacy (cline.rollout.nextActivationFailedVersion), reports a fallback telemetry event, and activates legacy — a crashed rollout self-heals without a marketplace re-publish. A new version gets to try next again.

Cohort rules

• Two-way, one knob. ext-sdk-bundle-rollout (percentage flag) is the entire remote control surface: each background refresh caches exactly what the flag says for the machine's next window. Dialing the percentage up promotes; dialing it down demotes on the next reload — the emergency lever is simply "set the rollout to 0%". Known demotion costs (accepted): tasks created on the SDK bundle are stored as SDK sessions the legacy bundle doesn't list (they reappear on re-promotion — nothing is deleted), and credentials rotated on next may require a re-login on legacy.
• The flag must stay a boolean flag. The loader only promotes on a literal true from /decide — a multivariate variant, number, or anything else fails safe to legacy (see parseRolloutAssignment + tests). Don't convert it to multivariate.
• The flag is evaluated against the same PostHog distinct id the extension's telemetry uses (machine id, mirroring src/services/logging/distinctId.ts), so cohort membership is correlatable with telemetry. Flag evaluation is always on (matching FeatureFlagsService); the loader's own extension.rollout.loader_decision event respects the user's telemetry opt-out and VS Code's global telemetry switch.
• Manual overrides, in either direction. The cline.rollout.bundleOverride user setting ("auto" | "next" | "legacy", editable straight from settings.json) forces a bundle for anyone — users in a pinch, or us debugging — beating the remote assignment both ways. Applies on window reload. CLINE_BUNDLE_OVERRIDE=next|legacy (env var) does the same for local dev and e2e and beats even the setting. Both are reported as override on the loader event so overridden machines don't pollute cohort comparisons.
• Crash pinning is local, not remote. If the next bundle throws during activation, the loader falls back to legacy in the same window and pins that VSIX version on this machine (cline.rollout.nextActivationFailedVersion); a new release gets to try next again. This safety net is independent of the flag.

The union manifest

package.json contributions are static — VS Code reads them before any code runs — so the shipped manifest must serve both cohorts. scripts/gen-manifest.mjs regenerates it at stitch time from both branches' real manifests:

• Contributions declared by both bundles pass through untouched.
• Menu entries / keybindings declared by only one get when AND-ed with cline.sdkBundle / !cline.sdkBundle, so a cohort never sees a button its bundle didn't register (and shared buttons that moved position don't render twice).
• Commands exclusive to one bundle are hidden from the other cohort's command palette.
• views / viewsContainers / configuration / walkthroughs must be identical in both manifests — they can't be safely gated at runtime, so divergence fails the build. Keep these static contributions in sync between the branches. engines may diverge: the union takes the newer requirement (which necessarily satisfies the older one).

Because the manifest is regenerated from both branches on every build, contribution drift between the branches can't ship silently — it either merges cleanly or the stitch fails.

Building locally

# 1. build both bundles (their own toolchains)
cd apps/vscode && bun run package                      # next
cd <legacy worktree>/apps/vscode && npm run package    # legacy (npm ci first)

# 2. build the loader + stitch + package
cd apps/vscode-rollout
bun run build                        # dev build; CI uses build:production with the PostHog key
node scripts/stitch.mjs \
  --next ../vscode --legacy <legacy worktree>/apps/vscode \
  --loader dist/extension.js --version 4.1.0 --out /tmp/cline-ab-staging
node scripts/smoke-loader.mjs /tmp/cline-ab-staging   # loader behavior smoke
cd /tmp/cline-ab-staging && vsce package --no-dependencies --allow-package-secrets sendgrid

The narrowly scoped sendgrid scanner exemption mirrors the existing next and legacy packaging workflows. This workflow supplies only the existing PostHog project-key inputs; it does not declare a SendGrid credential. Identify the exact matching string in production staging output before changing or broadening the exemption.

Local builds have no TELEMETRY_SERVICE_API_KEY, so the loader skips PostHog entirely and everyone stays on legacy unless CLINE_BUNDLE_OVERRIDE is set.

CI: the ext-vscode-ab-package workflow (manual dispatch) builds both refs, stitches, smoke-tests, uploads the .vsix artifact, and optionally publishes.

Nightly channel

The daily ext-vscode-publish-nightly workflow (cron + manual dispatch) publishes this same combined package as saoudrizwan.cline-nightly. Before each bundle builds, scripts/nightlify.mjs rewrites its manifest to the nightly identity — the same mutation the standalone nightly always applied (apps/vscode/scripts/publish-nightly.mjs on both branches is the source of truth), so nightly can be installed alongside stable:

The loader derives the namespace from its own packageJSON.name at runtime (idPrefix in src/cohort.ts), and gen-manifest derives it from the next manifest's name — no build flags involved. Nightly builds also show a status-bar indicator (Cline: Next / Cline: Legacy); stable builds never do.

Dispatching the nightly workflow from main with dry-run builds and uploads the installable .vsix without publishing or tagging. The publish job is intentionally restricted to main by both the workflow and the PublishNightly environment's deployment-branch policy.

Telemetry events

• extension.rollout.bundle_activated (authoritative, captured by the activated bundle's own telemetry via its reportRolloutActivation export; requires the bundle to be built with CLINE_ROLLOUT_VARIANT): attempted vs actual bundle, fallback flag, error details on fallback. Every other event from a rollout build carries extension_variant as a common property.
• extension.rollout.loader_decision (loader-owned, direct capture): the loader-side metadata the bundle event can't know — override source, launch cadence, loader version, extension_name (nightly vs stable) — and the only signal when BOTH bundles fail (double_failure: true).

Rollout runbook

Until the stable combined VSIX ships, the flag governs nightly installs only — dialing it is safe for production users and is the lever for moving nightly dogfooders onto next.

1. Create ext-sdk-bundle-rollout in PostHog before the first publish: a plain boolean release flag with a percentage rollout, starting at 0%. (There is deliberately no kill-switch flag — the assignment is two-way, so 0% is the kill switch.)
2. Publish the combined VSIX (version above every previously published one). With the rollout at 0% this release is behaviorally identical to legacy for everyone — it only validates the loader plumbing in the wild. Watch extension.rollout.bundle_activated and extension.rollout.loader_decision.
3. Dial ext-sdk-bundle-rollout up: 1% → 5% → 25% → 100%. Assignments apply on each machine's next window reload after its flag refresh, so propagation speed is bounded by how often people reload windows — watch the ms_since_last_activation distribution on loader events to see real uptake lag before deciding the next step, and compare cohorts by the bundle property.
4. Emergencies: dial the percentage down (0% pulls everyone back to legacy on their next reload). Demoted machines keep settings and creds; tasks created on the SDK bundle reappear when re-promoted. Ship the fix as a higher version, then dial back up. Machines whose next bundle crashed are additionally version-pinned to legacy locally, independent of the flag.
5. When next reaches 100% and soaks, retire the loader: publish a plain SDK extension build and delete this package.

Version numbering

The combined VSIX owns the marketplace version line and must always exceed the last version published from either branch (legacy stable was 4.0.x → start at 4.1.0). The bundles' own package.json versions ride along inside their subdirectories for provenance; the loader reports the combined version as loader_version.