VBARE: A simple alternative to Protobuf & Cap'n Proto for schema evolution

At Rivet, we're building an open-source alternative to Cloudflare Durable Objects — a tool for running stateful compute workloads. VBARE is a small but crucial component in meeting the demanding performance requirements of Rivet Actors.

Growing pains with Protobuf

We decided to adopt Protocol Buffers for any internal communications & data stored at rest. We've always believed it's a flawed technology, but it was the only mainstream and portable tool that provided binary serialization and schema evolution — or what we thought would be sufficient schema evolution.

The problem

Issues started arriving after our v1 launched and we started adding new features that required significant changes to the datastructures in our Protobufs. Eventually, all of our Protobuf files started generating bitrot of sorts. They became tedious to read, understand defaults, and understand migration paths. We frequently ended up with a handful of common issues:

• All new properties need to be optional: We had to litter our application logic with defaults if a property was none
• Properties cannot be moved or restructured: Say we want to change a bool to an enum or a struct to a union — good luck
• Datastructures need to be reshaped: The most simple example is switching from a list to a map or switching from a bool to a union

More importantly, the more we changed in Protobuf, more and more cruft had to be added to our application logic that made adding features a tedious process.

The crude solution

To solve this, we started following this pattern to allow us to clean up our application logic:

1. Copy and paste the entire Protobuf spec to a v2
2. Update the v2 to the ideal schema
3. Write a migration function between v1 and v2
4. Feed the final v2 version into the application logic

This became such a standard practice that most of our API saw a turnover to a new version once every 3-6 months.

The breaking point

While this process helped simplify our application logic significantly, it required a lot of manual effort to implement the Protobuf migrations. So we started evaluating other options that might provide better schema evolution.

Evaluating existing options

Goals

• Simple — Minimal features, can be easily forked and modified if needed
• Portable — Cross-language support with a well-defined standard
• Fast — Self-contained binary encoding (think: binary data without keys), ideally with zero-copy reads
• Clean SDK — Ser/de code has a huge impact on the legibility & friction of working with application logic

Non-Goals

• Data compactness — That's what gzip is for
• RPC layer — This is trivial to implement yourself based on your specific requirements

Exploring existing options

We evaluated numerous serialization protocols before deciding to build VBARE. Here's a brief overview of why each fell short:

• Protocol Buffers: Makes migrations your problem at runtime by making everything optional, with dangerous default values that lead to subtle bugs
• Flatbuffers: Similar issues to Protocol Buffers with field indexing requirements, awful codegen
• Cap'n Proto: Provides powerful schema migrations at the cost of complexity, focused on C++, poor generated clients for non-C++ languages
• CBOR/MessagePack/BSON: Self-describing formats, unsuitable for our performance needs
• Bebop/Borsh: Provides cross-language, self-contained binary encoding. We almost chose one of these instead of BARE, but BARE is simpler.
• Rust-specific options (postcard, bincode, etc.): Not cross-language, only supports Rust

(For a detailed comparison, see our full evaluation document.)

In the end, none of these solutions provided a compelling schema evolution mechanism.

VBARE overview

Enter, VBARE: a tiny extension of BARE to provide a version header and handle version migrations.

Instead of using an off-the-shelf solution, we opted to build a simple evolution system similar to the pattern we were already using: manually writing code to migrate between schemas. We would pair this technique with the BARE encoding to create VBARE.

Schema evolution in VBARE

VBARE's design philosophy centers on four core beliefs:

1. Make smaller, incremental schema changes instead of massive v1 to v2 overhauls. Build tools that make this easy and your schema will be easier to work with.

2. Manual evolution simplifies application logic by isolating schema evolution logic to a separate module before passing it to the application logic.

3. Real-world schema evolutions require more than simple property renaming: it involves complex reshaping and fetching data from remote sources, which automatic migration systems can't handle.

4. Manual evolution is less error prone by forcing developers to explicitly handle edge cases of migrations and breaking changes, trading verbosity for safety.

Versions

VBARE operates by declaring a schema file for every version of your protocol, then writing explicit conversion functions between adjacent versions.

Each message has an associated version number (unsigned 16-bit integer) that increases monotonically from 1. Versions are specified in the filename: my-schema/v1.bare, my-schema/v2.bare, etc.

Converters

Servers define conversion code that transforms between versions bidirectionally:

• Upgrade converters for deserialization (old → new)
• Downgrade converters for serialization (new → old)

There are no evolution semantics in the schema language itself. To create a new version, you simply copy the previous schema and make your changes.

Version Negotiation

Every message has an associated version, which can be either:

• Embedded in the message itself (first 2 bytes)
• Pre-negotiated via HTTP paths (POST /v3/users), query parameters, or handshakes

Servers vs Clients

• Servers must include converters between all versions to handle any client version.

• Clients only need to include a single version since the server handles all version conversion.

Use cases

Common use cases include:

• Network protocols — Allow the server to cleanly evolve the protocol version without breaking old clients
• Data at rest — Upgrade your file format without breaking old files

For examples, VBARE powers almost all of Rivet:

• Rivet Engine

  • Data at rest
  • Internal network protocols (Epoxy, UPS)
  • Public network protocols

• RivetKit

  • Client protocol
  • Persisted state
  • File system driver

Example Code

Here's a simple example demonstrating how VBARE handles a complex schema migration that is not possible with any existing tools:

type User struct {
  id: string
  name: string
}

type User struct {
  id: string
  // Split `name` in to 2 properties
  firstName: string
  lastName: string
}

import * as V1 from "./v1";
import * as V2 from "./v2";
import { createVersionedDataHandler } from "vbare";

// Converter from v1 to v2
function upgradeUserV1ToV2(v1: V1.User): V2.User {
  const [firstName, ...rest] = v1.name.split(' ');
  return {
    ...v1,
    firstName,
    lastName: rest.join(' ') || ''
  };
}

// Converter from v2 to v1
function downgradeUserV2ToV1(v2: V2.User): V1.User {
  return {
    ...v2,
    name: `${v2.firstName} ${v2.lastName}`.trim()
  };
}

// Create versioned data handler
export const USER_VERSIONED = createVersionedDataHandler<V2.User>({
  deserializeVersion: (bytes: Uint8Array, version: number): any => {
    switch (version) {
      case 1:
        return V1.decodeUser(bytes);
      case 2:
        return V2.decodeUser(bytes);
      default:
        throw new Error(`invalid version: ${version}`);
    }
  },
  serializeVersion: (data: any, version: number): Uint8Array => {
    switch (version) {
      case 1:
        return V1.encodeUser(data);
      case 2:
        return V2.encodeUser(data);
      default:
        throw new Error(`invalid version: ${version}`);
    }
  },
  deserializeConverters: () => [upgradeUserV1ToV2],
  serializeConverters: () => [downgradeUserV2ToV1],
});

Implementations

• TypeScript — Example Code
• Rust — Example Code

For a full list of BARE implementations, visit baremessages.org.

FAQ

Why is copying the entire schema for every version better than using decorators for gradual migrations?

• Decorators are limited and become very complicated over time
• It's unclear at what version of the protocol a decorator takes effect — explicit versions help clarify this
• Generated SDKs become more and more bloated with every change
• You need a validation build step for your validators
• Manual migrations provide more flexibility for complex transformations

Why not include RPC?

RPC interfaces are trivial to implement yourself. Libraries that provide RPC interfaces tend to add extra bloat and cognitive load through things like abstracting transports, compatibility with the language's async runtime, and complex codegen to implement handlers.

Usually, you just want a ToServer and ToClient union that looks like this:

• ToClient example
• ToServer example

Don't migration steps get repetitive?

Migration steps are fairly minimal to write. The most verbose migration steps will be for deeply nested structures that changed, but even that is relatively straightforward.

What are the downsides?

• More verbose migration code — but this is usually because VBARE forces you to handle all edge cases you wouldn't otherwise bother with
• The older the version, the more migration steps that need to run to bring it to the latest version — though migration steps are usually negligible in cost
• Migration steps are not portable across languages, but only the server needs the migration steps, so this is usually only implemented once

Getting started & source code

VBARE is open source and available on GitHub.

See the guides for getting started with TypeScript and Rust.