ContextQMD
Back to Rocksdb

RocksDB API Development Guide

claude_md/add_public_api.md

11.8.119.6 KB
Original Source

RocksDB API Development Guide

This document provides guidance for adding new public APIs to RocksDB, following the established patterns used by existing APIs like CompactRange.

API Layer Architecture

RocksDB exposes public APIs through multiple layers. Users can access RocksDB through any of the three public APIs: C++ headers, C headers, or Java bindings.

Here is an example for public header db.h:

┌─────────────────────────────────────────────────────────────────────────────┐
│                     Level 1: Public APIs (User Entry Points)                │
├───────────────────────┬─────────────────────────┬───────────────────────────┤
│   C++ Public API      │     C API Bindings      │       Java/JNI API        │
│ include/rocksdb/db.h  │   include/rocksdb/c.h   │ java/src/.../RocksDB.java │
│ include/rocksdb/*.h   │                         │ java/src/.../*.java       │
└───────────────────────┴────────────┬────────────┴───────────────────────────┘
                                     ↓
┌─────────────────────────────────────────────────────────────────────────────┐
│              Level 2: C++ Implementation (Internal Core)                    │
│              db/db_impl/db_impl*.cc, db/c.cc, java/rocksjni/*.cc            │
└─────────────────────────────────────────────────────────────────────────────┘

Step-by-Step Guide: Adding a New Public API

Step 1: Define the C++ Public Interface

File: include/rocksdb/db.h

Add the virtual method declaration in the DB class:

```cpp // Pure virtual - must be implemented by DBImpl virtual Status YourNewAPI(const YourAPIOptions& options, ColumnFamilyHandle* column_family, /* other params */) = 0;

// Convenience overload for default column family virtual Status YourNewAPI(const YourAPIOptions& options, /* other params /) { return YourNewAPI(options, DefaultColumnFamily(), / other params */); } ```

Key Patterns:

  • Use Status return type for error handling
  • Use OptSlice to avoid unnecessary levels of indirection and use of raw pointers.
  • Use ColumnFamilyHandle* for column family support
  • Provide convenience overloads for the default column family

Step 2: Define Options Struct (If Needed)

File: include/rocksdb/options.h

If your API has multiple configuration options, define an options struct:

```cpp struct YourAPIOptions { // Document each option with clear comments bool some_boolean_option = false;

// Default value explanation int some_int_option = -1;

// Pointer options require careful lifetime management std::atomic<bool>* canceled = nullptr;

// Enum options for multi-choice settings YourEnumType some_enum = YourEnumType::kDefault; }; ```

Key Patterns:

  • Use sensible default values specified inline (e.g., = false, = -1)
  • Do NOT redundantly document the default value in comments; instead, document the rationale (why this default), historical context, and how different values are interpreted
  • Group related options logically
  • Consider thread-safety for pointer options

Step 3: Implement in DBImpl

Header: db/db_impl/db_impl.h

```cpp using DB::YourNewAPI; Status YourNewAPI(const YourAPIOptions& options, ColumnFamilyHandle* column_family, /* other params */) override;

// Private internal implementation if needed Status YourNewAPIInternal(const YourAPIOptions& options, ColumnFamilyHandle* column_family, /* other params */); ```

Implementation: db/db_impl/db_impl_<category>.cc

Choose the appropriate implementation file based on functionality:

  • db_impl_compaction_flush.cc - Compaction and flush operations
  • db_impl_write.cc - Write operations
  • db_impl_open.cc - DB opening/closing
  • db_impl_files.cc - File operations
  • db_impl.cc - General operations

```cpp Status DBImpl::YourNewAPI(const YourAPIOptions& options, ColumnFamilyHandle* column_family, /* other params /) { // 1. Input validation if (/ invalid input */) { return Status::InvalidArgument("Error message"); }

// 2. Check for cancellation/abort conditions if (options.canceled && options.canceled->load(std::memory_order_acquire)) { return Status::Incomplete(Status::SubCode::kManualCompactionPaused); }

// 3. Get column family data auto cfh = static_cast<ColumnFamilyHandleImpl*>(column_family); auto cfd = cfh->cfd();

// 4. Core implementation logic // ...

return Status::OK(); } ```

Step 4: Handle Special DB Types

StackableDB (Wrapper DBs): File: include/rocksdb/utilities/stackable_db.h

```cpp using DB::YourNewAPI; Status YourNewAPI(const YourAPIOptions& options, ColumnFamilyHandle* column_family, /* other params /) override { return db_->YourNewAPI(options, column_family, / other params */); } ```

Secondary DB (Read-Only): File: db/db_impl/db_impl_secondary.h

```cpp using DBImpl::YourNewAPI; Status YourNewAPI(const YourAPIOptions& /options/, ColumnFamilyHandle* /column_family/, /* other params */) override { return Status::NotSupported("Not supported in secondary DB"); } ```

CompactedDB (Read-Only): File: db/db_impl/compacted_db_impl.h

```cpp using DBImpl::YourNewAPI; Status YourNewAPI(const YourAPIOptions& /options/, ColumnFamilyHandle* /column_family/, /* other params */) override { return Status::NotSupported("Not supported for read-only DB"); } ```

Step 5: Add C API Bindings

Important: include/rocksdb/c.h and db/c.cc are @generated and must NOT be edited by hand — your changes would be overwritten on the next regeneration. Choose the right path first (see "Before writing C API code by hand" below). For hand-written (manual) wrappers, edit the source templates tools/c_api_gen/c_base.h (declarations) and tools/c_api_gen/c_base.cc (implementations), then run python3 tools/c_api_gen/regen_all.py to produce c.h / c.cc. The snippets below show the declaration and implementation you would add to those templates.

Header (declaration → tools/c_api_gen/c_base.h):

c
// Basic version
extern ROCKSDB_LIBRARY_API void rocksdb_your_new_api(
    rocksdb_t* db,
    const char* start_key, size_t start_key_len,
    const char* limit_key, size_t limit_key_len);

// Column family version
extern ROCKSDB_LIBRARY_API void rocksdb_your_new_api_cf(
    rocksdb_t* db, rocksdb_column_family_handle_t* column_family,
    const char* start_key, size_t start_key_len,
    const char* limit_key, size_t limit_key_len);

// With options and error handling
extern ROCKSDB_LIBRARY_API void rocksdb_your_new_api_opt(
    rocksdb_t* db, rocksdb_your_api_options_t* opt,
    const char* start_key, size_t start_key_len,
    const char* limit_key, size_t limit_key_len,
    char** errptr);
\`\`\`

**Implementation (→ `tools/c_api_gen/c_base.cc`):**

\`\`\`cpp
void rocksdb_your_new_api(rocksdb_t* db, const char* start_key,
                          size_t start_key_len, const char* limit_key,
                          size_t limit_key_len) {
  Slice a, b;
  db->rep->YourNewAPI(
      YourAPIOptions(),  // Default options
      (start_key ? (a = Slice(start_key, start_key_len), &a) : nullptr),
      (limit_key ? (b = Slice(limit_key, limit_key_len), &b) : nullptr));
}

void rocksdb_your_new_api_cf(rocksdb_t* db,
                             rocksdb_column_family_handle_t* column_family,
                             const char* start_key, size_t start_key_len,
                             const char* limit_key, size_t limit_key_len) {
  Slice a, b;
  db->rep->YourNewAPI(
      YourAPIOptions(),
      column_family->rep,
      (start_key ? (a = Slice(start_key, start_key_len), &a) : nullptr),
      (limit_key ? (b = Slice(limit_key, limit_key_len), &b) : nullptr));
}
\`\`\`

**Before writing C API code by hand, choose the right path:**

- **Auto-managed simple struct fields:** If you added a new field to a managed
  public struct such as `ReadOptions`, selected option structs, or managed
  metadata structs, first check `tools/c_api_gen/auto_simple_bindings.py`.
  Simple scalar/enum/string fields should be regenerated with
  `python3 tools/c_api_gen/regen_all.py` rather than hand-editing
  `include/rocksdb/c.h` / `db/c.cc`.
  After regenerating, run
  `python3 tools/c_api_gen/verify_generated_up_to_date.py` to confirm the
  checked-in generated fragments are stable.
- **Spec-driven wrappers:** If the API is still mechanically generated but needs
  an explicit C shape, naming, or `Status`/`char** errptr` policy, add it to
  `tools/c_api_gen/spec.json`.
- **Fully manual wrappers:** If the API uses callbacks, ownership transfer,
  vectors/maps, open flows, or otherwise irregular marshalling, keep the C API
  hand-written.

**Temporary deferral for auto-managed families:**

If a new field lands in an auto-managed family but the C API shape is not ready
yet, add a checked-in entry to
`tools/c_api_gen/auto_simple_bindings_blocklist.json` with:

- `policy: "manual"` when the field is intentionally outside simple auto-gen
- `policy: "deferred"` when the field should be revisited later
- a concrete `reason`
- `tracking_issue` when available

If a field in an auto-managed family is not supported by the generator and is
not covered by the blocklist, regeneration should fail. That is intentional.

**If you have options, also add:**

\`\`\`cpp
// Options struct wrapper
struct rocksdb_your_api_options_t {
  YourAPIOptions rep;
};

rocksdb_your_api_options_t* rocksdb_your_api_options_create() {
  return new rocksdb_your_api_options_t;
}

void rocksdb_your_api_options_destroy(rocksdb_your_api_options_t* opt) {
  delete opt;
}

void rocksdb_your_api_options_set_some_option(
    rocksdb_your_api_options_t* opt, unsigned char value) {
  opt->rep.some_boolean_option = value;
}
\`\`\`

### Step 6: Add Java Bindings

**Java API:** `java/src/main/java/org/rocksdb/RocksDB.java`

\`\`\`java
// Basic version
public void yourNewAPI() throws RocksDBException {
  yourNewAPI(null);
}

// Column family version
public void yourNewAPI(ColumnFamilyHandle columnFamilyHandle)
    throws RocksDBException {
  yourNewAPI(nativeHandle_, null, -1, null, -1, 0,
      columnFamilyHandle == null ? 0 : columnFamilyHandle.nativeHandle_);
}

// Range version
public void yourNewAPI(final byte[] begin, final byte[] end)
    throws RocksDBException {
  yourNewAPI(null, begin, end);
}

// Full-featured version with options
public void yourNewAPI(ColumnFamilyHandle columnFamilyHandle,
                       final byte[] begin, final byte[] end,
                       final YourAPIOptions options)
    throws RocksDBException {
  yourNewAPI(nativeHandle_,
      begin, begin == null ? -1 : begin.length,
      end, end == null ? -1 : end.length,
      options.nativeHandle_,
      columnFamilyHandle == null ? 0 : columnFamilyHandle.nativeHandle_);
}

// Native method declaration
private static native void yourNewAPI(final long handle,
    /* @Nullable */ final byte[] begin, final int beginLen,
    /* @Nullable */ final byte[] end, final int endLen,
    final long optionsHandle,
    final long cfHandle);
\`\`\`

**Options Class:** `java/src/main/java/org/rocksdb/YourAPIOptions.java`

\`\`\`java
public class YourAPIOptions extends RocksObject {

  public YourAPIOptions() {
    super(newYourAPIOptions());
  }

  // Builder pattern setters
  public YourAPIOptions setSomeBooleanOption(boolean value) {
    setSomeBooleanOption(nativeHandle_, value);
    return this;
  }

  // Getters
  public boolean someBooleanOption() {
    return someBooleanOption(nativeHandle_);
  }

  // Native method declarations
  private static native long newYourAPIOptions();
  private static native void disposeInternalJni(long handle);
  private static native void setSomeBooleanOption(long handle, boolean value);
  private static native boolean someBooleanOption(long handle);

  @Override
  protected final void disposeInternal(final long handle) {
    disposeInternalJni(handle);
  }
}
\`\`\`

**JNI Implementation:** `java/rocksjni/rocksjni.cc`

\`\`\`cpp
void Java_org_rocksdb_RocksDB_yourNewAPI(
    JNIEnv* env, jclass,
    jlong jdb_handle, jbyteArray jbegin, jint jbegin_len,
    jbyteArray jend, jint jend_len,
    jlong joptions_handle, jlong jcf_handle) {

  // 1. Convert Java byte arrays to C++ strings
  jboolean has_exception = JNI_FALSE;
  std::string str_begin;
  if (jbegin_len > 0) {
    str_begin = ROCKSDB_NAMESPACE::JniUtil::byteString<std::string>(
        env, jbegin, jbegin_len,
        [](const char* str, const size_t len) { return std::string(str, len); },
        &has_exception);
    if (has_exception == JNI_TRUE) return;
  }

  std::string str_end;
  if (jend_len > 0) {
    str_end = ROCKSDB_NAMESPACE::JniUtil::byteString<std::string>(
        env, jend, jend_len,
        [](const char* str, const size_t len) { return std::string(str, len); },
        &has_exception);
    if (has_exception == JNI_TRUE) return;
  }

  // 2. Get or create options
  ROCKSDB_NAMESPACE::YourAPIOptions* options = nullptr;
  if (joptions_handle == 0) {
    options = new ROCKSDB_NAMESPACE::YourAPIOptions();
  } else {
    options = reinterpret_cast<ROCKSDB_NAMESPACE::YourAPIOptions*>(joptions_handle);
  }

  // 3. Unwrap handles
  auto* db = reinterpret_cast<ROCKSDB_NAMESPACE::DB*>(jdb_handle);
  ROCKSDB_NAMESPACE::ColumnFamilyHandle* cf_handle =
      jcf_handle == 0 ? db->DefaultColumnFamily()
                      : reinterpret_cast<ROCKSDB_NAMESPACE::ColumnFamilyHandle*>(jcf_handle);

  // 4. Create Slices
  std::unique_ptr<ROCKSDB_NAMESPACE::Slice> begin;
  std::unique_ptr<ROCKSDB_NAMESPACE::Slice> end;
  if (jbegin_len > 0) begin.reset(new ROCKSDB_NAMESPACE::Slice(str_begin));
  if (jend_len > 0) end.reset(new ROCKSDB_NAMESPACE::Slice(str_end));

  // 5. Call C++ API
  ROCKSDB_NAMESPACE::Status s = db->YourNewAPI(*options, cf_handle, begin.get(), end.get());

  // 6. Cleanup if we created options
  if (joptions_handle == 0) delete options;

  // 7. Throw Java exception on error
  ROCKSDB_NAMESPACE::RocksDBExceptionJni::ThrowNew(env, s);
}
\`\`\`

**Options JNI:** `java/rocksjni/your_api_options.cc`

\`\`\`cpp
jlong Java_org_rocksdb_YourAPIOptions_newYourAPIOptions(JNIEnv*, jclass) {
  auto* options = new ROCKSDB_NAMESPACE::YourAPIOptions();
  return GET_CPLUSPLUS_POINTER(options);
}

void Java_org_rocksdb_YourAPIOptions_disposeInternalJni(JNIEnv*, jclass, jlong jhandle) {
  auto* options = reinterpret_cast<ROCKSDB_NAMESPACE::YourAPIOptions*>(jhandle);
  delete options;
}

void Java_org_rocksdb_YourAPIOptions_setSomeBooleanOption(
    JNIEnv*, jclass, jlong jhandle, jboolean value) {
  auto* options = reinterpret_cast<ROCKSDB_NAMESPACE::YourAPIOptions*>(jhandle);
  options->some_boolean_option = static_cast<bool>(value);
}

jboolean Java_org_rocksdb_YourAPIOptions_someBooleanOption(JNIEnv*, jclass, jlong jhandle) {
  auto* options = reinterpret_cast<ROCKSDB_NAMESPACE::YourAPIOptions*>(jhandle);
  return static_cast<jboolean>(options->some_boolean_option);
}
\`\`\`

### Step 7: Update Build Files

**Java CMakeLists.txt:** `java/CMakeLists.txt`

Add your new Java source files:
\`\`\`cmake
src/main/java/org/rocksdb/YourAPIOptions.java
src/test/java/org/rocksdb/YourAPIOptionsTest.java
\`\`\`

### Step 8: Add Release Notes

**Directory:** `unreleased_history/`

RocksDB uses individual files in the `unreleased_history/` directory rather than directly editing `HISTORY.md`. This avoids merge conflicts and ensures changes are attributed to the correct release version.

Add a file to the appropriate subdirectory:
- `unreleased_history/new_features/` - For new functionality
- `unreleased_history/public_api_changes/` - For API changes
- `unreleased_history/behavior_changes/` - For behavior modifications
- `unreleased_history/bug_fixes/` - For bug fixes

**Example:** `unreleased_history/new_features/your_new_api.md`

\`\`\`markdown
Added `YourNewAPI()` to support [describe functionality]. See `YourAPIOptions` for configuration.
\`\`\`

**Example:** `unreleased_history/public_api_changes/your_api_options.md`

**Note:** Files should contain one line of markdown. The "* " prefix is automatically added if not included. These files are compiled into `HISTORY.md` during the release process.

### Step 9: Add Tests

**C++ Unit Tests:** `db/db_your_api_test.cc` or add to existing test file

\`\`\`cpp
TEST_F(DBTest, YourNewAPIBasic) {
  Options options = CurrentOptions();
  CreateAndReopenWithCF({"pikachu"}, options);

  // Setup test data
  ASSERT_OK(Put(1, "key1", "value1"));
  ASSERT_OK(Put(1, "key2", "value2"));

  // Test your API
  YourAPIOptions api_options;
  api_options.some_boolean_option = true;
  ASSERT_OK(db_->YourNewAPI(api_options, handles_[1], nullptr, nullptr));

  // Verify results
  // ...
}
\`\`\`

**Java Tests:** `java/src/test/java/org/rocksdb/YourAPIOptionsTest.java`

\`\`\`java
public class YourAPIOptionsTest {
  @Test
  public void yourAPIOptions() {
    try (final YourAPIOptions options = new YourAPIOptions()) {
      assertFalse(options.someBooleanOption());
      options.setSomeBooleanOption(true);
      assertTrue(options.someBooleanOption());
    }
  }
}
\`\`\`

## File Summary Checklist


| Component | File(s) | Required |
|-----------|---------|----------|
| C++ Public Interface | `include/rocksdb/db.h` | ✓ |
| Options Struct | `include/rocksdb/options.h` | If needed |
| DBImpl Declaration | `db/db_impl/db_impl.h` | ✓ |
| DBImpl Implementation | `db/db_impl/db_impl_*.cc` | ✓ |
| StackableDB | `include/rocksdb/utilities/stackable_db.h` | ✓ |
| Secondary DB | `db/db_impl/db_impl_secondary.h` | If not supported |
| Compacted DB | `db/db_impl/compacted_db_impl.h` | If not supported |
| C API Header (manual wrappers) | `tools/c_api_gen/c_base.h` → regen → `include/rocksdb/c.h` (`@generated`) | ✓ |
| C API Implementation (manual wrappers) | `tools/c_api_gen/c_base.cc` → regen → `db/c.cc` (`@generated`) | ✓ |
| Java API | `java/src/main/java/org/rocksdb/RocksDB.java` | ✓ |
| Java Options | `java/src/main/java/org/rocksdb/YourAPIOptions.java` | If needed |
| JNI Implementation | `java/rocksjni/rocksjni.cc` | ✓ |
| JNI Options | `java/rocksjni/your_api_options.cc` | If needed |
| Java CMake | `java/CMakeLists.txt` | If new files |
| Changelog | `unreleased_history/*.md` | ✓ |
| C++ Tests | `db/db_*_test.cc` | ✓ |
| Java Tests | `java/src/test/java/org/rocksdb/*Test.java` | ✓ |

## Best Practices

1. **Error Handling**: Always return `Status` objects in C++, throw exceptions in Java
2. **Default Values**: Provide sensible defaults for all options
3. **Documentation**: Add clear comments for all public methods and options
4. **Column Family Support**: Always support column family operations
5. **Thread Safety**: Document thread-safety guarantees
6. **Backward Compatibility**: Avoid breaking existing API contracts
7. **Testing**: Add comprehensive unit tests for all code paths