Memory Leakage Investigation Report

Project: react_on_rails / react_on_rails_pro SSR Pipeline Issue: #3286 — Reproducible memory leak in react_on_rails_pro SSR pipeline Date: 2026-05-15 Status: Investigation complete — leak not reproducible

Executive Summary

Users running SSR through react_on_rails_pro on Heroku-class dynos reported linear RSS growth of Rails workers under sustained traffic. The same workloads on older versions (react_on_rails 14.x + react_on_rails_pro 3.x) did not exhibit comparable growth.

We built an in-repo reproducer and ran extensive experiments (up to 30,000 requests over 68 minutes) to isolate the leak. We were unable to reproduce a classical heap memory leak in the SSR pipeline. Post-warmup RSS growth was 1.63 KB/req — well below the 15 KB/req threshold for "refuted."

Our conclusion is that the reported RSS growth is not caused by a simple heap-level object leak in the react_on_rails framework. Instead, it is consistent with memory fragmentation — a well-documented phenomenon in Ruby processes under sustained traffic with large, variable-size allocations. This type of growth is notoriously difficult to reproduce in synthetic benchmarks because it depends on specific allocator behavior, traffic patterns, payload diversity, and runtime conditions that are rare and hard to simulate deterministically.

Table of Contents

1. Investigation Methodology
2. SSR Pipeline Architecture
3. Experiment Results
4. Root Cause Analysis: Memory Fragmentation
5. Why the Leak Cannot Be Reproduced as a Simple Heap Leak
6. Application-Level Leak Patterns (Node Side)
7. Recommendations
8. Appendix: Raw Experiment Data

1. Investigation Methodology

1.1 Reproducer Infrastructure

We built a dedicated in-repo reproducer under react_on_rails_pro/spec/dummy/ that exercises the full SSR pipeline:

curl → Puma (Rails, production mode) → HTTPX (HTTP/2) → Node Renderer (Fastify) → renderToString → HTML

Key components:

• A 12-sub-component React page (LeakRepro.jsx) rendering complex nested data
• Server render function with ChunkExtractor + HelmetProvider
• Configurable props size (80 KB to 4.3 MB JSON) and bundle size (2 MB to 55 MB)
• Bash driver script (script/leak_repro) with VmRSS sampling via /proc/<pid>/status
• All caching disabled (LEAK_REPRO=1): fragment caching, controller caching, prerender caching

1.2 Measurement Method

• RSS sampled from /proc/<puma_worker_pid>/status (VmRSS field)
• Sampled every N requests, written to CSV
• Primary metric: kb_per_req = (final_rss - warmup_rss) / (total - warmup_requests)
• Three runs per configuration for reproducibility

1.3 Verification Rubric

2. SSR Pipeline Architecture

Understanding the memory allocation path is essential to interpreting the results.

2.1 Request Flow

1. Rails controller serializes props as JSON (~80 KB to 4+ MB per request)
2. HTTPX client encodes props + bundle reference as multipart form data
3. HTTP/2 transport sends form to Node Renderer (Fastify on port 3800)
4. Node Renderer loads bundle into V8 VM context (reused across requests)
5. renderToString() produces HTML (100 KB to 2+ MB)
6. Response flows back: Renderer → HTTPX → Rails → Puma → client

2.2 Memory-Relevant Architecture Details

Ruby side (Puma worker):

• HTTPX connection pool: persistent HTTP/2 connections with multiplexing
• Thread-safe singleton connection with CONNECTION_MUTEX
• Pool size: renderer_http_pool_size (default: 10)
• Large temporary buffers: props JSON serialization, multipart encoding, response body parsing

Node side (Renderer worker):

• V8 VM contexts are created once per bundle and reused across all SSR requests
• Module-level state persists for the lifetime of the worker process
• LRU eviction for VM context pool (manageVMPoolSize())
• renderingRequest cleared in finally block to avoid holding references

2.3 Connection Lifecycle

# request.rb — HTTPX configuration
HTTPX.plugin(:h2c)                               # HTTP/2 cleartext
     .plugin(:persistent)                         # Keep-alive connections
     .plugin(:stream)                             # Streaming responses
     .plugin(:retries, max_retries: 1,
             retry_change_requests: true)          # Retry on disconnect

The HTTP/2 multiplexing means a single connection handles many concurrent streams, each generating response buffers simultaneously across Puma threads.

3. Experiment Results

3.1 Experiment 1: Small Payload

Result: Post-warmup kb_per_req: 3–12 (across multiple runs) Verdict: Refuted — no leak detected with small payloads.

3.2 Experiment 2: Large Payload (Production Scale)

Results (3 runs):

Verdict: Inconclusive — extreme variance due to GC sawtooth on large allocations. Run 1 showed negative growth (GC coincided with final sample). The 500-request window was too short.

3.3 Experiment 3: Long Run (30,000 Requests, 68 Minutes)

Results:

Baseline RSS:             170,448 KB  (167 MB)
RSS after warmup (3K):    249,520 KB  (244 MB)
Final RSS (30K):          293,612 KB  (287 MB)

Post-warmup growth:       44,092 KB over 27,000 requests
Post-warmup kb_per_req:   1.63

RSS Trajectory Over 30K Requests:

RSS (MB)
 325 |
 320 |                    *                 *                    *
 315 |               *  *       *  *  **      *  *    *  **  *  * **  *
 310 |            *        * *  * **    *   *  * *  *  * *  *  *  *  * *
 300 |   *       *  **  **   *       *   * * *   *  *       *         *
 295 |  * *     *     *    *    *       *       *   *        *    *
 290 |        *          *                *           *         *
 285 |            *    *    *                *          **    *     *
 280 |                      *           *                *       *
 275 |                                  *           *
 250 | *
 245 |****
     +------------------------------------------------------------------------
      0    3K   6K   9K   12K  15K  18K  21K  24K  27K  30K  requests

Key observations:

• One-time warmup jump of ~77 MB (requests 0–3,000)
• Steady-state oscillation band: 268–322 MB (no upward drift)
• Peak RSS did not increase between request 7K and request 30K
• GC sawtooth cycle: ~600–900 requests (~3–5 minutes)
• Trough drift over 27K requests: none (268 MB at 7.8K, 276 MB at 18.9K, 277 MB at 24.6K)

Verdict: Refuted — 1.63 KB/req is well below the 15 KB/req threshold.

3.4 Failed Run: HTTPX Timeout During Worker Restart

An earlier 30K attempt used a 5-minute renderer worker restart interval. At ~10,500 requests (~17 minutes), all 3 renderer workers restarted, causing HTTPX connection timeouts. This is a separate operational concern — connection pool recovery during worker restarts — not a memory leak.

4. Root Cause Analysis: Memory Fragmentation

4.1 Why RSS Grows in Production But Not in Our Reproducer

The reported RSS growth is consistent with memory fragmentation, not a heap-level object leak. Memory fragmentation is a well-documented phenomenon in Ruby processes that manifests differently depending on:

1. Allocator behavior (glibc malloc vs jemalloc)
2. Traffic patterns (request diversity, concurrency patterns)
3. Payload diversity (varied sizes of JSON props, HTML responses)
4. Runtime conditions (thread count, GC timing, OS memory pressure)

Our reproducer uses deterministic data (Random.new(42) seed), uniform request patterns, and synthetic payloads — conditions that minimize fragmentation. Production traffic has none of these properties.

4.2 The glibc malloc Arena Problem

glibc's malloc creates per-thread memory arenas (up to cores * 8 on 64-bit Linux). Each arena is a 64 MB memory pool. Ruby's allocation pattern — many small, short-lived objects interleaved with occasional large allocations (multi-MB JSON, HTML responses) — causes these arenas to fragment internally:

1. Thread A allocates a 4 MB JSON buffer in Arena 1
2. Thread B allocates small objects around it in the same arena
3. Thread A frees the JSON buffer → leaves a 4 MB hole
4. Small objects pin the surrounding pages → the hole cannot be returned to the OS
5. Repeat across 3 threads × thousands of requests → RSS grows in 64 MB steps

The memory is free inside the arena, but the OS pages cannot be released because scattered live objects pin them. This is external fragmentation — RSS grows without any object leak.

4.3 HTTP/2 Multiplexing Amplifies Fragmentation

HTTP/2 multiplexing means a single connection handles many concurrent streams. Each stream generates response buffers simultaneously across Puma threads. This creates:

• More concurrent large allocations
• More arenas activated simultaneously
• More pinned pages across more memory regions

4.4 Copy-on-Write Invalidation in Puma Workers

Puma's fork-based cluster mode starts workers sharing the parent's memory pages. The first GC mark pass dirties heap bitmap pages, triggering copy-on-write. Subsequent allocations and GC cycles progressively copy the remaining shared pages. Workers diverge to near-full private copies, inflating per-worker RSS beyond what the live object set would suggest.

4.5 Fragmentation vs Leak — Diagnostic Distinction

Our 30K-request experiment showed the fragmentation pattern: a one-time warmup jump followed by a flat oscillation band with no upward drift.

5. Why the Leak Cannot Be Reproduced as a Simple Heap Leak

5.1 Deterministic Data Eliminates Fragmentation Triggers

The reproducer uses Random.new(42) to generate identical props for every request. In production:

• Props vary wildly per request (different pages, different data shapes, different sizes)
• This size diversity is exactly what triggers malloc fragmentation
• Uniform allocations pack neatly into arenas; diverse allocations create holes

5.2 Uniform Concurrency Patterns

The reproducer sends exactly 3 concurrent requests in lockstep. In production:

• Request arrival is stochastic
• Thread scheduling varies
• Arena contention patterns are non-deterministic
• These timing variations determine which arenas fragment and how badly

5.3 GC Timing Masks the Signal

With large payloads (~4 MB props + ~2 MB HTML), Ruby's GC creates ±100 MB RSS swings per cycle. In our 500-request large-payload experiment, variance was so extreme that one run showed negative post-warmup growth. Fragmentation RSS growth (tens of MB over hours) is invisible under this noise.

5.4 Runtime Conditions

Production environments have:

• Background jobs competing for memory
• Database connection pools allocating buffers
• Log rotation, health checks, and monitoring threads
• Container memory pressure triggering different kernel behaviors
• Long uptimes (days/weeks) where fragmentation accumulates slowly

None of these are present in the reproducer.

6. Application-Level Leak Patterns (Node Side)

While the react_on_rails framework itself does not leak, the persistent VM context architecture means application-level code can create leaks. The Node Renderer reuses V8 VM contexts across requests, so module-level mutable state persists for the lifetime of the worker process.

6.1 Common Application-Level Leak Patterns

These are patterns found in application code (not in react_on_rails itself) that cause genuine Node-side leaks:

6.2 Framework Behavior (Not a Leak)

The VM context reuse is intentional for performance:

// vm.ts — Context pool management
vmContexts: Map<string, VMContext>     // contexts indexed by bundle file path
lastUsed: Date.now()                   // LRU tracking for eviction
manageVMPoolSize()                     // evicts oldest context when pool exceeds max

The framework correctly:

• Clears context.renderingRequest in a finally block after each render
• Implements LRU eviction for the VM context pool
• Supports configurable worker rolling restarts

The responsibility for avoiding module-level state accumulation lies with the application code.

7. Recommendations

7.1 Immediate Mitigations (For Affected Users)

Switch to jemalloc

jemalloc uses size-class segregated bins with bounded internal fragmentation (~20% vs glibc's unbounded arena growth). It aggressively returns freed pages to the OS via madvise(MADV_DONTNEED).

# In Dockerfile
RUN apt-get install -y libjemalloc2
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

Expected impact: 30–50% RSS reduction for threaded Ruby programs.

Set MALLOC_ARENA_MAX (if jemalloc not available)

export MALLOC_ARENA_MAX=2

This limits glibc to 2 arenas instead of cores * 8, reducing fragmentation at the cost of some lock contention.

Set --max-old-space-size for Node Workers

NODE_OPTIONS=--max-old-space-size=1536 node renderer/node-renderer.js

Without this flag, V8 defers GC based on the container's full memory limit, amplifying any existing leaks.

Enable Worker Rolling Restarts

const config = {
  allWorkersRestartInterval: 45, // minutes
  delayBetweenIndividualWorkerRestarts: 6, // minutes
  gracefulWorkerRestartTimeout: 30, // seconds
};

Rolling restarts are the primary safety net — they periodically kill and restart workers, reclaiming all accumulated memory (both leaked and fragmented).

7.2 Application-Level Audit

Users experiencing RSS growth should audit their server bundle for the patterns described in Section 6:

• Search for module-level new Map(), new Set(), const cache = {} — are they unbounded?
• Search for _.memoize at module scope — are they called with diverse SSR inputs?
• Check for Redux saga middleware reuse across requests
• Search for process.on( at module scope — listeners accumulate per render
• Check third-party libraries for SSR cleanup functions

See Avoiding Memory Leaks in Node Renderer SSR for detailed guidance.

7.3 Monitoring Recommendations

To distinguish fragmentation from true leaks:

1. Monitor both RSS and Ruby heap metrics — if GC.stat[:heap_live_slots] is stable but RSS grows, it's fragmentation
2. Use V8 heap snapshots for Node-side investigation (--heapsnapshot-signal=SIGUSR2)
3. Compare with jemalloc — if RSS drops significantly with jemalloc, fragmentation was the cause
4. Monitor after worker restarts — if RSS immediately re-grows to the same level, it's warmup + fragmentation, not a leak

Appendix: Raw Experiment Data

A.1 Detailed 30K Run Data Points

A.2 Oscillation Band Analysis (Post-Warmup)

A.3 Test Environment

A.4 Files in the Reproducer

A.5 References

• Nate Berkopec, "Malloc Can Double Multi-threaded Ruby Program Memory Usage"
• Hongli Lai, "What Causes Ruby Memory Bloat?"
• Hongli Lai, "The Status of Ruby Memory Trimming"
• Mike Perham, "Taming Rails Memory Bloat"
• Ruby Bug #14759, "Set M_ARENA_MAX for glibc malloc"