Container Resource Detection

RustFS automatically detects container resource limits (CPU and memory) from cgroup v1/v2. This ensures correct resource allocation and accurate metrics in containerized environments (Kubernetes, Docker, etc.).

Problem

When RustFS runs in a container, the underlying system libraries report the host's total CPU cores and memory, not the container's limits. This leads to:

Over-provisioned Tokio threads: Too many worker and blocking threads
Incorrect memory metrics: rustfs_memory_usage_percent shows host-based percentage
Memory budget errors: Object data cache sized to host RAM instead of container limit
OOMKills: Container exceeds its memory limit and gets killed

Solution

RustFS now detects cgroup limits directly from the filesystem:

CPU: /sys/fs/cgroup/cpu.max (v2) or /sys/fs/cgroup/cpu/cpu.cfs_quota_us (v1)
Memory: /sys/fs/cgroup/memory.max (v2) or /sys/fs/cgroup/memory/memory.limit_in_bytes (v1)

The effective resource limits are the minimum of host and cgroup values.

Detection Logic

CPU Detection

Read cgroup v2 /sys/fs/cgroup/cpu.max
- Format: "$QUOTA $PERIOD" or "max" (unlimited)
- Calculate: cores = ceil(quota / period)
Fallback to cgroup v1 /sys/fs/cgroup/cpu/cpu.cfs_quota_us
- Calculate: cores = ceil(quota / period)
Fallback to host CPU count from sysinfo

Memory Detection

Read cgroup v2 /sys/fs/cgroup/memory.max
- Value in bytes or "max" (unlimited)
Fallback to cgroup v1 /sys/fs/cgroup/memory/memory.limit_in_bytes
- Very large values (≥2^62) indicate unlimited
Fallback to host memory from sysinfo

Environment Variables

Disable Cgroup Detection

bash

RUSTFS_DISABLE_CGROUP_DETECTION=1

Disables cgroup detection entirely. Useful for testing or when cgroup filesystem is not accessible.

Override CPU Cores

bash

RUSTFS_OVERRIDE_CPU_CORES=4

Overrides detected CPU cores. Takes precedence over cgroup detection.

Override Memory Limit

bash

RUSTFS_OVERRIDE_MEMORY_BYTES=2147483648

Overrides detected memory limit in bytes. Takes precedence over cgroup detection.

Metrics

New Metrics

Metric	Description
`rustfs_memory_effective_total_bytes`	Effective memory total (host or cgroup)
`rustfs_cgroup_detected`	Whether cgroup limits were detected (1=yes, 0=no)
`rustfs_cgroup_cpu_cores_limit`	Detected CPU cores limit
`rustfs_cgroup_memory_limit_bytes`	Detected memory limit

Updated Metrics

Metric	Change
`rustfs_memory_total_bytes`	Now uses effective memory (cgroup-aware)
`rustfs_memory_usage_percent`	Now calculated against effective memory

Startup Logging

RustFS logs detected container resources at startup:

INFO container resources (detected from cgroup) cpu_cores=2 memory_bytes=1073741824 memory_mib=1024

INFO container resources (overridden by environment variables) cpu_cores=4 memory_bytes=2147483648 memory_mib=2048

Examples

Kubernetes with Resource Limits

yaml

resources:
  limits:
    cpu: "2"
    memory: "1Gi"
  requests:
    cpu: "500m"
    memory: "512Mi"

RustFS will detect:

CPU cores: 2
Memory: 1 GiB (1073741824 bytes)

Docker with CPU and Memory Limits

bash

docker run --cpus=2 --memory=1g rustfs/rustfs:latest

RustFS will detect:

CPU cores: 2
Memory: 1 GiB

Manual Override

bash

export RUSTFS_OVERRIDE_CPU_CORES=4
export RUSTFS_OVERRIDE_MEMORY_BYTES=2147483648

RustFS will use:

CPU cores: 4
Memory: 2 GiB

Troubleshooting

Cgroup Detection Not Working

Check if cgroup filesystem is mounted:
bash
```
ls -la /sys/fs/cgroup/
```
Check cgroup version:
bash
```
stat -fc %T /sys/fs/cgroup/
```
- cgroup2fs = cgroup v2
- tmpfs = cgroup v1

Check if limits are set:

bash

# cgroup v2
cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/memory.max

# cgroup v1
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

Metrics Show Host Values

If rustfs_memory_effective_total_bytes shows host memory instead of cgroup limit:

Verify cgroup detection is not disabled:
bash
```
echo $RUSTFS_DISABLE_CGROUP_DETECTION
```
Check startup logs for cgroup detection:
bash
```
grep "container resources" /logs/rustfs.log
```
Use environment variable override as workaround:
bash
```
export RUSTFS_OVERRIDE_MEMORY_BYTES=1073741824
```

Implementation Details

Files Modified

rustfs/src/cgroup_resources.rs - Core cgroup detection logic
rustfs/src/container_config.rs - Container configuration with overrides
rustfs/src/memory_observability.rs - Updated memory metrics
rustfs/src/server/runtime.rs - Updated Tokio runtime configuration
rustfs/src/startup_entrypoint.rs - Startup logging

Performance Impact

Startup: One-time detection adds ~1ms overhead
Runtime: Cached values, no repeated filesystem reads
Memory: Negligible (<1KB for cached values)

Thread Safety

All detection functions are thread-safe and use OnceLock for caching.