docs/devcontainer/troubleshooting.md
This guide covers common issues and their solutions when working with MongoDB dev containers.
Symptoms
Docker version <version> or later is required
Solution
Restart VSCode. If you install Rancher Desktop while you already have VSCode open, it doesn't properly detect the Docker socket and prompts you to install Docker Desktop by mistake.
Symptoms:
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /Users/username/.ssh
Or on macOS/Linux systems using certain Docker providers:
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /socket_mnt/...
Root Cause:
The devcontainer configuration mounts your ~/.ssh directory to enable Git operations over SSH. If this directory doesn't exist on your host machine, the container fails to start. This directory is required even if you plan to use HTTPS instead of SSH for cloning.
Solutions:
Create the .ssh directory on your host machine:
# On your HOST machine (not in container)
mkdir -p ~/.ssh
Rebuild the container:
Note on SSH Agent Forwarding:
SSH agent forwarding behavior varies by Docker provider on macOS:
To use SSH agent forwarding, ensure your SSH keys are added to your host's SSH agent before starting the container:
ssh-add ~/.ssh/id_ed25519 # or your key name
ssh-add -l # verify keys are loaded
Symptoms:
Error: failed to solve: write /var/lib/docker/...: no space left on device
Solutions:
Clean up Docker resources:
# Remove unused containers, images, and volumes
docker system prune -a --volumes
# Check disk usage
docker system df
Increase Docker disk allocation:
Rancher Desktop:
Rancher Desktop does not have a UI for increasing disk size. To increase it:
On macOS or Linux:
~/Library/Application Support/rancher-desktop/lima/_config/override.yaml~/.config/rancher-desktop/lima/_config/override.yamldisk: 100GB
On Windows (WSL2):
The disk is managed by WSL2:
wsl --shutdownDocker Desktop:
Remove old dev containers:
# List all containers
docker ps -a
# Remove specific container
docker rm <container_id>
# Remove all stopped containers
docker container prune
Symptoms:
Error: curl: (22) The requested URL returned error: 404
Error: Failed to download toolchain
Solutions:
Check internet connection: Ensure you can access S3:
curl -I https://s3.amazonaws.com/boxes.10gen.com/
Verify toolchain URL:
# Check what's configured
cat .devcontainer/toolchain_config.env
# Try downloading manually to test
curl -I "$(grep TOOLCHAIN_URL .devcontainer/toolchain_config.env | cut -d'"' -f2)"
If toolchain URL is broken, report it to the MongoDB team. This is a devcontainer configuration issue that needs to be fixed upstream.
Symptoms:
Error: SHA256 checksum mismatch
Expected: abc123...
Got: def456...
This typically indicates the toolchain was updated but the config file wasn't.
Solutions:
Pull latest changes from the repository (the maintainers may have already fixed this):
git pull
# Then rebuild container
Clear Docker cache and rebuild:
# Command Palette → "Dev Containers: Rebuild Container Without Cache"
If problem persists, this is likely a devcontainer configuration issue - report it to the MongoDB team.
Symptoms:
Solutions:
Check Docker logs:
# Find container ID
docker ps -a
# View logs
docker logs <container_id>
Rebuild container:
Check Docker daemon status:
docker info
docker version
Symptoms:
Solutions:
Verify you're using a named volume (not bind mount):
# Inside container
df -h /workspaces/mongo
# Should NOT show a mount from host filesystem
# Should be part of container's internal filesystem
If using bind mount, migrate to named volume:
Increase Docker resources:
Check cache volume is mounted:
# Inside container
ls -la ~/.cache/bazel
# Should have bazel cache directory
Verify no antivirus scanning Docker:
Symptoms:
git status takes 5+ secondsRoot Cause: Bind mounts on macOS use osxfs which has high latency for filesystem operations.
Solution: ✅ Use named volumes instead of bind mounts (see Getting Started guide)
Symptoms:
Solutions:
Check for runaway processes:
# Inside container
top
htop # If available
Check for file watcher issues:
# Limit file watchers (Linux)
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
For additional VS Code-specific troubleshooting, see:
Symptoms:
Solutions:
Manually install extensions:
Check extension compatibility:
Reinstall extensions:
# Command Palette
> Developer: Reinstall Extension...
Symptoms:
Solutions:
Generate compile_commands.json:
bazel build compiledb --config=local
# Verify it exists
ls -lh compile_commands.json
Check clangd path:
# Verify the wrapper script exists
ls -l buildscripts/clangd_vscode.sh
# Test it
./buildscripts/clangd_vscode.sh --version
Restart clangd:
Check clangd output:
Clear clangd cache:
rm -rf ~/.cache/clangd
Symptoms:
Solutions:
Verify venv exists:
ls -la python3-venv/bin/python
source python3-venv/bin/activate
which python
Select interpreter in VS Code:
python3-venv/bin/pythonRebuild venv:
rm -rf python3-venv
/opt/mongodbtoolchain/v5/bin/python3.13 -m venv python3-venv
source python3-venv/bin/activate
poetry install --no-root --sync
Check settings.json:
{
"python.defaultInterpreterPath": "python3-venv/bin/python"
}
Symptoms:
Solutions:
Check settings:
{
"editor.formatOnSave": true,
"[cpp]": {
"editor.defaultFormatter": "xaver.clang-format",
"editor.formatOnSave": true
}
}
Verify formatter is installed:
Test formatter manually:
Symptoms:
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Solutions:
Verify SSH keys exist on host:
# On your HOST machine (not in container)
ls -la ~/.ssh/id_*.pub
# Should see id_ed25519.pub, id_rsa.pub, or similar
Test SSH connection to GitHub:
# On HOST machine
ssh -T [email protected]
# Should see: "Hi username! You've successfully authenticated..."
# If this fails, your SSH key isn't added to GitHub
Add SSH key to GitHub:
# Copy your public key
cat ~/.ssh/id_ed25519.pub # or id_rsa.pub
# Go to https://github.com/settings/keys
# Click "New SSH key" and paste
Ensure ssh-agent has your key:
# On HOST machine
ssh-add -l
# If empty or shows "Could not open connection"
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519 # or id_rsa
See Getting Started - SSH Setup for detailed instructions.
Symptoms:
Root Cause: SSH agent forwarding isn't working properly.
Solutions:
Verify agent forwarding requirements:
# On HOST machine (before opening container)
# SSH agent must be running
echo $SSH_AUTH_SOCK
# Should show a path, not empty
# Agent must have keys loaded
ssh-add -l
# Should list your SSH keys
Add keys to agent if missing:
# On HOST machine
ssh-add ~/.ssh/id_ed25519 # or id_rsa
# Verify
ssh-add -l
Restart VS Code and rebuild container:
Check SSH config (macOS):
# On HOST machine
# Add to ~/.ssh/config
Host *
AddKeysToAgent yes
UseKeychain yes
IdentityFile ~/.ssh/id_ed25519
Start ssh-agent automatically (Linux):
# Add to ~/.bashrc or ~/.zshrc on HOST
if [ -z "$SSH_AUTH_SOCK" ]; then
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
fi
Windows: Ensure ssh-agent service is running:
# In PowerShell as Administrator (on HOST)
Get-Service ssh-agent | Set-Service -StartupType Automatic
Start-Service ssh-agent
# Then add your key
ssh-add $env:USERPROFILE\.ssh\id_ed25519
Symptoms:
Username for 'https://github.com':
Password for 'https://[email protected]':
Causes:
Solutions:
Option 1: Switch to SSH (recommended):
# Check current remote URL
git remote -v
# If using HTTPS, switch to SSH
git remote set-url origin <ssh url>
# Verify
git remote -v
Option 2: Use Personal Access Token (for HTTPS):
# Generate token at https://github.com/settings/tokens
# Use token as password when prompted
# Or configure credential helper
git config --global credential.helper store
# Next time you enter credentials, they'll be saved
Option 3: Fix SSH agent forwarding: See "SSH Works on Host But Not in Container" section above.
Problem: Have multiple GitHub accounts or SSH keys
Solution: Use SSH config to manage multiple keys:
# On HOST machine, edit ~/.ssh/config
Host github.com-work
HostName github.com
User git
IdentityFile ~/.ssh/id_ed25519_work
Host github.com-personal
HostName github.com
User git
IdentityFile ~/.ssh/id_ed25519_personal
# Add both keys to agent
ssh-add ~/.ssh/id_ed25519_work
ssh-add ~/.ssh/id_ed25519_personal
# Clone using specific host alias
git clone [email protected]:<repo>
Symptoms:
error: gpg failed to sign the data
fatal: failed to write commit object
Solution:
GPG signing requires additional setup in devcontainers.
Use SSH signing (GitHub now supports this):
# Configure git to use SSH for signing
git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub
git config --global commit.gpgsign true
Symptoms:
ERROR: Bazel server terminated abruptly
Solutions:
Clean Bazel cache:
bazel clean --expunge
Check disk space:
df -h
Restart container:
Symptoms:
ERROR: No matching toolchains found
ERROR: Cannot find compiler
Solutions:
Verify toolchain installation:
ls -la /opt/mongodbtoolchain/revisions/
# Check compiler
/opt/mongodbtoolchain/v5/bin/gcc --version
Source toolchain environment:
source /opt/mongodbtoolchain/revisions/*/activate
Rebuild container to reinstall toolchain
Symptoms:
ERROR: Failed to authenticate with EngFlow
ERROR: Build Event Service upload failed
Solutions:
Check if credentials exist:
ls -la ~/.config/engflow_auth/
Re-authenticate with EngFlow:
rm -r ~/.config/engflow_auth/*
bazel run engflow_auth
Build without EngFlow:
bazel build --config=local install-mongod
Symptoms:
ERROR: Failed to install packages
KeyringError: ...
Solutions:
Set keyring backend:
export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring
poetry install --no-root --sync
Clear Poetry cache:
poetry cache clear --all pypi
poetry install --no-root --sync
Verify Poetry version:
poetry --version
# Should be version specified in poetry_requirements.txt
Symptoms:
(python3-venv) not in promptwhich python shows system PythonSolutions:
Manually activate:
source python3-venv/bin/activate
Check shell config:
cat ~/.bashrc | grep python3-venv
cat ~/.zshrc | grep python3-venv
Re-source config:
source ~/.bashrc # or ~/.zshrc
Symptoms:
ModuleNotFoundError: No module named 'pymongo'
Solutions:
Ensure venv is activated:
which python
# Should show: /workspaces/mongo/python3-venv/bin/python
Reinstall dependencies:
source python3-venv/bin/activate
poetry install --no-root --sync
Check Poetry lock file:
poetry check
poetry lock --check
Symptoms:
Root Cause: Volumes not mounting correctly
Solutions:
Check volumes are mounted:
docker inspect <container_id> | grep -A 10 Mounts
Verify volumes exist:
docker volume ls | grep mongo
Check devcontainer.json mounts:
"mounts": [
{
"source": "mongo-cache",
"target": "/home/youruser/.cache",
"type": "volume"
}
]
Symptoms:
Solution:
Files in named volumes are in Docker's VM, not directly accessible.
To access:
# Copy file from container to host
docker cp <container_id>:/workspaces/mongo/file.txt ~/Downloads/
# Or use VS Code
# Right-click file → Download...
To edit with external tools: Use bind mounts instead of named volumes (but sacrifices performance).
Symptoms:
docker system df
# Shows huge SIZE for volumes
Solutions:
Clean Bazel cache:
# Inside container
bazel clean --expunge
Remove old volumes:
# List volumes
docker volume ls
# Remove specific volume if needed (WARNING: loses data!)
docker volume rm old-cache-volume
Limit Bazel cache size:
# Add to ~/.bazelrc
echo "build --disk_cache=~/.cache/bazel --disk_cache_size=10G" >> ~/.bazelrc
Solutions:
Start Docker Desktop/Rancher Desktop:
Reset Docker:
Check Docker context:
docker context ls
docker context use default
Symptoms:
Solutions:
Verify base image supports ARM:
docker pull quay.io/mongodb/bazel-remote-execution:ubuntu24-...
docker inspect --format='{{.Architecture}}' <image_id>
Use platform flag if needed:
FROM --platform=linux/amd64 <base_image>
Check Rosetta 2 is enabled (Rancher Desktop)
Symptoms:
Solutions:
Enable WSL2 integration:
Use WSL2 terminal:
/mnt/c/Check WSL version:
wsl --list --verbose
# Should show VERSION 2
Symptoms:
permission denied while trying to connect to Docker daemon
Solutions:
Add user to docker group:
sudo usermod -aG docker $USER
newgrp docker # Or logout/login
Check Docker socket permissions:
ls -l /var/run/docker.sock
sudo chmod 666 /var/run/docker.sock # Temporary
Symptoms:
Solution:
Symptoms:
Solution: Go to Docker Desktop → Settings → Resources and allocate generously:
Note: MongoDB builds are resource-intensive. More resources = significantly faster builds.
Symptoms:
Solution: OrbStack has some limitations with devcontainer features. Try:
VS Code Dev Container logs:
Docker logs:
# Container logs
docker logs <container_id>
# Follow logs in real-time
docker logs -f <container_id>
Bazel verbose:
bazel build --verbose_failures --sandbox_debug install-mongod
# Get container ID
docker ps
# Exec into container
docker exec -it <container_id> /bin/bash
# Check processes
docker exec <container_id> ps aux
# Check environment
docker exec <container_id> env
# Inside container
df -h # Disk usage
free -h # Memory
top # CPU/Memory by process
# From host
docker stats # Live resource usage
Sometimes the best fix is a clean rebuild:
# Stop and remove container
docker stop <container_id>
docker rm <container_id>
# Rebuild without cache
# Command Palette → "Dev Containers: Rebuild Container Without Cache"
To isolate whether an issue is devcontainer-specific:
# Clone locally
git clone [email protected]:10gen/mongo.git
cd mongo
# Try building without devcontainer
# (Requires local toolchain setup)
If your issue isn't covered here:
See Also: