Data Migrations

```bash
git clone https://github.com/novuhq/novu.git
cd novu

git checkout <your-version-tag>  # e.g. v0.24.0
npm run clean
npm run setup:project # really important to run if you change tag
cd apps/api
```

<Warning>
Migrations are dependent on the underlying code (eg repositories):
- match the version of the code against the migration at the right point in time
- each version often requires the correct version of `node` and `pnpm` as per the `package.json`
</Warning>

<Note>
- Getting the code the run at a checked out version can be tricky
- If you want to know which tags are available: `git tag`
- Some migrations may fail because they rely on the underlying code rather than direct mongo queries
- Migrations are located in: `apps/api/migrations/`
- Checkout by tag stops accidentally running too many migrations that would normally be managed by migrations runner (see [migration versioning](#migration-versioning-and-idempotency))
</Note>

```bash
# Example: Tunnel MongoDB via SSH
ssh -L 27017:localhost:27017 your-user@your-db-host
```

Have your:

- MongoDB **connection string**
- **Username and password** (if required)

Configure access to the correct database:

```bash
npx cross-env \
MONGO_URL="mongodb://127.0.0.1:27017/novu-db" \
NEW_RELIC_ENABLED=false \
npm run migration -- ./migrations/normalize-users-email/normalize-users-email.migration.ts
```

Some notes:

- under the hood of the migration script ```npx ts-node --transpile-only ./apps/api/migrations/<script-name>.ts```
- some migrations require REDIS (ensure already running)
- some migrations require NEW_RELIC (so turn off)
- Run each script only once. Monitor logs or database changes for success.

- Tracking applied migrations in a collection (e.g., `migrations`).
- Guard clauses in code to check if a change has already been made.

Note:

- there are libraries that support idempotent Mongo migrations in TypeScript
- if you want to introduce versioned/idempotent migrations in your own deployment process, consider [
`migrate-mongo`](https://github.com/seppevs/migrate-mongo) or [
`mongodb-migrations`](https://github.com/emirotin/mongodb-migrations)

chmod +x run-migrations.sh

#!/bin/sh

###############################################################################
# 📦 NOVU MIGRATION RUNNER - POSIX SH EDITION (macOS Compatible)
#
# This script runs one or more database migration scripts for the Novu project.
# It supports:
#   - Selecting a version interactively or via argument
#   - Providing a custom MongoDB connection string
#   - Executing version-specific migration files in order
#   - Compatible with macOS default /bin/sh
#
# ⚠️ WARNING: some migrations do not exit properly ( eg < 0.19.0 ) and require Ctrl-C to finish and continue
#
# 🛠 REQUIREMENTS:
#   - Node.js (with npm and npx)
#   - ts-node and dependencies installed via your project
#
# ▶️ USAGE:
#   ./run-migrations.sh                       # Interactive version selection
#   ./run-migrations.sh v0.24.0               # Run migrations for version
#   ./run-migrations.sh mongodb://...         # Use custom Mongo URL (interactive version)
#   ./run-migrations.sh v0.18.0 mongodb://... # Version + custom Mongo URL
#   ./run-migrations.sh v0.24.0 --dry-run     # Dry run file check only
#
#  Atlas connection strings
#   - remove all & from query params
#      eg mongodb+srv://<USER>:<PASSWORD>@<INSTANCE>.mongodb.net/novu-db
#
# ⚠️ REDIS WARNING:
#   Some migrations require Redis (on 127.0.0.1:6379) to be running.
#   If Redis is not available, you may see ECONNREFUSED errors during execution.
#
#   To fix this:
#     • Start Redis
###############################################################################

set -e

DEFAULT_MONGO_URL="mongodb://127.0.0.1:27017/novu-db"
NEW_RELIC_ENABLED=false

VERSION=""
MONGO_URL=""
DRY_RUN=false

# Argument parsing
for arg in "$@"; do
  case "$arg" in
    v*) VERSION="$arg" ;;
    mongo*) MONGO_URL="$arg" ;;
    --dry-run) DRY_RUN=true ;;
  esac
done

[ -z "$MONGO_URL" ] && MONGO_URL="$DEFAULT_MONGO_URL"

# Define versions and associated keys
# versions must match suffix
VERSIONS="v0.17.2 v0.18.0 v0.21.0 v0.23.0 v0.24.0 v2.0.0 v2.0.1 Future"
MIGRATION_MAP_v0_17_2="novu-integrations"
MIGRATION_MAP_v0_18_0="changes-migration encrypt-credentials expire-at fcm-credentials in-app-integration normalize-users-email secure-to-boolean seen-read-support"
MIGRATION_MAP_v0_21_0="integration-scheme-update layout-identifier-update"
MIGRATION_MAP_v0_23_0="encrypt-api-keys"
MIGRATION_MAP_v0_24_0="normalize-message-template-cta-action topic-subscriber-normalize"
MIGRATION_MAP_v2_0_0="subscribers"
MIGRATION_MAP_v2_0_1="preference-centralization"
MIGRATION_MAP_Future="deleteLogs"

# Migration file paths
get_migration_path() {
  case "$1" in
    fcm-credentials) echo "fcm-credentials/fcm-credentials-migration.ts" ;;
    layout-identifier-update) echo "layout-identifier-update/add-layout-identifier-migration.ts" ;;
    normalize-message-template-cta-action) echo "normalize-message-template-cta-action/normalize-message-cta-action-migration.ts" ;;
    changes-migration) echo "changes-migration.ts" ;;
    seen-read-support) echo "seen-read-support/seen-read-support.migration.ts" ;;
    in-app-integration) echo "in-app-integration/in-app-integration.migration.ts" ;;
    novu-integrations) echo "novu-integrations/novu-integrations.migration.ts" ;;
    expire-at) echo "expire-at/expire-at.migration.ts" ;;
    secure-to-boolean) echo "secure-to-boolean/secure-to-boolean-migration.ts" ;;
    encrypt-api-keys) echo "encrypt-api-keys/encrypt-api-keys-migration.ts" ;;
    encrypt-credentials) echo "encrypt-credentials/encrypt-credentials-migration.ts" ;;
    topic-subscriber-normalize) echo "topic-subscriber-normalize/topic-subscriber-normalize.migration.ts" ;;
    subscriber-preferences-level) echo "subscriber-preferences-level/subscriber-preferences-level.migration.ts" ;;
    integration-scheme-update) echo "integration-scheme-update/add-primary-priority-migration.ts integration-scheme-update/update-primary-for-disabled-novu-integrations.ts integration-scheme-update/add-integration-identifier-migration.ts" ;;
    normalize-users-email) echo "normalize-users-email/normalize-users-email.migration.ts" ;;
    subscribers) echo "subscribers/remove-duplicated-subscribers/remove-duplicated-subscribers.migration.ts" ;;
    preference-centralization) echo "preference-centralization/preference-centralization-migration.ts" ;;
    deleteLogs) echo "deleteLogs/deleteLogsCollection.ts" ;;
    *) echo "" ;;
  esac
}

# Interactive version selector
if [ -z "$VERSION" ]; then
  echo "📦 No version provided. Please choose one:"
  i=1
  for v in $VERSIONS; do
    echo "  $i) $v"
    eval "VERSION_INDEX_$i=$v"
    i=$((i + 1))
  done

  echo
  printf "Enter the number of the version to run migrations for: "
  read choice

  eval "VERSION=\$VERSION_INDEX_$choice"

  if [ -z "$VERSION" ]; then
    echo "❌ Invalid selection."
    exit 1
  fi

  echo "✅ Selected version: $VERSION"
fi

# Normalize version string to match variable names
VERSION_VAR=$(echo "$VERSION" | tr '.' '_' | tr '-' '_')

eval "MIGRATION_KEYS=\$MIGRATION_MAP_$VERSION_VAR"

echo
echo "🌐 Using MongoDB: $MONGO_URL"
echo "🚀 Running migrations for version: $VERSION"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"

for key in $MIGRATION_KEYS; do
  paths=$(get_migration_path "$key")
  for path in $paths; do
    fullPath="./migrations/$path"
    if [ "$DRY_RUN" = true ]; then
      if [ -f "$fullPath" ]; then
        echo "✅ File exists: $fullPath"
      else
        echo "❌ File missing: $fullPath"
      fi
    else
      echo "🔁 Running migration: $fullPath"
      npx cross-env MONGO_URL="$MONGO_URL" NEW_RELIC_ENABLED="$NEW_RELIC_ENABLED" npm run migration -- "$fullPath"
    fi
  done
done

if [ "$DRY_RUN" = true ]; then
  echo "✅ Dry-run completed."
else
  echo "✅ All applicable migrations completed."
fi

for file in *; do
  echo "$(git log --diff-filter=A --follow --format=%ad --date=short -- $file | head -1) $file"
done | sort

2023-07-31 novu-integrations
2023-08-01 changes-migration.ts
2023-08-01 encrypt-credentials
...
2024-03-24 subscribers
2024-11-19 preference-centralization
2024-11-29 deleteLogs

git for-each-ref --sort=creatordate --format '%(creatordate:short) %(refname:short)' refs/tags

2023-08-13 v0.17.2
2023-08-14 v0.18.0
...
2024-02-07 v0.23.1
2024-03-06 v0.24.0
2024-04-05 v0.24.1
2024-10-07 v2.0.0
2024-11-11 v2.0.1

find . -type f -name "*.ts" ! -name "*.spec.ts"

./fcm-credentials/fcm-credentials-migration.ts
./layout-identifier-update/add-layout-identifier-migration.ts
./normalize-message-template-cta-action/normalize-message-cta-action-migration.ts
./normalize-message-template-cta-action/normalize-message-template-cta-action-migration.ts
./changes-migration.ts
./seen-read-support/seen-read-support.migration.ts
./in-app-integration/in-app-integration.migration.ts
./novu-integrations/novu-integrations.migration.ts
./expire-at/expire-at.migration.ts
./secure-to-boolean/secure-to-boolean-migration.ts
./encrypt-api-keys/encrypt-api-keys-migration.ts
./encrypt-credentials/encrypt-credentials-migration.ts
./topic-subscriber-normalize/topic-subscriber-normalize.migration.ts
./subscriber-preferences-level/subscriber-preferences-level.migration.ts
./integration-scheme-update/add-primary-priority-migration.ts
./integration-scheme-update/update-primary-for-disabled-novu-integrations.ts
./integration-scheme-update/add-integration-identifier-migration.ts
./normalize-users-email/normalize-users-email.migration.ts