@lowdefy/server-dev

Development server with hot reload and file watching.

Overview

The development server provides:

• Automatic rebuilding on config changes
• Hot reload without full page refresh
• Extended plugin set for development
• Environment file watching
• Process management
• In-browser ErrorBar for build errors and warnings

Key Differences from Production

Additional Dependencies

Beyond production server:

• @lowdefy/build - Build system
• chokidar (3.5.3) - File watcher
• dotenv (16.3.1) - Env loading
• opener (1.5.2) - Browser opener
• swr (2.2.4) - Data fetching
• postcss, tailwindcss, @tailwindcss/postcss - JIT CSS compilation (used by compileCss.mjs and lib/server/compileCss.js)

Additional block packages:

• @lowdefy/blocks-aggrid
• @lowdefy/blocks-echarts
• @lowdefy/blocks-markdown
• @lowdefy/blocks-qr

Additional operators:

• @lowdefy/operators-change-case
• @lowdefy/operators-diff
• @lowdefy/operators-dayjs
• @lowdefy/operators-mql
• @lowdefy/operators-nunjucks
• @lowdefy/operators-uuid
• @lowdefy/operators-yaml

CRITICAL: Singleton Packages in Local Dev

antd and @ant-design/cssinjs use React context for cross-component coordination. Multiple instances break CSS-in-JS context sharing between ConfigProvider, StyleProvider, and useDarkMode — dark mode and theming silently fail (only some antd components respond to theme changes, no console errors).

Both server and server-dev have antd and @ant-design/cssinjs as direct dependencies. This is correct — the published packages need them for pnpm strict mode resolution.

The singleton risk only exists in the local monorepo dev setup (scripts/dev.mjs), where rewriteDeps.mjs rewrites @lowdefy/* deps to link: paths. Without overrides, pnpm would install a separate npm copy of antd for the dev server while linked @lowdefy/client uses the monorepo's copy — two instances.

Fix: rewriteDeps.mjs has a SINGLETON_PACKAGES list (antd, @ant-design/cssinjs) that adds pnpm.overrides entries pointing to the monorepo's node_modules/ copies. This forces a single instance across the dev server and all linked packages.

If you add a new package that uses React context across components (like a UI library), add it to SINGLETON_PACKAGES in scripts/lib/rewriteDeps.mjs.

Symptoms of duplicate instances: Dark mode toggle only partially works — some antd components (like Menu) respond while the rest of the page stays in light mode. No errors in console.

Scripts

{
  "build": "cp package.json package.original.json",
  "start": "node manager/run.mjs"
}

Directory Structure

server-dev/
├── lib/
│   ├── build/                # Build artifact loaders (deserialize JSON)
│   ├── server/               # Server utilities
│   │   ├── jitPageBuilder.js # JIT page build on request
│   │   ├── pageCache.mjs     # PageCache class (shared by manager and server)
│   │   └── log/
│   │       └── createLogger.js
│   └── client/               # Client utilities (extended)
│       ├── App.js            # Dev app wrapper
│       ├── Page.js           # Page renderer (merges jsMap)
│       ├── Reload.js         # Hot reload component
│       ├── RestartingPage.js
│       └── Utils/
│           ├── usePageConfig.js       # SWR hook with versioned keys
│           ├── useRootConfig.js
│           ├── useMutateCache.js      # reloadVersion counter
│           └── waitForRestartedServer.js
├── manager/
│   ├── run.mjs               # Entry point
│   ├── getContext.mjs         # Context factory (stores JIT build state)
│   ├── processes/
│   │   ├── initialBuild.mjs
│   │   ├── lowdefyBuild.mjs  # Calls shallowBuild, captures result
│   │   ├── installPlugins.mjs
│   │   ├── nextBuild.mjs
│   │   ├── startServer.mjs
│   │   ├── restartServer.mjs
│   │   ├── shutdownServer.mjs
│   │   ├── readDotEnv.mjs
│   │   └── reloadClients.mjs
│   ├── utils/
│   │   └── loadSkeletonSourceFiles.mjs  # Read skeletonSourceFiles.json as Set
│   └── watchers/
│       ├── lowdefyBuildWatcher.mjs   # Skeleton vs page change classification
│       ├── moduleBuildWatcher.mjs    # Local module file change classification
│       ├── envWatcher.mjs
│       └── nextBuildWatcher.mjs
├── pages/
│   └── api/
│       ├── reload.js         # SSE endpoint
│       ├── ping.js           # Health check
│       ├── page/[pageId].js  # Page API (triggers JIT build)
│       ├── js/[env].js       # Serves clientJsMap.js or serverJsMap.js
│       ├── root.js
│       └── dev-tools.js
├── next.config.js
└── package.json

Manager System

Entry Point

File: manager/run.mjs

async function run() {
  const context = await getContext();

  // Initial build
  await context.initialBuild();

  // Start watchers (non-blocking)
  context.startWatchers();

  // Start server
  await context.startServer();

  // Open browser
  if (process.env.LOWDEFY_OPEN_BROWSER !== 'false') {
    opener(`http://localhost:${context.options.port}`);
  }
}

Context

File: manager/getContext.mjs

const context = {
  bin: { next: nextBin },
  directories: { build, config, server },
  logger,
  options: { port, refResolver, watch: [], watchIgnore: [] },
  version,
  packageManagerCmd,

  // JIT build state
  pageCache: new PageCache(), // Manager's PageCache instance
  pageRegistry: null, // Set after each skeleton build
  buildContext: null, // Build context from shallowBuild

  // Bound functions
  initialBuild,
  installPlugins,
  lowdefyBuild, // Wrapped to capture build result
  nextBuild,
  readDotEnv,
  reloadClients,
  restartServer,
  shutdownServer,
  startWatchers,
};

JIT (Just-In-Time) Build System

The dev server uses a two-phase build strategy to minimize rebuild times:

1. Skeleton build (shallowBuild): Resolves all _ref operators except page content (blocks, areas, events, requests, layout). Page content is left as _shallow markers.
2. JIT page build (buildPageJit): When a page is requested, resolves that page's _shallow markers, runs build steps, and writes page artifacts.

Two-Process Architecture

The manager process and the Next.js server run as separate processes with no shared memory:

Manager Process                    Next.js Server Process
┌────────────────────┐            ┌────────────────────┐
│ PageCache instance │            │ PageCache instance  │
│ pageRegistry       │            │ (jitPageBuilder.js) │
│ buildContext       │            │                     │
│                    │            │ cachedRegistry      │
│ Watcher → build    │            │ cachedBuildContext   │
│                    │            │ API → buildPageJit  │
└────────────────────┘            └────────────────────┘
         │                                  ↑
         │  invalidatePages (signal file)   │
         └──── (file on disk) ─────────────→┘

Cross-process communication uses files in the build directory:

• pageRegistry.json: Page metadata + raw content for JIT resolution
• refMap.json, keyMap.json, jsMap.json: Shared build state
• skeletonSourceFiles.json: Set of files that affect skeleton (read by watcher)
• invalidatePages: Timestamp signal file written by watcher for page-only changes

Skeleton Build Flow

File: manager/processes/lowdefyBuild.mjs

function lowdefyBuild({ directories, logger, options, pageCache }) {
  return async () => {
    logger.info('Building config...', { spin: true });
    const customTypesMap = await createCustomPluginTypesMap({ directories, logger });
    await pageCache.acquireSkeletonLock();
    try {
      const result = await shallowBuild({ customTypesMap, directories, logger, ... });
      logger.info('Built config.');
      return result; // { components, pageRegistry, context }
    } finally {
      pageCache.invalidateAll();
      pageCache.releaseSkeletonLock();
    }
  };
}

The manager wraps lowdefyBuild to capture and store the result on the manager context:

// In getContext.mjs
context.lowdefyBuild = async () => {
  const result = await buildFn();
  if (result) {
    context.pageRegistry = result.pageRegistry;
    context.buildContext = result.context;
  }
};

JIT Page Build Flow

File: lib/server/jitPageBuilder.js

When a page API request arrives (/api/page/[pageId]):

1. checkPageInvalidations() reads invalidatePages.json (with mtime caching)
2. loadPageRegistry() reads pageRegistry.json (with mtime caching)
3. pageCache.isCompiled(pageId) checks if page was already built
4. If not compiled, acquires build lock and calls buildPageJit()
5. getBuildContext() creates/caches a build context with restored refMap/keyMap/jsMap

async function buildPageIfNeeded({ pageId, buildDirectory, configDirectory }) {
  checkPageInvalidations(buildDirectory);
  const registry = loadPageRegistry(buildDirectory);
  if (!registry?.[pageId]) return false;

  if (pageCache.isCompiled(pageId)) return true;

  const shouldBuild = await pageCache.acquireBuildLock(pageId);
  if (!shouldBuild) return true; // Another request completed it

  try {
    const context = getBuildContext(buildDirectory, configDirectory);
    await buildPageJit({ pageId, pageRegistry: registry, context });
    pageCache.markCompiled(pageId);
    return true;
  } finally {
    pageCache.releaseBuildLock(pageId);
  }
}

PageCache

File: lib/server/pageCache.mjs

Tracks which pages have been JIT-compiled and provides concurrency control:

Two separate instances exist: one in the manager process (for watcher invalidation) and one in the Next.js server (for JIT build tracking).

File Dependency Map

File: packages/build/src/build/createFileDependencyMap.js

Maps config file paths → Set of page IDs that depend on them. Used for targeted invalidation.

Sources of dependencies:

1. Page source file: via pageEntry.refId → refMap[refId].path
2. Child _ref paths: Collected from _shallow markers in raw page content

Skeleton vs Page Change Classification

File: manager/watchers/lowdefyBuildWatcher.mjs

When a file changes, the watcher classifies it using the skeletonSourceFiles.json artifact (produced by the build's collectSkeletonSourceFiles):

The skeletonSourceFiles set is derived from ~r markers on non-page components during the shallow build. It includes every config file that contributes to non-page build artifacts (connections, API endpoints, auth, menus, etc.), traced through the refMap parent chain. This replaces the previous path-based heuristic (!f.startsWith('pages/')) which had false negatives for API files referenced from pages/ and false positives for page templates outside pages/.

Both lowdefyBuildWatcher and moduleBuildWatcher use the same loadSkeletonSourceFiles helper and the same classification logic. The set contains relative paths for main config refs and absolute paths for module refs — matching the path formats each watcher receives from chokidar.

Cross-Process Cache Invalidation

The manager and server run in separate processes with separate PageCache instances. When a file change only affects pages (not skeleton):

1. Manager writes invalidatePages signal file (timestamp)
2. Manager calls reloadClients() (SSE event)
3. On next page request, server's checkPageInvalidations() detects the signal file (mtime-based)
4. Server's pageCache.invalidateAll() clears all compiled pages
5. Server's cachedBuildContext is set to null to refresh maps

For skeleton changes, lowdefyBuild() triggers a full rebuild which calls pageCache.invalidateAll() on the manager side, and the server detects the new artifacts on next request.

Server Process and Logging

stdio: inherit

The manager spawns the Next.js server with stdio: ['ignore', 'inherit', 'pipe']:

• stdout is inherited — server pino JSON flows directly to the manager's stdout (which the CLI reads)
• stderr is piped — the manager formats stderr lines through its own logger
• stdin is ignored

This eliminates the need for a dev stdout line handler to parse and re-emit server logs. The server's pino logger emits JSON with optional color/spin/succeed fields, so the CLI can render each line correctly (error → red, blue → source link, spin → spinner, etc.).

// startServer.mjs
const nextServer = spawn('node', [context.bin.next, 'start'], {
  stdio: ['ignore', 'inherit', 'pipe'],
  env: {
    ...process.env,
    LOWDEFY_DIRECTORY_CONFIG: context.directories.config,
    PORT: context.options.port,
  },
});

Logger Setup

The server-dev uses two loggers:

Both emit pino JSON with optional color/spin/succeed fields to stdout. The CLI reads this JSON and renders it via createStdOutLineHandler → createCliLogger (ora spinners, colored output).

See @lowdefy/logger for details.

Build Processes

Initial Build

File: manager/processes/initialBuild.mjs

async function initialBuild(context) {
  await context.readDotEnv();
  await context.lowdefyBuild();
  await context.installPlugins();
  await context.nextBuild();
}

### Install Plugins

**File:** `manager/processes/installPlugins.mjs`

```javascript
async function installPlugins(context) {
  await spawnProcess({
    command: context.packageManagerCmd,
    args: ['install', '--no-frozen-lockfile'],
    cwd: context.directories.server,
    logger: context.logger
  });
}

File Watchers

Watcher Orchestration

File: manager/processes/startWatchers.mjs

function startWatchers(context) {
  return async () => {
    await Promise.all([
      envWatcher(context),          // .env changes → hard restart
      lowdefyBuildWatcher(context), // Config changes → soft reload
      moduleBuildWatcher(context),  // Local module changes → soft reload
      nextBuildWatcher(context),    // Plugin changes → rebuild + restart
    ]);
  };
}

Lowdefy Build Watcher

File: manager/watchers/lowdefyBuildWatcher.mjs

Watches config directory for changes. Decides between targeted invalidation (fast) and full skeleton rebuild based on file dependency map:

const callback = async (filePaths) => {
  const changedFiles = filePaths.map((f) => path.relative(configDir, f));

  // Check for version change in lowdefy.yaml
  if (lowdefyYamlModified) {
    /* exit if version changed */
  }

  const skeletonSourceFiles = loadSkeletonSourceFiles(context.directories.build);

  const isSkeletonChange =
    lowdefyYamlModified ||
    changedFiles.some((f) => skeletonSourceFiles.has(f));

  if (isSkeletonChange) {
    await context.lowdefyBuild(); // Full skeleton rebuild
  } else {
    // Page-only change: write signal file so server invalidates its page cache
    fs.writeFileSync(invalidatePath, String(Date.now()));
  }
  context.reloadClients();
};

Module Build Watcher

File: manager/watchers/moduleBuildWatcher.mjs

Watches local module directories (modules with isLocal: true in buildContext.modules). Uses the same skeletonSourceFiles.json artifact as the lowdefy build watcher to classify changes:

const callback = async (filePaths) => {
  const changedFiles = filePaths.flat(); // Absolute paths from chokidar

  const moduleYamlChanged = changedFiles.some(
    (filePath) => path.basename(filePath) === 'module.lowdefy.yaml'
  );

  const skeletonSourceFiles = loadSkeletonSourceFiles(context.directories.build);
  const hasSkeletonChanges = changedFiles.some((f) => skeletonSourceFiles.has(f));

  if (moduleYamlChanged || hasSkeletonChanges) {
    await context.lowdefyBuild();
    await context.compileCss();
  } else {
    fs.writeFileSync(invalidatePath, String(Date.now()));
  }
  context.reloadClients();
};

Module refs in the skeletonSourceFiles set are absolute paths (the walker resolves them via path.resolve), matching chokidar's absolute output. No path normalization needed.

The watcher only starts when local modules exist. If buildContext.modules is empty or has no local entries, moduleBuildWatcher returns immediately.

Environment Watcher

File: manager/watchers/envWatcher.mjs

Watches .env file:

const watcher = chokidar.watch('.env', { ignoreInitial: true });

watcher.on('all', async () => {
  await context.readDotEnv();
  await context.restartServer();
});

Next Build Watcher

File: manager/watchers/nextBuildWatcher.mjs

Watches build artifacts:

const watcher = chokidar.watch(['build/plugins/**', 'build/config.json', 'server/package.json'], {
  ignoreInitial: true,
});

watcher.on('all', async (event, filePath) => {
  if (filePath.includes('package.json')) {
    await context.installPlugins();
  }
  await context.nextBuild();
  await context.restartServer();
});

Hot Reload System

Server-Side (SSE)

File: pages/api/reload.js

export default function handler(req, res) {
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    Connection: 'keep-alive',
  });

  const watcher = chokidar.watch('./build/reload');

  watcher.on('change', () => {
    res.write('data: reload\n\n');
  });

  req.on('close', () => {
    watcher.close();
  });
}

Client-Side

File: lib/client/Reload.js

function Reload({ children, lowdefy, resetContext }) {
  const [restarting, setRestarting] = useState(false);

  useEffect(() => {
    const eventSource = new EventSource('/api/reload');

    eventSource.onmessage = (event) => {
      if (event.data === 'reload') {
        // Soft reload
        mutateCache();
        resetContext();
      }
    };

    eventSource.onerror = () => {
      // Server restarting
      setRestarting(true);
      waitForRestartedServer().then(() => {
        window.location.reload();
      });
    };

    return () => eventSource.close();
  }, []);

  if (restarting) {
    return <RestartingPage />;
  }

  return children;
}

Reload Trigger

File: manager/processes/reloadClients.mjs

async function reloadClients(context) {
  const reloadFile = path.join(context.directories.build, 'reload');
  fs.writeFileSync(reloadFile, Date.now().toString());
}

Dev-Specific API Routes

Page Config (JIT Build Trigger)

File: pages/api/page/[pageId].js

The page API endpoint triggers JIT page building before returning page config:

export default apiWrapper(async ({ context, req, res }) => {
  const { pageId } = req.query;

  // Trigger JIT build if page not yet compiled
  const built = await buildPageIfNeeded({
    pageId,
    buildDirectory: context.directories.build,
    configDirectory: context.directories.config,
  });

  if (!built) {
    return res.status(404).json({ message: 'Page not found' });
  }

  // Read and return built page config
  const pageConfig = await getPageConfig(context, { pageId });
  return res.json(pageConfig);
});

JS Map

File: pages/api/js/[env].js

Serves the client or server JS map as a JavaScript module. The client fetches this after a JIT build to get newly extracted _js function entries.

Health Check

File: pages/api/ping.js

export default function handler(req, res) {
  res.status(200).json({ status: 'ok' });
}

Client Components

App Wrapper

File: lib/client/App.js

function App({ children }) {
  const { data: rootConfig } = useRootConfig();

  return (
    <Reload lowdefy={lowdefy} resetContext={resetContext}>
      <Auth auth={auth}>{children}</Auth>
    </Reload>
  );
}

ErrorBar

File: lib/client/ErrorBar.js

Fixed bottom bar that displays build errors and warnings in the browser. Build warnings propagate from the build pipeline through the SSE reload channel to the client, giving developers immediate feedback without checking the terminal. Includes a copy-to-clipboard button for sharing error details with stack traces.

SWR Hooks

File: lib/client/Utils/usePageConfig.js

Uses SWR with a versioned key to support cache busting on hot reload:

import { getReloadVersion } from './useMutateCache.js';

async function fetchPageConfig(url) {
  const res = await fetch(url, { headers: { 'Content-Type': 'application/json' } });
  const data = await res.json();
  // After page config fetch (which triggers JIT build), fetch JS entries
  const basePath = url.replace(/\/api\/page\/.*$/, '');
  const jsEntries = await fetchJsEntries(basePath);
  data._jsEntries = jsEntries;
  return data;
}

function usePageConfig(pageId, basePath) {
  const url = `${basePath}/api/page/${pageId}`;
  // reloadVersion changes on hot reload, orphaning old SWR cache entries
  const { data } = useSWR([url, getReloadVersion()], ([fetchUrl]) => fetchPageConfig(fetchUrl), {
    suspense: true,
  });
  return { data };
}

File: lib/client/Utils/useMutateCache.js

Manages cache busting via a reloadVersion counter:

let reloadVersion = 0;

function getReloadVersion() {
  return reloadVersion;
}

function useMutateCache(basePath) {
  const { mutate } = useSWRConfig();
  return () => {
    reloadVersion += 1; // Orphans old SWR keys
    return mutate((key) => key === `${basePath}/api/root`); // Only revalidate root
  };
}

Why versioned keys instead of cache clearing:

• Clearing SWR entries to undefined causes React Suspense on currently mounted components
• This creates a three-request waterfall (/api/root → /api/page → /api/js) with visible delay
• Versioned keys orphan old entries without triggering Suspense, and new keys force fresh fetches

Why jsMap is fetched sequentially after page config:

• Page config fetch triggers JIT build which may extract new _js functions
• If jsMap is fetched in parallel, it returns stale data missing the new JS entries
• The _jsEntries are merged with the static jsMap in Page.js

Next.js Configuration

File: next.config.js

const nextConfig = {
  // Disable optimizations for faster dev builds
  swcMinify: false,
  compress: false,
  outputFileTracing: false,
  generateEtags: false,
  optimizeFonts: false,

  webpack: (config) => {
    // Same browser fallbacks as production
    return config;
  },
};

Key Files

Reload Types

Mock User for Testing

The dev server supports mock users for testing, bypassing the login flow.

Configuration

Environment Variable (takes precedence):

LOWDEFY_DEV_USER='{"sub":"test-user","email":"[email protected]","roles":["admin"]}'

Config File:

auth:
  dev:
    mockUser:
      sub: test-user
      email: [email protected]

Key Files

See Auth System Architecture for full details.

Plugin Strategy

The dev server uses a different plugin strategy than production to optimize for fast iteration.

Pre-installed Packages

The dev server's package.json includes a broad set of default plugin packages (blocks, operators, actions, connections) as dependencies. This means:

• No code build (Next.js rebuild) is needed when a user first uses a new block or action type
• Bundle size is not a concern in development — all installed types are available immediately
• The skeleton build reads the server's package.json to determine installed packages and includes all types from those packages in the generated import files

JIT Build and Type Counting

During development, the skeleton build (shallowBuild) stops at page content boundaries (pages.*.blocks, pages.*.events, etc.) and leaves _shallow markers. Page content is resolved just-in-time when requested. This means page-level types (actions, blocks, operators used inside pages) are NOT counted during the skeleton build.

To compensate, shallowBuild adds all types from installed packages to components.types after buildTypes runs. This ensures the generated plugin import files include all available types, not just those counted from non-page components (like connections and API config).

New Plugin Detection

If a user configures a plugin package that isn't installed in the dev server:

1. The build detects the new package via customTypesMap
2. installPlugins runs pnpm install to add the package
3. A Next.js rebuild is triggered to bundle the new imports
4. The server restarts with the new plugin available

Production Comparison

Key Files

Environment Variables