ContextQMD
Back to Spacedrive

Normalized Query

docs/react/ui/normalized-cache.mdx

0.4.318.0 KB
Original Source

Real-Time Normalized Cache with TanStack Query

The useNormalizedQuery hook provides instant, event-driven cache updates with server-side filtering. Built with 2025 best practices including runtime validation (Valibot), type-safe merging (ts-deepmerge), and proper subscription cleanup.

Overview

useNormalizedQuery wraps TanStack Query to add real-time capabilities:

  • Instant updates across all devices via WebSocket events
  • Server-side filtering reduces network traffic by 90%+
  • Client-side safety ensures correctness even with unrelated events
  • Proper cleanup prevents connection leaks
  • Runtime validation catches malformed events
  • Type-safe merging preserves data integrity

Architecture

import { FlowDiagram } from '/snippets/FlowDiagram.mdx';

<FlowDiagram steps={[ { title: "Device A: Create file", description: "User creates or modifies a file on their device" }, { title: "Backend: Emit event", description: "Backend detects change and emits events to all connected clients", metrics: { "During indexing": "10,000 events" } }, { title: "Server Filter: Per subscription", description: "Events are filtered server-side based on subscription criteria", items: [ "Desktop: 100 events (1%)", "Movies: 500 events (5%)", "Inspector: 1-5 events (0.05%)" ] }, { title: "Subscription Manager", description: "Multiplexes identical filters to optimize connections", items: [ "1 backend sub → N hooks", "Auto deduplication", "Reference counting" ] }, { title: "Client: Validate & filter", description: "Final validation and cache updates trigger React re-renders", items: [ "Valibot validation", "Client-side filtering", "Atomic cache updates" ] } ]} />

Basic Usage

Directory Listing

tsx
import { useNormalizedQuery } from "@sd/ts-client";

function DirectoryView({ path }: { path: SdPath }) {
	const { data, isLoading } = useNormalizedQuery({
		query: "files.directory_listing",
		input: { path },
		resourceType: "file",
		pathScope: path,
		includeDescendants: false, // Only direct children
	});

	if (isLoading) return <Spinner />;

	return (
		<div>
			{data?.files?.map((file) => <FileCard key={file.id} file={file} />)}
		</div>
	);
}

What happens:

  1. Initial query fetches directory listing
  2. Hook subscribes to file events for this path (exact mode)
  3. When files are created/updated, events arrive instantly
  4. Cache updates atomically
  5. UI re-renders with new data

Media View (Recursive)

tsx
function MediaGallery({ path }: { path: SdPath }) {
	const { data } = useNormalizedQuery({
		query: "files.media_listing",
		input: { path, include_descendants: true },
		resourceType: "file",
		pathScope: path,
		includeDescendants: true, // All media in subtree
	});

	return (
		<Grid>
			{data?.files?.map((file) => <MediaThumbnail key={file.id} file={file} />)}
		</Grid>
	);
}

Global Resources

tsx
function LocationsList() {
	const { data } = useNormalizedQuery({
		query: "locations.list",
		input: null,
		resourceType: "location",
		// No pathScope - locations are global resources
	});

	return (
		<ul>{data?.locations?.map((loc) => <li key={loc.id}>{loc.name}</li>)}</ul>
	);
}

Single Resource Queries

tsx
function FileInspector({ fileId }: { fileId: string }) {
	const { data: file } = useNormalizedQuery({
		query: "files.by_id",
		input: { file_id: fileId },
		resourceType: "file",
		resourceId: fileId, // Only events for this file
	});

	return (
		<div>
			<h1>{file?.name}</h1>
			{file?.sidecars?.map((sidecar) => (
				<Thumbnail key={sidecar.id} src={sidecar.url} />
			))}
		</div>
	);
}

API Reference

Options

tsx
interface UseNormalizedQueryOptions<I> {
	// Query method to call (e.g., "files.directory_listing")
	query: string;

	// Input for the query
	input: I;

	// Resource type for event filtering (e.g., "file", "location")
	resourceType: string;

	// Whether query is enabled (default: true)
	enabled?: boolean;

	// Optional path scope for server-side filtering
	pathScope?: SdPath;

	// Whether to include descendants (recursive) or only direct children (exact)
	// Default: false (exact matching)
	includeDescendants?: boolean;

	// Resource ID for single-resource queries
	resourceId?: string;
}

Path Filtering Modes

Exact Mode (Default)

Only events for files directly in the specified directory:

tsx
pathScope: { Physical: { device_slug: 'my-mac', path: '/Photos' } },
includeDescendants: false // or omit (default)

Behavior:

  • File in /Photos/image.jpg → ✓ Included
  • File in /Photos/Vacation/beach.jpg → ✗ Excluded
  • Directory /Photos/Vacation → ✗ Excluded

Recursive Mode

All events for files anywhere under the specified directory:

tsx
pathScope: { Physical: { device_slug: 'my-mac', path: '/Photos' } },
includeDescendants: true

Behavior:

  • File in /Photos/image.jpg → ✓ Included
  • File in /Photos/Vacation/beach.jpg → ✓ Included
  • File in /Photos/Vacation/Cruise/pic.jpg → ✓ Included

Server-Side Filtering

How It Works

Each hook creates a filtered subscription on the backend:

tsx
client.subscribeFiltered({
	resource_type: "file", // Only file events
	path_scope: "/Desktop", // Only this path
	include_descendants: false, // Exact mode
	library_id: "abc-123", // Current library
});

Backend applies filters before sending events:

  1. ✓ resource_type matches?
  2. ✓ library_id matches?
  3. ✓ path_scope matches? (with include_descendants mode)
  4. ✓ resourceId matches? (if specified)

Result: Only matching events are transmitted over the network.

Filter Logic

Exact Mode:

Event has affected_paths: [
  "/Desktop/file.txt",           // File path
  "/Desktop"                     // Parent directory
]

Subscription path_scope: "/Desktop"
include_descendants: false

Check: Does affected_paths contain "/Desktop" exactly?
Result: YES → Forward event

Recursive Mode:

Event has affected_paths: [
  "/Desktop/Subfolder/file.txt",
  "/Desktop/Subfolder"
]

Subscription path_scope: "/Desktop"
include_descendants: true

Check: Does "/Desktop/Subfolder" start with "/Desktop"?
Result: YES → Forward event

Client-Side Safety Filtering

Even with server-side filtering, the client applies a safety filter to batch events:

tsx
// Server forwards batch if ANY file matches
// Client filters to ONLY files that match

Batch has 100 files:
- 10 in /Desktop/ (direct children)
- 90 in /Desktop/Subfolder/ (subdirectories)

Server: Has 1 direct child → forward entire batch
Client: Filter batch → keep only 10 direct children
Cache: Contains only 10 files ✓

This ensures correctness even if server-side filtering has edge cases.

Event Types

ResourceChanged (Single)

tsx
{
  ResourceChanged: {
    resource_type: "location",
    resource: {
      id: "uuid",
      name: "Photos",
      path: "/Users/me/Photos",
      // ... full resource data
    },
    metadata: {
      no_merge_fields: ["sd_path"],
      affected_paths: [],
      alternate_ids: []
    }
  }
}

ResourceChangedBatch (Multiple)

tsx
{
  ResourceChangedBatch: {
    resource_type: "file",
    resources: [
      { id: "1", name: "photo1.jpg", ... },
      { id: "2", name: "photo2.jpg", ... }
    ],
    metadata: {
      no_merge_fields: ["sd_path"],
      affected_paths: [
        { Physical: { device_slug: "mac", path: "/Desktop/photo1.jpg" } },
        { Physical: { device_slug: "mac", path: "/Desktop" } },
        { Content: { content_id: "uuid" } }
      ],
      alternate_ids: []
    }
  }
}

ResourceDeleted

tsx
{
  ResourceDeleted: {
    resource_type: "location",
    resource_id: "uuid"
  }
}

Refresh (Invalidate All)

tsx
"Refresh";

Triggers queryClient.invalidateQueries() to refetch all data.

Deep Merge Behavior

Uses ts-deepmerge for type-safe, configurable merging:

tsx
// Existing cache
{
  id: "1",
  name: "Photos",
  metadata: { size: 1024, created_at: "2024-01-01" }
}

// Incoming event (partial update)
{
  id: "1",
  name: "My Photos",
  metadata: { size: 2048 }
}

// Result after merge
{
  id: "1",
  name: "My Photos",      // Updated
  metadata: {
    size: 2048,           // Updated
    created_at: "2024-01-01"  // Preserved ✓
  }
}

No-Merge Fields

Some fields should be replaced entirely, not merged:

tsx
metadata: {
	no_merge_fields: ["sd_path"];
}

// sd_path is replaced entirely, not deep merged
// This prevents incorrect path combinations

Runtime Validation

All events are validated with Valibot before processing:

tsx
const ResourceChangedSchema = v.object({
  ResourceChanged: v.object({
    resource_type: v.string(),
    resource: v.any(),
    metadata: v.nullish(v.object({ ... }))
  })
});

// Invalid events are logged and ignored
// Prevents crashes from malformed backend data

Subscription Multiplexing

Multiple hooks with identical filters automatically share a single backend subscription:

tsx
// Component A
function LocationsList() {
  useNormalizedQuery({
    query: 'locations.list',
    resourceType: 'location',
  });
}

// Component B (mounted at same time)
function LocationsDropdown() {
  useNormalizedQuery({
    query: 'locations.list',
    resourceType: 'location',
  });
}

// Result: Only 1 backend subscription created!
// Both hooks receive events from the same connection.

How it works:

  1. First hook creates subscription with filter {resource_type: "location", library_id: "abc"}
  2. Subscription manager generates key from filter: {"resource_type":"location","library_id":"abc"}
  3. Second hook with same filter reuses existing subscription
  4. Events broadcast to all listeners
  5. When both unmount, subscription cleaned up automatically

Benefits:

  • Eliminates duplicate subscriptions during render cycles
  • Reduces backend load (fewer Unix socket connections)
  • Faster subscription setup (reuses existing connection)
  • Automatic reference counting prevents premature cleanup

Subscription Cleanup

Subscriptions are properly cleaned up when components unmount:

tsx
useEffect(() => {
	let unsubscribe: (() => void) | undefined;

	client.subscribeFiltered(filter, handleEvent).then((unsub) => {
		unsubscribe = unsub;
	});

	return () => {
		unsubscribe?.(); // Closes WebSocket subscription
	};
}, [dependencies]);

Cleanup process:

  1. React calls cleanup function
  2. Frontend stops listening to events
  3. Tauri sends Unsubscribe request to daemon
  4. Daemon closes subscription
  5. Unix socket connection closed

Result: No connection leaks, no memory leaks.

Performance

Event Reduction

Indexing 10,000 files:

Without filtering:
- Each hook receives: 10,000 events
- Total transmitted: 50,000 events (5 hooks × 10,000)
- Result: UI lag, slow

With filtering:
- Desktop hook: 100 events (1%)
- Movies hook: 500 events (5%)
- Inspector: 1-5 events (0.05%)
- Total transmitted: ~600 events
- Result: Zero lag

Connection Management

  • Multiplexing: Multiple hooks with identical filters share one backend subscription
  • Reference counting: Subscriptions cleaned up when last hook unmounts
  • Deduplication: Eliminates duplicate subscriptions during render cycles
  • Monitoring: Check client.getSubscriptionStats() for active subscriptions

Testing

Test Coverage

Rust (Backend):

  • 9/9 event filtering tests passing
  • Validates exact vs recursive modes
  • Tests all path types (Physical, Content, Cloud, Sidecar)

TypeScript (Frontend):

  • 5/5 integration tests passing
  • Uses real backend event fixtures
  • Validates filtering and cache updates
  • Proves correctness with actual production code

Run Tests

bash
# Rust tests
cargo test --test event_filtering_test

# TypeScript tests
cd packages/ts-client && bun test

# Generate new fixtures from backend
cargo test --test normalized_cache_fixtures_test

Best Practices

Always Scope File Queries

tsx
// Good
const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path, // Server filters efficiently
});

// Bad - will skip subscription
const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	// Missing pathScope! Subscription skipped to prevent overload
});

Use Correct Mode for View Type

tsx
// Directory view - exact mode
includeDescendants: false; // Only direct children

// Media gallery - recursive mode
includeDescendants: true; // All media in subtree

// Search results - recursive mode
includeDescendants: true; // All matching files

Combine with TanStack Query Options

tsx
const { data } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path,
	// TanStack Query options
	enabled: !!path,
	staleTime: 5 * 60 * 1000,
	refetchOnWindowFocus: true,
});

Advanced Usage

Content-Addressed Files

Files use Content-based sd_path but have Physical paths in alternate_paths:

tsx
// File structure
{
  sd_path: { Content: { content_id: "uuid" } },
  alternate_paths: [
    { Physical: { device_slug: "mac", path: "/Desktop/file.txt" } }
  ]
}

// Client-side filtering uses alternate_paths for path matching
// This enables deduplication while maintaining path filtering

Multiple Instances

Multiple files with same content have different IDs:

tsx
// file1.txt (original)
{ id: "1", content_identity: { uuid: "abc" } }

// file2.txt (duplicate)
{ id: "2", content_identity: { uuid: "abc" } }

// Both update when content is processed

Debugging

Enable Logging

tsx
// Check console for:
// "[useNormalizedQuery] Invalid event: ..." - Validation failures
// "[TauriTransport] Unsubscribing: ..." - Cleanup events

Monitor Subscriptions

bash
# Backend logs show subscription lifecycle
RUST_LOG=sd_core::infra::daemon::rpc=debug cargo run -p spacedrive-tauri

# Look for:
# "New subscription created: ..." - Subscription started
# "Subscription cancelled: ..." - Cleanup triggered
# "Unsubscribe sent successfully" - Connection closed

Frontend subscription stats:

tsx
import { useSpacedriveClient } from '@sd/ts-client';

function DebugPanel() {
  const client = useSpacedriveClient();
  const stats = client.getSubscriptionStats();

  console.log(`Active subscriptions: ${stats.activeSubscriptions}`);
  stats.subscriptions.forEach(sub => {
    console.log(`  ${sub.key}: ${sub.refCount} hooks, ${sub.listenerCount} listeners`);
  });
}

Inspect Cache

tsx
import { useQueryClient } from "@tanstack/react-query";

const queryClient = useQueryClient();

// View all cached queries
console.log(queryClient.getQueryCache().getAll());

// View specific query
const queryKey = ["query:files.directory_listing", libraryId, { path }];
console.log(queryClient.getQueryData(queryKey));

Migration

From useLibraryQuery

tsx
// Before (no real-time updates)
const { data } = useLibraryQuery({
	type: "locations.list",
	input: {},
});

// After (instant updates)
const { data } = useNormalizedQuery({
	query: "locations.list",
	input: null,
	resourceType: "location",
});

Backward Compatibility

The old useNormalizedCache name is aliased:

tsx
// Both work identically
import { useNormalizedQuery } from "@sd/ts-client";
import { useNormalizedCache } from "@sd/ts-client"; // Alias

// Prefer useNormalizedQuery for new code

Technical Details

Exported Functions

Core logic is exported for testing:

tsx
import {
	filterBatchResources, // Filter resources by pathScope
	updateBatchResources, // Update cache with batch
	updateSingleResource, // Update single resource
	deleteResource, // Remove from cache
	safeMerge, // Deep merge utility
	handleResourceEvent, // Event dispatcher
} from "@sd/ts-client/hooks/useNormalizedQuery";

Runtime Dependencies

  • ts-deepmerge - Type-safe deep merging
  • valibot - Runtime event validation
  • tiny-invariant - Assertion helpers
  • type-fest - TypeScript utilities
  • @tanstack/react-query - Core caching

Subscription Lifecycle

1. Component mounts
   ↓
2. useNormalizedQuery creates subscription
   ↓
3. Backend creates filtered event stream
   ↓
4. Events flow: Backend → Tauri → Frontend → Hook → Cache
   ↓
5. Component unmounts
   ↓
6. Cleanup function called
   ↓
7. Tauri cancels background task
   ↓
8. Backend receives Unsubscribe
   ↓
9. Unix socket closed
   ↓
10. Connection freed

Common Patterns

List with Real-Time Updates

tsx
const { data: items } = useNormalizedQuery({
	query: "items.list",
	input: filters,
	resourceType: "item",
});

// Items list updates instantly when:
// - New items created
// - Existing items modified
// - Items deleted

Directory with Instant File Appearance

tsx
const { data: files } = useNormalizedQuery({
	query: "files.directory_listing",
	input: { path },
	resourceType: "file",
	pathScope: path,
});

// New files appear instantly:
// - Screenshot taken → appears immediately
// - File copied → shows up without refresh
// - File renamed → updates in real-time

Inspector with Sidecar Updates

tsx
const { data: file } = useNormalizedQuery({
	query: "files.by_id",
	input: { file_id },
	resourceType: "file",
	resourceId: file_id,
});

// Sidecars update as they're generated:
// - Thumbnail generated → appears instantly
// - Thumbstrip created → shows immediately
// - OCR extracted → updates in real-time

Summary

useNormalizedQuery provides production-grade real-time caching:

  • Server-side filtering (90%+ event reduction)
  • Client-side safety (validates and filters)
  • Proper cleanup (no connection leaks)
  • Runtime validation (catches bad events)
  • Type-safe merging (preserves data)
  • Comprehensive tests (9 Rust + 5 TypeScript)
  • TanStack Query compatible (all features work)
  • Cross-device sync (instant updates everywhere)

Use it for any query where data can change and you want instant updates without manual refetching.