Wekan Persistence Architecture Improvements

Changes Implemented

This document describes the architectural improvements made to Wekan's persistence layer to ensure proper separation between board-level data and per-user UI preferences.

1. Removed Board-Level UI State (✅ COMPLETED)

1.1 Collapsed State Removed from Schemas

Changes:

• ❌ Removed collapsed field from Swimlanes schema (models/swimlanes.js)
• ❌ Removed collapsed field from Lists schema (models/lists.js)
• ❌ Removed collapse() mutation from Swimlanes
• ❌ Removed collapsed field from REST API PUT /api/boards/:boardId/lists/:listId

Rationale: Collapsed state is a per-user UI preference and should never be stored at the board level. This prevents conflicts where one user collapses a swimlane/list and affects all other users.

Migration:

• Existing board-level collapsed values will be ignored
• Users' personal collapse preferences are stored in profile.collapsedSwimlanes and profile.collapsedLists
• For non-logged-in users, collapse state is stored in localStorage and cookies

2. LocalStorage Validation & Cleanup (✅ COMPLETED)

2.1 New Validation Utility

File: client/lib/localStorageValidator.js

Features:

• ✅ Validates all numbers (swimlane heights, list widths) are within valid ranges
• ✅ Validates all booleans (collapse states) are actual boolean values
• ✅ Removes corrupted or invalid data
• ✅ Limits stored data to prevent localStorage bloat:
  • Maximum 50 boards per key
  • Maximum 100 items per board
• ✅ Auto-cleanup on app startup (once per day)
• ✅ Validation ranges:
  • List widths: 100-1000 pixels
  • Swimlane heights: -1 (auto) or 50-2000 pixels
  • Collapsed states: boolean only

Usage:

import { validateAndCleanLocalStorage, shouldRunCleanup } from '/client/lib/localStorageValidator';

// Auto-runs on startup
Meteor.startup(() => {
  if (shouldRunCleanup()) {
    validateAndCleanLocalStorage();
  }
});

2.2 Updated User Storage Methods

File: models/lib/userStorageHelpers.js

Functions:

• getValidatedNumber(key, boardId, itemId, defaultValue, min, max) - Get with validation
• setValidatedNumber(key, boardId, itemId, value, min, max) - Set with validation
• getValidatedBoolean(key, boardId, itemId, defaultValue) - Get boolean
• setValidatedBoolean(key, boardId, itemId, value) - Set boolean

Validation Applied To:

• wekan-list-widths - List column widths
• wekan-list-constraints - List max-width constraints
• wekan-swimlane-heights - Swimlane row heights
• wekan-collapsed-lists - List collapse states
• wekan-collapsed-swimlanes - Swimlane collapse states

3. Per-User Position History System (✅ COMPLETED)

3.1 New Collection: UserPositionHistory

File: models/userPositionHistory.js

Purpose: Track all position changes (moves, reorders) per user with full undo/redo support.

Schema Fields:

• userId - User who made the change
• boardId - Board where change occurred
• entityType - Type: 'swimlane', 'list', 'card', 'checklist', 'checklistItem'
• entityId - ID of the moved entity
• actionType - Type: 'move', 'create', 'delete', 'restore', 'archive'
• previousState - Complete state before change (blackbox object)
• newState - Complete state after change (blackbox object)
• previousSort, newSort - Sort positions
• previousSwimlaneId, newSwimlaneId - Swimlane changes
• previousListId, newListId - List changes
• previousBoardId, newBoardId - Board changes
• isCheckpoint - User-marked savepoint
• checkpointName - Name for the savepoint
• batchId - Group related changes together
• createdAt - Timestamp

Key Features:

• ✅ Automatic tracking of all card movements
• ✅ Per-user isolation (users only see their own history)
• ✅ Checkpoint/savepoint system for marking important states
• ✅ Batch operations support (group related changes)
• ✅ Auto-cleanup (keeps last 1000 entries per user per board)
• ✅ Checkpoints are never deleted
• ✅ Full undo capability if entity still exists

Helpers:

• getDescription() - Human-readable change description
• canUndo() - Check if change can be undone
• undo() - Reverse the change

Indexes:

{ userId: 1, boardId: 1, createdAt: -1 }
{ userId: 1, entityType: 1, entityId: 1 }
{ userId: 1, isCheckpoint: 1 }
{ batchId: 1 }
{ createdAt: 1 }

3.2 Meteor Methods for History Management

Available Methods:

// Create a checkpoint/savepoint
Meteor.call('userPositionHistory.createCheckpoint', boardId, checkpointName);

// Undo a specific change
Meteor.call('userPositionHistory.undo', historyId);

// Get recent changes
Meteor.call('userPositionHistory.getRecent', boardId, limit);

// Get all checkpoints
Meteor.call('userPositionHistory.getCheckpoints', boardId);

// Restore to a checkpoint (undo all changes after it)
Meteor.call('userPositionHistory.restoreToCheckpoint', checkpointId);

3.3 Automatic Tracking Integration

Card Moves: models/cards.js

The card.move() method now automatically tracks changes:

// Capture previous state
const previousState = {
  boardId: this.boardId,
  swimlaneId: this.swimlaneId,
  listId: this.listId,
  sort: this.sort,
};

// After update, track in history
UserPositionHistory.trackChange({
  userId: Meteor.userId(),
  boardId: this.boardId,
  entityType: 'card',
  entityId: this._id,
  actionType: 'move',
  previousState,
  newState: { boardId, swimlaneId, listId, sort },
});

TODO: Add similar tracking for:

• List reordering
• Swimlane reordering
• Checklist/item reordering

4. SwimlaneId Validation & Rescue (✅ COMPLETED)

4.1 Migration: Ensure Valid Swimlane IDs

File: server/migrations/ensureValidSwimlaneIds.js

Purpose: Ensure all cards and lists have valid swimlaneId references, rescuing orphaned data.

Operations:

1. Fix Cards Without SwimlaneId

   • Finds cards with missing/null/empty swimlaneId
   • Assigns to board's default swimlane
   • Creates default swimlane if none exists

2. Fix Lists Without SwimlaneId

   • Finds lists with missing swimlaneId
   • Sets to empty string (for backward compatibility)

3. Rescue Orphaned Cards

   • Finds cards where swimlaneId points to deleted swimlane
   • Creates "Rescued Data (Missing Swimlane)" swimlane (red color, at end)
   • Moves orphaned cards there
   • Logs activity for transparency

4. Add Validation Hooks

   • Cards.before.insert - Auto-assign default swimlaneId
   • Cards.before.update - Prevent swimlaneId removal
   • Ensures swimlaneId is ALWAYS saved

Migration Tracking: Stored in migrations collection:

{
  name: 'ensure-valid-swimlane-ids',
  version: 1,
  completedAt: Date,
  results: {
    cardsFixed: Number,
    listsFixed: Number,
    cardsRescued: Number,
  }
}

5. TODO: Undo/Redo UI (⏳ IN PROGRESS)

5.1 Planned UI Components

Board Toolbar:

• Undo button (with keyboard shortcut Ctrl+Z)
• Redo button (with keyboard shortcut Ctrl+Shift+Z)
• History dropdown showing recent changes
• "Create Checkpoint" button

History Sidebar:

• List of recent changes with descriptions
• Visual timeline
• Checkpoint markers
• "Restore to This Point" buttons
• Search/filter history

5.2 Keyboard Shortcuts

// To implement in client/lib/keyboard.js
Mousetrap.bind('ctrl+z', () => {
  // Undo last change
});

Mousetrap.bind('ctrl+shift+z', () => {
  // Redo last undone change
});

Mousetrap.bind('ctrl+shift+s', () => {
  // Create checkpoint
});

6. TODO: Search History Feature (⏳ NOT STARTED)

6.1 Requirements

Per the user request:

"For board-level data, for each field (like description, comments etc) at Search All Boards have translatable options to also search from history of boards where user is member of board"

6.2 Proposed Implementation

New Collection: FieldHistory

{
  boardId: String,
  entityType: String, // 'card', 'list', 'swimlane', 'board'
  entityId: String,
  fieldName: String, // 'description', 'title', 'comments', etc.
  previousValue: String,
  newValue: String,
  changedBy: String, // userId
  changedAt: Date,
}

Search Enhancement:

• Add "Include History" checkbox to Search All Boards
• Search not just current field values, but also historical values
• Show results with indicator: "Found in history (changed 2 days ago)"
• Allow filtering by:
  • Current values only
  • Historical values only
  • Both current and historical

Translatable Field Options:

const searchableFieldsI18n = {
  'card-title': 'search-card-titles',
  'card-description': 'search-card-descriptions',
  'card-comments': 'search-card-comments',
  'list-title': 'search-list-titles',
  'swimlane-title': 'search-swimlane-titles',
  'board-title': 'search-board-titles',
  // Add i18n keys for each searchable field
};

6.3 Storage Considerations

Challenge: Field history can grow very large

Solutions:

1. Only track fields explicitly marked for history
2. Limit history depth (e.g., last 100 changes per field)
3. Auto-delete history older than X months (configurable)
4. Option to disable per board

Suggested Settings:

{
  enableFieldHistory: true,
  trackedFields: ['description', 'title', 'comments'],
  historyRetentionDays: 90,
  maxHistoryPerField: 100,
}

7. Data Validation Summary

7.1 Validation Applied

7.2 Auto-Cleanup Rules

LocalStorage:

• Corrupted data → Removed
• Invalid types → Removed
• Out-of-range values → Removed
• Excess boards (>50) → Oldest removed
• Excess items per board (>100) → Oldest removed
• Cleanup frequency → Daily (if needed)

UserPositionHistory:

• Keeps last 1000 entries per user per board
• Checkpoints never deleted
• Cleanup frequency → Daily
• Old entries (beyond 1000) → Deleted

8. Migration Guide

8.1 For Existing Installations

Automatic Migrations:

1. ✅ ensureValidSwimlaneIds - Runs automatically on server start
2. ✅ LocalStorage cleanup - Runs automatically on client start (once/day)

Manual Actions Required:

• None - all migrations are automatic

8.2 For Developers

When Adding New Per-User Preferences:

1. Add field to user profile schema:

'profile.myNewPreference': {
  type: Object,
  optional: true,
  blackbox: true,
}

2. Add validation function:

function validateMyNewPreference(data) {
  // Validate structure
  // Return cleaned data
}

3. Add localStorage support:

getMyNewPreferenceFromStorage(boardId, itemId) {
  if (this._id) {
    return this.getMyNewPreference(boardId, itemId);
  }
  return getValidatedData('wekan-my-preference', validators.myPreference);
}

4. Add to cleanup routine in localStorageValidator.js

9. Testing Checklist

9.1 Manual Testing

• Collapse swimlane → Reload → Should remain collapsed (logged-in)
• Collapse list → Reload → Should remain collapsed (logged-in)
• Resize list width → Reload → Should maintain width (logged-in)
• Resize swimlane height → Reload → Should maintain height (logged-in)
• Logout → Collapse swimlane → Reload → Should remain collapsed (cookies)
• Move card → Check UserPositionHistory created
• Move card → Click undo → Card returns to original position
• Create checkpoint → Move cards → Restore to checkpoint → Cards return
• Corrupted localStorage → Should be cleaned on next startup
• Card without swimlaneId → Should be rescued to rescue swimlane

9.2 Automated Testing

Unit Tests Needed:

• localStorageValidator.js - All validation functions
• userStorageHelpers.js - Get/set functions
• userPositionHistory.js - Undo logic
• ensureValidSwimlaneIds.js - Migration logic

Integration Tests Needed:

• Card move triggers history entry
• Undo actually reverses move
• Checkpoint restore works correctly
• localStorage validation on startup
• Rescue migration creates rescue swimlane

10. Performance Considerations

10.1 Indexes Added

// UserPositionHistory
{ userId: 1, boardId: 1, createdAt: -1 }
{ userId: 1, entityType: 1, entityId: 1 }
{ userId: 1, isCheckpoint: 1 }
{ batchId: 1 }
{ createdAt: 1 }

10.2 Query Optimization

• UserPositionHistory queries limited to 100 results max
• Auto-cleanup prevents unbounded growth
• Checkpoints indexed separately for fast retrieval

10.3 localStorage Limits

• Maximum 50 boards per key (prevents quota exceeded)
• Maximum 100 items per board
• Daily cleanup of excess data

11. Security Considerations

11.1 User Isolation

• ✅ UserPositionHistory isolated per-user (userId filter on all queries)
• ✅ Users can only undo their own changes
• ✅ Checkpoints are per-user
• ✅ History never shared between users

11.2 Validation

• ✅ All localStorage data validated before use
• ✅ Number ranges enforced
• ✅ Type checking on all inputs
• ✅ Invalid data rejected (not just sanitized)

11.3 Authorization

• ✅ Must be board member to create history entries
• ✅ Must be board member to undo changes
• ✅ Cannot undo other users' changes

12. Future Enhancements

12.1 Planned Features

1. Field-Level History

   • Track changes to card descriptions, titles, comments
   • Search across historical values
   • "What was this card's description last week?"

2. Collaborative Undo

   • See other users' recent changes
   • Undo with conflict resolution
   • Merge strategies for simultaneous changes

3. Export History

   • Export position history to CSV/JSON
   • Audit trail for compliance
   • Analytics on card movement patterns

4. Visual Timeline

   • Interactive timeline of board changes
   • Playback mode to see board evolution
   • Heatmap of frequently moved cards

12.2 Optimization Opportunities

1. Batch Operations

   • Group multiple card moves into single history entry
   • Reduce database writes

2. Compression

   • Compress old history entries
   • Store diffs instead of full states

3. Archival

   • Move very old history to separate collection
   • Keep last N months in hot storage

Document History

• Created: 2025-12-23
• Last Updated: 2025-12-23
• Status: Implementation In Progress
• Completed: Sections 1-4
• In Progress: Section 5-6
• Planned: Section 6.1-6.3