OpenSearch v3 Migration Implementation Plan

Goal

Migrate os-persistence-v3 from OpenSearch High-Level REST Client to opensearch-java 3.x client.

Current Status

• ✅ Module structure created
• ✅ Dependencies configured
• ✅ Config classes updated (OpenSearchProperties, OpenSearchConditions)
• ❌ DAO layer still uses old API (77+ compilation errors)
• ❌ Query builders not migrated

Migration Approach

Strategy: Incremental migration in small, testable commits.

Each commit should:

1. Compile successfully
2. Pass existing tests
3. Be reviewable independently

Phase 1: Foundation (Days 1-2)

Commit 1: Client Infrastructure

File: OpenSearchRestDAO.java (constructor + client init)

Tasks:

• Add new OpenSearchClient field
• Keep old RestHighLevelClient temporarily (dual-client mode)
• Add client initialization in constructor
• Add Jackson JSON mapper setup
• Add client close() method

Test: Verify server starts without errors

Commit 2: Query Builder Abstraction

New file: QueryHelper.java

Tasks:

• Create helper class for query building
• Implement buildBoolQuery(String structured, String freeText) → returns Query
• Implement buildMatchQuery(String field, String value) → returns Query
• Implement buildRangeQuery(String field, Object from, Object to) → returns Query
• Add unit tests for query building

Test: Unit tests pass

Phase 2: Search Operations (Days 3-4)

Commit 3: Core Search Method

File: OpenSearchRestDAO.java (new method)

Tasks:

• Create NEW method: searchObjectsV3(...) using new client
• Implement query building with lambda builders
• Implement sorting with new API
• Implement pagination
• Map results to SearchResult<T>

Test: Add integration test comparing v2 vs v3 search results

Commit 4: Migrate Search Methods (One at a Time)

Files: OpenSearchRestDAO.java

Order:

1. searchObjectsViaExpression() - use searchObjectsV3()
2. searchWorkflowSummary() - use searchObjectsV3()
3. searchTaskSummary() - use searchObjectsV3()

Test: Integration tests pass for each method

Commit 5: Count Operation

File: OpenSearchRestDAO.java

Tasks:

• Create countDocuments(String index, Query query) helper
• Update all count operations to use new method
• Fix CountResponse.count() vs old getCount()

Test: Count queries return correct values

Phase 3: Index Operations (Days 5-6)

Commit 6: Index/Update Operations

File: OpenSearchRestDAO.java

Tasks:

• Create indexDocumentV3(String index, String id, Object doc) helper
• Migrate indexObject() to use new method
• Migrate updateObject() to use new method
• Handle async updates

Test: Document indexing works correctly

Commit 7: Delete Operations

File: OpenSearchRestDAO.java

Tasks:

• Create deleteDocumentV3(String index, String id) helper
• Migrate deleteObject() to use new method

Test: Document deletion works correctly

Commit 8: Bulk Operations

File: OpenSearchRestDAO.java

Tasks:

• Create BulkHelper.java for bulk operation building
• Migrate bulkIndexObjects() to use lambda builders
• Migrate asyncBulkIndexObjects() to use lambda builders
• Update BulkProcessor initialization (if needed)

Test: Bulk operations work correctly

Phase 4: Specialized Operations (Day 7)

Commit 9: Task Logs

File: OpenSearchRestDAO.java

Tasks:

• Migrate addTaskExecutionLogs() to new API
• Migrate getTaskExecutionLogs() to new API

Test: Task logs index and retrieve correctly

Commit 10: Event Messages

File: OpenSearchRestDAO.java

Tasks:

• Migrate addMessage() to new API
• Migrate getMessages() to new API

Test: Event messages work correctly

Phase 5: Cleanup & Optimization (Day 8)

Commit 11: Remove Old Client

File: OpenSearchRestDAO.java

Tasks:

• Remove RestHighLevelClient field
• Remove dual-client code paths
• Clean up unused imports
• Remove old API dependencies from build.gradle (if possible)

Test: All tests still pass

Commit 12: Query Parser Migration

Files: os3/dao/query/parser/**/*.java

Tasks:

• Update Expression.java to use Query instead of QueryBuilder
• Update NameValue.java query building
• Update GroupedExpression.java query building
• Fix FilterProvider interface

Test: Query parsing works correctly

Commit 13: Spotless & Documentation

Tasks:

• Run Spotless formatting
• Update JavaDocs to reference new API
• Update README with migration notes
• Update build.gradle comments

Test: Build passes with no warnings

Phase 6: Testing & Validation (Days 9-10)

Commit 14: Integration Test Suite

New file: OpenSearchRestDAOV3IntegrationTest.java

Tasks:

• Test all CRUD operations
• Test search with complex queries
• Test bulk operations
• Test sorting and pagination
• Test task logs
• Test event messages
• Compare results with v2

Test: All integration tests pass

Commit 15: Side-by-Side Comparison

New file: os-persistence-v3/src/test/resources/comparison-tests.json

Tasks:

• Run test workflows on both v2 and v3
• Compare indexed documents
• Compare search results
• Document any differences

Test: No behavioral differences detected

Detailed Work Breakdown

Critical Files to Modify

1. OpenSearchRestDAO.java (~1343 lines)

   • Core DAO implementation
   • ~25 methods to migrate
   • Estimated: 20 hours

2. Query Parser Files (~300 lines total)

   • Expression.java
   • NameValue.java
   • GroupedExpression.java
   • FilterProvider.java
   • Estimated: 4 hours

3. Helper Classes (new)

   • QueryHelper.java (query building)
   • BulkHelper.java (bulk operations)
   • Estimated: 4 hours

4. Test Files (new)

   • Integration tests
   • Comparison tests
   • Estimated: 8 hours

Dependencies to Add/Remove

Keep:

implementation 'org.opensearch.client:opensearch-java:3.0.0'
implementation "org.opensearch.client:opensearch-rest-client:3.0.0"

Remove (after migration):

implementation "org.opensearch.client:opensearch-rest-high-level-client:3.0.0"

Risk Mitigation

Risk 1: Breaking API Changes

Mitigation: Side-by-side testing with v2

Risk 2: Performance Regression

Mitigation: Benchmark tests before/after

Risk 3: Serialization Issues

Mitigation: Test with real workflow data early

Risk 4: Unknown API Differences

Mitigation: Iterative approach, test each commit

Timeline Estimate

Optimistic: 1 week (40 hours) Realistic: 2 weeks (60-80 hours) Pessimistic: 3 weeks (if major blockers found)

Breakdown:

• Foundation: 2 days
• Search: 2 days
• CRUD: 2 days
• Specialized: 1 day
• Cleanup: 1 day
• Testing: 2 days
• Total: 10 working days

Success Criteria

• os-persistence-v3 compiles with 0 errors
• All unit tests pass
• All integration tests pass
• Side-by-side comparison shows identical behavior
• Performance within 10% of v2
• No deprecated API usage
• Code review approved
• Documentation updated

Next Steps

1. Start with Commit 1 (Client Infrastructure)
2. Create feature branch: feature/os-persistence-v3-migration
3. Work through commits sequentially
4. Test after each commit
5. Create PR when Phase 1-3 complete (minimal viable functionality)
6. Complete Phase 4-6 based on feedback

Questions to Resolve

1. Do we need to maintain backward compatibility with v2 config?
2. Should we support rolling upgrades from v2 to v3?
3. What's the deprecation timeline for v2?
4. Do we need separate Docker images for v2 vs v3?

Resources

• Migration Guide: os-persistence-v3/MIGRATION_GUIDE.md
• OpenSearch Java Docs: https://opensearch.org/docs/latest/clients/java/
• Example Code: https://github.com/opensearch-project/opensearch-java/tree/main/samples