docs/decisions/0073-linq-based-text-search-filtering.md
The Challenge: The existing ITextSearch interface uses clause-based TextSearchFilter for filtering, which creates runtime errors from property name typos, lacks IntelliSense support, and depends on obsolete VectorSearchFilter APIs. Modern .NET practices favor LINQ expressions for type safety and compile-time validation.
The Constraint: We cannot introduce breaking changes. Existing code using TextSearchFilter must continue working.
The Question: How do we migrate ITextSearch to modern LINQ-based filtering (Expression<Func<TRecord, bool>>) while maintaining backward compatibility?
Issue: https://github.com/microsoft/semantic-kernel/issues/10456
Chosen Option: "Dual Interface Pattern". Introduce generic ITextSearch<TRecord> with LINQ filtering alongside existing ITextSearch marked [Obsolete].
We introduce ITextSearch<TRecord> (modern, LINQ-based) alongside the existing ITextSearch (legacy, marked [Obsolete]). Both interfaces coexist temporarily to provide:
This is explicitly a temporary architectural state, not a permanent design. The dual interface pattern enables non-breaking migration while establishing a clear path to remove technical debt in a future major version.
Good, because:
[Obsolete] attribute signals deprecation and guides users to modern interfaceBad, because:
FilterClause to LINQ expression trees at runtime (temporary)Key Insight: The "bad" aspects are explicitly temporary. They exist only during the migration period and will be eliminated when the legacy interface is removed in a future major version.
This section documents specific implementation choices required to realize the dual interface pattern.
The dual interface pattern creates two parallel execution paths:
┌──────────────────────────────────────────────────────────────────────────────┐
│ ITextSearch Modernization │
└──────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────┐
│ Interface Layer │
├──────────────────────────────────────────────────────────────────────────────┤
│ │
│ [Obsolete] [Modern] │
│ ITextSearch ITextSearch<TRecord> │
│ ├─ TextSearchOptions ├─ TextSearchOptions<TRecord> │
│ │ └─ TextSearchFilter │ └─ Expression<Func<T, bool>> │
│ └─ No RequiresDynamicCode └─ No RequiresDynamicCode │
│ │
└──────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────┐
│ Implementation Layer: Two Patterns │
└──────────────────────────────────────────────────────────────────────────────┘
Pattern A: Direct LINQ Passthrough Pattern B: LINQ-to-Legacy Conversion
(VectorStoreTextSearch) (BingTextSearch, GoogleTextSearch, etc.)
┌──────────────────────────────┐ ┌──────────────────────────────────┐
│ VectorStoreTextSearch │ │ BingTextSearch │
│ : ITextSearch │ │ : ITextSearch │
│ : ITextSearch<TRecord> │ │ : ITextSearch<BingWebPage> │
├──────────────────────────────┤ ├──────────────────────────────────┤
│ Legacy Path: │ │ Legacy Path: │
│ TextSearchFilter │ │ TextSearchFilter │
│ ↓ │ │ ↓ │
│ BuildFilterExpression() │ │ Bing API parameters │
│ (clause → LINQ tree) │ │ ↓ │
│ ↓ │ │ HTTP GET request │
│ VectorSearchOptions.Filter │ │ │
│ ↓ │ │ Modern Path: │
│ Vector Store │ │ Expression<Func<T, bool>> │
│ │ │ ↓ │
│ Modern Path: │ │ LINQ tree analysis │
│ Expression<Func<T, bool>> │ │ ↓ │
│ ↓ │ │ TextSearchFilter (conversion) │
│ VectorSearchOptions.Filter │ │ ↓ │
│ (direct passthrough) │ │ Delegate to legacy path │
│ ↓ │ │ │
│ Vector Store │ │ │
└──────────────────────────────┘ └──────────────────────────────────┘
Key: Both paths use Key: Modern converts to legacy
VectorSearchOptions.Filter Reuses existing implementation
Key Architectural Characteristics:
ITextSearch) and modern (ITextSearch<TRecord>)VectorSearchOptions.Filter — legacy clauses are converted to LINQ expression trees via BuildFilterExpression(), modern path passes LINQ directlyTextSearchFilter, then delegated to existing implementation[RequiresDynamicCode] attributes on either interface or implementationsAll implementations follow the dual interface pattern, but with two different execution strategies based on underlying service capabilities:
VectorStoreTextSearch uses VectorSearchOptions.Filter (LINQ) for both code paths. The legacy path converts FilterClause values to a LINQ expression tree via BuildFilterExpression() — this is pure data-structure construction and fully AOT-compatible:
#pragma warning disable CS0618 // ITextSearch is obsolete - backward compatibility
public sealed class VectorStoreTextSearch<TRecord> : ITextSearch, ITextSearch<TRecord>
#pragma warning restore CS0618
{
// ===== LEGACY PATH (Non-Generic Interface) =====
public Task<KernelSearchResults<string>> SearchAsync(
string query,
TextSearchOptions? searchOptions = null,
CancellationToken cancellationToken = default)
{
var searchResponse = ExecuteVectorSearchAsync(query, searchOptions, cancellationToken);
return Task.FromResult(CreateStringSearchResponse(searchResponse));
}
// ===== MODERN PATH (Generic Interface) =====
Task<KernelSearchResults<string>> ITextSearch<TRecord>.SearchAsync(
string query,
TextSearchOptions<TRecord>? searchOptions,
CancellationToken cancellationToken)
{
var searchResponse = ExecuteVectorSearchAsync(query, searchOptions, cancellationToken);
return Task.FromResult(CreateStringSearchResponse(searchResponse));
}
// Legacy path: Converts FilterClauses to LINQ expression tree
private async IAsyncEnumerable<VectorSearchResult<TRecord>> ExecuteVectorSearchAsync(
string query, TextSearchOptions? searchOptions, ...)
{
var vectorSearchOptions = new VectorSearchOptions<TRecord> {
Filter = searchOptions.Filter?.FilterClauses is not null
? BuildFilterExpression(searchOptions.Filter.FilterClauses)
: null,
};
// ... execute
}
// Modern path: Direct LINQ passthrough - no obsolete API
private async IAsyncEnumerable<VectorSearchResult<TRecord>> ExecuteVectorSearchAsync(
string query, TextSearchOptions<TRecord>? searchOptions, ...)
{
var vectorSearchOptions = new VectorSearchOptions<TRecord> {
Filter = searchOptions.Filter, // Direct LINQ - no conversion
};
// ... execute
}
}
BingTextSearch, GoogleTextSearch, TavilyTextSearch, BraveTextSearch convert generic interface calls to legacy format:
#pragma warning disable CS0618 // ITextSearch is obsolete
public sealed class BingTextSearch : ITextSearch, ITextSearch<BingWebPage>
#pragma warning restore CS0618
{
// ===== LEGACY PATH (Non-Generic Interface) =====
public Task<KernelSearchResults<string>> SearchAsync(
string query,
TextSearchOptions? searchOptions = null,
CancellationToken cancellationToken = default)
{
// Direct Bing API call with TextSearchFilter
// ... existing logic
}
// ===== MODERN PATH (Generic Interface) =====
Task<KernelSearchResults<string>> ITextSearch<BingWebPage>.SearchAsync(
string query,
TextSearchOptions<BingWebPage>? searchOptions,
CancellationToken cancellationToken)
{
// Convert generic options to legacy format
var legacyOptions = searchOptions != null
? ConvertToLegacyOptions(searchOptions)
: new TextSearchOptions();
// Delegate to existing legacy implementation
return this.SearchAsync(query, legacyOptions, cancellationToken);
}
// LINQ-to-TextSearchFilter conversion
private static TextSearchOptions ConvertToLegacyOptions<TRecord>(
TextSearchOptions<TRecord> genericOptions)
{
return new TextSearchOptions
{
Top = genericOptions.Top,
Skip = genericOptions.Skip,
Filter = genericOptions.Filter != null
? ConvertLinqExpressionToBingFilter(genericOptions.Filter)
: null
};
}
// Expression tree analysis and mapping to Bing API syntax
private static TextSearchFilter ConvertLinqExpressionToBingFilter<TRecord>(
Expression<Func<TRecord, bool>> linqExpression)
{
var filter = new TextSearchFilter();
// Recursively process expression tree:
// - Equality (==) → language:en
// - Inequality (!=) → -language:fr
// - Contains() → intitle:"AI" or inbody:"term"
// - AND (&&) → multiple filter clauses
ProcessExpression(linqExpression.Body, filter);
return filter;
}
}
Key Differences:
| Aspect | Pattern A (VectorStoreTextSearch) | Pattern B (Web Connectors) |
|---|---|---|
| Execution Paths | Two independent paths | Modern converts to legacy |
| Conversion Layer | NO conversion | LINQ → TextSearchFilter |
| Legacy Path | Uses obsolete VectorSearchFilter.OldFilter | Uses existing TextSearchFilter directly |
| Modern Path | Uses VectorSearchOptions.Filter directly | Converts LINQ then delegates to legacy path |
| Performance | Zero overhead (direct passthrough) | Conversion overhead acceptable (network I/O) |
| Underlying Support | Native LINQ support | API-specific parameter mapping |
Why Two Patterns?
VectorSearchOptions<TRecord>.Filter. Direct passthrough eliminates overhead.Note: Both patterns maintain dual code paths (legacy + modern) as a temporary migration strategy. Once the obsolete ITextSearch interface is removed in a future major version, only the modern LINQ path will remain, eliminating the dual implementation complexity.
Both interfaces are designed to be AOT-compatible with no [RequiresDynamicCode] attributes:
Non-Generic Interface (ITextSearch):
TextSearchFilter (clause-based, no LINQ)Generic Interface (ITextSearch<TRecord>):
[RequiresDynamicCode] attribute requiredLINQ Expression Processing:
// Simple equality - AOT-compatible
filter = doc => doc.Department == "HR" && doc.IsActive
// Complex expressions - AOT-compatible (expression tree analysis)
filter = doc => doc.Tags.Any(tag => tag.Contains("urgent"))
AOT Compatibility Matrix:
| Scenario | ITextSearch | ITextSearch<TRecord> | Notes |
|---|---|---|---|
| Simple searches (no filtering) | ✅ AOT-compatible | ✅ AOT-compatible | No dynamic code needed |
| TextSearchFilter-based | ✅ AOT-compatible | N/A | Legacy clause-based filtering |
| Simple LINQ (equality) | N/A | ✅ AOT-compatible | Expression tree analysis |
| Complex LINQ (Contains, Any) | N/A | ✅ AOT-compatible | Expression tree analysis |
Context: The ITextSearch<TRecord> interface supports LINQ expressions, including Title.Contains("value") patterns. Different search engine APIs have varying capabilities:
intitle:, inbody:, url:)orTerms for additional search terms)Decision: Implement Title.Contains() support using query enhancement for Brave and Tavily search engines:
SearchQueryFilterClause instances and append to base search querySearchQueryFilterClause differently from regular filter clausesImplementation Pattern:
// LINQ Expression: results.Where(r => r.Title.Contains("AI"))
// Converts to: new SearchQueryFilterClause("AI")
// Query Enhancement: "original query" + " AI"
Alternatives Considered:
Consequences:
Context: SearchQueryFilterClause is used only by web search connectors (Brave, Tavily) in Plugins.Web. To minimize public API surface, it should reside in the same assembly as its consumers.
Problem: FilterClause base class originally had an internal constructor, preventing inheritance outside the VectorData.Abstractions assembly:
public abstract class FilterClause
{
internal FilterClause() // ← Blocked external inheritance
}
Moving SearchQueryFilterClause to Plugins.Web failed with:
error CS0122: 'FilterClause.FilterClause()' is inaccessible due to its protection level
Decision: Make FilterClause constructor protected and move SearchQueryFilterClause to Plugins.Web as internal sealed.
// In VectorData.Abstractions
public abstract class FilterClause
{
protected FilterClause() // internal → protected
}
// In Plugins.Web
internal sealed class SearchQueryFilterClause : FilterClause
Rationale:
SearchQueryFilterClause stays internal (not public)protected allows inheritance but maintains encapsulationPlugins.Web where it's actually usedprotected constructors are common for abstract base classesAlternatives Considered:
Consequences:
SearchQueryFilterClause remains internal implementation detailDecision: Mark the original ITextSearch interface with [Obsolete] attribute immediately:
[Obsolete("ITextSearch is deprecated. Use ITextSearch<TRecord> with LINQ filtering instead.")]
public interface ITextSearch
{
// Legacy implementation
}
Purpose of Obsolete Marking:
Why Mark as Obsolete Now (rather than waiting):
This decision implements a deliberate three-phase migration path from legacy clause-based filtering to modern LINQ-based filtering:
ITextSearch<TRecord> introduced with LINQ filtering (modern, recommended)ITextSearch marked [Obsolete] with deprecation warningKey Point: Marking ITextSearch as [Obsolete] serves dual purposes:
ObsoleteAttribute with error: true)ITextSearch interface entirelyTextSearchFilter in TextSearchOptionsVectorSearchFilter.OldFilterTextSearchFilter and FilterClause types retained internally as LINQ translation layer for web plugins only; vector stores use LINQ expressions directly via VectorSearchOptions<TRecord>.FilterEstimated Timeline: Phase 2 in next major version (e.g., SK 2.0), Phase 3 in subsequent major version (e.g., SK 3.0). This gives ecosystem minimum 1-2 years to migrate.
Phase 1 (Current):
├─ Both interfaces coexist
├─ Legacy ITextSearch marked [Obsolete]
├─ Deprecation warnings guide users to ITextSearch<TRecord>
└─ All implementations support both interfaces
Phase 2 (Future):
├─ Increase deprecation severity
├─ Add removal timeline to warnings
└─ Documentation emphasizes migration
Phase 3 (Eventually):
├─ Remove ITextSearch interface
├─ Remove TextSearchFilter class
├─ Remove VectorSearchFilter.OldFilter
└─ Single interface with LINQ expressions
The dual interface pattern is explicitly a temporary architectural state, not a permanent design. It provides:
This section documents alternative approaches that were evaluated but not selected.
Replace TextSearchFilter entirely with Expression<Func<T, bool>>. Remove non-generic interface completely.
Evaluation:
Why Not Chosen: Breaking change unacceptable for stable API.
Keep both interfaces but convert TextSearchFilter to LINQ internally.
Evaluation:
Update: This option was originally rejected based on an incorrect assessment that RequiresDynamicCode would cascade to all TextSearch APIs. In fact, building expression trees (Expression.Property, Expression.Equal, Expression.Lambda) is fully AOT-compatible — only compiling expression trees (Expression.Compile()) requires dynamic code generation. Since MEVD's VectorSearchOptions<TRecord>.Filter analyzes the expression tree without compiling it, there is no AOT incompatibility. This approach was adopted in the VectorStoreTextSearch legacy path to enable MEVD to remove its obsolete OldFilter property before publishing 1.0 provider versions.
Implement generic interface as wrapper over existing implementations.
Evaluation:
Why Not Chosen: Doesn't solve the core problem of obsolete API dependency.
Deprecate TextSearchFilter and introduce LINQ alongside within same interface.
Evaluation:
Why Not Chosen: Ambiguous API design and poor developer experience.
LINQ expressions processed on server side only. No user-supplied expression execution. Expression tree analysis validates supported operations before execution. Unsupported operations throw ArgumentException with clear error messages.
No immediate breaking changes: