QueryBuilder

The QueryBuilder interface provides a fluent, chainable API for constructing complex queries against your LocalVectorDB database. This approach offers a more structured and readable alternative to dictionary-based filters, especially for complex query conditions.

Core Concepts

The QueryBuilder pattern allows you to:

1. Chain multiple conditions with logical operators

2. Build complex nested queries

3. Mix different search types (vector, keyword, hybrid)

4. Apply metadata filters with precise control

5. Customize result handling and sorting

Basic Usage

Creating a QueryBuilder

from localvectordb import VectorDB

# Initialize your database
db = VectorDB("my_database", "./vector_data")

# Create a query builder from the database instance
query = db.query_builder()

# The following is equivalent:
from localvectordb.query_builder import QueryBuilder
query = QueryBuilder(db)

Simple Filters

# Find documents where category equals "tech"
results = (
    db.query_builder()
    .filter("category", "tech")  # Exact match on category field
    .execute()
)

# Find documents with priority of 3 or higher
results = (
    db.query_builder()
    .filter("priority", gte_=3)  # priority >= 3
    .execute()
)

# Multiple conditions combined with AND logic
results = (
    db.query_builder()
    .filter(category="tech", priority=5)  # category = "tech" AND priority = 5
    .execute()
)

Search Operations

# Perform vector search (semantic similarity)
results = (
    db.query_builder()
    .search("machine learning algorithms", search_type="vector")  # Find semantically similar content
    .execute()
)

# Perform keyword search (exact text matching)
results = (
    db.query_builder()
    .keyword("python programming")  # Look for exact keyword matches
    .execute()
)

# Combine vector and keyword search with custom weights
results = (
    db.query_builder()
    .hybrid("neural networks", vector_weight=0.5)  # 50% semantic similarity, 50% keyword matching
    .execute()
)

# Search within a specific metadata field (case-insensitive "contains" for
# string values, exact match otherwise) rather than the document content
results = (
    db.query_builder()
    .search_field("title", "transformer")  # title ILIKE '%transformer%'
    .execute()
)

Combining Search with Filters

# Search for documents about ML algorithms that are in the tech category and high priority
results = (
    db.query_builder()
    .search("machine learning algorithms")  # Find semantically similar documents
    .filter("category", "tech")  # Narrow down to tech category
    .filter("priority", gte_=3)  # Only include high priority (3+) documents
    .limit(5)  # Return only top 5 results
    .execute()
)

Note

Filters are pushed into the search, not applied afterwards. When you combine a metadata filter with a search, the filter is evaluated inside the vector and keyword search (a FAISS id-selector over the matching documents, or a subquery on the full-text index) so that a filter matching only a small fraction of the corpus still returns a full page of matches. A selective filter – say one that matches 0.1% of documents – therefore returns its best matches rather than only whatever happened to fall in the first candidate window.

This applies to query(), query_async(), the streaming cursor, and the section/document hierarchical searches. Two cases fall back to post-filtering a fixed candidate pool and log a warning if fewer than k results survive: a database built with faiss_index_type="IndexLSH" (which cannot take a selector – prefer a flat index for filtered search) and a filter that SQL cannot express, such as dot-notation into a JSON field ("author.name"). Results are always correct; only these two cases may under-return on a very selective filter.

Limiting and Pagination

# Limit to 10 results
results = (
    db.query_builder()
    .search("machine learning")
    .limit(10)  # Only return the top 10 most relevant results
    .execute()
)

# Pagination example - first page (results 1-20)
results_page1 = (
    db.query_builder()
    .filter(category="tech")  # Filter by category
    .limit(20)  # Page size of 20
    .offset(0)  # Start at the beginning (first page)
    .execute()
)

# Pagination example - second page (results 21-40)
results_page2 = (
    db.query_builder()
    .filter(category="tech")  # Same filter as above
    .limit(20)  # Same page size
    .offset(20)  # Skip the first 20 results (move to second page)
    .execute()
)

Advanced Usage

Semantic Filtering

Semantic filtering allows you to refine a search by other fields. This can be particularly useful when you have rich metadata, since you can do semantic matching of concepts across fields other than content.

For example, suppose you had a library of scientific publications where details about the methodology are extracted, and you want only papers about NLP using unsupervised techniques:

# Find documents about NLP that use unsupervised learning approaches
results = (
    db.query_builder()
    .search("natural language processing")  # Base query for NLP documents
    .semantic_filter(
        field="methodology",  # Look at the methodology field
        concept="unsupervised learning",  # Find semantic similarity to this concept
        threshold=0.75  # Must be at least 75% similar to match
    )
    .execute()
)

semantic_filter also accepts a metric argument selecting how similarity is measured. Pass a SimilarityMetric value — COSINE (default), EUCLIDEAN, DOT_PRODUCT, or MANHATTAN:

from localvectordb.query_builder import SimilarityMetric

results = (
    db.query_builder()
    .search("natural language processing")
    .semantic_filter(
        field="methodology",
        concept="unsupervised learning",
        threshold=0.75,
        metric=SimilarityMetric.DOT_PRODUCT,
    )
    .execute()
)

Return Type Customization

# Return the individual chunks instead of complete documents
results = (
    db.query_builder()
    .search("specific technical detail")
    .chunks()  # Return individual text chunks that match
    .execute()
)

# Return matching chunks with surrounding context
results = (
    db.query_builder()
    .search("neural network architecture")
    .context(window_size=3)  # Include 3 chunks before and after each match
    .execute()
)

# Control how document scores are calculated from multiple matching chunks
results = (
    db.query_builder()
    .search("machine learning")
    .documents(scoring_method="average")  # Use the mean of the matching chunk scores
    .execute()
)

Hierarchical Search

When the database was created with hierarchical_embeddings=True, two extra builder methods select the retrieval granularity (see Hierarchical Embeddings):

• .search_level(level, section_weight=None) — which retrieval level to query: "chunks" (default), "sections", "documents", or "fused". For "fused", section_weight (0–1) tunes the blend of chunk and section signal. See Hierarchical Embeddings.

• .sections() — return section-level results (type="section").

# Find the most relevant sections, not individual chunks
results = (
    db.query_builder()
    .search("network timeout errors")
    .search_level("sections")
    .filter("product", "gateway")
    .limit(5)
    .execute()
)

Ordering Results

# Order results by publication date (descending by default, newest first)
results = (
    db.query_builder()
    .filter(category="tech")
    .order_by("publish_date")  # Default direction is "desc" -> newest to oldest
    .execute()
)

# Order results by publication date in ascending order (oldest first)
results = (
    db.query_builder()
    .filter(category="tech")
    .order_by("publish_date", "asc")  # Sort from oldest to newest
    .execute()
)

# Order search results by relevance score
results = (
    db.query_builder()
    .search("artificial intelligence")
    .order_by_score("desc")  # Show most relevant results first
    .execute()
)

# Apply multiple ordering criteria (first by year, then by citations)
results = (
    db.query_builder()
    .search("research papers")
    .order_by("year", "desc")  # First sort by most recent year
    .order_by("citations", "desc")  # For papers from same year, sort by most citations
    .execute()
)

Grouping and Aggregation

# Group search results by category field
results = (
    db.query_builder()
    .search("machine learning")
    .group_by("category")  # Group matching documents by their category
    .execute()
)

# Count documents by category for documents from 2024
results = (
    db.query_builder()
    .filter(year=2024)  # Only include documents from 2024
    .group_by("category")  # Group by category
    .count_by("*", "document_count")  # Count documents in each group, call the results 'document_count'
    .execute()
)

# Calculate average priority by category and filter to high-priority categories
results = (
    db.query_builder()
    .group_by("category")  # Group by category
    .avg_by("priority", "avg_priority")  # Calculate average priority for each category
    .having("avg_priority", "gt", 3)  # Only include categories with average priority > 3
    .execute()
)

The full set of aggregation helpers mirrors SQL: count_by, sum_by, avg_by, min_by, and max_by. Each takes a field and an optional alias (defaulting to e.g. sum_<field>):

# Total and lowest word_count per category
results = (
    db.query_builder()
    .group_by("category")
    .sum_by("word_count", "total_words")   # SUM(word_count)
    .min_by("word_count", "shortest")      # MIN(word_count)
    .execute()
)

For HAVING on a count aggregation, having_count(operator, value, alias="count") is a shorthand for having(alias, operator, value). HAVING clauses support only the comparison operators eq, ne, gt, gte, lt, and lte:

# Only keep categories with more than 5 documents
results = (
    db.query_builder()
    .group_by("category")
    .count_by("*", "count")
    .having_count("gt", 5)
    .execute()
)

Result Reranking

The QueryBuilder provides powerful reranking capabilities to improve search relevance by reordering results based on various criteria. This is particularly useful for enhancing the quality of search results beyond simple relevance scoring.

Basic Reranking

Rerank results using different strategies:

# Rerank by recency (newer documents ranked higher)
results = (
    db.query_builder()
    .search("machine learning research")
    .rerank_by_recency(date_field="publish_date", weight=0.3)  # 30% weight to recency
    .execute()
)

# Rerank by diversity (promote variety in specified field)
results = (
    db.query_builder()
    .search("AI applications")
    .rerank_by_diversity(field="category", weight=0.5)  # Promote diverse categories
    .execute()
)

Cross-Encoder Reranking

Use a cross-encoder or reranking model to re-score results for higher relevance. Cross-encoders evaluate query-document pairs jointly, producing more accurate relevance scores than bi-encoder (embedding) similarity alone.

# Rerank using a local cross-encoder model (requires sentence-transformers)
results = (
    db.query_builder()
    .search("machine learning optimization")
    .rerank_by_model(
        provider="sentence_transformers",
        model="cross-encoder/ms-marco-MiniLM-L-6-v2",
        top_k=10
    )
    .execute()
)

# Rerank using Jina AI reranker API
results = (
    db.query_builder()
    .search("neural network architectures")
    .rerank_by_model(
        provider="jina",
        model="jina-reranker-v2-base-multilingual",
        top_k=5
    )
    .execute()
)

# Rerank using HuggingFace Inference API
results = (
    db.query_builder()
    .search("deep learning")
    .rerank_by_model(
        provider="huggingface",
        model="BAAI/bge-reranker-v2-m3"
    )
    .execute()
)

Cross-encoder reranking can also be applied directly via the query() method:

# Using reranker_config dict
results = db.query(
    "machine learning",
    reranker_config={
        "provider": "mock",
        "model": "mock-reranker"
    }
)

# Or pass a reranker instance
from localvectordb.reranking import create_reranker

reranker = create_reranker("sentence_transformers")
results = db.query("machine learning", reranker=reranker)

Over-fetching for reranking (rerank_k)

A reranker can only improve results it actually sees. When a reranker is configured, query() fetches a larger candidate pool – rerank_k, default 5 * k – re-scores that pool, and returns the top k. Without this over-fetch the reranker would only ever reorder the k results the search already selected, so it could never pull a stronger match up from just below the cutoff.

# Fetch 100 candidates, rerank them, return the best 10.
results = db.query(
    "machine learning",
    k=10,
    rerank_k=100,
    reranker_config={"provider": "jina", "model": "jina-reranker-v2-base-multilingual"},
)

rerank_k is clamped to at most 200 (a cross-encoder pass costs one model call per candidate) and is never smaller than k. It has no effect when no reranker is supplied. The same parameter is accepted by query_async() and by the HTTP API. Reranking is not available on the streaming/cursor path – use query()/query_async() when a reranker is configured.

Note

Each reranker leaves result.score as an absolute value in [0, 1] (comparable across queries and usable with score_threshold), and preserves the model’s raw output in result.metadata["rerank_raw_score"] and the pre-rerank search score in result.metadata["original_score"]. Jina uses its API’s native relevance score; the SentenceTransformers and HuggingFace cross-encoders squash their logits with a logistic sigmoid. None uses a per-batch min-max, which would be pool-relative rather than absolute.

Reranker Provider Options

The built-in providers accept a few construction options (passed either through reranker_config or as keyword arguments to create_reranker):

• All rerankers accept timeout (seconds, default 90) and max_retries (default 3).

• sentence_transformers accepts device (e.g. "cpu", "cuda", "mps") to select where the local cross-encoder runs.

• huggingface accepts base_url to target a self-hosted Text-Embeddings- Inference / custom endpoint instead of the hosted Inference API.

from localvectordb.reranking import create_reranker, list_rerankers

print(list_rerankers())   # ['jina', 'sentence_transformers', 'huggingface', 'mock', ...]

reranker = create_reranker(
    "sentence_transformers",
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
    device="cuda",
)

Custom Rerankers

Rerankers use the same plugin architecture as embedding providers. Subclass Reranker, then either register it in-process or expose it as an entry point:

from localvectordb.reranking import Reranker, RerankerRegistry

class MyReranker(Reranker):
    ...

# In-process registration
RerankerRegistry.register("my_reranker", MyReranker)
reranker = create_reranker("my_reranker")

To make a reranker discoverable across installs, add an entry point under the localvectordb.reranker_providers group in your pyproject.toml:

[project.entry-points."localvectordb.reranker_providers"]
my_reranker = "my_package.rerankers:MyReranker"

Advanced Reranking Configuration

Use the generic rerank() method for more control:

# Custom recency reranking with specific parameters
results = (
    db.query_builder()
    .search("recent developments")
    .rerank("recency",
            date_field="updated_at",
            weight=0.4,
            decay_factor=0.95)  # How quickly recency importance decays
    .execute()
)

# Diversity reranking with threshold
results = (
    db.query_builder()
    .search("research papers")
    .rerank("diversity",
            field="author",
            weight=0.6,
            max_per_group=2)  # Maximum 2 papers per author
    .execute()
)

Combining Multiple Reranking Strategies

Apply multiple reranking strategies in sequence:

# First promote diversity, then favor recent content
results = (
    db.query_builder()
    .search("technology trends")
    .filter("category", "tech")
    .rerank_by_diversity(field="subcategory", weight=0.3)  # Promote diverse subcategories
    .rerank_by_recency(date_field="publish_date", weight=0.2)  # Then favor recent
    .order_by("score", "desc")  # Final ordering by combined score
    .limit(20)
    .execute()
)

Use Cases for Reranking

Recency Reranking - Ideal for:

• News and content feeds where freshness matters

• Documentation where newer versions are preferred

• Research databases prioritizing recent findings

• Time-sensitive information retrieval

# News feed with recency boost
news_results = (
    db.query_builder()
    .search("economic forecast")
    .filter("type", "news")
    .rerank_by_recency(
        date_field="publish_date",
        weight=0.5  # Strong recency preference
    )
    .limit(10)
    .execute()
)

Diversity Reranking - Ideal for:

• Avoiding echo chambers by promoting varied perspectives

• Ensuring broad coverage across categories or sources

• Reducing redundancy in search results

• Providing comprehensive overviews of topics

# Diverse perspectives on a topic
diverse_results = (
    db.query_builder()
    .search("climate change solutions")
    .rerank_by_diversity(
        field="source_type",  # Mix academic, news, government sources
        weight=0.4
    )
    .limit(15)
    .execute()
)

Understanding Reranking Impact

Monitor how reranking affects your results:

# Compare results with and without reranking
base_results = (
    db.query_builder()
    .search("artificial intelligence")
    .limit(10)
    .execute()
)

reranked_results = (
    db.query_builder()
    .search("artificial intelligence")
    .rerank_by_recency(date_field="publish_date", weight=0.3)
    .limit(10)
    .execute()
)

# Analyze the differences
print("Base results order:")
for i, result in enumerate(base_results):
    print(f"{i+1}. {result.metadata.get('title', 'No title')} "
          f"(score: {result.score:.3f}, date: {result.metadata.get('publish_date')})")

print("\nReranked results order:")
for i, result in enumerate(reranked_results):
    print(f"{i+1}. {result.metadata.get('title', 'No title')} "
          f"(score: {result.score:.3f}, date: {result.metadata.get('publish_date')})")

Reranking with Query Explanation

Use query explanation to understand reranking decisions:

# Get detailed explanation of reranking process
results = (
    db.query_builder()
    .search("data science techniques")
    .rerank_by_diversity(field="methodology", weight=0.4)
    .explain(detailed=True)
    .execute()
)

# Check execution plan
for result in results:
    if "_execution_plan" in result.metadata:
        plan = result.metadata["_execution_plan"]
        if "reranking" in plan["steps"]:
            # detailed=True adds a "details" entry; "steps" is always present.
            print(f"Reranking applied. Plan details: {plan.get('details')}")

Best Practices for Reranking

1. Start with Low Weights: Begin with reranking weights of 0.1-0.3 and adjust based on results

2. Test with Real Queries: Use actual user queries to evaluate reranking effectiveness

3. Monitor User Engagement: Track click-through rates and user satisfaction

4. Combine Strategically: Use multiple reranking strategies thoughtfully to avoid conflicts

5. Consider Field Quality: Ensure reranking fields have good data quality and coverage

# Example of gradual reranking tuning
def test_reranking_weights(query, weights=[0.1, 0.2, 0.3, 0.4, 0.5]):
    results_by_weight = {}

    for weight in weights:
        results = (
            db.query_builder()
            .search(query)
            .rerank_by_recency(date_field="publish_date", weight=weight)
            .limit(5)
            .execute()
        )
        results_by_weight[weight] = results

    return results_by_weight

# Test different weights to find optimal setting
test_results = test_reranking_weights("machine learning frameworks")

Optimizing Performance

# Process a large result set in manageable batches. batch_size is honoured by
# stream() (which pages through a cursor); execute() always returns the full
# list, so pass batch_size to stream() rather than execute().
for batch in (
    db.query_builder()
    .search("common term")  # Potentially matches many documents
    .filter(is_public=True)
    .stream(batch_size=200)  # Yields lists of up to 200 results at a time
):
    for result in batch:
        ...  # process each result

# Remove semantically similar results to reduce redundancy and increase diversity of returned documents
results = (
    db.query_builder()
    .search("neural networks")
    .semantic_dedup(threshold=0.92)  # Remove results that are >92% semantically similar
    .execute()
)

# Get detailed information about query execution for performance tuning
results = (
    db.query_builder()
    .search("complex query")
    .filter(category="research")
    .explain(detailed=True)  # Include execution time and query plan details
    .execute()
)

Streaming Results

The stream() method uses a cursor internally: FAISS/FTS is queried once and results are hydrated from SQLite in batches. This is much more efficient than re-executing the full query for each page.

# Stream results in batches (single FAISS/FTS search, lazy SQLite loading)
query = db.query_builder().search("machine learning").limit(200)

for batch in query.stream(batch_size=50):
    for result in batch:
        print(f"Processing {result.id}")

For more control over the iteration lifecycle, create a QueryCursor directly:

# Create a cursor with explicit lifecycle management
cursor = (
    db.query_builder()
    .hybrid("neural networks", vector_weight=0.5)
    .filter("year", gte_=2023)
    .limit(100)
    .cursor(batch_size=25, cursor_ttl=600.0)
)

with cursor:
    print(f"Found {cursor.total_candidates} candidates")
    first_batch = cursor.fetch_batch()
    remaining = cursor.fetch_all()  # Fetch everything else

See Streaming Query Results for the full streaming and cursor API reference.

Counting Results

# Get the count of matching documents without retrieving all data
count = (
    db.query_builder()
    .filter(category="tech", year=2024)  # Documents from tech category in 2024
    .count()  # Return only the count, not the actual documents
)

print(f"Found {count} matching documents")

Async Support

# Use async execution for better performance
results = await (
    db.query_builder()
    .search("machine learning")
    .filter(is_published=True)
    .execute_async()
)

# Async streaming with cursor-based batching (single search, lazy hydration)
async for batch in (
    db.query_builder()
    .search("deep learning")
    .limit(100)
    .stream_async(batch_size=25)
):
    for result in batch:
        await process_result(result)

# Async cursor with explicit lifecycle
cursor = await (
    db.query_builder()
    .hybrid("transformers", vector_weight=0.8)
    .limit(50)
    .cursor_async(batch_size=10)
)

async with cursor:
    async for result in cursor.stream_individual_async():
        await handle(result)

# Get count asynchronously
count = await db.query_builder().filter(year=2024).count_async()

Complex Queries

Combined Features

# Complex multi-criteria search with filtering and ranking
results = (
    db.query_builder()
    .hybrid("deep learning applications", vector_weight=0.5)  # Use hybrid search (50% vector, 50% keyword)
    .filter(year=2024)  # Only from 2024
    .filter("citations", gte_=10)  # With at least 10 citations
    .semantic_filter("abstract", "supervised learning", threshold=0.8)  # Using supervised learning approaches
    .documents()  # Return full documents
    .order_by("publish_date", "desc")  # Most recent first
    .limit(20)  # Top 20 results only
    .execute()
)

# Advanced analysis with grouping, aggregation and filtering
results = (
    db.query_builder()
    .search("climate change")  # Find climate change related documents
    .filter("publish_date", gte_="2020-01-01")  # Published since 2020
    .group_by("author", "institution")  # Group by author and institution
    .count_by("*", "publication_count")  # Count publications per group
    .max_by("citations", "max_citations")  # Find maximum citations per group
    .having("publication_count", "gte", 5)  # Only include prolific authors (5+ publications)
    .order_by("max_citations", "desc")  # Order by most cited
    .limit(10)  # Top 10 groups only
    .execute()
)

Debug and Diagnostics

The QueryBuilder provides comprehensive debugging and diagnostic capabilities to help you understand query execution, optimize performance, and troubleshoot issues.

Query Explanation

The explain() method provides detailed information about how your query will be executed. It can be used in two ways: traditional mode (returns QueryBuilder with explanation enabled) or direct mode (returns execution plan immediately).

Traditional Usage (returns QueryBuilder with explanation enabled):

# Enable explanation for query execution
results = (
    db.query_builder()
    .search("machine learning")
    .filter(category="tech")
    .rerank_by_recency(date_field="publish_date", weight=0.3)
    .explain(detailed=True)  # Enable detailed explanation
    .execute()
)

# Execution plan is included in result metadata
for result in results:
    if "_execution_plan" in result.metadata:
        plan = result.metadata["_execution_plan"]
        print(f"Query type: {plan['query_type']}")
        print(f"Execution steps: {plan['steps']}")
        print(f"Estimated cost: {plan['estimated_cost']}")

Direct Usage (returns execution plan immediately):

# Get execution plan directly without executing the query
plan = (
    db.query_builder()
    .search("machine learning")
    .filter(category="tech", year=2024)
    .rerank_by_recency(date_field="publish_date", weight=0.3)
    .explain(detailed=True, return_plan=True)  # Return plan directly
)

print(f"Query will execute with these steps: {plan['steps']}")
print(f"Estimated relative cost: {plan['estimated_cost']}")
print(f"Query type: {plan['query_type']}")
print(f"Optimizations: {plan['optimizations']}")

# With detailed=True, get additional information
if 'details' in plan:
    details = plan['details']
    print(f"Search clauses: {details['search_clauses']}")
    print(f"Exact filters: {details['exact_filters']}")
    print(f"Semantic filters: {details['semantic_filters']}")
    print(f"Aggregations: {details['aggregations']}")

Execution Plan Analysis

Understanding the execution plan helps optimize query performance:

# Analyze different query variations
queries = [
    db.query_builder().search("AI research"),
    db.query_builder().search("AI research").filter(year=2024),
    db.query_builder().search("AI research").semantic_filter("methodology", "deep learning", 0.8)
]

for i, query in enumerate(queries):
    plan = query.explain(return_plan=True)
    print(f"\nQuery {i+1} execution plan:")
    print(f"  Steps: {' -> '.join(plan['steps'])}")
    print(f"  Estimated cost: {plan['estimated_cost']}")

    # Identify expensive operations
    if plan['estimated_cost'] > 100:
        print(f"  ⚠️  High cost query - consider optimization")

    if 'semantic_filtering' in plan['steps']:
        print(f"  🧠 Semantic filtering detected - will use embeddings")

Async Execution Plans

Get execution plans for async queries:

# Get execution plan for async query
async def analyze_async_query():
    plan = await (
        db.query_builder()
        .search("complex async query")
        .semantic_filter("field", "concept", 0.75)
        .get_execution_plan_async(detailed=True)
    )

    print(f"Async execution plan: {plan}")
    return plan

# Use in async context
import asyncio
plan = asyncio.run(analyze_async_query())

Performance Profiling

Monitor query execution time and performance:

import time

# Compare execution times with and without optimizations
def profile_query_variations():
    base_query = (
        db.query_builder()
        .search("machine learning algorithms")
        .filter(year=2024)
    )

    # Test different configurations
    test_cases = [
        ("Base query", base_query),
        ("With semantic filter", base_query.semantic_filter("topic", "neural networks", 0.8)),
        ("With reranking", base_query.rerank_by_recency("publish_date", 0.3)),
        ("With both", base_query.semantic_filter("topic", "neural networks", 0.8)
                               .rerank_by_recency("publish_date", 0.3))
    ]

    for name, query in test_cases:
        # Get execution plan first
        plan = query.explain(return_plan=True)

        # Time the actual execution
        start_time = time.time()
        results = query.execute()
        execution_time = time.time() - start_time

        print(f"\n{name}:")
        print(f"  Estimated cost: {plan['estimated_cost']}")
        print(f"  Actual time: {execution_time:.3f}s")
        print(f"  Results count: {len(results)}")
        print(f"  Steps: {' -> '.join(plan['steps'])}")

profile_query_variations()

Debug Information

Get comprehensive debug information about query state:

# Build a complex query for debugging
query = (
    db.query_builder()
    .search("complex query")
    .filter(category="tech")
    .semantic_filter("abstract", "machine learning", 0.7)
    .rerank_by_diversity(field="author", weight=0.4)
    .group_by("institution")
    .count_by("*", "paper_count")
    .having("paper_count", "gte", 5)
    .order_by("paper_count", "desc")
    .limit(20)
)

# Get detailed debug information
debug_info = query.debug_info()
print(f"Query complexity: {debug_info}")

# Debug information includes:
# - Number of search clauses, filters, aggregations
# - Current query configuration
# - Return type and limits
# - Ordering and grouping information

Query Validation

Validate query configuration before execution:

# Build a potentially problematic query
query = (
    db.query_builder()
    .semantic_filter("field", "concept", 0.8)  # Semantic filter without search
    .group_by("category")
    .having("count", "gte", 10)  # Having without aggregation
    .limit(10000)  # Very large limit
)

# Validate the query
validation = query.validate()

print(f"Query valid: {validation['valid']}")

if validation['issues']:
    print("Issues found:")
    for issue in validation['issues']:
        print(f"  ❌ {issue}")

if validation['warnings']:
    print("Warnings:")
    for warning in validation['warnings']:
        print(f"  ⚠️  {warning}")

if validation['recommendations']:
    print("Recommendations:")
    for rec in validation['recommendations']:
        print(f"  💡 {rec}")

print(f"Query complexity: {validation['query_complexity']}")

Debugging Common Issues

Use diagnostics to troubleshoot common problems:

# Debug slow queries
def debug_slow_query(query_builder):
    plan = query_builder.explain(detailed=True, return_plan=True)

    # Check for expensive operations
    expensive_ops = []
    if plan['estimated_cost'] > 200:
        expensive_ops.append("High overall cost")

    if 'semantic_filtering' in plan['steps']:
        semantic_count = plan['details']['semantic_filters']
        if semantic_count > 2:
            expensive_ops.append(f"Multiple semantic filters ({semantic_count})")

    if expensive_ops:
        print("⚠️  Performance concerns:")
        for op in expensive_ops:
            print(f"   - {op}")

        print("\n💡 Optimization suggestions:")
        print("   - Reduce semantic filter count")
        print("   - Add exact filters before semantic filters")
        print("   - Consider using streaming for large result sets")

# Example usage
slow_query = (
    db.query_builder()
    .search("broad topic")
    .semantic_filter("field1", "concept1", 0.7)
    .semantic_filter("field2", "concept2", 0.8)
    .semantic_filter("field3", "concept3", 0.75)
    .limit(1000)
)

debug_slow_query(slow_query)