Benchmark

The benchmark_search tool runs a battery of test queries across different search modes and reports comparative performance metrics. One command gives you a full performance comparison table.

What It Measures

• Keyword search — FTS5 + BM25 performance
• Hybrid search — RRF fusion of keyword + semantic
• Regex search — Pattern matching over code
• Fuzzy search — Levenshtein distance re-ranking
• Streaming vs non-streaming — Speedup comparison

Usage

benchmark_search(project_name: "my-app")

Output

The benchmark produces a comparison table with these metrics:

Test Queries

The benchmark runs multiple queries designed to stress different search paths:

• Short keyword queries (exact symbol matching)
• Natural language queries (semantic understanding)
• Regex pattern queries (pattern matching)
• Fuzzy queries with misspellings
• Each query is run with and without streaming for comparison

When to Use

• After indexing a new project — verify search performance
• Tuning configuration — compare before/after env changes
• Troubleshooting — identify slow search paths
• Hardware changes — assess impact of more CPUs/memory

Interpreting Results

Streaming Speedup

Streaming search typically shows ~1.5-2× speedup for hybrid mode since keyword and semantic searches run in parallel instead of sequentially.

Mode Performance

Keyword search is typically fastest (no embedding generation). Hybrid adds ~30-50% overhead for embedding but provides better semantic relevance. Regex performance depends on pattern complexity and codebase size.

Tips

• Run benchmarks with SEARCH_CACHE_TTL_MS=0 to avoid cache skew
• Compare streaming enable/disable: set SEARCH_STREAM_ENABLED then re-run
• Results vary with project size — larger projects benefit more from parallel processing