claude-code-facts/code-search-vex-vs-ast-index.md

Code Search: vex vs ast-index — Benchmark Notes

Snapshot: 2026-05-26 · Tested versions: vex 1.9.1, ast-index 3.41.0

These tools evolve quickly. Results below are point-in-time and only describe the versions and the single repo tested. Re-run the benchmarks before citing them on a different repo, on later versions, or after either tool changes its index format.

Heads-up: Several conclusions from the 2026-05-18 snapshot have flipped on this revision. See "Changes since the 2026-05-18 snapshot" at the bottom for a summary.

Capability update — vex 1.16.0 (capabilities only, NOT re-benchmarked): the latency and quality tables below remain pinned to vex 1.9.1, but vex's feature surface has moved on. New since 1.9.1 (no fresh measurements taken here):

• vex history <Symbol> (v1.16) — query-time git-log walker returning every historical version of a symbol (--diff, --since/--until, --author, --kind); opt-in indexed sidecar via vex index --history. No ast-index equivalent.
• vex mcp install / uninstall / list (v1.15.0) — idempotent MCP-server registration for Claude Code / Cursor (replaces hand-editing the agent config).
• Scope & metadata search filters on vex search / vex usages — --kind, --include/--exclude, --since/--since-branched/--changed-only, --visibility, --async-only, --static-only, --sealed-only, --why.
• vex gpu execution-provider probe; vex watch continuous reindex; vex init --agents-md; prebuilt Windows vex + vex-mcp binaries.

See vex.md for the full current reference (install, GPU/CUDA, config, command set). Re-run the tables below on 1.16.0 before citing the numbers.

Test environment

Aspect	Value
Repo	led-grab (private, mixed-language LED capture/streaming app)
Total files indexed	555 (ast-index); vex indexes a similar set
Total symbols indexed	~15,596 (vex) / ~18,226 (ast-index)
Reference edges	n/a in vex status output / 62,625 refs (ast-index)
Languages present	Python, Kotlin (Android), TypeScript, JavaScript, plus PowerShell/Bash scripts
Host	Single Windows 10 workstation, Git Bash, SSD
Index storage	%LOCALAPPDATA%\vex\<hash>\index.vex on Windows (~/.cache/vex/ on Unix) / %LOCALAPPDATA%\ast-index\<hash>\index.db

The repo size is "small/medium" by both tools' definitions. Numbers on a 10× larger repo will not scale linearly — semantic embeddings in particular grow with symbol count, and call-graph construction grows with edge count.

Indexing & footprint

Aspect	vex (structural)	vex (--semantic)	ast-index
Cold build time	~1–2 s	5 m 20 s (one-time embeddings)	~1–2 s
Symbols	15,596	15,596	18,226
Index size on disk	(structural-only smaller)	26.4 MB (with embeddings)	10.3 MB
Incremental update	vex update, or auto_update = true in .vex.toml (auto-runs before queries when stale)	same	ast-index update (incremental) or rebuild
Call graph	Built into index, ~4 ms queries	same	Now populated for Python (was broken in 3.27)
Multi-language	18+ via tree-sitter	same	13+
Branch-diff (symbol-level vs git rev)	vex diff --base <rev> (NEW in 1.7+)	same	ast-index changed --base <rev>
Self-update	vex self-update (NEW; works on Windows/macOS/Linux)	same	— (manual install)

Query latency (warm, sub-100 ms is "fast enough")

Operation	vex	ast-index	Notes
Symbol definition	~100 ms	~30–90 ms	Both fast
Usages	~80 ms	~35 ms	See "Notable findings" #2 for precision flip
Callers (direct)	~45 ms	~50 ms	Both now resolve real call sites for Python
Implementations / subclasses	~80 ms	~50 ms	Both work; vex generics gap fixed in v1.7.0
Existence check (vex check)	~30 ms	— (use symbol)	vex-only fast multi-symbol existence
Semantic (NL → symbol)	~300 ms	—	only vex (requires --semantic index)
similar SymName	~110 ms	—	only vex
Near-duplicate scan	~18 s whole-repo	—	only vex
Multi-hop callers (paths, reachable)	hundreds of ms	—	only vex (transitive call graph)
Bundle (1-shot show + callers + callees + similar)	~150 ms	—	only vex (bundle --mode symbol)

Query quality findings

Three real queries from the test repo (re-run on 2026-05-26):

Query	vex 1.9.1	ast-index 3.41.0	Better fit
usages BaseJsonStore	0 hits (no structural refs — subclasses caught by implementations)	4 hits, all in comments/docstrings	depends on intent — vex if you want structural-only, ast-index if you want any textual mention
callers get_latest_frame	6 real call sites	9 hits (incl. 3 false positives in docstrings)	vex (cleaner)
implementations BaseJsonStore	2 hits (_TestStore, _LegacyStore)	n/a (class is closest)	vex
diff --base HEAD~5	211 changes (incl. heading/markdown moves)	"No supported files changed" (Python/Kotlin/TS only)	depends — vex broader, ast-index narrower-by-design
Semantic "WLED device discovery over mDNS"	finds wled_provider.discover, wled_client	n/a	vex only
Semantic "JSON storage migration logic"	finds BaseJsonStore, TestLegacyKeyMigration, _LegacyStore	n/a	vex only

Notable findings

1. ast-index's Python call graph now works. In the 2026-05-18 run on v3.27.0 it returned 0 for several functions; on v3.41.0 it returns real call sites for the same queries. Whatever was broken upstream is fixed. (Watch out: it still includes textual mentions in docstrings as "callers" — see #2.)

2. usages precision is now the inverse of last snapshot.

   • vex 1.9.1: T1-language usages (Python/TS/Rust/C#/C++) is an AST identifier walk, optionally backed by persisted reference edges with --strict (Phase 11.1 / v1.8.0). It returned 0 hits for usages BaseJsonStore — correctly, since there are 0 structural usages outside subclass declarations (those are picked up by implementations).
   • ast-index 3.41.0: returned 4 hits, all of which are in comments or docstrings.
   • The old "vex catches comments and docstrings, ast-index doesn't" advice has swapped. Today, vex is the stricter tool by default and ast-index is the textual one. Use vex grep (regex) or ast-index usages if you actually want comment/docstring mentions.

3. vex's implementations for generic-parameterized subclasses now works. class Foo(Base[T]) is detected as of v1.7.0. The old workaround (vex pattern 'class $NAME($BASE[$$_]):' --lang python) is no longer needed in this case. Remaining gap (per CLAUDE.md): decorator-based dispatch is not linked.

4. vex now has diff --base <rev> (symbol-level git rev diff). This replaces the previous "ast-index has branch-diff, vex does not" finding. The two tools differ in scope, not in capability:

   • vex diff covers all parsed symbol kinds across all indexed languages, including headings in markdown — broader and noisier.
   • ast-index changed covers Python/Kotlin/TS/JS code symbols only — narrower and cleaner if you only care about code changes. Use vex when you want everything; use ast-index when you only want code-symbol churn.

5. vex's semantic index still has a one-time setup cost. ~5 minutes to embed ~15k symbols and ~86 MB ONNX model download on first run. Worth it for natural-language queries and similar/duplicates, but you must commit to it upfront. After that it lives in the same index.vex (~26 MB total with embeddings included).

6. vex now ships prebuilt Windows binaries. vex self-update works on Windows/macOS/Linux in v1.9.1 — no more building from source on first install. Update claude-code-tools.md § vex accordingly.

7. New vex commands worth knowing (added since v1.5):

   • vex diff --base <rev> — symbol-level branch diff (#4 above).
   • vex paths --from A --to B — enumerate caller chains between two symbols (multi-hop callers).
   • vex reachable --target T — find all symbols that transitively reach T via the call graph.
   • vex check sym1,sym2,... — fast multi-symbol existence check.
   • vex bundle --mode {symbol,pr-impact,project} — one call replaces the show → callers → callees → similar round trip; pr-impact mode bundles changed symbols + transitive callers + reachable tests for code review.
   • vex eval — built-in ranking eval harness for CI regression.
   • vex capabilities — machine-readable feature matrix.
   • Since v1.10–v1.16 (see the capability-update note at the top + vex.md): vex history (symbol-level git archaeology), vex mcp install/uninstall/list (idempotent MCP registration), vex gpu (EP probe), vex watch (continuous reindex), and scope/metadata search filters (--kind, --include/--exclude, --since/--changed-only, --visibility, --async-only, --why).

Practical recommendation

The chain in the recommended global CLAUDE.md snippet still applies:

vex → ast-index → Grep/Glob

…but the reasons for the fallback have shifted on this version:

• Default to vex for symbol search, usages, callers, callees, implementations, semantic, similar, duplicates, bundle, and branch-diff. The call graph, the precision improvements (T1 AST walk + --strict), and the new bundle/paths/reachable/diff primitives cover most agent code-search needs in one tool.
• Fall back to ast-index when:
  • You want only code-symbol churn on a branch (changed --base is narrower than vex diff).
  • You want textual matches in comments/docstrings (vex T1 usages is now strict and may miss those — vex grep is the alternative but ast-index usages is sometimes more convenient).
  • vex is not installed on the host (rare now that prebuilt binaries exist).
• Fall through to Grep/Glob for regex, config files (YAML/JSON/TOML), pure prose, or unparsed languages.

Neither tool fully resolves: decorator-based dispatch, string-resolved targets (uvicorn factory strings, Celery task names), reflection / getattr, dynamic imports, and macro-expanded references. Before any rename or delete, backstop structural results with vex grep '\bName\b' regardless of which tool you started with.

Re-running these benchmarks

If you want to validate on a different repo or newer versions:

# 1. Build both indices fresh
vex init && vex index                            # vex structural (set auto_update = true)
vex index --semantic                             # vex semantic (slow, one-time)
ast-index rebuild                                # ast-index

# 2. Run identical queries through both
SYM="SomeClassInYourRepo"
vex search "$SYM" --format compact   ; ast-index symbol "$SYM"
vex usages "$SYM" --format compact   ; ast-index usages "$SYM"
vex callers "$SYM" --format compact  ; ast-index callers "$SYM"

# 3. Branch-diff — both tools now support this
vex diff --base master --format compact
ast-index changed --base master

# 4. Multi-hop and bundle (vex-only)
vex paths --from caller_fn --to callee_fn
vex reachable --target some_critical_fn
vex bundle --mode symbol --symbol "$SYM"
vex bundle --mode pr-impact --base master

Record the tool versions and timestamps alongside the numbers — see this document's header for the template.

Changes since the 2026-05-18 snapshot

What flipped, what got fixed, what's new:

Finding from 2026-05-18	Status on 2026-05-26
ast-index's Python call graph is empty	Fixed — now returns real call sites in v3.41.0
vex's implementations misses generic-parameterized subclasses	Fixed — works as of v1.7.0
vex's usages is text-flavored (catches comments/docstrings)	Reversed — T1 usages is now AST-precise (Phase 11.1 / v1.8.0); ast-index is now the textual one
ast-index has changed --base, vex does not	Obsolete — vex diff --base <rev> shipped in 1.7+; tools differ in scope, not capability
Windows install of vex requires building from source	Fixed — prebuilt Windows binaries; vex self-update works
4-round-trip agent loop show → callers → callees → similar	Collapsed — vex bundle --mode symbol is one call
Multi-hop call-graph queries unsupported	Added — vex paths and vex reachable