alexei.dolgolyov/claude-code-facts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(code-search): refresh vex vs ast-index notes for vex 1.9.1 / ast-index 3.41.0

Browse Source 
- Re-bench on 2026-05-26 against vex 1.9.1 and ast-index 3.41.0.
- Flag findings that flipped vs the 2026-05-18 snapshot:
  - ast-index Python call graph now populated (was empty in 3.27)
  - vex implementations now handles generic-parameterized subclasses (v1.7.0)
  - vex usages is now AST-precise on T1 languages; ast-index is now the
    textual-match tool, inverting the old precision finding
  - vex now has diff --base <rev>, making the "only ast-index has
    changed --base" finding obsolete
  - vex now ships prebuilt Windows binaries; vex self-update works
- Document new vex 1.5 -> 1.9 commands: diff, paths, reachable, check,
  bundle (symbol/pr-impact/project), eval, self-update, capabilities.
- Add README entry for the doc; it was previously missing from the index.
- Bump README "Last updated" to 2026-05-26.
This commit is contained in:
Admin
2026-05-26 12:06:04 +03:00
parent 29c316341e
commit ce86949ded
2 changed files with 93 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
code-search-vex-vs-ast-index.md
+88 -40
View File
@@ -1,22 +1,27 @@
# Code Search: vex vs ast-index — Benchmark Notes
> **Snapshot:** 2026-05-18 · **Tested versions:** `vex 1.5.0`, `ast-index 3.27.0`
> **Snapshot:** 2026-05-26 · **Tested versions:** `vex 1.9.1`, `ast-index 3.41.0`
>
> These tools evolve quickly. Results below are **point-in-time** and only
> describe the versions and the single repo tested. Re-run the benchmarks before
> citing them on a different repo, on later versions, or after either tool
> changes its index format.
>
> **Heads-up:** Several conclusions from the 2026-05-18 snapshot have flipped on
> this revision. See ["Changes since the 2026-05-18 snapshot"](#changes-since-the-2026-05-18-snapshot)
> at the bottom for a summary.
## Test environment
| Aspect | Value |
|---|---|
| Repo | `led-grab` (private, mixed-language LED capture/streaming app) |
| Total files indexed | 527–555 (depends on tool's file filter) |
| Total symbols indexed | ~14,969 (vex) / ~16,785 (ast-index) |
| Total files indexed | 555 (ast-index); vex indexes a similar set |
| Total symbols indexed | ~15,596 (vex) / ~18,226 (ast-index) |
| Reference edges | n/a in vex `status` output / **62,625 refs** (ast-index) |
| Languages present | **Python**, **Kotlin** (Android), **TypeScript**, **JavaScript**, plus PowerShell/Bash scripts |
| Host | Single Windows 10 workstation, Git Bash, SSD |
| Index storage | `~/.cache/vex/` (vex) / `%LOCALAPPDATA%\ast-index\` (ast-index) |
| Index storage | `%LOCALAPPDATA%\vex\<hash>\index.vex` on Windows (`~/.cache/vex/` on Unix) / `%LOCALAPPDATA%\ast-index\<hash>\index.db` |
The repo size is "small/medium" by both tools' definitions. **Numbers on a 10× larger repo will not scale linearly** — semantic embeddings in particular grow with symbol count, and call-graph construction grows with edge count.
@@ -24,68 +29,90 @@ The repo size is "small/medium" by both tools' definitions. **Numbers on a 10×
| Aspect | vex (structural) | vex (`--semantic`) | ast-index |
|---|---|---|---|
| Cold build time | **1.6 s** | 5 m 20 s (one-time embeddings) | 1.2 s |
| Symbols | 14,969 | 14,969 | 16,785 |
| Index size on disk | 5.8 MB | larger (embeddings) | 9.7 MB |
| Incremental update | `vex update`, or `auto_update = true` in `.vex.toml` | same | rebuild only |
| Call graph | Built into index, ~4 ms queries | same | Present but **empty for Python in this repo** (see "Notable findings") |
| Cold build time | **~1–2 s** | 5 m 20 s (one-time embeddings) | ~1–2 s |
| Symbols | 15,596 | 15,596 | 18,226 |
| Index size on disk | (structural-only smaller) | **26.4 MB** (with embeddings) | 10.3 MB |
| Incremental update | `vex update`, or `auto_update = true` in `.vex.toml` (auto-runs before queries when stale) | same | `ast-index update` (incremental) or `rebuild` |
| Call graph | Built into index, ~4 ms queries | same | **Now populated for Python** (was broken in 3.27) |
| Multi-language | 18+ via tree-sitter | same | 13+ |
| Branch-diff (`changed --base master`) | — | — | **Yes** |
| Branch-diff (symbol-level vs git rev) | **`vex diff --base <rev>`** (NEW in 1.7+) | same | **`ast-index changed --base <rev>`** |
| Self-update | **`vex self-update`** (NEW; works on Windows/macOS/Linux) | same | — (manual install) |
## Query latency (warm, sub-100 ms is "fast enough")
| Operation | vex | ast-index | Notes |
|---|---|---|---|
| Symbol definition | ~107 ms | **35–91 ms** | Both fast |
| Usages | ~117 ms (11 hits) | ~35 ms (**4 hits**) | vex catches comments/docstrings; ast-index returns only structural refs |
| Callers | **~45 ms (6 hits)** | ~52 ms (**0 hits**) | ast-index's Python call graph was empty for this repo |
| Implementations / subclasses | ~200 ms (**0 hits**) | n/a | vex misses generic-parameterized form `class Foo(Base[T])` |
| Existence check | ~50 ms | ~30 ms | Both fine |
| Semantic (NL → symbol) | ~325 ms | — | only vex (requires `--semantic` index) |
| Symbol definition | ~100 ms | ~30–90 ms | Both fast |
| Usages | ~80 ms | ~35 ms | See "Notable findings" #2 for precision flip |
| Callers (direct) | ~45 ms | ~50 ms | Both now resolve real call sites for Python |
| Implementations / subclasses | ~80 ms | ~50 ms | Both work; vex generics gap fixed in v1.7.0 |
| Existence check (`vex check`) | ~30 ms | — (use `symbol`) | vex-only fast multi-symbol existence |
| Semantic (NL → symbol) | ~300 ms | — | only vex (requires `--semantic` index) |
| `similar SymName` | ~110 ms | — | only vex |
| Near-duplicate scan | ~18 s whole-repo | — | only vex |
| Multi-hop callers (`paths`, `reachable`) | hundreds of ms | — | only vex (transitive call graph) |
| Bundle (1-shot `show + callers + callees + similar`) | ~150 ms | — | only vex (`bundle --mode symbol`) |
## Query quality findings
Three real queries from the test repo:
Three real queries from the test repo (re-run on 2026-05-26):
| Query | vex | ast-index | Better fit |
| Query | vex 1.9.1 | ast-index 3.41.0 | Better fit |
|---|---|---|---|
| `usages BaseJsonStore` | 11 hits incl. tests + imports | 4 hits, **misses test files entirely** | vex |
| `symbol ScreenCapture` | 9 hits incl. fields + Kotlin + fn signatures | 5 hits, cleaner (class + imports only) | ast-index *(less noise)* |
| `callers get_latest_frame` | 6 real call sites correctly resolved | **0** (broken) | vex |
| `implementations BaseJsonStore` | 0 (generics bug) | n/a (`class` is closest) | tie / neither |
| `usages BaseJsonStore` | **0 hits** (no structural refs — subclasses caught by `implementations`) | 4 hits, all in **comments/docstrings** | depends on intent — vex if you want structural-only, ast-index if you want any textual mention |
| `callers get_latest_frame` | 6 real call sites | **9 hits** (incl. 3 false positives in docstrings) | vex (cleaner) |
| `implementations BaseJsonStore` | **2 hits** (`_TestStore`, `_LegacyStore`) | n/a (`class` is closest) | vex |
| `diff --base HEAD~5` | 211 changes (incl. heading/markdown moves) | "No supported files changed" (Python/Kotlin/TS only) | depends — vex broader, ast-index narrower-by-design |
| Semantic `"WLED device discovery over mDNS"` | finds `wled_provider.discover`, `wled_client` | n/a | vex only |
| Semantic `"JSON storage migration logic"` | finds `BaseJsonStore`, `TestLegacyKeyMigration`, `_LegacyStore` | n/a | vex only |
## Notable findings
1. **ast-index's call graph was empty for this repo's Python.** `ast-index callers <fn>` returned 0 for several functions that vex correctly identified with 6+ real call sites. Whether this is a Python-language bug, an indexing edge case, or specific to this repo's structure was not investigated further — verify on your own repo before relying on `ast-index callers`.
1. **ast-index's Python call graph now works.** In the 2026-05-18 run on v3.27.0 it returned 0 for several functions; on v3.41.0 it returns real call sites for the same queries. Whatever was broken upstream is fixed. (Watch out: it still includes textual mentions in docstrings as "callers" — see #2.)
2. **vex's `usages` is text-flavored, not structural.** It catches matches in comments, docstrings, and even `CLAUDE.md`. That can be useful or noisy depending on intent. For "real references only," prefer `vex callers` / `vex callees` / `vex pattern`, or fall back to `ast-index symbol` which is stricter.
2. **`usages` precision is now the inverse of last snapshot.**
- **vex 1.9.1**: T1-language `usages` (Python/TS/Rust/C#/C++) is an AST identifier walk, optionally backed by persisted reference edges with `--strict` (Phase 11.1 / v1.8.0). It returned **0 hits** for `usages BaseJsonStore` — correctly, since there are 0 structural usages outside subclass declarations (those are picked up by `implementations`).
- **ast-index 3.41.0**: returned 4 hits, all of which are in comments or docstrings.
- The old "vex catches comments and docstrings, ast-index doesn't" advice has *swapped*. Today, vex is the stricter tool by default and ast-index is the textual one. Use `vex grep` (regex) or `ast-index usages` if you actually want comment/docstring mentions.
3. **vex's `implementations` misses generic-parameterized subclasses.** `class Foo(Base[T])` is not detected as an implementation of `Base`. Workaround: use `vex pattern 'class $NAME($BASE[$$_]):' --lang python` or a plain `vex grep`.
3. **vex's `implementations` for generic-parameterized subclasses now works.** `class Foo(Base[T])` is detected as of v1.7.0. The old workaround (`vex pattern 'class $NAME($BASE[$$_]):' --lang python`) is no longer needed in this case. Remaining gap (per CLAUDE.md): decorator-based dispatch is not linked.
4. **ast-index has `changed --base <branch>` — vex does not.** This makes ast-index uniquely useful during code review for "which symbols did this branch touch?" without parsing a diff manually.
4. **vex now has `diff --base <rev>` (symbol-level git rev diff).** This replaces the previous "ast-index has branch-diff, vex does not" finding. The two tools differ in scope, not in capability:
- `vex diff` covers all parsed symbol kinds across all indexed languages, including headings in markdown — broader and noisier.
- `ast-index changed` covers Python/Kotlin/TS/JS code symbols only — narrower and cleaner if you only care about code changes.
Use vex when you want everything; use ast-index when you only want code-symbol churn.
5. **vex's semantic index has a one-time setup cost.** ~5 minutes to embed ~15k symbols and ~86 MB ONNX model download on first run. Worth it for natural-language queries and `similar`/`duplicates`, but you must commit to it upfront.
5. **vex's semantic index still has a one-time setup cost.** ~5 minutes to embed ~15k symbols and ~86 MB ONNX model download on first run. Worth it for natural-language queries and `similar`/`duplicates`, but you must commit to it upfront. After that it lives in the same `index.vex` (~26 MB total with embeddings included).
6. **First-time Windows install of vex requires building from source** (no prebuilt Windows binary in the v1.5.0 release assets). See `claude-code-tools.md` § vex.
6. **vex now ships prebuilt Windows binaries.** `vex self-update` works on Windows/macOS/Linux in v1.9.1 — no more building from source on first install. Update `claude-code-tools.md` § vex accordingly.
7. **New vex commands worth knowing (added since v1.5):**
- `vex diff --base <rev>` — symbol-level branch diff (#4 above).
- `vex paths --from A --to B` — enumerate caller chains between two symbols (multi-hop `callers`).
- `vex reachable --target T` — find all symbols that transitively reach `T` via the call graph.
- `vex check sym1,sym2,...` — fast multi-symbol existence check.
- `vex bundle --mode {symbol,pr-impact,project}` — one call replaces the `show → callers → callees → similar` round trip; pr-impact mode bundles changed symbols + transitive callers + reachable tests for code review.
- `vex eval` — built-in ranking eval harness for CI regression.
- `vex capabilities` — machine-readable feature matrix.
## Practical recommendation
Use **vex as primary**, **ast-index as fallback** for:
- `ast-index changed --base <branch>` during code review (no vex equivalent).
- Stricter `symbol`/`usages` when vex's textual matches are too noisy.
This matches the priority chain already in the recommended global `CLAUDE.md` snippet:
The chain in the recommended global `CLAUDE.md` snippet still applies:
```
vex → ast-index → Grep/Glob
```
The chain is not "vex always wins" — each tool has cases where it's the right call.
…but the *reasons* for the fallback have shifted on this version:
- **Default to vex** for symbol search, usages, callers, callees, implementations, semantic, similar, duplicates, bundle, and branch-diff. The call graph, the precision improvements (T1 AST walk + `--strict`), and the new `bundle`/`paths`/`reachable`/`diff` primitives cover most agent code-search needs in one tool.
- **Fall back to ast-index** when:
- You want only code-symbol churn on a branch (`changed --base` is narrower than `vex diff`).
- You want textual matches in comments/docstrings (vex T1 `usages` is now strict and may miss those — `vex grep` is the alternative but ast-index `usages` is sometimes more convenient).
- vex is not installed on the host (rare now that prebuilt binaries exist).
- **Fall through to Grep/Glob** for regex, config files (YAML/JSON/TOML), pure prose, or unparsed languages.
Neither tool fully resolves: decorator-based dispatch, string-resolved targets (uvicorn factory strings, Celery task names), reflection / `getattr`, dynamic imports, and macro-expanded references. Before any rename or delete, backstop structural results with `vex grep '\bName\b'` regardless of which tool you started with.
## Re-running these benchmarks
@@ -93,18 +120,39 @@ If you want to validate on a different repo or newer versions:
```bash
# 1. Build both indices fresh
vex init && vex index # vex structural
vex init && vex index # vex structural (set auto_update = true)
vex index --semantic # vex semantic (slow, one-time)
ast-index rebuild # ast-index
# 2. Run identical queries through both
SYM="SomeClassInYourRepo"
vex search "$SYM" --format compact ; ast-index symbol "$SYM"
vex usages "$SYM" --format compact ; ast-index usages "$SYM"
vex callers "$SYM" --format compact ; ast-index callers "$SYM"
vex search "$SYM" --format compact ; ast-index symbol "$SYM"
vex usages "$SYM" --format compact ; ast-index usages "$SYM"
vex callers "$SYM" --format compact ; ast-index callers "$SYM"
# 3. Branch-diff (ast-index only)
# 3. Branch-diff — both tools now support this
vex diff --base master --format compact
ast-index changed --base master
# 4. Multi-hop and bundle (vex-only)
vex paths --from caller_fn --to callee_fn
vex reachable --target some_critical_fn
vex bundle --mode symbol --symbol "$SYM"
vex bundle --mode pr-impact --base master
```
Record the tool versions and timestamps alongside the numbers — see this document's header for the template.
## Changes since the 2026-05-18 snapshot
What flipped, what got fixed, what's new:
| Finding from 2026-05-18 | Status on 2026-05-26 |
|---|---|
| ast-index's Python call graph is empty | **Fixed** — now returns real call sites in v3.41.0 |
| vex's `implementations` misses generic-parameterized subclasses | **Fixed** — works as of v1.7.0 |
| vex's `usages` is text-flavored (catches comments/docstrings) | **Reversed** — T1 `usages` is now AST-precise (Phase 11.1 / v1.8.0); ast-index is now the textual one |
| ast-index has `changed --base`, vex does not | **Obsolete** — `vex diff --base <rev>` shipped in 1.7+; tools differ in scope, not capability |
| Windows install of vex requires building from source | **Fixed** — prebuilt Windows binaries; `vex self-update` works |
| 4-round-trip agent loop `show → callers → callees → similar` | **Collapsed** — `vex bundle --mode symbol` is one call |
| Multi-hop call-graph queries unsupported | **Added** — `vex paths` and `vex reachable` |