From ce86949ded9bb915d928ef8feb15a6b9d155c950 Mon Sep 17 00:00:00 2001 From: "alexei.dolgolyov" Date: Tue, 26 May 2026 12:06:04 +0300 Subject: [PATCH] docs(code-search): refresh vex vs ast-index notes for vex 1.9.1 / ast-index 3.41.0 - Re-bench on 2026-05-26 against vex 1.9.1 and ast-index 3.41.0. - Flag findings that flipped vs the 2026-05-18 snapshot: - ast-index Python call graph now populated (was empty in 3.27) - vex implementations now handles generic-parameterized subclasses (v1.7.0) - vex usages is now AST-precise on T1 languages; ast-index is now the textual-match tool, inverting the old precision finding - vex now has diff --base , making the "only ast-index has changed --base" finding obsolete - vex now ships prebuilt Windows binaries; vex self-update works - Document new vex 1.5 -> 1.9 commands: diff, paths, reachable, check, bundle (symbol/pr-impact/project), eval, self-update, capabilities. - Add README entry for the doc; it was previously missing from the index. - Bump README "Last updated" to 2026-05-26. --- README.md | 6 +- code-search-vex-vs-ast-index.md | 128 ++++++++++++++++++++++---------- 2 files changed, 93 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 3d39bfb..f816dd9 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Claude Code Facts -> Last updated: 2026-05-18 +> Last updated: 2026-05-26 A collection of interesting and useful facts, tips, tools, and guides for working with Claude and Claude Code. @@ -25,3 +25,7 @@ Review of code signing options for Windows executables — Azure Trusted Signing ### [Gitea Act Runner — Worker Capacity (TrueNAS)](gitea-runner.md) How to configure parallel job capacity for Gitea Act Runner on TrueNAS. Covers the `CONFIG_FILE` env var requirement, custom `config.yaml` mount, and capacity tuning. + +### [Code Search: vex vs ast-index — Benchmark Notes](code-search-vex-vs-ast-index.md) + +Point-in-time benchmark of `vex` and `ast-index` on a real mixed-language repo — indexing footprint, query latency, precision findings, and when to fall back from one to the other. Refreshed for `vex 1.9.1` / `ast-index 3.41.0`, with a summary of what flipped since the previous snapshot. diff --git a/code-search-vex-vs-ast-index.md b/code-search-vex-vs-ast-index.md index 703da81..94e1141 100644 --- a/code-search-vex-vs-ast-index.md +++ b/code-search-vex-vs-ast-index.md @@ -1,22 +1,27 @@ # Code Search: vex vs ast-index — Benchmark Notes -> **Snapshot:** 2026-05-18 · **Tested versions:** `vex 1.5.0`, `ast-index 3.27.0` +> **Snapshot:** 2026-05-26 · **Tested versions:** `vex 1.9.1`, `ast-index 3.41.0` > > These tools evolve quickly. Results below are **point-in-time** and only > describe the versions and the single repo tested. Re-run the benchmarks before > citing them on a different repo, on later versions, or after either tool > changes its index format. +> +> **Heads-up:** Several conclusions from the 2026-05-18 snapshot have flipped on +> this revision. See ["Changes since the 2026-05-18 snapshot"](#changes-since-the-2026-05-18-snapshot) +> at the bottom for a summary. ## Test environment | Aspect | Value | |---|---| | Repo | `led-grab` (private, mixed-language LED capture/streaming app) | -| Total files indexed | 527–555 (depends on tool's file filter) | -| Total symbols indexed | ~14,969 (vex) / ~16,785 (ast-index) | +| Total files indexed | 555 (ast-index); vex indexes a similar set | +| Total symbols indexed | ~15,596 (vex) / ~18,226 (ast-index) | +| Reference edges | n/a in vex `status` output / **62,625 refs** (ast-index) | | Languages present | **Python**, **Kotlin** (Android), **TypeScript**, **JavaScript**, plus PowerShell/Bash scripts | | Host | Single Windows 10 workstation, Git Bash, SSD | -| Index storage | `~/.cache/vex/` (vex) / `%LOCALAPPDATA%\ast-index\` (ast-index) | +| Index storage | `%LOCALAPPDATA%\vex\\index.vex` on Windows (`~/.cache/vex/` on Unix) / `%LOCALAPPDATA%\ast-index\\index.db` | The repo size is "small/medium" by both tools' definitions. **Numbers on a 10× larger repo will not scale linearly** — semantic embeddings in particular grow with symbol count, and call-graph construction grows with edge count. @@ -24,68 +29,90 @@ The repo size is "small/medium" by both tools' definitions. **Numbers on a 10× | Aspect | vex (structural) | vex (`--semantic`) | ast-index | |---|---|---|---| -| Cold build time | **1.6 s** | 5 m 20 s (one-time embeddings) | 1.2 s | -| Symbols | 14,969 | 14,969 | 16,785 | -| Index size on disk | 5.8 MB | larger (embeddings) | 9.7 MB | -| Incremental update | `vex update`, or `auto_update = true` in `.vex.toml` | same | rebuild only | -| Call graph | Built into index, ~4 ms queries | same | Present but **empty for Python in this repo** (see "Notable findings") | +| Cold build time | **~1–2 s** | 5 m 20 s (one-time embeddings) | ~1–2 s | +| Symbols | 15,596 | 15,596 | 18,226 | +| Index size on disk | (structural-only smaller) | **26.4 MB** (with embeddings) | 10.3 MB | +| Incremental update | `vex update`, or `auto_update = true` in `.vex.toml` (auto-runs before queries when stale) | same | `ast-index update` (incremental) or `rebuild` | +| Call graph | Built into index, ~4 ms queries | same | **Now populated for Python** (was broken in 3.27) | | Multi-language | 18+ via tree-sitter | same | 13+ | -| Branch-diff (`changed --base master`) | — | — | **Yes** | +| Branch-diff (symbol-level vs git rev) | **`vex diff --base `** (NEW in 1.7+) | same | **`ast-index changed --base `** | +| Self-update | **`vex self-update`** (NEW; works on Windows/macOS/Linux) | same | — (manual install) | ## Query latency (warm, sub-100 ms is "fast enough") | Operation | vex | ast-index | Notes | |---|---|---|---| -| Symbol definition | ~107 ms | **35–91 ms** | Both fast | -| Usages | ~117 ms (11 hits) | ~35 ms (**4 hits**) | vex catches comments/docstrings; ast-index returns only structural refs | -| Callers | **~45 ms (6 hits)** | ~52 ms (**0 hits**) | ast-index's Python call graph was empty for this repo | -| Implementations / subclasses | ~200 ms (**0 hits**) | n/a | vex misses generic-parameterized form `class Foo(Base[T])` | -| Existence check | ~50 ms | ~30 ms | Both fine | -| Semantic (NL → symbol) | ~325 ms | — | only vex (requires `--semantic` index) | +| Symbol definition | ~100 ms | ~30–90 ms | Both fast | +| Usages | ~80 ms | ~35 ms | See "Notable findings" #2 for precision flip | +| Callers (direct) | ~45 ms | ~50 ms | Both now resolve real call sites for Python | +| Implementations / subclasses | ~80 ms | ~50 ms | Both work; vex generics gap fixed in v1.7.0 | +| Existence check (`vex check`) | ~30 ms | — (use `symbol`) | vex-only fast multi-symbol existence | +| Semantic (NL → symbol) | ~300 ms | — | only vex (requires `--semantic` index) | | `similar SymName` | ~110 ms | — | only vex | | Near-duplicate scan | ~18 s whole-repo | — | only vex | +| Multi-hop callers (`paths`, `reachable`) | hundreds of ms | — | only vex (transitive call graph) | +| Bundle (1-shot `show + callers + callees + similar`) | ~150 ms | — | only vex (`bundle --mode symbol`) | ## Query quality findings -Three real queries from the test repo: +Three real queries from the test repo (re-run on 2026-05-26): -| Query | vex | ast-index | Better fit | +| Query | vex 1.9.1 | ast-index 3.41.0 | Better fit | |---|---|---|---| -| `usages BaseJsonStore` | 11 hits incl. tests + imports | 4 hits, **misses test files entirely** | vex | -| `symbol ScreenCapture` | 9 hits incl. fields + Kotlin + fn signatures | 5 hits, cleaner (class + imports only) | ast-index *(less noise)* | -| `callers get_latest_frame` | 6 real call sites correctly resolved | **0** (broken) | vex | -| `implementations BaseJsonStore` | 0 (generics bug) | n/a (`class` is closest) | tie / neither | +| `usages BaseJsonStore` | **0 hits** (no structural refs — subclasses caught by `implementations`) | 4 hits, all in **comments/docstrings** | depends on intent — vex if you want structural-only, ast-index if you want any textual mention | +| `callers get_latest_frame` | 6 real call sites | **9 hits** (incl. 3 false positives in docstrings) | vex (cleaner) | +| `implementations BaseJsonStore` | **2 hits** (`_TestStore`, `_LegacyStore`) | n/a (`class` is closest) | vex | +| `diff --base HEAD~5` | 211 changes (incl. heading/markdown moves) | "No supported files changed" (Python/Kotlin/TS only) | depends — vex broader, ast-index narrower-by-design | | Semantic `"WLED device discovery over mDNS"` | finds `wled_provider.discover`, `wled_client` | n/a | vex only | | Semantic `"JSON storage migration logic"` | finds `BaseJsonStore`, `TestLegacyKeyMigration`, `_LegacyStore` | n/a | vex only | ## Notable findings -1. **ast-index's call graph was empty for this repo's Python.** `ast-index callers ` returned 0 for several functions that vex correctly identified with 6+ real call sites. Whether this is a Python-language bug, an indexing edge case, or specific to this repo's structure was not investigated further — verify on your own repo before relying on `ast-index callers`. +1. **ast-index's Python call graph now works.** In the 2026-05-18 run on v3.27.0 it returned 0 for several functions; on v3.41.0 it returns real call sites for the same queries. Whatever was broken upstream is fixed. (Watch out: it still includes textual mentions in docstrings as "callers" — see #2.) -2. **vex's `usages` is text-flavored, not structural.** It catches matches in comments, docstrings, and even `CLAUDE.md`. That can be useful or noisy depending on intent. For "real references only," prefer `vex callers` / `vex callees` / `vex pattern`, or fall back to `ast-index symbol` which is stricter. +2. **`usages` precision is now the inverse of last snapshot.** + - **vex 1.9.1**: T1-language `usages` (Python/TS/Rust/C#/C++) is an AST identifier walk, optionally backed by persisted reference edges with `--strict` (Phase 11.1 / v1.8.0). It returned **0 hits** for `usages BaseJsonStore` — correctly, since there are 0 structural usages outside subclass declarations (those are picked up by `implementations`). + - **ast-index 3.41.0**: returned 4 hits, all of which are in comments or docstrings. + - The old "vex catches comments and docstrings, ast-index doesn't" advice has *swapped*. Today, vex is the stricter tool by default and ast-index is the textual one. Use `vex grep` (regex) or `ast-index usages` if you actually want comment/docstring mentions. -3. **vex's `implementations` misses generic-parameterized subclasses.** `class Foo(Base[T])` is not detected as an implementation of `Base`. Workaround: use `vex pattern 'class $NAME($BASE[$$_]):' --lang python` or a plain `vex grep`. +3. **vex's `implementations` for generic-parameterized subclasses now works.** `class Foo(Base[T])` is detected as of v1.7.0. The old workaround (`vex pattern 'class $NAME($BASE[$$_]):' --lang python`) is no longer needed in this case. Remaining gap (per CLAUDE.md): decorator-based dispatch is not linked. -4. **ast-index has `changed --base ` — vex does not.** This makes ast-index uniquely useful during code review for "which symbols did this branch touch?" without parsing a diff manually. +4. **vex now has `diff --base ` (symbol-level git rev diff).** This replaces the previous "ast-index has branch-diff, vex does not" finding. The two tools differ in scope, not in capability: + - `vex diff` covers all parsed symbol kinds across all indexed languages, including headings in markdown — broader and noisier. + - `ast-index changed` covers Python/Kotlin/TS/JS code symbols only — narrower and cleaner if you only care about code changes. + Use vex when you want everything; use ast-index when you only want code-symbol churn. -5. **vex's semantic index has a one-time setup cost.** ~5 minutes to embed ~15k symbols and ~86 MB ONNX model download on first run. Worth it for natural-language queries and `similar`/`duplicates`, but you must commit to it upfront. +5. **vex's semantic index still has a one-time setup cost.** ~5 minutes to embed ~15k symbols and ~86 MB ONNX model download on first run. Worth it for natural-language queries and `similar`/`duplicates`, but you must commit to it upfront. After that it lives in the same `index.vex` (~26 MB total with embeddings included). -6. **First-time Windows install of vex requires building from source** (no prebuilt Windows binary in the v1.5.0 release assets). See `claude-code-tools.md` § vex. +6. **vex now ships prebuilt Windows binaries.** `vex self-update` works on Windows/macOS/Linux in v1.9.1 — no more building from source on first install. Update `claude-code-tools.md` § vex accordingly. + +7. **New vex commands worth knowing (added since v1.5):** + - `vex diff --base ` — symbol-level branch diff (#4 above). + - `vex paths --from A --to B` — enumerate caller chains between two symbols (multi-hop `callers`). + - `vex reachable --target T` — find all symbols that transitively reach `T` via the call graph. + - `vex check sym1,sym2,...` — fast multi-symbol existence check. + - `vex bundle --mode {symbol,pr-impact,project}` — one call replaces the `show → callers → callees → similar` round trip; pr-impact mode bundles changed symbols + transitive callers + reachable tests for code review. + - `vex eval` — built-in ranking eval harness for CI regression. + - `vex capabilities` — machine-readable feature matrix. ## Practical recommendation -Use **vex as primary**, **ast-index as fallback** for: - -- `ast-index changed --base ` during code review (no vex equivalent). -- Stricter `symbol`/`usages` when vex's textual matches are too noisy. - -This matches the priority chain already in the recommended global `CLAUDE.md` snippet: +The chain in the recommended global `CLAUDE.md` snippet still applies: ``` vex → ast-index → Grep/Glob ``` -The chain is not "vex always wins" — each tool has cases where it's the right call. +…but the *reasons* for the fallback have shifted on this version: + +- **Default to vex** for symbol search, usages, callers, callees, implementations, semantic, similar, duplicates, bundle, and branch-diff. The call graph, the precision improvements (T1 AST walk + `--strict`), and the new `bundle`/`paths`/`reachable`/`diff` primitives cover most agent code-search needs in one tool. +- **Fall back to ast-index** when: + - You want only code-symbol churn on a branch (`changed --base` is narrower than `vex diff`). + - You want textual matches in comments/docstrings (vex T1 `usages` is now strict and may miss those — `vex grep` is the alternative but ast-index `usages` is sometimes more convenient). + - vex is not installed on the host (rare now that prebuilt binaries exist). +- **Fall through to Grep/Glob** for regex, config files (YAML/JSON/TOML), pure prose, or unparsed languages. + +Neither tool fully resolves: decorator-based dispatch, string-resolved targets (uvicorn factory strings, Celery task names), reflection / `getattr`, dynamic imports, and macro-expanded references. Before any rename or delete, backstop structural results with `vex grep '\bName\b'` regardless of which tool you started with. ## Re-running these benchmarks @@ -93,18 +120,39 @@ If you want to validate on a different repo or newer versions: ```bash # 1. Build both indices fresh -vex init && vex index # vex structural +vex init && vex index # vex structural (set auto_update = true) vex index --semantic # vex semantic (slow, one-time) ast-index rebuild # ast-index # 2. Run identical queries through both SYM="SomeClassInYourRepo" -vex search "$SYM" --format compact ; ast-index symbol "$SYM" -vex usages "$SYM" --format compact ; ast-index usages "$SYM" -vex callers "$SYM" --format compact ; ast-index callers "$SYM" +vex search "$SYM" --format compact ; ast-index symbol "$SYM" +vex usages "$SYM" --format compact ; ast-index usages "$SYM" +vex callers "$SYM" --format compact ; ast-index callers "$SYM" -# 3. Branch-diff (ast-index only) +# 3. Branch-diff — both tools now support this +vex diff --base master --format compact ast-index changed --base master + +# 4. Multi-hop and bundle (vex-only) +vex paths --from caller_fn --to callee_fn +vex reachable --target some_critical_fn +vex bundle --mode symbol --symbol "$SYM" +vex bundle --mode pr-impact --base master ``` Record the tool versions and timestamps alongside the numbers — see this document's header for the template. + +## Changes since the 2026-05-18 snapshot + +What flipped, what got fixed, what's new: + +| Finding from 2026-05-18 | Status on 2026-05-26 | +|---|---| +| ast-index's Python call graph is empty | **Fixed** — now returns real call sites in v3.41.0 | +| vex's `implementations` misses generic-parameterized subclasses | **Fixed** — works as of v1.7.0 | +| vex's `usages` is text-flavored (catches comments/docstrings) | **Reversed** — T1 `usages` is now AST-precise (Phase 11.1 / v1.8.0); ast-index is now the textual one | +| ast-index has `changed --base`, vex does not | **Obsolete** — `vex diff --base ` shipped in 1.7+; tools differ in scope, not capability | +| Windows install of vex requires building from source | **Fixed** — prebuilt Windows binaries; `vex self-update` works | +| 4-round-trip agent loop `show → callers → callees → similar` | **Collapsed** — `vex bundle --mode symbol` is one call | +| Multi-hop call-graph queries unsupported | **Added** — `vex paths` and `vex reachable` |