Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Symbol Extraction Specification

1. Purpose

Define language-pack extraction contracts backed by Tree-sitter grammar queries. Each language pack covers symbols, imports, and references. The set of supported languages grows over time; see ROADMAP.md Phase 6 for the expansion plan.

2. Shared extraction rules

  • Prefer grammar field-based extraction where available.
  • Record both byte and line/column ranges.
  • Continue extraction in presence of parser recovery nodes when safe.

3. Language packs

Python

  • Extract: functions, classes, imports, module-level assignments.
  • Handle decorated definitions and async functions.

JavaScript

  • Extract: function declarations, function expressions, arrow functions, classes, methods, imports/exports.

TypeScript / TSX

  • Extract JS symbols plus interfaces, type aliases, enums.
  • Use TSX grammar for JSX-bearing files.

C

  • Extract: function definitions/declarations, structs, enums, typedefs, includes.
  • Extensions: .c and .h.
  • Distinguish declaration vs definition ‘where possible’.

C++

  • Extract C symbols plus classes, namespaces, templates, aliases, method definitions.
  • Extensions: .cc, .cpp, .cxx and .hpp.

Rust

  • Extract: functions, structs, enums, traits, impl blocks, use declarations, modules, const/static/type aliases.

Go

  • Extract: functions, methods with receiver, types, interfaces, imports, const/var declarations.

4. Output normalization

Each extracted symbol maps to canonical shape:

  • name
  • kind
  • language
  • file
  • source_range
  • optional: name_range (the identifier range when the query captured @name), signature, visibility, docstring, async

The extraction also carries the analyzed source text when the caller sets ExtractOptions::keep_text. Ranges index those bytes, so a consumer that needs them for position conversion must use that text rather than re-reading the file. Shard records never persist it.

5. Error tolerance policy

  • Keep partial output if recoverable parse exists.
  • Emit diagnostics for query compile failures or unsupported grammar drift.

6. Version policy

Query packs are tied to grammar versions in Cargo.toml. Any grammar upgrade requires:

  1. Query validation pass.
  2. Fixture/snapshot refresh.
  3. Update to this spec.