Code Structure and Design Plan

This document defines the module layout, data structures, design patterns, language features, testing strategy, and implementation order for meta-ast. It is the authoritative reference for how code is organized and why.

1. Module Structure

src/
├── cache.rs                BLAKE3 fingerprinting and the extraction cache
├── error.rs                Error + Diagnostic types (thiserror)
├── lib.rs                  Public API re-exports
├── main.rs                 CLI entry point: parse, subscriber setup, dispatch
├── pipeline.rs             Full graph analysis orchestration, one assembly path for both entry points
├── reanalyze.rs            Incremental re-analysis: overlays, diffs, cache reuse
├── deploy/
│   ├── check.rs                check_cut_fairness() - bijection check between cuts and rpc_stub edges
│   ├── client_call.rs          Client call resolution: load aware first, then the global name index
│   ├── config.rs               DeployConfig and its defaults
│   ├── cut.rs                  find_cross_language_cuts(), find_oversized_pod_cut(), CutEdge, CutAnnotation
│   ├── dependency.rs           One table of lockfile and manifest sources plus one reader per format
│   ├── manifest.rs             generate_pod_manifest(), PodManifest, ManifestEdge
│   ├── mesh.rs                 generate_mesh_annotation(), DeploymentUnit, CrossLanguageEdge
│   ├── metrics.rs              compute_file_metrics(), compute_pod_metrics(), FileMetrics
│   ├── mod.rs                  Entry: run_deploy() in named stages, DeployConfig, add_metacall_edge()
│   ├── pod.rs                  Union-Find partition_into_pods(), PodPartition, InterPodEdge
│   ├── scanner.rs              tree-sitter call-site detection, CallSite, CallSiteVariant, confidence
│   └── tags.rs                 LangId <-> MetaCall runtime tag mapping
├── extractor/
│   └── mod.rs                  Pipeline orchestration: parallel parse + extract per file (symbols, imports, references, call sites, dataflow)
├── graph/
│   ├── builder.rs              GraphBuilder: named stages for files, symbols, dataflow, imports, references and client calls; AnalysisParts (graph, scc, scope, references, client-call records)
│   ├── edge.rs                 EdgeKind enum (Ownership / Import / Reference / Flow) with confidence + flow_kind, one merge rule, confidence_tier
│   ├── mod.rs                  CodeGraph (DiGraph), add_edge_normalized_with_flow, re-exports
│   ├── naming.rs               Node display name and kind name; one authority for the graph output
│   ├── node.rs                 NodeData enum (File / Symbol / External / Data)
│   ├── resolver.rs             FlattenedScopeCache, ResolutionContext, resolve_references_detailed, ResolvedReference, resolve_all_references
│   └── scc.rs                  Tarjan SCC + DeployabilityHint in a single edge walk
├── input/
│   └── mod.rs                  File discovery, filtering, language routing, portable paths
├── interface/
│   ├── args.rs                 Clap derive structs (Inspect, Graph, Deploy and their flags)
│   ├── banner.rs               Startup banner
│   ├── commands.rs             Command implementations: inspect, graph, watch and deploy
│   ├── mod.rs                  CLI module root
│   └── report.rs               FailOn policy and diagnostic reporting
├── language/
│   ├── c.rs                    C queries + extraction
│   ├── common.rs               extract_with_spec, extract_imports_and_references_with_spec, associate_docstrings
│   ├── cpp.rs                  C++ queries + extraction
│   ├── dataflow.rs             extract_dataflow() dispatcher (feature: dataflow; Rust impl in rust.rs)
│   ├── go.rs                   Go queries + extraction
│   ├── import_resolver.rs      ImportResolver trait, stateful resolvers (Python, Go, JS, TS)
│   ├── javascript.rs           JavaScript queries + extraction
│   ├── mod.rs                  LangId enum, LanguageSpec struct, DefaultVisibility, DocCommentConfig, the query registry
│   ├── pack.rs                 define_language_pack! macro: query statics, fallible accessors, spec literal and snapshot scaffolding
│   ├── python.rs               Python queries + extraction
│   ├── ruby.rs                 Ruby queries + extraction
│   ├── rust.rs                 Rust queries + extraction
│   ├── tsx.rs                  TSX queries + extraction (separate grammar from TS)
│   └── typescript.rs           TypeScript queries + extraction
├── model/
│   ├── ids.rs                  FileId, SymbolId, SnapshotId, DataNodeId (newtyped NonZeroU32; generator starts at 1)
│   ├── mod.rs                  Symbol, SymbolKind, SourceRange, UnresolvedImport, UnresolvedReference, FileExtraction, DataNode, DataScope, FlowEdge, FlowKind (feature: dataflow)
│   └── output.rs               InspectOutput, FuncEntry, ClassEntry, ObjectEntry
├── output/
│   ├── dashboard.rs            Interactive HTML dashboard with the vendored Cytoscape bundle (--html, --open)
│   ├── emitter.rs              EmitConfig, emit_inspect(), emit_graph() - CLI output dispatch
│   ├── graph.rs                Unified GraphOutput (schema_version, metadata, nodes, edges, sccs, deployability)
│   ├── inspect.rs              Inspect-compatible JSON/YAML emission
│   ├── mod.rs                  OutputFormat and the single default output path
│   └── shard/
│       ├── edge.rs                 ShardEdge, ShardEdgeKind, restore_shard_edges()
│       ├── error.rs                ShardError enum
│       ├── file.rs                 ShardFile, ShardSymbol, write_shard(), read_shard()
│       ├── header.rs               ShardHeader, write_header(), read_header()
│       ├── index.rs                load_index(), hardened index verification
│       ├── manifest.rs             ShardManifestRecord, write_manifest(), read_manifest()
│       ├── mod.rs                  Module root, re-exports, unit tests
│       └── name.rs                 Portable shard naming: the name plan, the collision key and parent hierarchy resolution
├── parser/
│   └── mod.rs                  Tree-sitter parser lifecycle, parse function
├── sink/
│   └── mod.rs                  GraphSink trait + JsonSink
└── watch/
    ├── config.rs               WatchConfig: debounce, emit configuration and the diagnostic policy
    ├── mod.rs                  IncrementalCache, WatchState, re-analysis entry point, run_watch
    └── watcher.rs              Debounced notify loop with a stop flag and a failure count

Module dependency direction

CLI (interface/)
  → Pipeline (pipeline.rs)  → orchestrates the full graph analysis
    → Extractor (extractor/) → depends on model + language + parser
      → Parser (parser/)     → depends on language (grammar dispatch)
    → Graph (graph/)         → depends on model + petgraph
      → Resolver (graph/resolver.rs) → cross-file reference resolution
    → Input (input/)         → depends on language (detection)
  → Output (output/)         → depends on model + graph
  → Deploy (deploy/)         → depends on pipeline + graph + input [feature: metacall-deploy]
  → Sink (sink/)             → depends on output/graph [feature: dataflow]
  → Error (error.rs)         ← cross-cutting

Outer layers depend on inner layers. The model layer has zero knowledge of parsing, I/O, or language specifics.

2. Core Data Structures

2.1 ID Types

Newtyped NonZeroU32 values generated by define_id_type! and allocated by IdGenerator<T> (an AtomicU32 wrapper) for lock-free, thread-safe, session-deterministic allocation. Type-safe against mixing.

The generator starts at 1: 0 is the permanently invalid niche value, so Option<Id> niche-optimizes to 4 bytes (the size of Id itself) instead of the 8 bytes an Option<u32> would cost. This benefits structures that store an optional id, e.g. DataNode.symbol_id: Option<SymbolId>.

#![allow(unused)]
fn main() {
define_id_type!(FileId);
define_id_type!(SymbolId);
define_id_type!(SnapshotId);
define_id_type!(DataNodeId);
}

Construction is fallible: Id::new(u32) -> Option<Self> returns None for 0 (rejecting the niche value at the type boundary, including deserialization). The raw value is reachable via Id::to_raw() -> u32 and From<NonZeroU32>.

2.2 Source Location

#![allow(unused)]
fn main() {
pub struct LineColumn {
    pub line: usize,    // 0-indexed
    pub column: usize,  // 0-indexed, byte offset within line
}

pub struct SourceRange {
    pub byte_start: usize,
    pub byte_end: usize,
    pub start: LineColumn,
    pub end: LineColumn,
}
}

2.3 Symbol Model

Immutable IR - constructed once during extraction, never mutated.

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Serialize)]
#[non_exhaustive]
pub enum SymbolKind {
    Function,
    Method,
    Class,
    Struct,
    Interface,
    Trait,
    Enum,
    Object,
    Constant,
    Static,
    Module,
    Namespace,
    TypeAlias,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[non_exhaustive]
pub enum Visibility {
    Public,
    Private,
}

#[derive(Debug, Clone, Serialize)]
pub struct Symbol {
    pub id: SymbolId,
    pub name: String,
    pub kind: SymbolKind,
    pub language: LangId,
    pub file_path: PathBuf,
    pub source_range: SourceRange,
    /// Identifier range when the language pack captured one; editors reveal
    /// this range instead of the whole declaration.
    #[serde(skip_serializing_if = "Option::is_none")]
    pub name_range: Option<SourceRange>,
    pub visibility: Option<Visibility>,
    pub signature: Option<String>,
    pub docstring: Option<String>,
    pub is_async: bool,
}
}

FileExtraction also carries text: Option<Arc<str>>, filled only when ExtractOptions::keep_text asks for it. Shards never persist the text, and the field is not serialized. The full symbol and extraction contract lives in docs/src/specs/symbol-extraction.md.

2.4 Graph Model

Node and edge types:

Graph invariants:

1. Every SymbolNode maps to exactly one FileNode.
2. Ownership edges form an acyclic containment structure.
3. SCC applies to dependency/reference subgraph only (Ownership excluded).
4. Duplicate edges normalized by (src, dst, edge_kind).
5. External dependencies get NodeData::External placeholder nodes.

2.5 Inspect Output

Stable contract:

#![allow(unused)]
fn main() {
pub struct InspectOutput {
    pub funcs: Vec<FuncEntry>,
    pub classes: Vec<ClassEntry>,
    pub objects: Vec<ObjectEntry>,
}
}

Each entry type includes: name, source_range, optional signature, visibility, docstring. FuncEntry additionally includes an async flag.

3. Language System Design

3.1 LanguageSpec Struct

Each language is a static LanguageSpec constant with function pointers (not a trait):

#![allow(unused)]
fn main() {
pub struct LanguageSpec {
    pub extensions: &'static [&'static str],
    pub grammar_fn: fn() -> tree_sitter::Language,
    pub query_fn: fn() -> &'static Query,
    pub import_path_resolver: fn(&str, &Path, &Path) -> Option<PathBuf>,
    pub import_ref_query_fn: fn() -> &'static Query,
    pub class_like_parents: &'static [&'static str],
    pub ancestor_visibility_rules: &'static [(&'static str, Visibility)],
    pub visibility_from_name: Option<fn(&str) -> Option<Visibility>>,
    pub import_statement_kinds: &'static [&'static str],
    pub default_visibility: DefaultVisibility,
    pub doc_comment_config: Option<DocCommentConfig>,
}
}

3.2 LangId Enum

The aggregate dispatch enum. #[non_exhaustive] for forward compatibility:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, strum::Display, strum::AsRefStr)]
#[non_exhaustive]
#[serde(rename_all = "snake_case")]
#[strum(serialize_all = "snake_case")]
#[repr(usize)]
pub enum LangId {
    Python,
    JavaScript,
    TypeScript,
    Tsx,
    C,
    Cpp,
    Rust,
    Go,
    Ruby,
}
}

3.3 Stateful Import Resolution Seam

To support complex stateful import resolution (e.g. resolving paths using configuration files like tsconfig.json or module boundary scanning like go.mod), meta-ast implements a hybrid seam combining static LanguageSpec specs with a stateful ImportResolver trait:

#![allow(unused)]
fn main() {
pub trait ImportResolver: Send + Sync {
    fn resolve(
        &self,
        raw: &str,
        source_dir: &Path,
        project_root: &Path,
    ) -> Option<PathBuf>;
}
}

Hybrid Resolution Bridge

1. LanguageSpec remains static and const (containing a stateless import_path_resolver fn pointer).
2. ImportResolver represents a stateful trait interface.
3. Concrete adapters bridge the two:
   • StatelessResolver: Zero-cost wrapper delegating to static fn pointers.
   • PythonResolver, GoModResolver, NodeResolver: concrete structs implementing ImportResolver that memoize filesystem probes.
4. make_resolver(LangId) -> Box<dyn ImportResolver>: Factory function constructing the stateful resolver for each language dynamically.

Stateful Caching and Memoization Engines

To guarantee maximum throughput and avoid redundant filesystem traversal during large-scale workspace parsing, the stateful resolvers employ optimized, thread-safe caching strategies:

• OnceLock Module Boundary Scanning (GoModResolver): Scans for the root go.mod file and parses the module path at most once per execution using a standard OnceLock. Subsequent resolution calls query the in-memory boundary in $O(1)$ time.
• RwLock File Existence Memoization (PythonResolver, NodeResolver): memoizes exists() and is_file() checks in an RwLock<HashMap<PathBuf, bool>>, which removes repeated system calls during candidate extension resolution while staying safe for concurrency.
• Stateless Fallback: When candidate paths do not match or cannot be resolved using stateful logic, all resolvers gracefully fallback to their underlying stateless LanguageSpec function pointer, ensuring 100% backward compatibility.

During graph assembly, make_resolver builds one resolver set per graph build, so a memoized probe never outlives the build that created it.

3.4 Adding a New Language

The process is:

1. Add the tree-sitter grammar crate to Cargo.toml.
2. Create src/language/<name>.rs with query constants, extraction function, and LanguageSpec constant.
3. Add a variant to LangId enum.
4. Add a match arm in spec_for().
5. Add fixture files and tests.

No trait objects, no runtime plugins. Compile-time completeness checking via exhaustive match.

3.5 Language Detection

detect_language(path: &Path) -> Option<LangId> maps file extensions to LangId variants. Lives in input/mod.rs.

4. Design Patterns

4.1 Enum Static Dispatch (Language System)

All language-specific behavior dispatches through match on LangId. No vtables, no dyn - full monomorphization and inline optimization.

4.2 Pipeline Pattern

The analysis pipeline is orchestrated by pipeline.rs:

Source Discovery -> Parallel Parse + Extract -> Graph Assembly -> Import Resolution -> Reference Resolution -> SCC -> Output
   (sequential)       (rayon par_iter)         (sequential)       (sequential)          (sequential)      (sequential)

Parse and extract are combined per-file to avoid materializing all tree-sitter trees simultaneously.

4.3 Newtype Pattern

FileId, SymbolId, SnapshotId are newtyped u32 values via define_id_type! macro. The compiler prevents mixing them, and #[serde(transparent)] keeps serialization clean.

4.4 Recoverable Error Accumulation

Parse errors do not abort extraction. The pipeline accumulates Vec<Diagnostic> alongside results. Tree-sitter ERROR and MISSING nodes are skipped during extraction. Diagnostics are a separate concern from the symbol model.

4.5 Immutable IR

Symbol structs are constructed during extraction and never mutated. Downstream consumers (graph assembly, output serialization) read them immutably.

5. Parallelism Strategy

5.1 rayon Integration

rayon = "1.12" is used for file-level parallelism in the parse + extract phase.

• A thread-local pool of Parser instances (one per language) is maintained within each worker thread via thread_local! and RefCell caching. This avoids sharing the non-Sync Parser across threads.
• Emitted Tree and symbol models are Send and are safely returned from rayon workers to the main thread for graph assembly.
• Caching Parser instances avoids redundant grammar re-initialization and allocation overhead on every task.

5.2 Pipeline Phases

6. Error Handling

6.1 Error Type Hierarchy

#![allow(unused)]
fn main() {
#[derive(Debug, thiserror::Error)]
pub enum Error {
    #[error("IO: {0}")]
    Io(#[from] std::io::Error),

    #[error("parse error in {path}: {message}")]
    Parse { path: PathBuf, message: String },

    #[error("query error ({language}): {message}")]
    Query { language: LangId, message: String },

    #[error("config: {0}")]
    Config(String),

    #[error("graph error: {0}")]
    Graph(String),
}
}

Library uses Result<T, Error> with ? propagation. Application boundary (CLI) uses anyhow::Result.

6.2 Diagnostics

#![allow(unused)]
fn main() {
pub struct Diagnostic {
    pub path: PathBuf,
    pub severity: Severity,  // Warning, Error
    pub message: String,
    pub source_range: Option<SourceRange>,
}
}

Diagnostics are accumulated in a Vec<Diagnostic> separate from the symbol model. Extraction continues on recoverable errors.

6.3 Error Recovery Rules

1. Tree-sitter ERROR and MISSING nodes are skipped during extraction.
2. Partial extraction is allowed and expected for malformed source files.
3. Any tree with an error node emits one Warning diagnostic that carries the error ratio. Extraction still returns partial results and never aborts the pipeline.
4. Fatal errors are reserved for invalid configuration or unrecoverable I/O failures.

6.4 Query Compilation Failure Strategy

Tree-sitter queries are hardcoded constants in each language pack. If a query fails to compile, it indicates a programmer bug in the shipped query text, not a runtime input error.

Strategy: compile_query uses panic!() rather than std::process::abort() or Result propagation.

Why not abort(): panic!() runs destructors, is propagated by rayon, and integrates with Rust’s panic infrastructure. abort() skips all cleanup.

Why not Result: Queries are compiled inside LazyLock<T>::new() closures which require FnOnce() -> T (infallible return).

Mitigation: language::validate_queries() eagerly initializes all 16 LazyLock statics at startup, ensuring any query bug panics immediately rather than after processing files.

7. Rust Language Features Used

8. Dependencies

8.1 Runtime Dependencies

8.2 Development Dependencies

8.3 Feature Flags

9. Test Structure

tests/
├── integration.rs                   Integration test module root
├── integration/
│   ├── pipeline_test.rs             End-to-end: discover -> parse -> extract -> graph -> output
│   ├── dashboard_test.rs            HTML dashboard generation tests
│   ├── output_format_test.rs        JSON/YAML output format tests
│   └── inspect_output_test.rs       Inspect-compatible output validation
└── fixtures/
    ├── python/
    │   ├── simple_functions.py
    │   ├── classes.py
    │   ├── async_decorators.py
    │   ├── deep_nesting.py
    │   ├── partial_syntax_error.py
    │   └── sample.py
    ├── javascript/
    │   ├── functions.js
    │   ├── classes.js
    │   └── large_classes.js
    ├── typescript/
    │   └── interfaces.ts
    ├── tsx/
    │   └── components.tsx
    ├── c/
    │   ├── functions.c
    │   └── structs_enums.c
    ├── cpp/
    │   ├── classes.cpp
    │   └── namespaces.cpp
    ├── rust/
    │   ├── functions.rs
    │   ├── structs_enums.rs
    │   └── large_file.rs
    ├── go/
    │   ├── functions.go
    │   ├── methods.go
    │   └── deep_nesting.go
    ├── mixed/                      Multi-language single-directory fixtures
    │   ├── app.py, index.js, main.rs, test.generated.py
    │   └── auth_microservice{,_level2,_level3}  Deploy edge-case fixtures
    │       (star / cross-language SCC cycle / full-module stress)
    └── multi/                      Multi-file cross-language fixtures
        ├── main.py, lib.py, app.js, util.js
        ├── c_app/, cpp_app/, go_app/, rust_crate/, ts_app/, tsx_app/
        └── edge_*/                 Edge case fixtures (circular, alias, shadowing, etc.)

Snapshot policy

Insta snapshot files live in src/language/snapshots/ as .snap files (not under fixture directories). Each language module generates snapshots via inline unit tests. Update workflow: cargo insta test then cargo insta review then commit accepted .snap files.

Testing Strategy