Code Structure and Design Plan

This document defines the module layout, data structures, design patterns, language features, testing strategy, and implementation order for meta-ast. It is the authoritative reference for how code is organized and why.

1. Module Structure

src/
├── lib.rs                    Public API re-exports
├── main.rs                   CLI entrypoint
├── error.rs                  Error + Diagnostic types (thiserror)
├── pipeline.rs               Full graph analysis orchestration
│
├── model/
│   ├── mod.rs                Symbol, SymbolKind, SourceRange, UnresolvedImport, UnresolvedReference, FileExtraction, DataNode, DataScope, FlowEdge, FlowKind (feature: dataflow)
│   ├── ids.rs                FileId, SymbolId, SnapshotId, DataNodeId (newtyped NonZeroU32 via define_id_type! macro; generator starts at 1)
│   └── output.rs             InspectOutput, FuncEntry, ClassEntry, ObjectEntry
│
├── language/
│   ├── mod.rs                LangId enum, LanguageSpec struct, DefaultVisibility, DocCommentConfig
│   ├── common.rs             extract_with_spec, extract_imports_and_references_with_spec, associate_docstrings
│   ├── dataflow.rs           extract_dataflow() dispatcher (feature: dataflow; Rust impl in rust.rs)
│   ├── python.rs             Python queries + extraction
│   ├── javascript.rs         JavaScript queries + extraction
│   ├── typescript.rs         TypeScript queries + extraction
│   ├── tsx.rs                TSX queries + extraction (separate grammar from TS)
│   ├── c.rs                  C queries + extraction
│   ├── cpp.rs                C++ queries + extraction
│   ├── rust.rs               Rust queries + extraction
│   ├── go.rs                 Go queries + extraction
│   ├── ruby.rs               Ruby queries + extraction
│   └── import_resolver.rs    ImportResolver trait, stateful resolvers (Python, Go, JS, TS)
│
├── input/
│   └── mod.rs                File discovery, filtering, language routing
│
├── parser/
│   └── mod.rs                Tree-sitter parser lifecycle, parse function
│
├── extractor/
│   └── mod.rs                Pipeline orchestration: parallel parse + extract per-file (symbols + imports + references)
│
├── graph/
│   ├── mod.rs                CodeGraph (DiGraph), add_edge_normalized_with_flow, re-exports
│   ├── node.rs               NodeData enum (File / Symbol / External / Data)
│   ├── edge.rs               EdgeKind enum (Ownership / Import / Reference / Flow) with confidence + flow_kind
│   ├── builder.rs            GraphBuilder, from_extractions, add_data_node, add_flow_edge, import_adjacency
│   ├── scc.rs                Tarjan SCC + DeployabilityHint
│   └── resolver.rs           FlattenedScopeCache, ResolutionContext, resolve_all_references
│
├── output/
│   ├── mod.rs                OutputFormat enum (Json / Yaml) with serialize dispatch
│   ├── emitter.rs            EmitConfig, emit_inspect(), emit_graph() - CLI output dispatch
│   ├── inspect.rs            Inspect-compatible JSON/YAML emission
│   ├── graph.rs              Unified GraphOutput (schema_version, metadata, nodes, edges, sccs, deployability)
│   ├── shard/                `.metast` v2 stable-name JSONL shard and index persistence
│   │   ├── mod.rs            Module root, re-exports, unit tests
│   │   ├── error.rs          ShardError enum
│   │   ├── file.rs           ShardFile, ShardSymbol, write_shard(), read_shard()
│   │   ├── edge.rs           ShardEdge, ShardEdgeKind, restore_shard_edges()
│   │   ├── manifest.rs       ShardManifestRecord, write_manifest(), read_manifest()
│   │   ├── header.rs         ShardHeader, write_header(), read_header()
│   │   └── name.rs           Stable naming, descriptors, and parent hierarchy resolution
│   └── dashboard.rs          Interactive HTML dashboard (Cytoscape.js via CDN, --html)
│
└── sink/                     [feature: dataflow] GraphSink trait + JsonSink
    └── mod.rs                GraphSink trait, JsonSink (file/stdout)
│
└── interface/                CLI layer
    ├── mod.rs                CLI module root
    └── args.rs               Clap derive structs (Inspect, Graph, Deploy + -l, --format, --html, --datagraph, --watch, --watch-debounce, -o, --check)
│
├── watch/                     [feature: watch]
│   └── mod.rs                 IncrementalCache, WatchState, incremental_reanalyze, run_watch
│
└── deploy/                   [feature: metacall-deploy] See [DEPLOY.md](DEPLOY.md)
    ├── mod.rs                Entry: run_deploy(), DeployConfig, add_metacall_edge()
    ├── scanner.rs            tree-sitter call-site detection, CallSite, CallSiteVariant, confidence
    ├── pod.rs                Union-Find partition_into_pods(), PodPartition, InterPodEdge
    ├── cut.rs                find_cross_language_cuts(), find_oversized_pod_cut(), CutEdge
    ├── dependency.rs         classify_external(), resolve_dependencies(), per-language resolvers
    ├── metrics.rs            compute_file_metrics(), compute_pod_metrics(), FileMetrics
    ├── manifest.rs           generate_pod_manifest(), PodManifest, ManifestEdge
    ├── mesh.rs               generate_mesh_annotation(), DeploymentUnit, CrossLanguageEdge
    ├── check.rs              check_cut_fairness() - bijection check between cuts and rpc_stub edges
    └── tags.rs               LangId <-> MetaCall runtime tag mapping

Module dependency direction

CLI (interface/)
  → Pipeline (pipeline.rs)  → orchestrates the full graph analysis
    → Extractor (extractor/) → depends on model + language + parser
      → Parser (parser/)     → depends on language (grammar dispatch)
    → Graph (graph/)         → depends on model + petgraph
      → Resolver (graph/resolver.rs) → cross-file reference resolution
    → Input (input/)         → depends on language (detection)
  → Output (output/)         → depends on model + graph
  → Deploy (deploy/)         → depends on pipeline + graph + input [feature: metacall-deploy]
  → Sink (sink/)             → depends on output/graph [feature: dataflow]
  → Error (error.rs)         ← cross-cutting

Outer layers depend on inner layers. The model layer has zero knowledge of parsing, I/O, or language specifics.

2. Core Data Structures

2.1 ID Types

Newtyped NonZeroU32 values generated by define_id_type! and allocated by IdGenerator<T> (an AtomicU32 wrapper) for lock-free, thread-safe, session-deterministic allocation. Type-safe against mixing.

The generator starts at 1: 0 is the permanently invalid niche value, so Option<Id> niche-optimizes to 4 bytes (the size of Id itself) instead of the 8 bytes an Option<u32> would cost. This benefits structures that store an optional id, e.g. DataNode.symbol_id: Option<SymbolId>.

#![allow(unused)]
fn main() {
define_id_type!(FileId);
define_id_type!(SymbolId);
define_id_type!(SnapshotId);
define_id_type!(DataNodeId);
}

Construction is fallible: Id::new(u32) -> Option<Self> returns None for 0 (rejecting the niche value at the type boundary, including deserialization). The raw value is reachable via Id::to_raw() -> u32 and From<NonZeroU32>.

2.2 Source Location

#![allow(unused)]
fn main() {
pub struct LineColumn {
    pub line: usize,    // 0-indexed
    pub column: usize,  // 0-indexed, byte offset within line
}

pub struct SourceRange {
    pub byte_start: usize,
    pub byte_end: usize,
    pub start: LineColumn,
    pub end: LineColumn,
}
}

2.3 Symbol Model

Immutable IR - constructed once during extraction, never mutated.

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Serialize)]
#[non_exhaustive]
pub enum SymbolKind {
    Function,
    Method,
    Class,
    Struct,
    Interface,
    Trait,
    Enum,
    Object,
    Constant,
    Static,
    Module,
    Namespace,
    TypeAlias,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[non_exhaustive]
pub enum Visibility {
    Public,
    Private,
}

#[derive(Debug, Clone, Serialize)]
pub struct Symbol {
    pub id: SymbolId,
    pub name: String,
    pub kind: SymbolKind,
    pub language: LangId,
    pub file_path: PathBuf,
    pub source_range: SourceRange,
    pub visibility: Option<Visibility>,
    pub signature: Option<String>,
    pub docstring: Option<String>,
    pub is_async: bool,
}
}

2.4 Graph Model

Node and edge types:

Graph invariants:

1. Every SymbolNode maps to exactly one FileNode.
2. Ownership edges form an acyclic containment structure.
3. SCC applies to dependency/reference subgraph only (Ownership excluded).
4. Duplicate edges normalized by (src, dst, edge_kind).
5. External dependencies get NodeData::External placeholder nodes.

2.5 Inspect Output

Stable contract:

#![allow(unused)]
fn main() {
pub struct InspectOutput {
    pub funcs: Vec<FuncEntry>,
    pub classes: Vec<ClassEntry>,
    pub objects: Vec<ObjectEntry>,
}
}

Each entry type includes: name, source_range, optional signature, visibility, docstring. FuncEntry additionally includes an async flag.

3. Language System Design

3.1 LanguageSpec Struct

Each language is a static LanguageSpec constant with function pointers (not a trait):

#![allow(unused)]
fn main() {
pub struct LanguageSpec {
    pub extensions: &'static [&'static str],
    pub grammar_fn: fn() -> tree_sitter::Language,
    pub query_fn: fn() -> &'static Query,
    pub import_path_resolver: fn(&str, &Path, &Path) -> Option<PathBuf>,
    pub import_ref_query_fn: fn() -> &'static Query,
    pub class_like_parents: &'static [&'static str],
    pub ancestor_visibility_rules: &'static [(&'static str, Visibility)],
    pub visibility_from_name: Option<fn(&str) -> Option<Visibility>>,
    pub import_statement_kinds: &'static [&'static str],
    pub default_visibility: DefaultVisibility,
    pub doc_comment_config: Option<DocCommentConfig>,
}
}

3.2 LangId Enum

The aggregate dispatch enum. #[non_exhaustive] for forward compatibility:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, strum::Display, strum::AsRefStr)]
#[non_exhaustive]
#[serde(rename_all = "snake_case")]
#[strum(serialize_all = "snake_case")]
#[repr(usize)]
pub enum LangId {
    Python,
    JavaScript,
    TypeScript,
    Tsx,
    C,
    Cpp,
    Rust,
    Go,
    Ruby,
}
}

3.3 Stateful Import Resolution Seam

To support complex stateful import resolution (e.g. resolving paths using configuration files like tsconfig.json or module boundary scanning like go.mod), meta-ast implements a hybrid seam combining static LanguageSpec specs with a stateful ImportResolver trait:

#![allow(unused)]
fn main() {
pub trait ImportResolver: Send + Sync {
    fn resolve(
        &self,
        raw: &str,
        source_dir: &Path,
        project_root: &Path,
    ) -> Option<PathBuf>;
}
}

Hybrid Resolution Bridge

1. LanguageSpec remains static and const (containing a stateless import_path_resolver fn pointer).
2. ImportResolver represents a stateful trait interface.
3. Concrete adapters bridge the two:
   • StatelessResolver: Zero-cost wrapper delegating to static fn pointers.
   • PythonResolver, GoModResolver, JsResolver, TsConfigResolver: Concrete structs implementing ImportResolver, prepped to hold caches or parse configs.
4. make_resolver(LangId) -> Box<dyn ImportResolver>: Factory function constructing the stateful resolver for each language dynamically.

Stateful Caching and Memoization Engines

To guarantee maximum throughput and avoid redundant filesystem traversal during large-scale workspace parsing, the stateful resolvers employ optimized, thread-safe caching strategies:

• OnceLock Module Boundary Scanning (GoModResolver): Scans for the root go.mod file and parses the module path at most once per execution using a standard OnceLock. Subsequent resolution calls query the in-memory boundary in $O(1)$ time.
• RwLock File Existence Memoization (PythonResolver, JsResolver, TsConfigResolver): Memoizes exists() and is_file() filesystem checks using an RwLock<HashMap<PathBuf, bool>>. This minimizes expensive system calls during TypeScript candidate extensions resolution (e.g. trying .ts, .tsx, .js) and Python relative path matching, while remaining safe for concurrency.
• Stateless Fallback: When candidate paths do not match or cannot be resolved using stateful logic, all resolvers gracefully fallback to their underlying stateless LanguageSpec function pointer, ensuring 100% backward compatibility.

During graph assembly, resolvers are created once per run and cached inside the builder to ensure O(1) config-file reading and caching properties.

3.4 Adding a New Language

The process is:

1. Add the tree-sitter grammar crate to Cargo.toml.
2. Create src/language/<name>.rs with query constants, extraction function, and LanguageSpec constant.
3. Add a variant to LangId enum.
4. Add a match arm in spec_for().
5. Add fixture files and tests.

No trait objects, no runtime plugins. Compile-time completeness checking via exhaustive match.

3.5 Language Detection

detect_language(path: &Path) -> Option<LangId> maps file extensions to LangId variants. Lives in input/mod.rs.

4. Design Patterns

4.1 Enum Static Dispatch (Language System)

All language-specific behavior dispatches through match on LangId. No vtables, no dyn - full monomorphization and inline optimization.

4.2 Pipeline Pattern

The analysis pipeline is orchestrated by pipeline.rs:

Source Discovery -> Parallel Parse + Extract -> Graph Assembly -> Import Resolution -> Reference Resolution -> SCC -> Output
   (sequential)       (rayon par_iter)         (sequential)       (sequential)          (sequential)      (sequential)

Parse and extract are combined per-file to avoid materializing all tree-sitter trees simultaneously.

4.3 Newtype Pattern

FileId, SymbolId, SnapshotId are newtyped u32 values via define_id_type! macro. The compiler prevents mixing them, and #[serde(transparent)] keeps serialization clean.

4.4 Recoverable Error Accumulation

Parse errors do not abort extraction. The pipeline accumulates Vec<Diagnostic> alongside results. Tree-sitter ERROR and MISSING nodes are skipped during extraction. Diagnostics are a separate concern from the symbol model.

4.5 Immutable IR

Symbol structs are constructed during extraction and never mutated. Downstream consumers (graph assembly, output serialization) read them immutably.

5. Parallelism Strategy

5.1 rayon Integration

rayon = "1.10" is used for file-level parallelism in the parse + extract phase.

• A thread-local pool of Parser instances (one per language) is maintained within each worker thread via thread_local! and RefCell caching. This avoids sharing the non-Sync Parser across threads.
• Emitted Tree and symbol models are Send and are safely returned from rayon workers to the main thread for graph assembly.
• Caching Parser instances avoids redundant grammar re-initialization and allocation overhead on every task.

5.2 Pipeline Phases

6. Error Handling

6.1 Error Type Hierarchy

#![allow(unused)]
fn main() {
#[derive(Debug, thiserror::Error)]
pub enum Error {
    #[error("IO: {0}")]
    Io(#[from] std::io::Error),

    #[error("parse error in {path}: {message}")]
    Parse { path: PathBuf, message: String },

    #[error("query error ({language}): {message}")]
    Query { language: LangId, message: String },

    #[error("config: {0}")]
    Config(String),

    #[error("graph error: {0}")]
    Graph(String),
}
}

Library uses Result<T, Error> with ? propagation. Application boundary (CLI) uses anyhow::Result.

6.2 Diagnostics

#![allow(unused)]
fn main() {
pub struct Diagnostic {
    pub path: PathBuf,
    pub severity: Severity,  // Warning, Error
    pub message: String,
    pub source_range: Option<SourceRange>,
}
}

Diagnostics are accumulated in a Vec<Diagnostic> separate from the symbol model. Extraction continues on recoverable errors.

6.3 Error Recovery Rules

1. Tree-sitter ERROR and MISSING nodes are skipped during extraction.
2. Partial extraction is allowed and expected for malformed source files.
3. If > 50% of a file’s nodes are errors, the file is marked as unparseable but does not abort the pipeline.
4. Fatal errors are reserved for invalid configuration or unrecoverable I/O failures.

6.4 Query Compilation Failure Strategy

Tree-sitter queries are hardcoded constants in each language pack. If a query fails to compile, it indicates a programmer bug in the shipped query text, not a runtime input error.

Strategy: compile_query uses panic!() rather than std::process::abort() or Result propagation.

Why not abort(): panic!() runs destructors, is propagated by rayon, and integrates with Rust’s panic infrastructure. abort() skips all cleanup.

Why not Result: Queries are compiled inside LazyLock<T>::new() closures which require FnOnce() -> T (infallible return).

Mitigation: language::validate_queries() eagerly initializes all 16 LazyLock statics at startup, ensuring any query bug panics immediately rather than after processing files.

7. Rust Language Features Used

8. Dependencies

8.1 Runtime Dependencies

8.2 Development Dependencies

8.3 Feature Flags

9. Test Structure

tests/
├── integration.rs                   Integration test module root
├── integration/
│   ├── pipeline_test.rs             End-to-end: discover -> parse -> extract -> graph -> output
│   ├── dashboard_test.rs            HTML dashboard generation tests
│   ├── output_format_test.rs        JSON/YAML output format tests
│   └── inspect_output_test.rs       Inspect-compatible output validation
└── fixtures/
    ├── python/
    │   ├── simple_functions.py
    │   ├── classes.py
    │   ├── async_decorators.py
    │   ├── deep_nesting.py
    │   ├── partial_syntax_error.py
    │   └── sample.py
    ├── javascript/
    │   ├── functions.js
    │   ├── classes.js
    │   └── large_classes.js
    ├── typescript/
    │   └── interfaces.ts
    ├── tsx/
    │   └── components.tsx
    ├── c/
    │   ├── functions.c
    │   └── structs_enums.c
    ├── cpp/
    │   ├── classes.cpp
    │   └── namespaces.cpp
    ├── rust/
    │   ├── functions.rs
    │   ├── structs_enums.rs
    │   └── large_file.rs
    ├── go/
    │   ├── functions.go
    │   ├── methods.go
    │   └── deep_nesting.go
    ├── mixed/                      Multi-language single-directory fixtures
    │   ├── app.py, index.js, main.rs, test.generated.py
    │   └── auth_microservice{,_level2,_level3}  Deploy edge-case fixtures
    │       (star / cross-language SCC cycle / full-module stress)
    └── multi/                      Multi-file cross-language fixtures
        ├── main.py, lib.py, app.js, util.js
        ├── c_app/, cpp_app/, go_app/, rust_crate/, ts_app/, tsx_app/
        └── edge_*/                 Edge case fixtures (circular, alias, shadowing, etc.)

Snapshot policy

Insta snapshot files live in src/language/snapshots/ as .snap files (not under fixture directories). Each language module generates snapshots via inline unit tests. Update workflow: cargo insta test then cargo insta review then commit accepted .snap files.

Testing Strategy