meta-ast Documentation

This directory contains the implementation-facing technical documentation for meta-ast.

The structure is intentionally traceable: specs -> architecture -> structure -> ADRs -> roadmap -> validation artifacts, so design decisions are easy to trace back to requirements and forward to tests.

Document map

• ARCHITECTURE.md - system architecture, component boundaries, runtime flow, output formats, and the metacall-deploy feature layer.
• STRUCTURE.md - code structure, data structures, design patterns, module layout, and implementation order.
• DEPLOY.md - MetaCall deploy manifests: scanner, pod partitioning, mesh annotation.
• DEV_CRATE_DECISIONS.md - crate selection rationale and trade-offs.
• CI_CD.md - CI/CD architecture and quality gates.
• ROADMAP.md - phase-aligned implementation milestones and measurable exit gates.
• BENCHMARKS.md - criterion benchmark results for pipeline, graph, and incremental.
• DEMO.md - recorded CLI walkthroughs (GIFs) aligned to the delivered artifacts.
• FINAL_REPORT.md - GSoC 2026 completion report.

Specs

• specs/requirements.md - normative requirements and acceptance criteria.
• specs/graph-model.md - symbol graph and datagraph contracts, including language_id, project-root-relative path, snapshot_id, file_id, visibility, and DataNode semantics.
• specs/symbol-extraction.md - language-pack extraction contracts.
• specs/traceability.md - mapping from deliverables to implementation/docs/tests.

Architecture Decision Records (ADRs)

• adr/0001-stateful-import-resolver.md - Stateful import resolver trait seam
• adr/0002-scope-resolution-heuristics.md - Scope resolution heuristics
• adr/0003-unresolved-import-policy.md - Unresolved import handling policy
• adr/0004-global-scope-synthetic-symbols.md - Global scope synthetic symbol generation

Requests for Comments (RFCs)

• rfcs/0001-language-loading-model.md - Language loading model
• rfcs/0002-error-semantics-and-recovery.md - Error semantics and recovery
• rfcs/0003-incremental-parsing-strategy.md - Incremental parsing strategy
• rfcs/0004-graph-representation-and-scc.md - Graph representation and SCC
• rfcs/0005-output-contract-policy.md - Output contract policy
• rfcs/0006-type-inference-scope.md - Type inference scope
• rfcs/0007-dgraph-integration-scope.md - Dgraph integration scope
• rfcs/0008-graph-module.md - Graph module design
• rfcs/0009-cross-file-dependency-mapping.md - Cross-file dependency mapping
• rfcs/0010-deploy-manifests.md - MetaCall deploy manifests (superseded by the pod model)
• rfcs/0011-metacall-client-api-support.md - MetaCall client call support: metacall() and the full invocation surface
• rfcs/0012-polyglot-lsp-server.md - Polyglot LSP Server proposal: architecture, protocol, and deployment

Scope policy

• MVP (must ship): symbol extraction, dependency graph + SCC/Deployment Unit analysis, cross-platform CI.
• metacall-deploy feature: Cross-Language Call Site detection, Deploy Manifest generation, Root Manifest assembly, Mesh Annotation from SCC.
• Stretch: intra-procedural dataflow beyond simple def-use, live Dgraph sink, advanced cross-language type matching, expanded language support.

Update policy

When implementation changes any public contract (schema, CLI behavior, graph semantics, language support), update the corresponding file in this directory in the same pull request.

Architecture

1. Design goals

• Standalone-first static analysis (no runtime execution).
• Deterministic and resilient extraction under partial syntax errors.
• Incremental-by-design workflow for watch/update scenarios.
• Language-agnostic core; MetaCall deployment support is an opt-in feature.

2. High-level pipeline

1. Source discovery and language detection.
2. Tree-sitter parse per file.
3. Query-based symbol extraction per language pack.
4. Intermediate symbol model normalization.
5. Dependency graph construction (initial node + file edges).
6. Import path resolution via stateful resolvers implementing the ImportResolver trait (mapping import strings to file paths/IDs, supporting stateful configs like tsconfig.json and disk caches).
7. Cross-file reference resolution via FlattenedScopeCache (DFS the import graph once per file, then O(1) scope lookups).
8. SCC analysis (Tarjan) and Deployment Unit annotation.
9. Output emission (JSON, YAML, or interactive HTML dashboard).
10. (Requires metacall-deploy feature) Cross-language call-site scanning, pod partitioning, dependency resolution from lockfiles, pod-and-mesh manifest generation, and CI fairness checking. See DEPLOY.md.

3. Component boundaries

• Input layer: path discovery, filtering, language routing.
• Parser layer: Tree-sitter parser lifecycle and tree ownership.
• Extractor layer: language-specific query packs and capture mapping.
• Model layer: normalized symbol/domain structs.
• Graph layer: directed graph assembly + SCC algorithms. External dependencies (stdlib, third-party packages) that are referenced but not part of the project are represented as ExternalNode entries (graph/node.rs:85), carrying the raw import path and language. They appear in the graph but have no file-backed symbol data.
• Pipeline layer: full graph analysis orchestration (pipeline.rs).
• Resolver layer: cross-file reference resolution via FlattenedScopeCache (graph/resolver.rs).
• Output layer: serialization and optional adapters.
• Interface layer: CLI + library API (future: C ABI).
• Deploy layer (feature-gated: metacall-deploy): Cross-language call-site scanner (scanner.rs), pod partitioning via Union-Find over same-language edges (pod.rs), cross-language SCC cut detection and oversized-pod rebalancing (cut.rs), per-language external dependency resolution from lockfiles and manifests (dependency.rs), pod manifest generation (manifest.rs), Function Mesh annotation (mesh.rs), and CI fairness checking for RPC-converted cut edges (check.rs). See DEPLOY.md.

Detailed module layout, data structures, and dependency direction are defined in STRUCTURE.md.

4. Data contracts (summary)

Primary symbol extraction output:

• funcs
• classes
• objects

Static extensions:

• source_range
• docstring (where available)

Deploy output (feature-gated: metacall-deploy):

• metacall.pods.json - pod manifest with per-pod deployments, inter-pod edges, dependency lists, and AST node metrics
• metacall.mesh.json - Function Mesh topology annotation with SCC-derived deployment units and cross-language call-site attribution

See DEPLOY.md for schema details and the call site scanner reference.

Detailed graph contract is defined in specs/graph-model.md.

5. Error handling model

• Parse errors are recoverable when Tree-sitter yields partial trees.
• Extraction errors are scoped to file/language unit where possible.
• Unresolvable Cross-Language Call Sites (dynamic tag/path arguments) are annotated as low-confidence entries in the Mesh Annotation, not silently dropped.
• Fatal process-level errors are reserved for invalid configuration or unrecoverable IO/system failures.

6. Incremental analysis model

• Baseline mode: re-parse changed file and recompute the full graph.
• Optimized mode: apply InputEdit + changed range reduction (planned, benchmark-triggered).

Current status: Baseline incremental analysis is implemented behind the watch feature flag (--features watch). The watch module (src/watch/mod.rs) provides:

• incremental_reanalyze() - pure, deterministic, testable re-analysis step.
• run_watch() - debounced OS-level watcher loop (via notify + notify-debouncer-mini).
• BLAKE3 cryptographic content-hash fingerprinting (Fingerprint([u8; 32])) for change detection.
• Cached Arc<FileExtraction> per file: zero-allocation pointer sharing for unchanged files; graph rebuilt from scratch each tick (graph + SCC rebuild is sub-ms).
• CLI integration via meta-ast graph <path> --watch [--watch-debounce <ms>].

The InputEdit / changed-range narrowing optimization is deferred per RFC 0003.

Parallel parse + extract uses rayon per-file; graph assembly is sequential. See STRUCTURE.md section 5 for pipeline phase details.

7. Output formats

The CLI supports JSON and YAML for programmatic consumption, plus an interactive HTML dashboard for visual analysis and datagraph JSON exports.

• JSON / YAML: Controlled by the -f, --format flag. JSON is the default. YAML requires no extra setup - just pass --format yaml.
• HTML dashboard: Separate concern, activated with --html. Generates a single .html file with an interactive Cytoscape.js graph loaded from a CDN (cached by the browser after first fetch). The browser auto-opens unless you redirect.
• Datagraph JSON: Activated with --datagraph (requires --features dataflow). Exports detailed data/flow node definitions and def-use relations.
• Language Filter: Only analyze files detected as this language with -l, --language <lang>.

The dashboard turns SCC analysis into something you can actually see. Nodes in cyclic clusters (co-deployment required) are colored red. Independent Deployment Units are green. This is the difference between “your code has cycles” and “here is the exact knot you need to untangle before you can split this into independent mesh units.”

8. Compatibility and integration

• Optional integration layers (C ABI, metacall-deploy, Dgraph) are feature-scoped and do not block standalone operation. The metacall-deploy feature is implemented (see DEPLOY.md). C ABI is planned but not yet implemented.
• Discussion and contributions: Discord

Code Structure and Design Plan

This document defines the module layout, data structures, design patterns, language features, testing strategy, and implementation order for meta-ast. It is the authoritative reference for how code is organized and why.

1. Module Structure

src/
├── lib.rs                    Public API re-exports
├── main.rs                   CLI entrypoint
├── error.rs                  Error + Diagnostic types (thiserror)
├── pipeline.rs               Full graph analysis orchestration
│
├── model/
│   ├── mod.rs                Symbol, SymbolKind, SourceRange, UnresolvedImport, UnresolvedReference, FileExtraction, DataNode, DataScope, FlowEdge, FlowKind (feature: dataflow)
│   ├── ids.rs                FileId, SymbolId, SnapshotId, DataNodeId (newtyped NonZeroU32 via define_id_type! macro; generator starts at 1)
│   └── output.rs             InspectOutput, FuncEntry, ClassEntry, ObjectEntry
│
├── language/
│   ├── mod.rs                LangId enum, LanguageSpec struct, DefaultVisibility, DocCommentConfig
│   ├── common.rs             extract_with_spec, extract_imports_and_references_with_spec, associate_docstrings
│   ├── dataflow.rs           extract_dataflow() dispatcher (feature: dataflow; Rust impl in rust.rs)
│   ├── python.rs             Python queries + extraction
│   ├── javascript.rs         JavaScript queries + extraction
│   ├── typescript.rs         TypeScript queries + extraction
│   ├── tsx.rs                TSX queries + extraction (separate grammar from TS)
│   ├── c.rs                  C queries + extraction
│   ├── cpp.rs                C++ queries + extraction
│   ├── rust.rs               Rust queries + extraction
│   ├── go.rs                 Go queries + extraction
│   ├── ruby.rs               Ruby queries + extraction
│   └── import_resolver.rs    ImportResolver trait, stateful resolvers (Python, Go, JS, TS)
│
├── input/
│   └── mod.rs                File discovery, filtering, language routing
│
├── parser/
│   └── mod.rs                Tree-sitter parser lifecycle, parse function
│
├── extractor/
│   └── mod.rs                Pipeline orchestration: parallel parse + extract per-file (symbols + imports + references)
│
├── graph/
│   ├── mod.rs                CodeGraph (DiGraph), add_edge_normalized_with_flow, re-exports
│   ├── node.rs               NodeData enum (File / Symbol / External / Data)
│   ├── edge.rs               EdgeKind enum (Ownership / Import / Reference / Flow) with confidence + flow_kind
│   ├── builder.rs            GraphBuilder, from_extractions, add_data_node, add_flow_edge, import_adjacency
│   ├── scc.rs                Tarjan SCC + DeployabilityHint
│   └── resolver.rs           FlattenedScopeCache, ResolutionContext, resolve_all_references
│
├── output/
│   ├── mod.rs                OutputFormat enum (Json / Yaml) with serialize dispatch
│   ├── emitter.rs            EmitConfig, emit_inspect(), emit_graph() - CLI output dispatch
│   ├── inspect.rs            Inspect-compatible JSON/YAML emission
│   ├── graph.rs              Unified GraphOutput (schema_version, metadata, nodes, edges, sccs, deployability)
│   ├── shard/                `.metast` v2 stable-name JSONL shard and index persistence
│   │   ├── mod.rs            Module root, re-exports, unit tests
│   │   ├── error.rs          ShardError enum
│   │   ├── file.rs           ShardFile, ShardSymbol, write_shard(), read_shard()
│   │   ├── edge.rs           ShardEdge, ShardEdgeKind, restore_shard_edges()
│   │   ├── manifest.rs       ShardManifestRecord, write_manifest(), read_manifest()
│   │   ├── header.rs         ShardHeader, write_header(), read_header()
│   │   └── name.rs           Stable naming, descriptors, and parent hierarchy resolution
│   └── dashboard.rs          Interactive HTML dashboard (Cytoscape.js via CDN, --html)
│
└── sink/                     [feature: dataflow] GraphSink trait + JsonSink
    └── mod.rs                GraphSink trait, JsonSink (file/stdout)
│
└── interface/                CLI layer
    ├── mod.rs                CLI module root
    └── args.rs               Clap derive structs (Inspect, Graph, Deploy + -l, --format, --html, --datagraph, --watch, --watch-debounce, -o, --check)
│
├── watch/                     [feature: watch]
│   └── mod.rs                 IncrementalCache, WatchState, incremental_reanalyze, run_watch
│
└── deploy/                   [feature: metacall-deploy] See [DEPLOY.md](DEPLOY.md)
    ├── mod.rs                Entry: run_deploy(), DeployConfig, add_metacall_edge()
    ├── scanner.rs            tree-sitter call-site detection, CallSite, CallSiteVariant, confidence
    ├── pod.rs                Union-Find partition_into_pods(), PodPartition, InterPodEdge
    ├── cut.rs                find_cross_language_cuts(), find_oversized_pod_cut(), CutEdge
    ├── dependency.rs         classify_external(), resolve_dependencies(), per-language resolvers
    ├── metrics.rs            compute_file_metrics(), compute_pod_metrics(), FileMetrics
    ├── manifest.rs           generate_pod_manifest(), PodManifest, ManifestEdge
    ├── mesh.rs               generate_mesh_annotation(), DeploymentUnit, CrossLanguageEdge
    ├── check.rs              check_cut_fairness() - bijection check between cuts and rpc_stub edges
    └── tags.rs               LangId <-> MetaCall runtime tag mapping

Module dependency direction

CLI (interface/)
  → Pipeline (pipeline.rs)  → orchestrates the full graph analysis
    → Extractor (extractor/) → depends on model + language + parser
      → Parser (parser/)     → depends on language (grammar dispatch)
    → Graph (graph/)         → depends on model + petgraph
      → Resolver (graph/resolver.rs) → cross-file reference resolution
    → Input (input/)         → depends on language (detection)
  → Output (output/)         → depends on model + graph
  → Deploy (deploy/)         → depends on pipeline + graph + input [feature: metacall-deploy]
  → Sink (sink/)             → depends on output/graph [feature: dataflow]
  → Error (error.rs)         ← cross-cutting

Outer layers depend on inner layers. The model layer has zero knowledge of parsing, I/O, or language specifics.

2. Core Data Structures

2.1 ID Types

Newtyped NonZeroU32 values generated by define_id_type! and allocated by IdGenerator<T> (an AtomicU32 wrapper) for lock-free, thread-safe, session-deterministic allocation. Type-safe against mixing.

The generator starts at 1: 0 is the permanently invalid niche value, so Option<Id> niche-optimizes to 4 bytes (the size of Id itself) instead of the 8 bytes an Option<u32> would cost. This benefits structures that store an optional id, e.g. DataNode.symbol_id: Option<SymbolId>.

#![allow(unused)]
fn main() {
define_id_type!(FileId);
define_id_type!(SymbolId);
define_id_type!(SnapshotId);
define_id_type!(DataNodeId);
}

Construction is fallible: Id::new(u32) -> Option<Self> returns None for 0 (rejecting the niche value at the type boundary, including deserialization). The raw value is reachable via Id::to_raw() -> u32 and From<NonZeroU32>.

2.2 Source Location

#![allow(unused)]
fn main() {
pub struct LineColumn {
    pub line: usize,    // 0-indexed
    pub column: usize,  // 0-indexed, byte offset within line
}

pub struct SourceRange {
    pub byte_start: usize,
    pub byte_end: usize,
    pub start: LineColumn,
    pub end: LineColumn,
}
}

2.3 Symbol Model

Immutable IR - constructed once during extraction, never mutated.

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Serialize)]
#[non_exhaustive]
pub enum SymbolKind {
    Function,
    Method,
    Class,
    Struct,
    Interface,
    Trait,
    Enum,
    Object,
    Constant,
    Static,
    Module,
    Namespace,
    TypeAlias,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[non_exhaustive]
pub enum Visibility {
    Public,
    Private,
}

#[derive(Debug, Clone, Serialize)]
pub struct Symbol {
    pub id: SymbolId,
    pub name: String,
    pub kind: SymbolKind,
    pub language: LangId,
    pub file_path: PathBuf,
    pub source_range: SourceRange,
    pub visibility: Option<Visibility>,
    pub signature: Option<String>,
    pub docstring: Option<String>,
    pub is_async: bool,
}
}

2.4 Graph Model

Node and edge types:

Graph invariants:

1. Every SymbolNode maps to exactly one FileNode.
2. Ownership edges form an acyclic containment structure.
3. SCC applies to dependency/reference subgraph only (Ownership excluded).
4. Duplicate edges normalized by (src, dst, edge_kind).
5. External dependencies get NodeData::External placeholder nodes.

2.5 Inspect Output

Stable contract:

#![allow(unused)]
fn main() {
pub struct InspectOutput {
    pub funcs: Vec<FuncEntry>,
    pub classes: Vec<ClassEntry>,
    pub objects: Vec<ObjectEntry>,
}
}

Each entry type includes: name, source_range, optional signature, visibility, docstring. FuncEntry additionally includes an async flag.

3. Language System Design

3.1 LanguageSpec Struct

Each language is a static LanguageSpec constant with function pointers (not a trait):

#![allow(unused)]
fn main() {
pub struct LanguageSpec {
    pub extensions: &'static [&'static str],
    pub grammar_fn: fn() -> tree_sitter::Language,
    pub query_fn: fn() -> &'static Query,
    pub import_path_resolver: fn(&str, &Path, &Path) -> Option<PathBuf>,
    pub import_ref_query_fn: fn() -> &'static Query,
    pub class_like_parents: &'static [&'static str],
    pub ancestor_visibility_rules: &'static [(&'static str, Visibility)],
    pub visibility_from_name: Option<fn(&str) -> Option<Visibility>>,
    pub import_statement_kinds: &'static [&'static str],
    pub default_visibility: DefaultVisibility,
    pub doc_comment_config: Option<DocCommentConfig>,
}
}

3.2 LangId Enum

The aggregate dispatch enum. #[non_exhaustive] for forward compatibility:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, strum::Display, strum::AsRefStr)]
#[non_exhaustive]
#[serde(rename_all = "snake_case")]
#[strum(serialize_all = "snake_case")]
#[repr(usize)]
pub enum LangId {
    Python,
    JavaScript,
    TypeScript,
    Tsx,
    C,
    Cpp,
    Rust,
    Go,
    Ruby,
}
}

3.3 Stateful Import Resolution Seam

To support complex stateful import resolution (e.g. resolving paths using configuration files like tsconfig.json or module boundary scanning like go.mod), meta-ast implements a hybrid seam combining static LanguageSpec specs with a stateful ImportResolver trait:

#![allow(unused)]
fn main() {
pub trait ImportResolver: Send + Sync {
    fn resolve(
        &self,
        raw: &str,
        source_dir: &Path,
        project_root: &Path,
    ) -> Option<PathBuf>;
}
}

Hybrid Resolution Bridge

1. LanguageSpec remains static and const (containing a stateless import_path_resolver fn pointer).
2. ImportResolver represents a stateful trait interface.
3. Concrete adapters bridge the two:
   • StatelessResolver: Zero-cost wrapper delegating to static fn pointers.
   • PythonResolver, GoModResolver, JsResolver, TsConfigResolver: Concrete structs implementing ImportResolver, prepped to hold caches or parse configs.
4. make_resolver(LangId) -> Box<dyn ImportResolver>: Factory function constructing the stateful resolver for each language dynamically.

Stateful Caching and Memoization Engines

To guarantee maximum throughput and avoid redundant filesystem traversal during large-scale workspace parsing, the stateful resolvers employ optimized, thread-safe caching strategies:

• OnceLock Module Boundary Scanning (GoModResolver): Scans for the root go.mod file and parses the module path at most once per execution using a standard OnceLock. Subsequent resolution calls query the in-memory boundary in $O(1)$ time.
• RwLock File Existence Memoization (PythonResolver, JsResolver, TsConfigResolver): Memoizes exists() and is_file() filesystem checks using an RwLock<HashMap<PathBuf, bool>>. This minimizes expensive system calls during TypeScript candidate extensions resolution (e.g. trying .ts, .tsx, .js) and Python relative path matching, while remaining safe for concurrency.
• Stateless Fallback: When candidate paths do not match or cannot be resolved using stateful logic, all resolvers gracefully fallback to their underlying stateless LanguageSpec function pointer, ensuring 100% backward compatibility.

During graph assembly, resolvers are created once per run and cached inside the builder to ensure O(1) config-file reading and caching properties.

3.4 Adding a New Language

The process is:

1. Add the tree-sitter grammar crate to Cargo.toml.
2. Create src/language/<name>.rs with query constants, extraction function, and LanguageSpec constant.
3. Add a variant to LangId enum.
4. Add a match arm in spec_for().
5. Add fixture files and tests.

No trait objects, no runtime plugins. Compile-time completeness checking via exhaustive match.

3.5 Language Detection

detect_language(path: &Path) -> Option<LangId> maps file extensions to LangId variants. Lives in input/mod.rs.

4. Design Patterns

4.1 Enum Static Dispatch (Language System)

All language-specific behavior dispatches through match on LangId. No vtables, no dyn - full monomorphization and inline optimization.

4.2 Pipeline Pattern

The analysis pipeline is orchestrated by pipeline.rs:

Source Discovery -> Parallel Parse + Extract -> Graph Assembly -> Import Resolution -> Reference Resolution -> SCC -> Output
   (sequential)       (rayon par_iter)         (sequential)       (sequential)          (sequential)      (sequential)

Parse and extract are combined per-file to avoid materializing all tree-sitter trees simultaneously.

4.3 Newtype Pattern

FileId, SymbolId, SnapshotId are newtyped u32 values via define_id_type! macro. The compiler prevents mixing them, and #[serde(transparent)] keeps serialization clean.

4.4 Recoverable Error Accumulation

Parse errors do not abort extraction. The pipeline accumulates Vec<Diagnostic> alongside results. Tree-sitter ERROR and MISSING nodes are skipped during extraction. Diagnostics are a separate concern from the symbol model.

4.5 Immutable IR

Symbol structs are constructed during extraction and never mutated. Downstream consumers (graph assembly, output serialization) read them immutably.

5. Parallelism Strategy

5.1 rayon Integration

rayon = "1.10" is used for file-level parallelism in the parse + extract phase.

• A thread-local pool of Parser instances (one per language) is maintained within each worker thread via thread_local! and RefCell caching. This avoids sharing the non-Sync Parser across threads.
• Emitted Tree and symbol models are Send and are safely returned from rayon workers to the main thread for graph assembly.
• Caching Parser instances avoids redundant grammar re-initialization and allocation overhead on every task.

5.2 Pipeline Phases

6. Error Handling

6.1 Error Type Hierarchy

#![allow(unused)]
fn main() {
#[derive(Debug, thiserror::Error)]
pub enum Error {
    #[error("IO: {0}")]
    Io(#[from] std::io::Error),

    #[error("parse error in {path}: {message}")]
    Parse { path: PathBuf, message: String },

    #[error("query error ({language}): {message}")]
    Query { language: LangId, message: String },

    #[error("config: {0}")]
    Config(String),

    #[error("graph error: {0}")]
    Graph(String),
}
}

Library uses Result<T, Error> with ? propagation. Application boundary (CLI) uses anyhow::Result.

6.2 Diagnostics

#![allow(unused)]
fn main() {
pub struct Diagnostic {
    pub path: PathBuf,
    pub severity: Severity,  // Warning, Error
    pub message: String,
    pub source_range: Option<SourceRange>,
}
}

Diagnostics are accumulated in a Vec<Diagnostic> separate from the symbol model. Extraction continues on recoverable errors.

6.3 Error Recovery Rules

1. Tree-sitter ERROR and MISSING nodes are skipped during extraction.
2. Partial extraction is allowed and expected for malformed source files.
3. If > 50% of a file’s nodes are errors, the file is marked as unparseable but does not abort the pipeline.
4. Fatal errors are reserved for invalid configuration or unrecoverable I/O failures.

6.4 Query Compilation Failure Strategy

Tree-sitter queries are hardcoded constants in each language pack. If a query fails to compile, it indicates a programmer bug in the shipped query text, not a runtime input error.

Strategy: compile_query uses panic!() rather than std::process::abort() or Result propagation.

Why not abort(): panic!() runs destructors, is propagated by rayon, and integrates with Rust’s panic infrastructure. abort() skips all cleanup.

Why not Result: Queries are compiled inside LazyLock<T>::new() closures which require FnOnce() -> T (infallible return).

Mitigation: language::validate_queries() eagerly initializes all 16 LazyLock statics at startup, ensuring any query bug panics immediately rather than after processing files.

7. Rust Language Features Used

8. Dependencies

8.1 Runtime Dependencies

8.2 Development Dependencies

8.3 Feature Flags

9. Test Structure

tests/
├── integration.rs                   Integration test module root
├── integration/
│   ├── pipeline_test.rs             End-to-end: discover -> parse -> extract -> graph -> output
│   ├── dashboard_test.rs            HTML dashboard generation tests
│   ├── output_format_test.rs        JSON/YAML output format tests
│   └── inspect_output_test.rs       Inspect-compatible output validation
└── fixtures/
    ├── python/
    │   ├── simple_functions.py
    │   ├── classes.py
    │   ├── async_decorators.py
    │   ├── deep_nesting.py
    │   ├── partial_syntax_error.py
    │   └── sample.py
    ├── javascript/
    │   ├── functions.js
    │   ├── classes.js
    │   └── large_classes.js
    ├── typescript/
    │   └── interfaces.ts
    ├── tsx/
    │   └── components.tsx
    ├── c/
    │   ├── functions.c
    │   └── structs_enums.c
    ├── cpp/
    │   ├── classes.cpp
    │   └── namespaces.cpp
    ├── rust/
    │   ├── functions.rs
    │   ├── structs_enums.rs
    │   └── large_file.rs
    ├── go/
    │   ├── functions.go
    │   ├── methods.go
    │   └── deep_nesting.go
    ├── mixed/                      Multi-language single-directory fixtures
    │   ├── app.py, index.js, main.rs, test.generated.py
    │   └── auth_microservice{,_level2,_level3}  Deploy edge-case fixtures
    │       (star / cross-language SCC cycle / full-module stress)
    └── multi/                      Multi-file cross-language fixtures
        ├── main.py, lib.py, app.js, util.js
        ├── c_app/, cpp_app/, go_app/, rust_crate/, ts_app/, tsx_app/
        └── edge_*/                 Edge case fixtures (circular, alias, shadowing, etc.)

Snapshot policy

Insta snapshot files live in src/language/snapshots/ as .snap files (not under fixture directories). Each language module generates snapshots via inline unit tests. Update workflow: cargo insta test then cargo insta review then commit accepted .snap files.

Testing Strategy

Deploy Module (metacall-deploy)

Feature-gated. Build with --features metacall-deploy.

Overview

The deploy subcommand scans polyglot projects for MetaCall load and client call sites. It partitions files into same-language pods, resolves external dependencies from lockfiles, and generates two deployment artifacts:

Usage

# Build with the feature enabled
cargo build --release --features metacall-deploy

# Generate manifests
meta-ast deploy <path> --out <output_dir>

# CI validation: verify every cut edge has an RPC stub entry
meta-ast deploy <path> --check

Options

Pipeline

run_deploy()
  1. discover_files()                               - language-routed file list
  2. pipeline::analyze_graph()                      - symbol, import, and SCC analysis
  3. scanner::scan_file() per file (rayon parallel) - MetaCall call-site detection
  4. inject MetaCall import edges into graph        - add_metacall_edge with path resolution
  5. resolve client calls (two-phase)               - client_call::resolve_client_calls
  6. SCC recompute with new edges                   - update SCC analysis
  7. pod::partition_into_pods()                     - Union-Find over same-language edges
  8. metrics::compute_file_metrics()                - AST node counts per file/pod
  9. cut::find_cross_language_cuts()                - cheapest-edge split for cross-lang SCCs
 10. cut::find_oversized_pod_cut() per pod          - second-pass rebalancing
 11. dependency::resolve_dependencies()             - lockfile and manifest parsing
 12. manifest::generate_pod_manifest()              - PodManifest serialization
 13. mesh::generate_mesh_annotation()               - SCC-derived topology
 14. write artifacts or check::check_cut_fairness() in --check mode

Module map

src/deploy/
├── mod.rs          Entry point: run_deploy(), DeployConfig, add_metacall_edge()
├── scanner.rs      tree-sitter call-site detection, CallSite, CallSiteVariant
├── client_call.rs  resolve_client_calls(), resolve_script_to_file() - two-phase client invocation resolution
├── pod.rs          Union-Find partition_into_pods(), PodPartition, InterPodEdge
├── cut.rs          find_cross_language_cuts(), find_oversized_pod_cut(), CutEdge
├── dependency.rs   classify_external(), resolve_dependencies(), per-language resolvers
├── metrics.rs      compute_file_metrics(), compute_pod_metrics(), FileMetrics
├── manifest.rs     generate_pod_manifest(), PodManifest, ManifestEdge
├── mesh.rs         generate_mesh_annotation(), DeploymentUnit, CrossLanguageEdge
├── check.rs        check_cut_fairness() - bijection check between cuts and rpc_stub edges
└── tags.rs         LangId <-> MetaCall tag mapping (py, node, ts, c, cpp, rs, go)

Call Site Scanner

scanner::scan_file runs tree-sitter queries to detect MetaCall API load variants and client calls.

Supported variants

Confidence scoring

Language coverage

Queries cover all 9 supported languages: Python, JavaScript, TypeScript, TSX, C, C++, Rust, Go, and Ruby.

Client invocation resolution

client_call::resolve_client_calls resolves ClientCall targets after scanning:

• Phase A (Load-aware): Maps caller load sites to candidate target files, matching function names within loaded scope.
• Phase B (Global fallback): Searches a project-wide index of symbol names when Phase A finds no match. Emits a Warning diagnostic if no match exists.

Ambiguity resolution is deterministic: one edge per matching symbol in path-sorted order. Client-call edges are EdgeKind::Reference from calling file to target symbol node.

run_deploy also warns on orphaned metacall.json files that no LoadFromConfiguration call site references.

Pod Partitioning

pod::partition_into_pods uses Union-Find to group files into same-language deployment units.

Files sharing the same LangId and connected by Import or Reference edges join the same pod. Cross-language edges remain inter-pod edges. Ownership edges are excluded because they express file structure, not dependency.

Confidence fusion

When both an Import edge and a Reference edge connect the same pod pair, confidence scores multiply to form a combined weight in [0.0, 1.0]. If only one edge type exists, its confidence is used directly.

Language tag mapping

Cut Detection

cut.rs implements two cut rules:

1. Cross-language SCC cuts: Cuts the lowest-confidence internal edge when an SCC spans multiple languages (CutReason::CrossLanguageScc). The manifest marks these as RPC stubs.
2. Oversized pod cuts: Greedy single-pass cut on pods exceeding the pod size limit (DEFAULT_MAX_POD_SIZE, 20 files by default). Override the limit with --max-pod-size <N>.

External Dependency Resolution

dependency::resolve_dependencies collects external imports per pod and inspects project lockfiles and manifests.

Lockfiles supply pinned versions (source: "Lockfile"). Manifest fallbacks set version: None (source: "Manifest").

Pod Manifest Schema

manifest::generate_pod_manifest writes metacall.pods.json:

{
  "version": "1.0",
  "deployments": [
    {
      "id": 0,
      "language": "py",
      "files": ["auth.py", "__init__.py"],
      "metrics": {
        "total_ast_nodes": 63,
        "file_count": 2,
        "symbol_count": 2
      },
      "dependencies": [
        {
          "name": "requests",
          "version": "2.32.3",
          "language": "python",
          "source": "Lockfile"
        }
      ]
    }
  ],
  "edges": [
    {
      "from_pod": 0,
      "to_pod": 1,
      "kind": "import",
      "confidence": 1.0,
      "is_cross_language": true,
      "cut_annotation": null
    }
  ],
  "metrics": {
    "total_pods": 2,
    "cross_language_edges": 1,
    "total_ast_nodes": 112
  }
}

Mesh Annotation

mesh::generate_mesh_annotation exports SCC analysis to metacall.mesh.json.

#![allow(unused)]
fn main() {
MeshAnnotation {
    version: String,
    deployment_units: Vec<DeploymentUnit>,
    cross_language_edges: Vec<CrossLanguageEdge>,
    stats: MeshStats,
}

DeploymentUnit {
    id: usize,
    symbols: Vec<UnitSymbol>,
    is_cross_language: bool,
    is_mesh_candidate: bool,
    deployability: String,
}

CrossLanguageEdge {
    from_unit: usize,
    to_unit: usize,
    from_language: String,
    to_language: String,
    call_site: Option<String>,
    confidence: f64,
}
}

Units with is_mesh_candidate = true and is_cross_language = false deploy independently as Function Mesh services.

Check Mode (Fairness)

check::check_cut_fairness validates RPC stub contracts:

1. Every cut edge appears in manifest.edges[] with a cut_annotation.
2. Cut edges have kind: "rpc_stub".
3. Non-cut edges omit cut_annotation.

run_deploy exits non-zero if fairness checks fail.

Edge-case Fixtures

Integration tests use fixtures in tests/fixtures/mixed/:

Extending the Scanner

To add call site detection for a new language:

1. Add a static <LANG>_QUERY: LazyLock<Query> in scanner.rs.
2. Add a dispatch arm to scan_file.
3. Map tags in tags.rs.
4. Add unit tests in scanner.rs.

CI/CD Architecture

1. Objectives

• Enforce correctness and portability.
• Catch regressions early (lint, tests, benchmarks).
• Produce deterministic release artifacts.

2. Workflow layout

• ci.yml - single workflow with three jobs: test (nextest + doc tests), build (release artifacts), lint (fmt + clippy + cargo-deny).
• benchmark.yml - criterion benchmarks and trend tracking (uploads raw reports as artifacts).
• docs.yml - rustdoc build (deny warnings) and mdbook site publication to GitHub Pages.
• release.yml - tag-driven release and package publication with changelog generation.

3. Quality gates

Required for protected branch merge (all within ci.yml):

1. Lint job green (fmt + clippy + cargo-deny).
2. Test matrix green (nextest + doc tests across OS/toolchain).
3. Build artifacts generated (release binaries uploaded per OS).

4. Matrix strategy

OS targets:

• Linux (ubuntu-latest)
• macOS (macos-latest)
• Windows (windows-latest)
• Windows ARM64 (windows-11-arm)

Rust channels:

• stable (required)
• nightly (advisory compatibility signal)

5. Caching and artifacts

• Use cargo target cache for CI acceleration.
• Persist benchmark reports as artifacts.
• Publish release binaries per target triple.

6. Branch protection

• Require status checks before merge.
• Require branch up-to-date with target branch.
• Require at least one review approval.
• Require resolved review conversations.

7. Security posture

• Dependency vulnerability scanning on schedule and push.
• Keep lockfile current and reviewed.
• Do not suppress warnings as default policy.

8. Release policy

• Semantic version tags trigger release workflow.
• Generate changelog summary from merged PRs/issues.
• Publish binary artifacts and crate package (when ready).

9. Documentation policy in CI

Any change that touches output contract, graph semantics, or language extraction must update relevant docs under docs/ in the same PR.

10. Toolchain configuration

• rust-toolchain.toml pins channel 1.94.0 with rustfmt and clippy components.
• deny.toml enforces license allowlist (MIT, Apache-2.0, BSD, ISC, etc.) and bans wildcard dependencies.
• clippy.toml sets MSRV and complexity thresholds.
• rustfmt.toml sets edition 2024 formatting rules.

11. Pre-commit hooks (lefthook)

Managed via lefthook.yml. Install with:

https://lefthook.dev/install/

lefthook install

a dev script will be provided later to automate this process

pre-commit (parallel, fast)

1. cargo fmt --all -- --check - formatting gate.
2. Trailing whitespace check (*.rs, *.toml, *.md, *.yml).
3. Merge conflict marker detection.
4. Large file guard (>512KB).

pre-push (sequential, thorough)

1. cargo clippy --all-targets --all-features -- -D warnings - lint gate.
2. cargo nextest run --all-features - local test gate.

12. Test runner (nextest)

CI and local dev use cargo-nextest for faster test execution.

• Config: .config/nextest.toml
• CI profile: fail-fast = false (full matrix visibility).
• Slow-timeout: 60s per test, terminate after 3 periods.

13. Benchmarks (criterion)

• Dev dependency: criterion 0.8 with html_reports feature.
• Benchmark targets: benches/pipeline.rs (extraction throughput per language fixture), benches/graph.rs (graph operation throughput).
• Profile: opt-level = 3, LTO enabled (see [profile.bench] in Cargo.toml).

14. Release targets

15. Snapshot policy

• Insta .snap files are committed to the repository.
• Pending snapshots (.snap.new) are gitignored.
• Update workflow: cargo insta test → cargo insta review → commit accepted .snap files.

Roadmap

Phase 1 - Core & MVP symbols [COMPLETE]

Goals:

• Parser lifecycle implementation for all initial languages: Python, JavaScript, TypeScript, TSX, C, C++, Rust, Go.
• Symbol extraction and normalized IR.
• Structured JSON/YAML output (funcs, classes, objects).

Exit gates:

1. All target languages parse on fixtures.
2. Stable JSON output for representative projects.
3. Contract tests for required keys pass.

Phase 2 - Dependency graph & SCC [COMPLETE]

Goals:

• Build directed dependency/reference graph.
• Compute SCCs and annotate Deployment Units (independent vs. co-deployment required).

Exit gates:

1. SCC results match fixture expectations.
2. Cross-file dependency mapping validated on mixed-language samples.
3. ReferenceEdges appear in graph output with confidence scores in cross-file resolution tests.

Phase 3 - Datagraph & optional sink [COMPLETE]

Goals:

• Extend model with optional data/flow nodes (DataNode, FlowEdge, DataScope, FlowKind).
• Implement intra-procedural def-use extraction for Rust (let bindings, parameters).
• Provide portable graph export contract with schema versioning (v1).
• Pluggable sink adapters (GraphSink trait + JsonSink).
• CLI integration: --datagraph flag on graph subcommand.
• Unified GraphOutput serialization (replaces separate datagraph module).

Exit gates:

1. Export format validated via integration tests (JSON roundtrip, field checks).
2. Snapshot/version semantics documented and tested (SCHEMA_VERSION = 2).
3. End-to-end pipeline extracts data nodes from real Rust fixtures.
4. Flow edges created for def-use chains (param→usage, let→let shadowing).

Phase 4 - CLI polish, output formats, visualization [COMPLETE]

Goals:

• Structured output (JSON + YAML) with --format flag.
• Interactive HTML dashboard with Cytoscape.js via --html flag.
• Watch mode and incremental-update strategy.
• C ABI scaffolding and header generation (scoped out, see below).

Exit gates:

1. --format json|yaml works for analysis output. DONE
2. --html generates a dashboard with SCC/Deployment Unit coloring, auto-opens in browser. DONE
3. Watch-mode stability tests pass. DONE
4. Incremental performance target evidence captured. DONE
5. C ABI smoke tests. DROPPED - issue #21 closed NOT_PLANNED; the C ABI interface was proposed in RFC 0011 but not implemented. The exit gate is removed from scope and tracked as post-GSoC future work in issue #63.

Phase 5 - MetaCall Deploy Manifests [COMPLETE]

Requires --features metacall-deploy. Full documentation in DEPLOY.md.

Goals:

• Implement cross-language call-site detection across all 9 supported language ports (metacall_load_from_file, metacall_load_from_memory, metacall_load_from_package, metacall_load_from_configuration), including CommonJS require() for JS/TS and bare-name call detection for Rust after use import.
• Partition files into same-language pods via Union-Find over dependency edges.
• Resolve external dependencies per-language from lockfiles (preferred for exact version pinning) and package manifests (fallback).
• Generate pod manifest (metacall.pods.json) with per-pod deployments, inter-pod edges with fused confidence scores, and scoped dependency lists.
• Emit mesh annotation (metacall.mesh.json) from SCC deployment unit analysis, classifying independent Function Mesh candidates vs. co-deployment-required groups with cross-language call-site attribution.
• Implement --check validation mode: fairness check ensuring every cut edge has a corresponding RPC stub entry in the manifest (bijection check, ADR 0003 pattern).

Exit gates:

1. Pod manifests generated match expected fixtures for all projects in tests/fixtures/mixed/. DONE
2. Mesh annotation correctly classifies deployment units for auth-function-mesh fixture with call-site attribution. DONE
3. --check detects missing RPC stubs for cut edges and reports structured diagnostics. DONE
4. Dynamic call-site arguments emit low-confidence annotation rather than hard failure. DONE
5. External dependency resolution identifies jsonwebtoken from package.json/lockfile in the auth-function-mesh fixture with exact version pinning. DONE

Phase 6 - Language expansion [COMPLETE]

Goals:

• Extend language support beyond the initial 8, prioritizing C# and Java.
• Each new language requires: grammar crate, query pack (symbols + imports + references), import resolver, visibility rules, and fixture tests.
• Cross-language Call Site detection extended to new language ports as they ship.

Outcome:

• Ruby shipped end to end: grammar, query pack, resolver, visibility rules, fixtures, snapshots, and metacall-deploy call-site and lockfile coverage.
• C# (issue #23) and Java (issue #24) were evaluated and closed NOT_PLANNED. Ruby was the third language added, bringing the catalog to nine.

Exit gates:

1. New language parses on fixtures. DONE (Ruby)
2. New language pack passes extraction and cross-file dependency tests. DONE
3. metacall-deploy feature detects call sites in the new port bindings. DONE

Phase 7 - Validation and delivery [COMPLETE]

Goals:

• CI/CD hardening.
• Documentation completion.
• Benchmark and portability evidence.

Exit gates:

1. Green CI matrix on Linux/macOS/Windows. DONE
2. Benchmarks and docs published. DONE - see BENCHMARKS.md and the mdbook site (GitHub Pages).
3. Candidate demo narrative aligns with delivered artifacts. DONE - see DEMO.md.
4. Release artifacts (binaries, crates) published and verified. DONE - v0.5.0 on GitHub Releases (7 targets x core + deploy binaries) and crates.io.
5. Release announcement drafted and scheduled. DONE - v0.5.0 release notes and the Final Report.

Phase 8 - Polyglot LSP Server & Shard Indexing (metacall/lsp) [IN PROGRESS]

Goals:

• Implement Phase 0 engine prerequisites: symbol coordinates (source_range, file_path), in-memory buffer extraction seam (extract_text_with_id_gen), and modular .metast v2 shard and index persistence (ShardFile, ShardEdge, ShardManifestRecord, ShardHeader).
• Implement dynamic cache invalidation across all import resolvers (clear_cache).
• Enable downstream metacall/lsp development for single-language and polyglot navigation:
  • Phase 8a: Synchronous language server (goto-definition, hover, diagnostics).
  • Phase 8b: Cross-language jump-to-definition and reference resolution over metacall() boundaries.
  • Phase 8c: Signature enrichment and cross-language stub generation.

Exit gates:

1. Phase 0 engine seams implemented, tested, and schema version bumped to 2. DONE
2. .metast v2 modular shards, headers, and manifest files persist and restore graph topology. DONE
3. Resolver cache invalidation handles dynamic configuration updates. DONE
4. metacall/lsp language server crate operational against meta-ast core library.

Phase 9 - Engine Refactoring & Graph Reuse [PLANNED]

Goals:

• Zero-allocation resolver dispatch: replace Box<dyn ImportResolver> trait objects with an enum dispatch model (Resolver) to eliminate heap allocation during pipeline runs (issue #41).
• Language module deduplication: introduce declarative macros (define_language_pack!) to eliminate repetitive spec and query boilerplate across language packs (issue #39).
• Deploy pipeline modularization: extract DeployOrchestrator struct from run_deploy for single-responsibility and independent step reuse by downstream tools (issue #40).
• Reusable graph visitor interfaces over CodeGraph for custom static analysis passes.

Exit gates:

1. Zero heap allocations during per-file import resolution dispatch.
2. Language pack boilerplate reduced across Python, Ruby, C, C++, Rust, Go, JS, TS, and TSX.
3. DeployOrchestrator exposes individual pipeline stages (scan, partition, cuts, manifests, mesh).

Phase 10 - Polyglot Security & Taint Flow Analysis (SAST) [PLANNED]

Goals:

• Deliver cross-language taint-flow analysis across MetaCall FFI boundaries (issue #29, metacall/polyglot-sast).
• Detect untrusted inputs in one language reaching dangerous execution sinks in another language.
• Classify findings into Common Weakness Enumeration (CWE) categories.
• Output native SARIF (v2.1.0) reports for GitHub/GitLab Security tab integration.
• Integrate with MetaSSR as deployment-blocking middleware and dashboard visualization.

Exit gates:

1. Cross-language taint flow correctly traces from Python/JS inputs into C/Rust sinks.
2. Deterministic rule-based engine emits valid SARIF v2.1.0 reports.
3. MetaSSR deploy middleware blocks deployments with critical security findings.

Phase 11 - Developer Ecosystem & Community Tooling [IN PROGRESS]

Goals:

• Cross-platform distribution scripts: Unix scripts/install.sh (issue #46) and Windows scripts/install.ps1.
• Property-based testing with proptest for Tarjan SCC, cycle detection, and edge normalization invariants (issue #48).
• Streamline contributor experience: curated “Good First Issues” with detailed task guides.
• CLI output ergonomics: JSON error reporting and enhanced diagnostic formatting (issue #47).

Exit gates:

1. Verified curl/PowerShell installation scripts published for all release artifacts.
2. proptest suites validating graph normalization and SCC determinism.
3. Active contributor onboarding through structured issue templates.

Phase 12 - Deep Expression AST & Full Syntax Trees [PLANNED]

Goals:

• Extend meta-ast beyond coarse symbol-level IR into fine-grained expression syntax trees and intra-procedural Control Flow Graphs (CFG).
• Extract statement nodes, binary operations, control flow branches, and expression terms across all 9 supported languages.
• Maintain a layered representation:
  • Layer 1 (Default): Fast, lightweight symbol & reference graph.
  • Layer 2 (Opt-in): Full expression-level AST with lexical scopes and operator nodes.
• Generate intra-procedural CFGs for abstract interpretation, dead branch elimination, and fine-grained taint propagation.

Exit gates:

1. Full expression AST extractable via --depth full or extract_full_ast.
2. Control Flow Graph (CFG) generated with branch conditions and join nodes.
3. Zero performance regression on default symbol-only extraction passes.

Phase 13 - Polyglot Code Transformation & Refactoring Engine [PLANNED]

Goals:

• Evolve meta-ast from a read-only static analyzer into a bidirectional polyglot code transformation and refactoring engine.
• Implement lossless Concrete Syntax Tree (CST) rewriting, preserving whitespace, formatting, and comments.
• Deliver cross-language atomic symbol renaming:
  • Renaming a function or method in C, C++, or Rust automatically rewrites and updates all cross-language caller sites in Python, JavaScript, and Ruby.
• Implement automated polyglot code migrations, AST rewrite recipes, and FFI/RPC stub generation (meta-ast refactor, meta-ast codegen).
• Provide programmatic transformation APIs for language migration tools and automated refactorings.

Exit gates:

1. Lossless round-trip source rewriting verified across all 9 languages without formatting loss.
2. Cross-language atomic symbol renaming verified on mixed Python/JS/Rust/C fixture codebases.
3. Automated refactoring CLI (meta-ast refactor) and FFI stub generator (meta-ast codegen).

Strategic Architecture Evolution

meta-ast follows a phased strategic evolution from lightweight symbol graph to a full polyglot transformation engine:

1. Current Foundation (Phases 1-11):
   • High-speed, read-only static analysis and symbol-level IR.
   • Cross-language dependency graph, import resolution, and Tarjan SCC cycle detection.
   • Language Server (LSP) seams, shard index persistence (.metast v2), and security analysis (SAST).
2. Deep Syntax Expansion (Phase 12):
   • Full expression-level syntax trees and Control Flow Graphs (CFG) layered over the symbol graph.
3. Bidirectional Transformation (Phase 13):
   • Lossless CST source rewriting, cross-language atomic refactoring, and automated code generation.

Scope boundaries

• Core priority: general-purpose symbol extraction, cross-language dependency graph, cycle detection, shard persistence, zero-cost abstractions.
• Tooling priority: Polyglot LSP server (metacall/lsp), IDE integration, general-purpose CI gates.
• Evolution priority: Full expression AST (Phase 12), Polyglot code transformation & refactoring (Phase 13), SAST security analysis (metacall/polyglot-sast).

Crate Decisions

1. Decision principles

• Correctness and stability over novelty.
• Keep runtime dependencies minimal for CLI/library users.
• Use ecosystem-standard crates with strong maintenance signals.

2. Selected crates by concern

Parsing

• tree-sitter + language crates (c, cpp, python, javascript, typescript, rust, go)
• Rationale: robust incremental parsing and grammar-level extraction.
• Language crates provide battle-tested queries and node definitions.
• python, javascript, typescript as a start in every iteration.

Graph and SCC

• petgraph
• Rationale: mature directed graph algorithms and built-in Tarjan SCC.

Serialization

• serde, serde_json, yaml_serde
• Rationale: stable, standard JSON and YAML contract tooling.

CLI and watch

• clap, notify, notify-debouncer-mini, blake3
• Rationale: battle-tested CLI ergonomics with color, derive, and env support. notify and notify-debouncer-mini provide OS file events for watch mode. blake3 provides fast, deterministic cryptographic content hashing for file change fingerprinting.

Parallelism

• rayon
• Rationale: data-parallel file processing with work-stealing. A thread-local pool of Parser instances (thread_local!) enables safe parallel parse + extract per-file without non-Sync parser contention.

Enum utilities

• strum
• Rationale: derive macros for Display, EnumIter, EnumString on enums.

Filesystem

• ignore, dunce
• Rationale: gitignore-aware file walking (respecting .gitignore and .ignore files) and cross-platform path canonicalization.

Browser

• webbrowser
• Rationale: auto-open HTML dashboard in the user’s default browser.

Error handling

• thiserror (library errors), anyhow (application boundary)
• Rationale: explicit typed errors + practical context propagation.

3. Development dependencies

• insta - snapshot testing for JSON output contracts.
• criterion - benchmark gating (pipeline, graph, and incremental).
• tempfile - isolated filesystem fixtures for integration testing.

4. Feature flags

• watch - debounced file-system watch mode with incremental re-analysis and BLAKE3 fingerprinting.
• dataflow - data/flow node tracking and def-use graph extraction.
• metacall-deploy - includes deploy scanner/manifest/mesh generators for MetaCall deployment manifest generation.

5. Recommended additions

• tracing, tracing-subscriber - structured observability.
• cbindgen - C ABI header generation when ABI phase begins.

6. Alternatives and trade-offs

• Graph: custom adjacency maps can be faster but increase maintenance cost.
• CLI: smaller parsers reduce binary size but lose feature depth.
• JSON: high-performance serializers are unnecessary before proven bottleneck.
• Parallelism: crossbeam scopes are an alternative but rayon’s work-stealing is better suited for file-level data parallelism.
• Language dispatch: trait objects allow runtime plugins but lose compile-time completeness checking; enum dispatch chosen (see STRUCTURE.md).

7. Risk register

• Grammar drift risk (low): mitigate with fixtures + snapshots.
• Watch-mode debounce edge cases (low): mitigate with integration tests.
• Over-scoping optional sinks (medium): keep feature-gated.

8. Policy

Crate upgrades that affect behavior must include:

1. CI pass on all platforms.
2. Snapshot/fixture update.
3. Documentation update in this file and specs/symbol-extraction.md.

Benchmarks

Measured with criterion on the CI runner (ubuntu-latest) and on a local developer machine. Results below are the local run from 2026-08-03, machine: x86_64 Linux, release profile, --features watch.

Raw criterion reports are uploaded as CI artifacts from .github/workflows/benchmark.yml. To reproduce locally:

cargo bench --features watch

Pipeline (extraction)

End-to-end extraction across the per-language fixture suites:

Graph

Graph construction, Tarjan SCC, edge deduplication, and node lookup at scale:

Datagraph and dataflow

Incremental re-analysis

Requires --features watch. Cold analyze_graph vs warm incremental_reanalyze after a single-file change (FR-5 target: <100 ms for files under 5k LOC):

Demo

This page walks the delivered artifacts end to end. The animated GIFs below were recorded from the real CLI against the fixture trees in this repository. Each GIF shows the command being typed and its actual output.

Intro

Version, help, and a first inspect run over the Python fixtures:

Graph

Cross-language dependency graph with SCC analysis over the tests/fixtures/mixed/three_lang_math project (Python orchestrator, JS and Rust workers), first as JSON then as YAML:

Deploy

metacall-deploy manifest generation for the same project, producing metacall.pods.json and metacall.mesh.json:

Watch mode

Debounced incremental re-analysis: the watch loop prints a fresh graph snapshot on every tick. An edit to main.py adds a function, and the next snapshot picks up the new symbol (node_count grows, snapshot_id increments) without a full re-parse of unchanged files:

Try it yourself

git clone https://github.com/metacall/meta-ast.git
cd meta-ast
cargo build --release --all-features

./target/release/meta-ast inspect tests/fixtures/python
./target/release/meta-ast graph tests/fixtures/mixed/three_lang_math --html
./target/release/meta-ast deploy tests/fixtures/mixed/three_lang_math --out ./deploy-out

Final Report: meta-ast, GSoC 2026

• Author: Khaled Alam
• Organization: MetaCall
• Project: meta-ast - Standalone Polyglot Static Analysis Engine
• Program: Google Summer of Code (GSoC) 2026
• Status: Completed, v0.5.0

1. Project Summary and Goals

meta-ast is a fast, standalone static analysis engine written in Rust. The project parses source code across nine programming languages, extracts a normalized symbol Intermediate Representation (IR), builds cross-language dependency graphs, detects import cycles with Tarjan Strongly Connected Components (SCC), and generates deployment manifests for the MetaCall Function Mesh runtime. The engine never executes target code.

Original Problem Statement

MetaCall supports polyglot architectures where functions written in different languages call each other seamlessly. However, developers lacked a fast, unified static analysis tool to:

1. Map cross-language dependencies without running arbitrary user code.
2. Detect cyclic imports that block deployment decomposition.
3. Automatically partition polyglot applications into optimal, language-specific deployment pods.
4. Pin external package dependencies across multiple language package managers.

Project Goals

1. Polyglot Parsing: Parse 9 languages (Python, JavaScript, TypeScript, TSX, C, C++, Rust, Go, Ruby) using a unified tree-sitter pipeline.
2. Normalized Symbol IR: Extract functions, classes, methods, structs, enums, and interfaces into a language-agnostic intermediate representation.
3. Cross-Language Dependency Graph: Resolve imports, symbol references, and cross-language call sites with confidence-weighted edges.
4. Cycle Detection and Pod Partitioning: Identify cyclic clusters using Tarjan SCC and partition code into same-language deployment units.
5. MetaCall Manifest Generation: Scan cross-language metacall_load_from_* and metacall() invocations, generate pod manifests (metacall.pods.json) and mesh annotations (metacall.mesh.json), and validate cut fairness.
6. High-Performance Watch Mode: Deliver sub-100 ms incremental re-analysis using cryptographic content hashing and zero-allocation cache reuse.
7. Production Quality: Provide comprehensive test coverage, robust CI/CD, cross-platform release binaries, and complete documentation.

2. What Was Accomplished (Phase-by-Phase)

The project executed across seven planned phases. All milestones were delivered, tested, and released.

Phase 1: Unified Parser Lifecycle and Symbol Extraction

• Built thread-local tree-sitter parser pools for zero-overhead multi-threaded parsing.
• Implemented uniform AST query packs across Python, JavaScript, TypeScript, TSX, C, C++, Rust, and Go.
• Normalized declarations into the Symbol IR with visibility, signature, and docstring metadata.
• Implemented robust error recovery: malformed source files emit structured Diagnostic records without stopping the pipeline.

Phase 2: Dependency Graph and Tarjan SCC

• Implemented the CodeGraph directed graph model over petgraph.
• Implemented cross-file import and symbol resolution with confidence scoring (1.0 for own file / direct import, 0.8 for transitive import, 0.6 for cross-language).
• Integrated Tarjan Strongly Connected Components (SCC) algorithm with EdgeFiltered views to isolate cyclic clusters while excluding ownership and flow edges.
• Classified graph components into independent deployment units vs. co-deployment clusters.

Phase 3: Datagraph and Dataflow Sinks

• Extended the IR with intra-procedural def-use dataflow nodes (DataNode, FlowEdge, DataScope).
• Implemented def-use extraction for parameter bindings and variable declarations.
• Defined a schema-versioned export format (schema version 1) and the GraphSink pluggable adapter trait.
• Added the --datagraph CLI flag for dataflow export.

Phase 4: CLI Polish, Visualization, and Incremental Watch Mode

• Added structured output formats (--format json|yaml) across all subcommands.
• Implemented an interactive Cytoscape.js HTML visualization dashboard (--html).
• Built incremental watch mode (--watch):
  • File-system monitoring with debounced event processing.
  • BLAKE3 cryptographic content hashing to detect modified files.
  • Zero-allocation cache reuse for unchanged files via Arc<FileExtraction>.
  • Collision-free ID generation using IdGenerator::with_start.

Phase 5: MetaCall Deployment Manifest Generator (metacall-deploy)

• Implemented AST scanners for MetaCall call sites (metacall_load_from_file, metacall_load_from_memory, metacall_load_from_package, metacall_load_from_configuration, and metacall() client calls per RFC 0011).
• Implemented same-language pod partitioning using Union-Find over resolved dependency edges.
• Added external package dependency resolution from lockfiles (package-lock.json, Cargo.lock, go.sum, requirements.txt, Gemfile.lock) for exact version pinning.
• Generated deployment artifacts:
  • metacall.pods.json: Pod manifests with deployment units, dependency lists, and AST metrics.
  • metacall.mesh.json: Function Mesh topology annotations with cross-language edge attribution.
• Added --check mode: verifies cut fairness (every cut edge across pods has a corresponding RPC stub entry, fulfilling ADR 0003).

Phase 6: Language Expansion (Ruby)

• Implemented full Ruby support: tree-sitter grammar integration, symbol query pack, require/require_relative import resolver, reference detection, and lockfile resolution (Gemfile.lock).
• Evaluated C# and Java; concluded Ruby provided the highest immediate utility for MetaCall Function Mesh targets.

Phase 7: Validation, CI/CD, Benchmarking, and Delivery

• Hardened CI matrix across 4 operating systems (Linux glibc/musl, macOS x86_64/ARM64, Windows x86_64/ARM64) on stable and nightly Rust toolchains.
• Added automated linting, formatting, cargo-deny license/vulnerability audits, and cargo-nextest execution.
• Configured Criterion benchmark suite tracking pipeline, graph, and incremental performance.
• Published mdbook documentation to GitHub Pages.
• Published crate releases to crates.io and multi-platform binaries to GitHub Releases.

3. Code Merged Upstream

All work was developed in pull requests, reviewed, and merged into the main branch of metacall/meta-ast.

Merged Pull Requests

Notes

• there was multiple commits unlinked with issues or PRs “Mostly optimizations”.

4. Key Metrics and Project Numbers

• Languages Supported: 9 languages (Python, JavaScript, TypeScript, TSX, C, C++, Rust, Go, Ruby).
• Test Suite: 430+ automated tests (327 unit tests, 106 integration tests, doc tests).
• Test Matrix: 4 Operating Systems (Linux, macOS, Windows, Windows ARM) across 2 Rust toolchains (stable and nightly).
• Performance Benchmark:
  • Incremental warm re-analysis: 1.16 ms for single-file changes (target was <100 ms).
  • Graph construction & Tarjan SCC: <0.5 ms for 1,000 nodes.
  • End-to-end extraction across fixture suite: 16.8 ms.
• Releases:
  • 5 GitHub releases (v0.1.0 through v0.5.0).
  • 14 pre-compiled binary packages per release (covering Linux glibc/musl, macOS x86/ARM, Windows x86/ARM across core and deploy configurations).
  • 2 published crates.io versions (0.4.0, 0.5.0).

5. Technical Challenges and Engineering Insights

A. Polyglot Semantic Gap and Import Resolution

Challenge: Each programming language uses different module and symbol resolution semantics. C/C++ relies on header inclusion; Python uses runtime sys.path and relative imports; JavaScript/TypeScript uses ESM, CommonJS, and tsconfig.json path mappings; Rust uses hierarchical crate paths; Ruby uses require and require_relative. Solution: Designed the ImportResolver abstraction. Each language provides a stateful resolver that caches directory hierarchies and configuration files. Unresolved imports are recorded with confidence penalties (0.6 cross-language vs 1.0 same-file) and surface as non-fatal diagnostics.

B. Collision-Free Incremental State in Watch Mode

Challenge: When re-analyzing modified files in watch mode, creating new symbol IDs could collide with cached IDs from unchanged files or force expensive whole-graph re-allocations. Solution: Implemented an explicit ID generation seam using IdGenerator::with_start(max_cached_id + 1). Unchanged files retain their existing Arc<FileExtraction> references with zero allocations (verified via Arc::ptr_eq), while newly extracted files receive strictly monotonic, non-overlapping IDs.

C. Deployment Cut Fairness and Invariant Verification

Challenge: Partitioning polyglot applications into separate deployment pods can sever dependencies. If an edge across pods lacks an RPC stub, the deployed application fails at runtime. Solution: Implemented the cut fairness validation algorithm (ADR 0003). In --check mode, the analyzer constructs a bijection between inter-pod cut edges and declared RPC stubs. Any missing stub produces an immediate diagnostic error with the exact source location of the call site.

D. Graph Assembly Performance at Scale

Challenge: Initial graph construction used repeated edge scans to deduplicate multi-language references, causing quadratic slowdowns on large graphs. Solution: Replaced linear scans with an indexed (src, dst, kind) lookup map in CodeGraph, delivering O(1) edge deduplication and max-confidence fusion. This reduced graph construction time for 10,000 duplicate edges to 486 microseconds.

E. Cross-Platform Path Normalization

Challenge: File paths and snapshots differed across Linux, macOS, and Windows due to backslashes and case sensitivity. Solution: Enforced universal forward-slash path normalization across all internal data structures, JSON serializers, and snapshot tests, ensuring deterministic CI verification across all four operating systems.

6. How to Build, Test, and Verify

Prerequisites

• Rust 1.94.0 or newer (rustup.rs)

Build from Source

git clone https://github.com/metacall/meta-ast.git
cd meta-ast

# Build core analyzer
cargo build --release

# Build with deployment manifest generator and watch mode
cargo build --release --features metacall-deploy --features watch --features dataflow

Run Test Suite

# Run all tests across all feature flags
cargo test --all-features

# Run linter and formatting checks
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --check

Run Benchmarks

cargo bench --features watch

Run CLI Commands

# Inspect declarations in a project
./target/release/meta-ast inspect ./tests/fixtures/python/ -f json

# Build graph with interactive HTML dashboard
./target/release/meta-ast graph ./tests/fixtures/mixed/ --html -o dashboard.html

# Generate MetaCall deployment manifests and check cut fairness
./target/release/meta-ast deploy ./tests/fixtures/mixed/auth_microservice --check -o ./deploy_out

7. Current State and Future Work

Current State

meta-ast is feature-complete for its GSoC 2026 milestones and project goals. The engine is released as v0.5.0 on crates.io and GitHub Releases, with documentation published at metacall.github.io/meta-ast.

Future Work (Post-GSoC Roadmap)

• C ABI and Header Generation: Implement automatic C header generation for exported polyglot functions (RFC 0011).
• Deeper Intra-Procedural Dataflow: Expand dataflow node extraction from Rust to JavaScript, TypeScript, Python, and Go.
• Additional Language Packs: Add grammars for PHP, Java, and C# based on user demand.
• Polyglot SAST Integration: Integrate static application security testing rules and machine learning assisted anomaly detection (RFC 0029).

8. Important Links

• Source Code Repository: https://github.com/metacall/meta-ast
• crates.io Package: https://crates.io/crates/meta-ast
• Documentation Book: https://metacall.github.io/meta-ast/
• GitHub Releases & Binaries: https://github.com/metacall/meta-ast/releases
• Architecture Overview: docs/src/ARCHITECTURE.md
• Deployment Manifest Specification: docs/src/DEPLOY.md
• Benchmark Suite: docs/src/BENCHMARKS.md
• Recorded Demonstrations: docs/src/DEMO.md

Requirements Specification

1. Purpose

Define normative requirements for meta-ast, a standalone Rust static analyzer for polyglot source trees. The tool extracts symbol surfaces, builds cross-file dependency graphs, and computes SCCs. MetaCall FaaS deployment manifest generation is an optional capability behind the metacall-deploy feature flag.

2. Functional requirements

FR-1: Parsing and language support

The analyzer shall parse source files for a growing set of languages, starting with:

• Python
• JavaScript
• TypeScript (including TSX)
• C
• C++
• Rust
• Go

Additional languages (C#, Java, and others) are planned in later phases.

FR-2: Symbol extraction

The analyzer shall extract top-level symbols and language-appropriate nested symbols into a normalized intermediate representation:

• Functions
• Classes / structs / interfaces / traits
• Objects (constants, globals, module-level bindings)

FR-3: Dependency graph

The analyzer shall construct a directed graph of symbol-level dependencies:

• File import/usage edges
• Intra-project symbol references
• Cross-language reference candidates

FR-4: SCC and Deployment Unit identification

The analyzer shall compute Strongly Connected Components (Tarjan) and annotate each SCC as a Deployment Unit, classifying it as:

• Independent (acyclic, Function Mesh separation candidate)
• Co-deployment required (cyclic, must remain grouped)

FR-5: Incremental updates (Planned)

The analyzer is designed to support update workflows from file changes with a target incremental response of under 100ms for files below 5k LOC. Note: This is planned for Phase 4 of the roadmap and is not yet implemented.

FR-6: CLI and library modes

The project shall expose:

• Rust library interface
• CLI entrypoint for project analysis and output emission

FR-7: C ABI (Planned)

The project will provide a stable C ABI header (mc_ast.h) for embedding scenarios in a later phase.

FR-8: Datagraph export (Planned)

The project will support exporting a datagraph model suitable for external graph sinks (including Dgraph) in a later phase.

FR-9: Deploy Manifest generation (feature-gated: metacall-deploy, Implemented)

When built with --features metacall-deploy, the deploy subcommand:

• Scans source files for cross-language call sites (metacall_load_from_file, metacall_load_from_memory, metacall_load_from_package, metacall_load_from_configuration)
• Extracts (language_tag, script_paths[]) pairs from call-site arguments
• Partitions files into same-language pods via Union-Find over dependency edges
• Resolves external dependencies per-language from lockfiles and package manifests
• Generates a pod manifest (metacall.pods.json) with per-pod deployments, inter-pod edges with fused confidence scores, and scoped dependency lists with pinned versions
• Inlines referenced metacall_load_from_configuration targets when present; emits low-confidence annotations for computed/dynamic arguments
• Runs fairness checks when --check is passed: every cut edge must have a corresponding RPC stub entry in the manifest

FR-10: Mesh Annotation (feature-gated: metacall-deploy, Implemented)

When built with --features metacall-deploy, the deploy subcommand also emits metacall.mesh.json containing:

• SCC-derived deployment units with constituent symbol lists
• Cross-language boundary flags per unit
• Independent mesh candidate classification per unit
• Cross-language edges with call-site file attribution

3. Non-functional requirements

NFR-1: Correctness

Incorrect symbol labeling is a high-severity defect. Correctness takes priority over throughput.

NFR-2: Resilience

Malformed source files shall not crash analysis. Partial extraction shall be allowed when parser recovery is possible. Unresolvable Cross-Language Call Site arguments (dynamic values) shall be annotated, not silently discarded.

NFR-3: Portability

Build and test shall pass on Linux, macOS, and Windows.

NFR-4: Determinism

Given identical input set and tool version, emitted output shall be deterministic.

NFR-5: Observability

The implementation shall provide structured diagnostics suitable for CI and local debugging.

4. Acceptance criteria

1. Parse all supported languages and emit valid symbol JSON.
2. Correct SCC identification for multi-language project fixtures.
3. CI pipeline green on Linux/macOS/Windows.
4. (Planned) Incremental-update target under 100ms for files below 5k LOC.
5. (Planned with metacall-deploy) Deploy Manifests generated match expected fixtures for all example projects in tests/fixtures/mixed/.
6. (Planned with metacall-deploy) Mesh Annotation correctly classifies Deployment Units for the auth-function-mesh fixture.

5. Out-of-scope for MVP

• Incremental watch-mode and C ABI embedding (post-MVP).
• metacall-deploy manifest/mesh generation and --check mode (post-MVP).
• Dgraph sink/export adapter (post-MVP).
• Full inter-procedural global dataflow with alias analysis.
• Sound cross-language type inference.
• Mandatory online graph database dependency.
• Dynamic Cross-Language Call Site resolution (runtime tag/path values).

6. Versioning policy

Breaking changes to output schema or graph semantics require:

1. ADR update.
2. Traceability matrix update.
3. Migration note in roadmap/changelog.

Graph Model Specification

1. Purpose

Define the normalized graph model used for dependency analysis and SCC computation.

2. Node categories

FileNode

• id: stable identifier
• path: normalized path relative to the analyzed project root
• language_id: configured language/runtime identifier
• snapshot_id: snapshot identifier for the analysis version this file belongs to

SymbolNode

• id: stable identifier
• name: symbol name
• kind: source symbol category such as function | class | object | interface | trait | struct | enum | method
• file_id: references the owning FileNode.id
• visibility: optional, when applicable: public | private
• source_range: byte/line range

DataNode

• id: stable identifier for a value-bearing node
• symbol_id: optional symbol reference when the data node is derived from a named symbol
• scope: local, parameter, closure, member, or temporary scope classification
• type_hint: optional inferred or declared type information

DataNode represents a value or variable instance used for def-use and flow analysis rather than a declaration boundary.

3. Edge categories

ImportEdge

FileNode -> FileNode representing import/include/use relationships.

ReferenceEdge

SymbolNode -> SymbolNode representing symbol usage/call/reference candidates.

OwnershipEdge

FileNode -> SymbolNode and optional SymbolNode -> SymbolNode for nesting.

FlowEdge

DataNode -> DataNode for def-use transitions.

4. Graph invariants

1. Every SymbolNode must map to exactly one FileNode.
2. Ownership edges must form an acyclic containment structure.
3. SCC computation applies to dependency/reference subgraph, not ownership edges. Self-loop detection and independence classification follow the same subgraph rule.
4. Duplicate edges should be normalized by (src, dst, edge_kind) key. Client-call edges (FileNode to SymbolNode) can only merge with other client-call edges; scope-resolved references (SymbolNode to SymbolNode) never collide with them. Strongest evidence wins within a triple.

5. SCC semantics

Tarjan SCC runs on directed dependency/reference graph.

• SCC size = 1 with no self-loop => acyclic unit.
• SCC size > 1 or self-loop => cyclic unit.

Deployability hint policy:

• Acyclic SCCs are preferred deployment candidates.
• Cyclic SCCs require grouped deployment or refactor guidance.

6. Serialization contract

Graph serialization for external consumers shall preserve:

• snapshot-local node IDs
• edge kinds
• defining file paths and source ranges for symbol nodes

Sink adapters (e.g., Dgraph) that must preserve semantic equivalence.

The serialized language field on file and external nodes uses the canonical lowercase language names, in enum declaration order:

python, javascript, typescript, tsx, c, cpp, rust, go, ruby

These values also parse back through the CLI --language flag. The same names apply to the strum Display/AsRefStr and serde representations of LangId.

Graph output schema version 2 adds file_path and source_range to serialized symbol nodes. .metast v2 shards do not persist numeric IDs. They store stable language-scoped endpoint names and regenerate symbol IDs when loaded.

7. Known limitations

• Cross-language resolution is initially best-effort string/scope matching.
• Full semantic type equivalence across languages is deferred.
• DataNode/FlowEdge extraction is implemented for Rust (let bindings, parameters, def-use chains) behind the dataflow feature flag; other languages return empty vectors. Python/JS/TS/Go/C/C++ are stubbed with TODO markers. Full coverage is tracked in Phase 6.

Symbol Extraction Specification

1. Purpose

Define language-pack extraction contracts backed by Tree-sitter grammar queries. Each language pack covers symbols, imports, and references. The set of supported languages grows over time; see ROADMAP.md Phase 6 for the expansion plan.

2. Shared extraction rules

• Prefer grammar field-based extraction where available.
• Record both byte and line/column ranges.
• Continue extraction in presence of parser recovery nodes when safe.

3. Language packs

Python

• Extract: functions, classes, imports, module-level assignments.
• Handle decorated definitions and async functions.

JavaScript

• Extract: function declarations, function expressions, arrow functions, classes, methods, imports/exports.

TypeScript / TSX

• Extract JS symbols plus interfaces, type aliases, enums.
• Use TSX grammar for JSX-bearing files.

C

• Extract: function definitions/declarations, structs, enums, typedefs, includes.
• Distinguish declaration vs definition ‘where possible’.

C++

• Extract C symbols plus classes, namespaces, templates, aliases, method definitions.

Rust

• Extract: functions, structs, enums, traits, impl blocks, use declarations, modules, const/static/type aliases.

Go

• Extract: functions, methods with receiver, types, interfaces, imports, const/var declarations.

4. Output normalization

Each extracted symbol maps to canonical shape:

• name
• kind
• language
• file
• source_range
• optional: signature, visibility, docstring, async

5. Error tolerance policy

• Keep partial output if recoverable parse exists.
• Emit diagnostics for query compile failures or unsupported grammar drift.

6. Version policy

Query packs are tied to grammar versions in Cargo.toml. Any grammar upgrade requires:

1. Query validation pass.
2. Fixture/snapshot refresh.
3. Update to this spec.

Traceability Matrix

Deliverables to docs mapping

Acceptance criteria mapping

Decision traceability

0001-stateful-import-resolver

We introduced a stateful ImportResolver trait utilizing thread-safe interior mutability (RwLock and OnceLock) for caching module boundaries and file existence checks, replacing the previous stateless function pointer resolution.

This design satisfies Rust’s thread-safety constraints (Send + Sync) to allow gradual migration of 8 languages, prepares the pipeline for parallelized import resolution under Rayon, and supports persistent cache reuse across incremental compilation runs.

0002-scope-resolution-heuristics

We resolve symbol references across files using a single, unified scope resolution strategy: transitive BFS lookup over the import graph.

Rather than maintaining separate “strict” and “heuristic” CLI flags, we build a flattened scope cache for each file. The resolver crawls the import graph using a breadth-first search (BFS). Confidence scores decay based on import distance and language boundaries:

• 1.0: Reference is local (distance = 0) or direct, same-language (distance = 1).
• 0.8: Reference is transitive, same-language (distance > 1).
• 0.6: Reference is cross-language (distance >= 1).

This keeps the CLI interface clean and guarantees security or dependency tools high recall by default, while still capturing precision through the decayed confidence scores.

0003-unresolved-import-policy

We defined a strict policy for handling unresolved imports to prevent silent failures and preserve dependency context.

Unresolved Relative Imports (specifiers starting with . or /) are treated as configuration errors and emit a warning Diagnostic containing the source range and path. Unresolved Non-Relative Imports are treated as third-party package dependencies and are mapped to a placeholder External Node in the directed graph rather than being silently discarded, preserving the complete structural architecture.

0004-global-scope-synthetic-symbols

We proposed introducing a synthetic Module Body Symbol (named "<global>" or "<module>") for files containing top-level global execution blocks to represent module-level dependencies. Under that proposed schema, references occurring outside any class, function, or method boundary would be owned by this synthetic symbol.

Current MVP Status: We do not yet generate this synthetic symbol. In the current implementation, references that occur at the top-level (outside of functions or classes) have a None source symbol and are skipped during scope resolution.

Implementing the synthetic module body symbol is deferred to a future phase to maintain simplicity in the initial graph schema.

RFC 0001: Language Loading Model

Status

Accepted

Context

The analyzer must support multiple Tree-sitter grammars with predictable behavior and low operational complexity.

Decision

Use compile-time language crate integration with explicit language dispatch.

Alternatives considered

1. Runtime-loaded grammars via dynamic libraries.
2. Hybrid runtime plugin registry.

Rationale

Compile-time dispatch maximizes type safety and build determinism for MVP scope.

Consequences

• New language support requires source-level addition and release.
• Lower runtime complexity and fewer deployment surprises.

RFC 0002: Error Semantics and Recovery

Status

Accepted

Context

Static analysis should remain useful even on partially invalid source code.

Decision

Treat parse errors as recoverable when Tree-sitter yields partial trees; treat unrecoverable extraction/config errors as scoped failures with diagnostics.

Alternatives considered

1. Fail-fast on first parse anomaly.
2. Silent permissive mode with minimal diagnostics.

Rationale

Recoverable parsing preserves developer feedback loops while maintaining explicit failure signaling.

Consequences

• Partial results are possible and expected.
• Diagnostics become part of core product quality.

RFC 0003: Incremental Parsing Strategy

Status

Accepted

Context

Proposal target includes incremental responsiveness (<100ms for files under 5k LOC).

Decision

Adopt a staged approach:

1. Correct baseline implementation first.
2. Incremental optimization with InputEdit and changed-range narrowing behind benchmark evidence.

Alternatives considered

1. Full incremental complexity from day one.
2. Full reparse forever.

Rationale

Correctness-first minimizes early defect risk; benchmark-driven optimization avoids premature complexity.

Consequences

• Early versions may reparse more broadly.
• Optimization milestones are explicitly measurable.

RFC 0004: Graph Representation and SCC

Status

Accepted

Context

The project requires dependency analysis and SCC computation across polyglot symbol sets.

Decision

Use a directed graph representation with explicit node/edge kinds and Tarjan SCC as the canonical cycle analysis algorithm.

Alternatives considered

1. Custom adjacency structures.
2. Relational-first representation.

Rationale

A typed directed graph model matches analysis semantics and keeps SCC computation straightforward and testable.

Consequences

• Graph model contracts must stay stable (specs/graph-model.md).
• SCC behavior is deterministic and auditable.

RFC 0005: Output Contract Policy

Status

Accepted

Context

Compatibility with MetaCall-style inspect consumers is a central project requirement.

Decision

Make inspect-compatible JSON the primary output contract, preserving stable keys:

• funcs
• classes
• objects

Alternatives considered

1. New schema first, with adapters later.
2. Multiple equal-priority formats.

Rationale

Contract stability reduces integration risk and aligns with proposal goals.

Consequences

• Breaking output changes require versioning, migration note, and traceability update.
• Snapshot/schema validation becomes mandatory in CI.

RFC 0006: Type Inference Scope

Status

Accepted

Context

Cross-language full type inference is high complexity and not required for MVP parity.

Decision

Limit MVP to extraction of declared/type-hint metadata and best-effort symbol linking. Defer sound cross-language inference to future milestones.

Alternatives considered

1. Full type system in MVP.
2. No type metadata at all.

Rationale

Keeps scope feasible while preserving useful metadata for analysis.

Consequences

• Some cross-language links remain heuristic.
• Future advanced inference can be introduced without breaking core contracts.

RFC 0007: Dgraph Integration Scope

Status

Accepted

Context

The proposal includes Dgraph sink capability, but standalone analyzer operation is mandatory.

Decision

Treat Dgraph integration as optional/export-layer capability. Core analysis must not depend on external graph database availability.

Alternatives considered

1. Dgraph as required runtime dependency.
2. No graph sink pathway.

Rationale

Optional integration preserves standalone usability and reduces operational burden in MVP.

Consequences

• Export contracts must remain sink-agnostic.
• Dgraph adapter can evolve independently behind feature boundaries.

RFC 0008:Graph Module and SCC Analysis

Status

Accepted

Context

Phase 1 of meta-ast established the foundation: file discovery, parsing, symbol extraction, and inspect-compatible JSON output. Phase 2 extends this with dependency graph construction and Strongly Connected Component (SCC) analysis to provide deployability insights for polyglot codebases.

The graph model was specified in specs/graph-model.md and the implementation approach using petgraph was validated through research. This RFC defines the concrete module structure, APIs, and implementation strategy.

Goals

1. Build a directed dependency/reference graph from extracted symbols
2. Implement Tarjan SCC algorithm for cycle detection
3. Provide deployability hints based on SCC analysis
4. Maintain stable inspect output contract (funcs, classes, objects)
5. Support cross-file dependency mapping for mixed-language projects

Non-Goals

1. Full inter-procedural dataflow analysis (Phase 3)
2. Live graph database integration (Phase 4)
3. Real-time incremental graph updates (Phase 4)
4. Cross-language type inference (out of scope for MVP)

Design

1. Module Structure

New modules under src/graph/:

Output extension: src/output/graph.rs for graph serialization.

2. Graph Types

Node storage uses petgraph DiGraph<NodeData, EdgeData> with stable node indices. NodeData is an enum for heterogeneous node types:

#![allow(unused)]
fn main() {
pub enum NodeData {
    File(FileNode),
    Symbol(SymbolNode),
}

pub struct FileNode {
    pub id: FileId,
    pub path: PathBuf,           // Project-relative as discussed with vicente
    pub language_id: LangId,
    pub snapshot_id: SnapshotId,
}

pub struct SymbolNode {
    pub id: SymbolId,
    pub name: String,
    pub kind: SymbolKind,
    pub file_id: FileId,
    pub visibility: Option<Visibility>,
    pub source_range: SourceRange,
}
}

Edges carry kind and metadata:

#![allow(unused)]
fn main() {
pub enum EdgeKind {
    Import,      // File imports another file
    Reference,   // Symbol references another symbol
    Ownership,   // File owns symbol, or symbol contains nested symbol
}

pub struct EdgeData {
    pub kind: EdgeKind,
    pub strength: EdgeStrength,  // Strong, Weak, or Dynamic
}

pub enum EdgeStrength {
    Strong,      // Direct, resolvable dependency
    Weak,        // Optional or conditional
    Dynamic,     // Runtime-resolved
}
}

3. Graph Construction

Two-phase construction from extraction results:

Phase A - Ownership graph (always acyclic by construction):

• Add FileNode for each processed file
• Add SymbolNode for each extracted symbol
• Add Ownership edges: FileNode -> SymbolNode

Phase B - Dependency graph:

• Add Import edges: FileNode -> FileNode (cross-file imports)
• Add Reference edges: SymbolNode -> SymbolNode (symbol usage)

Import extraction extends the language pack system with import-specific tree-sitter queries per language.

4. SCC Analysis

SCC computation follows specs/graph-model.md invariants:

1. SCC runs on dependency subgraph only (Import + Reference edges)
2. Ownership edges are explicitly excluded from SCC computation
3. Duplicate edges normalized by (src, dst, edge_kind)

Tarjan’s algorithm via petgraph::algo::tarjan_scc produces components in reverse topological order.

SccAnalysis result structure:

#![allow(unused)]
fn main() {
pub struct SccAnalysis {
    pub components: Vec<Scc>,
    pub node_to_component: HashMap<NodeIndex, usize>,
}

pub struct Scc {
    pub index: usize,
    pub nodes: Vec<NodeIndex>,
    pub is_cyclic: bool,
    pub deployability_hint: DeployabilityHint,
}

pub enum DeployabilityHint {
    Independent,        // Size=1, no self-loop
    AcyclicDependency,  // Size=1, depends on other components
    CyclicCluster,      // Size>1 or self-loop present
}
}

5. CLI Integration

New subcommand graph alongside existing inspect:

#![allow(unused)]
fn main() {
pub enum Cli {
    Inspect(InspectArgs),
    Graph(GraphArgs),  // New
}
}

GraphArgs accepts same path/language filters as InspectArgs plus output format options.

Output format (JSON):

{
  "meta": {
    "snapshot_id": 1,
    "file_count": 10,
    "symbol_count": 150,
    "edge_count": 200
  },
  "nodes": [
    {"id": "F0", "kind": "file", "path": "src/main.py", "language": "python"},
    {"id": "S42", "kind": "symbol", "name": "main", "kind": "function", "file_id": "F0"}
  ],
  "edges": [
    {"source": "F0", "target": "F1", "kind": "import"},
    {"source": "S42", "target": "S43", "kind": "reference"}
  ],
  "sccs": [
    {
      "index": 0,
      "nodes": ["S10", "S11"],
      "is_cyclic": true,
      "deployability": "cyclic_cluster",
      "size": 2
    }
  ],
  "deployability_report": {
    "independent_units": 120,
    "cyclic_clusters": 5,
    "total_components": 125
  }
}

6. Error Handling

Graph construction errors are recoverable and emit diagnostics:

• Missing import target: warning, edge not added
• Duplicate edge: deduplicated silently
• Cycle in ownership edges: error (violates invariant)

SCC computation is infallible for valid graphs.

7. Testing Strategy

1. Unit tests for GraphBuilder with known input/output
2. Unit tests for SCC computation on synthetic cyclic/acyclic graphs
3. Fixture tests for cross-file import detection per language
4. Integration tests for mixed-language dependency chains
5. Snapshot tests for graph JSON output format

8. Performance Considerations

• Graph construction: sequential (linear in symbols + edges)
• Symbol extraction remains parallel (rayon)
• SCC computation: O(V + E) via Tarjan
• Memory: adjacency list via petgraph (O(V + E))
• No incremental updates in Phase 2 (full graph rebuild)

Migration Path

This RFC introduces new modules without breaking existing Phase 1 functionality:

• inspect subcommand unchanged
• Symbol extraction interface unchanged
• New graph functionality additive only

Existing tests continue to pass. New tests validate graph-specific behavior.

Alternatives Considered

Alternative 1: Relational-first representation

Instead of adjacency list, use relational tables (Vec of nodes, Vec of edges with indices).

Rejected: petgraph provides battle-tested algorithms and the graph is inherently graph-structured. Relational adds indirection without benefit for SCC computation.

Alternative 2: Custom SCC implementation

Implement Tarjan from scratch instead of using petgraph.

Rejected: petgraph’s implementation is optimized and widely tested. No performance justification for custom implementation.

Alternative 3: Separate ownership and dependency graphs

Maintain two distinct graph structures.

Rejected: Single graph with edge kind filtering is simpler and memory-efficient. SCC explicitly filters by edge kind.

Open Questions

1. Should we include intra-file reference edges (symbol calls within same file) in SCC? yes, for completeness.

2. How to handle unresolved imports (external dependencies)? Skip with warning,? Decision: Skip with warning for Phase2.

3. Should deployability hints include suggested entry points for cyclic clusters? Deferred to Phase 3 when call graph is richer.

References

• specs/graph-model.md - Graph semantics and invariants
• specs/requirements.md - FR-4, FR-5 for graph and SCC requirements
• ADR 0004 - Graph representation decision record
• petgraph documentation: https://docs.rs/petgraph

RFC 0009: Cross-File Dependency Mapping

Status

Accepted

Context

The current meta-ast architecture successfully extracts definitions (functions, classes) and builds a foundational CodeGraph. However, it does not extract or resolve cross-file dependencies (imports and references) in the main pipeline. For meta-ast to serve as a robust backbone for SAST tools, it must accurately map how data and control flow across file boundaries.

This RFC defines the implementation of a graph-driven scope resolution strategy with high-performance optimizations to bridge the gap between extraction and dependency mapping.

Goals

1. Extract import statements and symbol references across supported languages.
2. Resolve symbol references to their definitions using graph-aware scoping.
3. Validate cross-file dependency mapping on mixed-language projects.
4. Maintain high precision and performance for large-scale codebases.

5. Multi-Pass Resolution Strategy

To ensure all definitions are available before resolution, the pipeline follows four sequential passes:

1. Pass 1: Extract symbols, imports, and references (Parallelizable).
2. Pass 2: Build File Nodes and resolve/add ImportEdges.
3. Pass 3: Build Export Map Cache (BFS/DFS with circular dependency guards).
4. Pass 4: Resolve References using local scope + Export Map.

Design

1. Data Model Extensions

New Types in src/model/mod.rs:

#![allow(unused)]
fn main() {
pub struct UnresolvedImport {
    pub target_path: String, // Raw path from source
    pub namespace: Option<String>,
    pub alias: Option<String>,
    pub is_star: bool,
    pub range: SourceRange,
}

pub struct UnresolvedReference {
    pub source_symbol: Option<SymbolId>, // None if module-level
    pub name: String,
    pub range: SourceRange,
}
}

2. Extraction Pipeline Update

The ExtractionResult carries these unresolved items. Note that source_path is omitted from individual structs as it is stored at the container level.

#![allow(unused)]
fn main() {
pub struct ExtractionResult {
    pub symbols: Vec<Symbol>,
    pub imports: Vec<UnresolvedImport>,
    pub references: Vec<UnresolvedReference>,
    pub diagnostics: Vec<Diagnostic>,
}
}

Testing Strategy

1. Unit Tests: Verify GraphBuilder resolution with shadowing, circular dependencies, and transitive imports.
2. Integration Tests: Use tests/fixtures/mixed/ to validate cross-language resolution precision.

Key Design Decisions

1. Decoupled Extraction Identifiers

UnresolvedImport and UnresolvedReference use name-based/path-based identification during extraction. This ensures the parser remains stateless and testable in isolation from the global FileId state.

2. Graph-Driven Scope Resolution with Export Maps

To avoid the $O(R \times (V + E))$ bottleneck of naive BFS traversal, resolution uses a caching strategy:

• Export Maps: Once ImportEdges are established, the builder pre-computes a “Flattened Export Map” for each file, caching public symbols available via direct and transitive imports.
• Resolution pass: Symbol references are resolved against the local file scope first (handling shadowing), then against the cached Export Map ($O(1)$ lookup).

3. Star Import Handling

The resolution logic explicitly handles “opaque” or star imports (import *). Search paths are flagged as “exhausted” if a star import is encountered, triggering a fallback search against the target file’s full public export set.

4. Separate Queries Per Language

Each language implements isolated tree-sitter queries for imports and references via import_query_fn() and reference_query_fn() in LanguageSpec.

References

• RFC 0008 - Graph Module and SCC Analysis
• specs/graph-model.md

MetaCall Deploy Manifests

Research synthesis. 2026-06-24.

Status

Superseded. The original per-language manifest model (metacall.json + metacall.{tag}.json) was replaced by a pod-based model (metacall.pods.json) that partitions files by language via Union-Find and scopes dependencies per pod. The pod model is documented in DEPLOY.md. This RFC is retained as a design record of the original rationale and trade-offs.

1. What We Know

1.1 The Gap MetaCall Has

MetaCall’s runtime makes cross-language calls transparent at runtime but invisible at deployment time. The canonical metacall.json manifest is deliberately minimal:

{ "language_id": "py", "path": ".", "scripts": ["__init__.py"] }

It declares what files to load but not how they relate to each other. Call topology is discovered at runtime by the core loader/inspector via reflection. This means:

• No pre-deployment validation of cross-language call paths.
• No topology visualization before deploy.
• No SCC detection across language boundaries.
• No static check that referenced scripts actually exist.

meta-ast already produces everything needed to fill this gap: cross-file dependency graphs, SCC analysis, language identification, symbol extraction with types, and deployability classification. Phase 5 wires this data into deploy manifest generation.

1.2 What the Codebase Already Has

1.3 The Reference Fixture

tests/fixtures/mixed/auth_microservice/ demonstrates the target pattern:

• auth.py (Python) calls metacall_load_from_file('node', ['auth/auth.js']) to load Node.js functions, then calls metacall('sign', text) and metacall('verify', token) to invoke them.
• auth/auth.js (Node.js) exports sign and verify using jsonwebtoken.

The deploy tool must detect this metacall_load_from_file('node', [...]) call and generate:

1. A Python-side manifest listing the scripts.
2. A Node.js-side manifest listing auth/auth.js.
3. A root manifest composing both.
4. A mesh annotation showing two cross-language deployment units.

1.4 MetaCall Language Tags vs meta-ast LangId

MetaCall uses short runtime tags. meta-ast uses descriptive LangId variants. A mapping is required:

Note: Ruby (rb) and Java (java) are MetaCall-supported but not currently in meta-ast’s 8-language extraction engine. Deployment manifests for these will be generated if they are targets of metacall_load_from_* calls, but meta-ast will not extract symbols from their source files.

2. Architecture

2.1 Data Flow

Input Discovery
     |
     v
Parallel Parse + Extract (rayon)
     |
     v
Graph Assembly (GraphBuilder::from_extractions)
     |                                     \
     v                                      \
SCC Analysis (Tarjan)                        v
     |                               [NEW] Preserve Vec<FileExtraction>
     v                                      |
Mesh Annotation Emitter                      v
from SCC Deployment Units         [NEW] Call Site Scanner
     |                             (per-file metacall_load_from_* detection)
     v                                      |
Deploy Manifest Generator <-----------------+
     |
     v
Root Manifest Assembler
     |
     v
[optional] --check validation against existing metacall.json

2.2 Module Layout

src/deploy/
  mod.rs           DeployConfig, DeployError, public API
  tags.rs          LangId -> MetaCall tag mapping, tag validation
  scanner.rs       Cross-Language Call Site detection (tree-sitter queries)
  manifest.rs      DeployManifest, RootManifest types + generation
  mesh.rs          MeshAnnotation type + SCC -> deployment unit converter
  check.rs         --check validation mode (diff against existing manifests)

2.3 Feature Gate

# Cargo.toml
[features]
default = []
embed-cytoscape = ["dep:handlebars"]
metacall-deploy = []

All src/deploy/ code is behind #[cfg(feature = "metacall-deploy")]. The CLI Deploy variant is gated. Zero compile-time cost when disabled.

3. Type System Design

3.1 Tag Mapping (src/deploy/tags.rs)

Maps meta-ast’s LangId to MetaCall runtime tags (py, node, ts, c, cpp, rs, go).

3.2 Cross-Language Call Site (src/deploy/scanner.rs)

The scanner uses tree-sitter queries per language pack to find calls to metacall_load_from_file, metacall_load_from_memory, etc. It extracts:

• The first argument (language tag string literal or enum) -> target_lang.
• The second argument (array of script paths) -> scripts.

Literals receive 1.0 confidence. Computed arguments (identifiers, calls, etc.) receive 0.4 confidence and are captured as their raw text.

3.2.1 Language Specific Patterns

• Python: metacall_load_from_file(tag, scripts)
• JS/TS/Tsx: metacall_load_from_file(tag, scripts)
• C/C++: metacall_load_from_file(tag, paths, size) or metacall_load_from_file_ex(...)
• Rust: metacall::load::from_files(Tag::NodeJS, ...)
• Go: metacall.LoadFromFile("node", ...)

3.3 Deploy Manifest (src/deploy/manifest.rs)

Defines DeployManifest and RootManifest structs compatible with MetaCall’s format, plus an extension for multi-language projects.

3.4 Mesh Annotation (src/deploy/mesh.rs)

Mesh annotation emitted as metacall.mesh.json. Maps SCC-derived deployment units to a mesh topology with cross-language boundaries and independence classification.

4. Scanner Implementation Strategy

4.1 Tree-Sitter Queries

Implemented queries for all 8 supported languages using an “anchor and capture arguments node” strategy to ensure robust extraction of language tags and script path arrays.

4.2 Data Preservation

The current pipeline discards Vec<FileExtraction> after graph building. The implementation uses a dedicated lightweight scanner pass that re-parses files only when the deploy command is used, avoiding memory overhead on the primary analysis path.

5. Output Formats

5.1 Per-Language Manifest (metacall.{tag}.json)

Groups discovered project files and detected call sites by target language.

5.2 Root Manifest (metacall.json)

Identifies the primary entry point language and composes all per-language packages.

5.3 Mesh Annotation (metacall.mesh.json)

Emits deployment unit topology derived from SCC analysis, identifying cyclic clusters and independent candidates across language boundaries.

6. Check Mode (--check)

Diffs generated manifests against any existing metacall.json in the project tree and reports divergences (missing/extra scripts, language mismatches) as structured diagnostics.

7. Testing Strategy

7.1 Example Validation Matrix

Verified against all fixtures in tests/fixtures/mixed/:

7.2 Integration Tests

Implemented in tests/integration/deploy_test.rs, verifying manifest correctness, root composition, and --check failure detection.

8. Risks and Tradeoffs

8.1 No Upstream MetaCall Extension

The enriched manifest format (with packages field and mesh annotation) is a meta-ast extension, not an upstream MetaCall feature. MetaCall core will ignore unknown fields. We target the meshfunction project’s deployment needs rather than the core upstream loader.

8.2 Extraction Data Loss

Re-scanning files for call sites adds slight latency ‘1 ms’ but preserves a lean memory model for the primary GraphAnalysis pipeline.

8.3 Unsupported Languages

Symbol extraction is limited to the 8 languages in meta-ast’s core engine. Other languages (Ruby, Java) are handled as external nodes.

8.4 Dynamic Call Sites

Unresolvable arguments are assigned 0.4 confidence and annotated, aligning with NFR-2 (resilience).

RFC 0011: MetaCall Client API Support (metacall())

Status

Accepted and Implemented.

Implementation Notes

Implemented as designed with these notes:

• metacall_handle is excluded because argument layout varies per port (tag first in C/Node, handle first in Rust).
• Phase A and Phase B index all extracted symbols regardless of visibility flag.
• Rust metacall_no_arg and metacall_untyped map to ClientCall; load::from_single_file maps to LoadFromFile.
• Client-call reference edges flow as distinct inter-pod reference edges without altering load confidence.

1. Problem

The deploy scanner previously detected only metacall_load_from_* calls. Client function calls were not tracked:

from metacall import metacall_load_from_file, metacall

metacall_load_from_file('node', ['auth/auth.js'])

def encrypt(text):
    return metacall('sign', text) # Untracked client call

Without client call tracking, Function Mesh topology missed function-level cross-language dependencies and call-site attribution.

2. API Surface

The scanner detects client invocation APIs across all supported ports:

3. Design

3.1 Model

CallSite includes ClientCall variant fields:

#![allow(unused)]
fn main() {
pub enum CallSiteVariant {
    LoadFromFile,
    LoadFromMemory,
    LoadFromPackage,
    LoadFromConfiguration,
    ClientCall,
}

pub struct CallSite {
    pub function_name: Option<String>,
    pub is_async: bool,
    // existing fields...
}
}

3.2 Two-Phase Function Resolution

client_call::resolve_client_calls resolves target functions:

1. Phase A (Load-aware): Matches call names against public symbols in files explicitly loaded by the calling file. Matches score 1.0 (unique) or 0.8 (ambiguous).
2. Phase B (Global fallback): Searches all project symbols if Phase A finds no match. Matches score 0.6 (unique) or 0.5 (ambiguous).
3. Computed names: Dynamic arguments cap edge confidence at 0.4. Unresolved names produce a Warning diagnostic.

3.3 Graph Integration

Client-call edges are EdgeKind::Reference from calling file node to target symbol node. They participate in SCC analysis before pod partitioning, ensuring cross-language call cycles create proper cuts.

4. Impact

• metacall.pods.json: Includes reference edges for cross-language client calls.
• metacall.mesh.json: cross_language_edges attributes target deployment units and call-site files.

RFC 0012: Polyglot LSP Server over the .metast Index

Status

Approved.

1. Problem

MetaCall projects mix languages in one workspace. A function defined in Python is called from TypeScript through metacall(). No editor sees this: editors run one language server per file type, and each server knows nothing about the other languages. Cross-language navigation, hover, and completion are invisible.

Two past attempts failed:

• intellisense: a VSCode extension, never a real language server. Per-request Python subprocesses for goto-definition (with an async race that missed on first use), placeholder types injected destructively into user .ts files, hardcoded type maps, and a committed cache with absolute Windows paths “first try”. The vscode-languageclient dependency was dead weight.
• vscode-extension: deploy tooling and snippets only, no code intelligence “which doesn’t make sense”.

Meanwhile meta-ast already computes everything a code-intelligence backend needs: tree-sitter extraction for 9 languages, import resolution, a cross-language dependency graph with a confidence ladder, Tarjan SCC, call-site detection, and incremental re-analysis. But its output is a CLI artifact with no consumer loop except Function Mesh for now.

2. Proposal

The etags shape, upgraded:

1. meta-ast emits a versioned, incrementally updatable index artifact (.metast v2).
2. A new LSP server consumes that index and serves requests for all supported languages from one process.
3. Open-editor buffers feed the same extraction path, so unsaved edits stay consistent.
4. Runtime metadata from metacall inspect enriches hover when available; static features never depend on it.

The LSP does not re-parse anything. It serves queries from an immutable index snapshot and coordinates re-indexing when files change.

Home: the server lives in its own repository, metacall/lsp, not in this one. meta-ast stays a library dependency as discussed, Phase 0 patches below are the only changes required here.

3. Architecture

┌──────────────────────────────────────────────────────┐
│  Editor (VSCode / Neovim / emacs)                    │
│    thin client: launch server, forward buffers       │
├─────────────────────────── LSP (stdio) ──────────────┤
│  meta-ast-lsp server                                 │
│  ┌────────────────┐   crossbeam   ┌───────────────┐  │
│  │ sync main loop │<-------------│ reindex worker│  │
│  │ answers from   │   swap Arc   │ runs engine   │  │
│  │ Arc<IndexSnap> │------------->│ incremental   │  │
│  └────────────────┘              └───────────────┘  │
│                          │ embeds                    │
│  ┌───────────────────────▼──────────────────────┐    │
│  │ meta-ast engine (library, not CLI)           │    │
│  │ WatchState + incremental_reanalyze           │    │
│  │ BLAKE3 diff, per-file FileExtraction deltas  │    │
│  └──────────────────────────────────────────────┘    │
├──────────────────────────────────────────────────────┤
│  Disk state                                          │
│    .meta-ast/manifest.jsonl + per-file shards        │
│    (cold start without full re-analysis)             │
└──────────────────────────────────────────────────────┘

3.1 Engine layer

Embed meta-ast as a library inside the server process. Reuse directly:

• incremental_reanalyze + WatchState: BLAKE3 fingerprint diff, re-extract only changed files, unchanged files keep their cached Arc<FileExtraction>.
• IdGenerator::with_start(max_cached_id + 1) seam: no ID collisions across ticks.
• GraphBuilder::from_extractions + add_edge_normalized: dedup and confidence rules stay canonical.

New seams required:

• In-memory ingestion: extract from (uri, text, version) for open buffers instead of disk reads. This extends extract_with_id_gen, it does not replace disk discovery.
• Resolver invalidation: PythonResolver, TsConfigResolver, GoModResolver cache filesystem state for the process lifetime. An LSP is long-lived, so these caches must invalidate on relevant config-file events. Until then, config changes require a server restart (documented limitation).
• Debounce: reuse the notify-debouncer pattern from src/watch/watcher.rs, driven by both OS events and textDocument/didChange flushes.

3.2 Index artifact (.metast v2)

Layout under .meta-ast/:

manifest.jsonl     one line per file:
                   { path, content_hash (BLAKE3 hex), size,
                     mtime, shard, schema_version }
shards/<n>.jsonl   one block per file: symbols (full ranges),
                   imports, references, diagnostics, ast_node_count,
                   edges whose source or target belongs to the file
                   (endpoints as stable names)
header.json        { schema_version, tool_version, created_at }

Rules:

• Hash bytes, never trust mtime alone. mtime and size are a fast negative check only (git checkout and rsync lie about mtime).
• Stable join key: language-scoped qualified name (SCIP-style descriptor, e.g. python module . encrypt .). Numeric SymbolIds are per-run and rayon-nondeterministic; they are regenerated in memory at load and never persisted.
• Edge rows carry (source_name, target_name, kind, confidence, flow_kind) so a merged load reproduces add_edge_normalized semantics exactly.
• Cold start loads manifest.jsonl + shards; warm path skips unchanged files entirely.

Optional exchange layer (Phase 3): meta-ast export scip writes a standard SCIP index so Sourcegraph-style consumers can read the same data “not needed for now”.

3.3 Server layer

Framework: lsp-server (rust-analyzer’s sync framework, actively released from the rust-analyzer monorepo).

Rationale for this specific shape:

• Indexed queries are hash-map lookups. A synchronous main loop answering from Arc<IndexSnapshot> needs no async runtime.
• Reindex coordination is plain threads: worker mutates nothing shared, then hands the new snapshot over the channel; main loop swaps the Arc.
• This mirrors rust-analyzer’s main_loop.rs task-passing design without salsa.
• Cancellation: $/cancelRequest becomes a token registry checked between handler phases. With sub-millisecond queries, cancellation pressure is minimal.

3.4 Client layer

Thin clients hosted under clients/ in metacall/lsp: VSCode and Zed first. Each client (~200 lines) activates on supported languages, spawns the server binary over stdio, forwards didOpen/didChange/didSave/didClose, and surfaces status. No intelligence lives in a client.

Editors without an extension story (emacs, vim, helix) work with zero client code: any LSP configuration pointing at the binary is enough. The TypeScript/Rust clients are deliberately simple onboarding tasks for new contributors arriving through Discord.

3.5 Distribution and packaging

The end user installs one extension and sees none of this machinery:

1. CI publishes static server binaries per platform to GitHub releases of metacall/lsp.
2. The client extension downloads the matching binary on activation and checks releases for updates automatically.
3. If MetaCall itself is missing, the extension guides installation instead of failing silently. Static features never require MetaCall; only runtime enrichment (Phase 3) does.
4. Optional later step: bundle the server into the existing MetaCall installer ‘not decided’, which already ships deploy and FaaS components. The maintainer confirmed this path stays open. Default stays GitHub releases because it decouples tooling releases from runtime releases.

4. Feature Phasing

Phase 0: meta-ast prep (prerequisite, small diffs)

1. serialize_symbol_node emits source_range and file_path for symbol nodes (currently dropped, which makes the graph export unusable for navigation).
2. Expose extract_with_id_gen over in-memory text.
3. Shard writer/reader module behind no new feature flag (it is pure output).

Phase 1: single-language correctness (not alive in this crate)

• initialize, capabilities, workspace root handling (one root per server instance).
• Incremental document sync + buffer overlay.
• textDocument/documentSymbol (tree-sitter symbols already carry ranges).
• textDocument/hover: signature + docstring markdown.
• textDocument/definition within one language via resolved references.
• Publish diagnostics on re-extract.

Phase 2: the polyglot payoff

• Cross-language definition and references routed through CodeGraph edges, including metacall() client-call edges with the RFC 0011 confidence ladder.
• workspace/symbol.
• textDocument/completion: bucketed symbol list with kind, signature, defining file; cross-language candidates ranked by edge confidence.
• Debounced background reindex on file change with snapshot swap.

Phase 3: enrichment and ecosystem

• Hover merge with runtime metadata. Two sources, both optional enrichment:
  1. Local: parse metacall inspect JSON (parameter and return annotations where the loader captured them, e.g. Python type hints).
  2. Deployed: query a live FaaS deployment through the same metacall/protocol API that metacall/deploy-mcp-server wraps, so signatures reflect the running runtime. Absence of either changes nothing statically.
• Semantic tokens (symbol kinds map cleanly).
• Stub generation (recycled concept from deprecated/intellisense, done right): emit .pyi / .d.ts from the static model so native per-language servers also see cross-language signatures. Never write into user source files; write sibling stub directories configured as extra paths. Stubs are a one-way complement that feeds foreign signatures to native servers, they never carry navigation or diagnostics. The two-way channel is this LSP itself.
• meta-ast export scip.

5. Non-Goals

• Full type inference (RFC 0006 scope stands).
• Renaming across languages (write path; later decision if demand appears).
• Running or loading user code. Static analysis only; runtime inspection happens out-of-band via metacall inspect.
• Replacing native per-language servers. This server coexists; editors keep their TypeScript or Python LSP for deep single-language semantics and gain cross-language features from this one. We do not rebuild nine type systems: pyright and tsserver stay authoritative for deep inference.

6. Failure Modes We Will Not Repeat

From the old forensics:

1. No destructive edits of user source. Stubs go to generated directories.
2. No per-request subprocesses. Everything answers from the loaded snapshot.
3. No placeholder or fabricated types. Absent metadata means absent hover detail.
4. No committed caches with machine-specific paths.
5. No VSCode-only logic. Intelligence lives in the server; clients stay thin.

7. Deliverables

1. meta-ast Phase 0 patches in this repository: serializer ranges, memory-text extraction seam, shard IO. These land first so metacall/lsp starts against a capable library.
2. metacall/lsp repository: server crate (meta-ast-lsp: engine host, sync server loop, index store) plus thin TypeScript clients under clients/vscode and clients/zed.
3. Release engineering: per-platform binary builds published to GitHub releases; extension-side download and auto-update flow.

Verification for metacall/lsp follows the same repo standards:

cargo build --features watch --features metacall-deploy --features dataflow
cargo test --all-features
cargo clippy --all-targets --all-features -- -D warnings
cargo +1.94.0 fmt --check

Plus new integration tests: shard round-trip, snapshot swap under concurrent queries, cross-language definition fixtures under tests/fixtures/mixed/.