Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Final Report: meta-ast, GSoC 2026

  • Author: Khaled Alam
  • Organization: MetaCall
  • Project: meta-ast - Standalone Polyglot Static Analysis Engine
  • Program: Google Summer of Code (GSoC) 2026
  • Status: Completed, v0.5.0

1. Project Summary and Goals

meta-ast is a fast, standalone static analysis engine written in Rust. The project parses source code across nine programming languages, extracts a normalized symbol Intermediate Representation (IR), builds cross-language dependency graphs, detects import cycles with Tarjan Strongly Connected Components (SCC), and generates deployment manifests for the MetaCall Function Mesh runtime. The engine never executes target code.

Original Problem Statement

MetaCall supports polyglot architectures where functions written in different languages call each other seamlessly. However, developers lacked a fast, unified static analysis tool to:

  1. Map cross-language dependencies without running arbitrary user code.
  2. Detect cyclic imports that block deployment decomposition.
  3. Automatically partition polyglot applications into optimal, language-specific deployment pods.
  4. Pin external package dependencies across multiple language package managers.

Project Goals

  1. Polyglot Parsing: Parse 9 languages (Python, JavaScript, TypeScript, TSX, C, C++, Rust, Go, Ruby) using a unified tree-sitter pipeline.
  2. Normalized Symbol IR: Extract functions, classes, methods, structs, enums, and interfaces into a language-agnostic intermediate representation.
  3. Cross-Language Dependency Graph: Resolve imports, symbol references, and cross-language call sites with confidence-weighted edges.
  4. Cycle Detection and Pod Partitioning: Identify cyclic clusters using Tarjan SCC and partition code into same-language deployment units.
  5. MetaCall Manifest Generation: Scan cross-language metacall_load_from_* and metacall() invocations, generate pod manifests (metacall.pods.json) and mesh annotations (metacall.mesh.json), and validate cut fairness.
  6. High-Performance Watch Mode: Deliver sub-100 ms incremental re-analysis using cryptographic content hashing and zero-allocation cache reuse.
  7. Production Quality: Provide comprehensive test coverage, robust CI/CD, cross-platform release binaries, and complete documentation.

2. What Was Accomplished (Phase-by-Phase)

The project executed across seven planned phases. All milestones were delivered, tested, and released.

Phase 1: Unified Parser Lifecycle and Symbol Extraction

  • Built thread-local tree-sitter parser pools for zero-overhead multi-threaded parsing.
  • Implemented uniform AST query packs across Python, JavaScript, TypeScript, TSX, C, C++, Rust, and Go.
  • Normalized declarations into the Symbol IR with visibility, signature, and docstring metadata.
  • Implemented robust error recovery: malformed source files emit structured Diagnostic records without stopping the pipeline.

Phase 2: Dependency Graph and Tarjan SCC

  • Implemented the CodeGraph directed graph model over petgraph.
  • Implemented cross-file import and symbol resolution with confidence scoring (1.0 for own file / direct import, 0.8 for transitive import, 0.6 for cross-language).
  • Integrated Tarjan Strongly Connected Components (SCC) algorithm with EdgeFiltered views to isolate cyclic clusters while excluding ownership and flow edges.
  • Classified graph components into independent deployment units vs. co-deployment clusters.

Phase 3: Datagraph and Dataflow Sinks

  • Extended the IR with intra-procedural def-use dataflow nodes (DataNode, FlowEdge, DataScope).
  • Implemented def-use extraction for parameter bindings and variable declarations.
  • Defined a schema-versioned export format (schema version 1) and the GraphSink pluggable adapter trait.
  • Added the --datagraph CLI flag for dataflow export.

Phase 4: CLI Polish, Visualization, and Incremental Watch Mode

  • Added structured output formats (--format json|yaml) across all subcommands.
  • Implemented an interactive Cytoscape.js HTML visualization dashboard (--html).
  • Built incremental watch mode (--watch):
    • File-system monitoring with debounced event processing.
    • BLAKE3 cryptographic content hashing to detect modified files.
    • Zero-allocation cache reuse for unchanged files via Arc<FileExtraction>.
    • Collision-free ID generation using IdGenerator::with_start.

Phase 5: MetaCall Deployment Manifest Generator (metacall-deploy)

  • Implemented AST scanners for MetaCall call sites (metacall_load_from_file, metacall_load_from_memory, metacall_load_from_package, metacall_load_from_configuration, and metacall() client calls per RFC 0011).
  • Implemented same-language pod partitioning using Union-Find over resolved dependency edges.
  • Added external package dependency resolution from lockfiles (package-lock.json, Cargo.lock, go.sum, requirements.txt, Gemfile.lock) for exact version pinning.
  • Generated deployment artifacts:
    • metacall.pods.json: Pod manifests with deployment units, dependency lists, and AST metrics.
    • metacall.mesh.json: Function Mesh topology annotations with cross-language edge attribution.
  • Added --check mode: verifies cut fairness (every cut edge across pods has a corresponding RPC stub entry, fulfilling ADR 0003).

Phase 6: Language Expansion (Ruby)

  • Implemented full Ruby support: tree-sitter grammar integration, symbol query pack, require/require_relative import resolver, reference detection, and lockfile resolution (Gemfile.lock).
  • Evaluated C# and Java; concluded Ruby provided the highest immediate utility for MetaCall Function Mesh targets.

Phase 7: Validation, CI/CD, Benchmarking, and Delivery

  • Hardened CI matrix across 4 operating systems (Linux glibc/musl, macOS x86_64/ARM64, Windows x86_64/ARM64) on stable and nightly Rust toolchains.
  • Added automated linting, formatting, cargo-deny license/vulnerability audits, and cargo-nextest execution.
  • Configured Criterion benchmark suite tracking pipeline, graph, and incremental performance.
  • Published mdbook documentation to GitHub Pages.
  • Published crate releases to crates.io and multi-platform binaries to GitHub Releases.

3. Code Merged Upstream

All work was developed in pull requests, reviewed, and merged into the main branch of metacall/meta-ast.

Merged Pull Requests

PRTitleDescription
#1SkeletonInitial project structure, tree-sitter integration, base CLI.
#3Refactor - 1Parser lifecycle and normalized symbol data model.
#4Ref2Extractor modularization and multi-language query packs.
#5Phase2/dep-graphDependency graph construction, petgraph integration, Tarjan SCC.
#6Unify Serialization and Introduce HTML DashboardJSON/YAML emitters, Cytoscape.js interactive visualization.
#7Cross-file dependency mapping (RFC 0009)Cross-file import and reference resolution with confidence scoring.
#8Scope wrapper enhancementsSymbol lookup scoping and namespace qualification.
#9Combined query optimizationConsolidated AST queries for improved parsing throughput.
#10Merge CI workflowsUnified CI pipeline (build, test, lint, deny, fmt).
#11Group and traceTraceability matrices and graph node groupings.
#12Deep graph assemblyEdge normalization, confidence fusion, and diagnostics propagation.
#13Deploy documentationSpecifications and user guides for deployment manifest generation.
#14Deploy module foundationCall-site scanner and initial manifest generator.
#16Architecture and ADRs updateDocumentation of ADR 0001-0004 and RFC 0001-0010.
#17Complete MetaCall deployment manifest generatorPod partitioning, lockfile resolution, mesh annotation, --check mode.
#30Optimization 1Parser reuse and memory allocation optimizations.
#31Phase 3 datagraph optional sinkDataflow IR, def-use analysis, pluggable sink trait, --datagraph.
#32ID generation optimizationAtomic ID generation and newtype validation.
#33Watch mode and incremental update strategyDebounced watcher, BLAKE3 change detection, incremental re-analysis.
#34Add Ruby supportRuby grammar, symbol queries, import resolver, lockfile parsing.
#35Phase 7 completionmdbook documentation site, benchmark suites, demo recordings.
#36Release v0.5.0Release automation, packaging, and announcements.
#37GitHub Pages deployment workflowAutomated mdbook deployment to GitHub Pages.
#53CLI & graph hardening--language filtering, --max-pod-size, O(1) edge normalization.

Notes

  • there was multiple commits unlinked with issues or PRs “Mostly optimizations”.

4. Key Metrics and Project Numbers

  • Languages Supported: 9 languages (Python, JavaScript, TypeScript, TSX, C, C++, Rust, Go, Ruby).
  • Test Suite: 430+ automated tests (327 unit tests, 106 integration tests, doc tests).
  • Test Matrix: 4 Operating Systems (Linux, macOS, Windows, Windows ARM) across 2 Rust toolchains (stable and nightly).
  • Performance Benchmark:
    • Incremental warm re-analysis: 1.16 ms for single-file changes (target was <100 ms).
    • Graph construction & Tarjan SCC: <0.5 ms for 1,000 nodes.
    • End-to-end extraction across fixture suite: 16.8 ms.
  • Releases:
    • 5 GitHub releases (v0.1.0 through v0.5.0).
    • 14 pre-compiled binary packages per release (covering Linux glibc/musl, macOS x86/ARM, Windows x86/ARM across core and deploy configurations).
    • 2 published crates.io versions (0.4.0, 0.5.0).

5. Technical Challenges and Engineering Insights

A. Polyglot Semantic Gap and Import Resolution

Challenge: Each programming language uses different module and symbol resolution semantics. C/C++ relies on header inclusion; Python uses runtime sys.path and relative imports; JavaScript/TypeScript uses ESM, CommonJS, and tsconfig.json path mappings; Rust uses hierarchical crate paths; Ruby uses require and require_relative. Solution: Designed the ImportResolver abstraction. Each language provides a stateful resolver that caches directory hierarchies and configuration files. Unresolved imports are recorded with confidence penalties (0.6 cross-language vs 1.0 same-file) and surface as non-fatal diagnostics.

B. Collision-Free Incremental State in Watch Mode

Challenge: When re-analyzing modified files in watch mode, creating new symbol IDs could collide with cached IDs from unchanged files or force expensive whole-graph re-allocations. Solution: Implemented an explicit ID generation seam using IdGenerator::with_start(max_cached_id + 1). Unchanged files retain their existing Arc<FileExtraction> references with zero allocations (verified via Arc::ptr_eq), while newly extracted files receive strictly monotonic, non-overlapping IDs.

C. Deployment Cut Fairness and Invariant Verification

Challenge: Partitioning polyglot applications into separate deployment pods can sever dependencies. If an edge across pods lacks an RPC stub, the deployed application fails at runtime. Solution: Implemented the cut fairness validation algorithm (ADR 0003). In --check mode, the analyzer constructs a bijection between inter-pod cut edges and declared RPC stubs. Any missing stub produces an immediate diagnostic error with the exact source location of the call site.

D. Graph Assembly Performance at Scale

Challenge: Initial graph construction used repeated edge scans to deduplicate multi-language references, causing quadratic slowdowns on large graphs. Solution: Replaced linear scans with an indexed (src, dst, kind) lookup map in CodeGraph, delivering O(1) edge deduplication and max-confidence fusion. This reduced graph construction time for 10,000 duplicate edges to 486 microseconds.

E. Cross-Platform Path Normalization

Challenge: File paths and snapshots differed across Linux, macOS, and Windows due to backslashes and case sensitivity. Solution: Enforced universal forward-slash path normalization across all internal data structures, JSON serializers, and snapshot tests, ensuring deterministic CI verification across all four operating systems.

6. How to Build, Test, and Verify

Prerequisites

Build from Source

git clone https://github.com/metacall/meta-ast.git
cd meta-ast

# Build core analyzer
cargo build --release

# Build with deployment manifest generator and watch mode
cargo build --release --features metacall-deploy --features watch --features dataflow

Run Test Suite

# Run all tests across all feature flags
cargo test --all-features

# Run linter and formatting checks
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --check

Run Benchmarks

cargo bench --features watch

Run CLI Commands

# Inspect declarations in a project
./target/release/meta-ast inspect ./tests/fixtures/python/ -f json

# Build graph with interactive HTML dashboard
./target/release/meta-ast graph ./tests/fixtures/mixed/ --html -o dashboard.html

# Generate MetaCall deployment manifests and check cut fairness
./target/release/meta-ast deploy ./tests/fixtures/mixed/auth_microservice --check -o ./deploy_out

7. Current State and Future Work

Current State

meta-ast is feature-complete for its GSoC 2026 milestones and project goals. The engine is released as v0.5.0 on crates.io and GitHub Releases, with documentation published at metacall.github.io/meta-ast.

Future Work (Post-GSoC Roadmap)

  • C ABI and Header Generation: Implement automatic C header generation for exported polyglot functions (RFC 0011).
  • Deeper Intra-Procedural Dataflow: Expand dataflow node extraction from Rust to JavaScript, TypeScript, Python, and Go.
  • Additional Language Packs: Add grammars for PHP, Java, and C# based on user demand.
  • Polyglot SAST Integration: Integrate static application security testing rules and machine learning assisted anomaly detection (RFC 0029).