Empirical Evaluation of Topology-Aware Continual Learning

Abstract

We present the first empirical evaluation of Graphonomous, a topology-aware continual learning engine, on a real-world multi-domain codebase. The corpus is the full [&] Protocol portfolio — 18,165 source files across 14 projects ingested via the engine’s native scan_directory feature. This includes Elixir, TypeScript, JavaScript, HTML, CSS, JSON, Markdown, and YAML files spanning agent orchestration, governance, spatial/temporal intelligence, knowledge graph editing, and the engine’s own source code. The self-referential property (the engine processes its own implementation) creates genuine cyclic knowledge structures (κ>0), enabling the first naturalistic test of κ-aware routing and deliberation.

We evaluate all 22 MCP tools across eight dimensions: (1) ingestion throughput via filesystem traversal, (2) cross-domain retrieval quality, (3) topological cycle detection (κ), (4) the full learning loop (outcome, feedback, novelty, interaction), (5) goal lifecycle and coverage-driven review, (6) graph operations and specialized retrieval (BFS traversal, graph stats, episodic/procedural retrieval, deliberation), (7) memory consolidation dynamics, and (8) attention-driven goal prioritization.

Key findings: (1) automated edge extraction creates 12,871 edges from imports/requires/references; (2) the graph contains 22 naturally occurring SCCs with max κ=27; (3) graph-expanded retrieval outperforms flat baseline by +0.024 F1 and +0.103 recall (F1=0.415 vs 0.391); (4) deliberation achieves 100% pass rate (2/2); (5) all ~75 tests pass across 22 MCP tools; (6) consolidation throughput reaches ~2 µs/cycle (27.1M nodes/sec); (7) domain-aware re-ranking promotes cross-domain diversity; (8) orphan node rate is 80.5%.

1. Motivation

1.1 The Gap

Agent memory systems are evaluated primarily through synthetic benchmarks: random fact insertion, isolated retrieval, or toy knowledge bases. No published evaluation tests a memory system on a real multi-domain corpus where:

• Cross-domain dependencies exist (governance specs reference memory specs which reference governance)
• Cyclic knowledge is natural (a spec about cycle detection contains cycles about itself)
• Multiple abstraction levels coexist (architecture specs, API contracts, implementation code, decision records)
• The evaluation corpus is the system’s own codebase (genuine dogfooding)
• All skill surfaces are exercised (not just store/retrieve, but learning, goals, topology, consolidation, attention)

1.2 Why This Matters

Continual learning engines claim to support multi-domain reasoning, but without empirical evidence on complex real-world corpora, these claims are untestable. This protocol establishes:

1. A reproducible benchmark anyone can run (mix benchmark.run)
2. Baseline measurements across eight evaluation dimensions covering all 22 MCP tools
3. Identified gaps that guide engineering priorities
4. A methodology for evaluating topology-aware memory systems

1.3 Related Work

2. Experimental Setup

2.1 System Configuration

2.2 Corpus Description

The [&] Protocol Portfolio is a full multi-project codebase:

Total: 18,165 files ingested from 14 project directories spanning the full [&] ecosystem.

2.3 Known Cross-Domain Dependencies

The ampersand ↔ graphonomous bidirectional relationship creates a genuine κ=1 cycle: the ampersand spec defines κ routing, Graphonomous implements it, and the spec references Graphonomous as the implementation target.

2.4 MCP Tool Coverage

3. Results

3.1 Ingestion Performance

Neural embedding cost: The 7.4 files/sec throughput is attributable to neural embedding computation (~87ms per file via EXLA+CUDA GPU with batch_size=8). This is a deliberate quality-for-speed tradeoff — neural embeddings produce meaningful semantic retrieval (F1=0.415) where trigram hashing yields F1=0.0.

3.2 Retrieval Quality

13 queries tested across 4 categories with neural embeddings:

Per-Category Breakdown

Notable Query Results

3.3 Topology & κ Detection

Synthetic Cycle Tests (4/4 passed)

Edge Impact Prediction (2/2 passed)

3.4 Learning Loop

Outcome Learning (4/4 passed)

The asymmetric confidence adjustment is correct: failure has larger magnitude than success (Bayesian prior favoring caution), and timeout is penalized less severely than explicit failure.

Feedback Learning (3/3 passed)

Interaction Learning (2/2 passed)

3.5 Goal Lifecycle & Coverage

Goal Lifecycle (4/4 passed)

Goal Review (2/2 passed)

3.6 Graph Operations

3.7 Consolidation Dynamics

Confidence Decay Trajectory (5 cycles, 27,111 nodes)

Decay curve: c(n) = c(0) × (1 − r)n where r=0.02. After 5 cycles, average confidence drops from 0.650 to 0.600 — a 9.6% total loss. No nodes pruned (minimum 0.452 > prune threshold 0.10). Throughput: 27.1M nodes/sec (~2 µs/cycle).

3.8 Attention Engine

The attention engine correctly returns 0 items — freshly created goals with no outcome history should not trigger dispatch. The “learn before act” gate works at 18K-node scale.

4. Discussion

4.1 What Works

1. Neural embeddings enable meaningful retrieval. F1=0.415 (graph-expanded) with neural embeddings vs F1=0.0 with trigram hashing. Single-domain queries achieve 0.667 recall. Graph expansion adds +0.024 F1 and +0.103 recall over flat baseline.
2. κ detection scales correctly. 100% accuracy at 18K-node scale. Signal is robust to massive graph growth.
3. Full skill surface is functional. 20/22 MCP tools exercised. The learning loop works end-to-end.
4. scan_directory is production-quality. 18,165 files with 0 failures (100% success rate).
5. Learning confidence adjustments are correct. Bayesian asymmetry (failure > success magnitude) and timeout/failure distinction work.
6. Consolidation is extremely fast. ~2 µs/cycle at 27K nodes (27.1M nodes/sec throughput).
7. The “learn before act” gate works at scale. Attention engine correctly refuses to dispatch when epistemic coverage is insufficient.

4.2 Known Limitations

1. Ingestion throughput. 7.4 files/sec with neural embeddings. Batch embedding (batch_size=8) is implemented but GPU memory constraints limit further gains at this corpus scale.
2. Cross-domain precision. 0.320 vs single-domain 0.590. Domain-aware re-ranking (0.95 decay) partially addresses this but a gap remains.
3. Attention survey latency. 51 seconds for 5 goals. Pre-seeded outcome histories provide meaningful prioritization signal but latency remains high.
4. Orphan rate. 80.5% of nodes lack edges. EdgeExtractor covers import/require/reference patterns; additional heuristics (e.g., co-location, semantic similarity) could reduce this further.

4.3 The Self-Referential Observation

The most intellectually interesting result: the corpus naturally contains a κ=1 cycle between the [&] protocol spec (which defines κ routing) and Graphonomous (which implements κ routing). At 18K-node scale this cycle is found identically — the signal is robust to massive graph growth.

This validates the core thesis: cyclic knowledge structures arise naturally in complex multi-domain systems, and a memory engine that can detect and route around them has a structural advantage over flat retrieval systems.

4.4 Summary of Results

4.5 LongMemEval Competitive Benchmark (Phase 9)

LongMemEval (ICLR 2025) is the standard benchmark for long-term memory in chat assistants, testing 5 core abilities across 500 questions: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.

Neural Results (Bumblebee/all-MiniLM-L6-v2)

By Ability

Competitive Comparison

5. Reproduction

5.1 Running the Benchmark

cd graphonomous
source .envrc          # sets LD_PRELOAD and LD_LIBRARY_PATH for CUDA/EXLA
mix deps.get
mix benchmark.run --neural --cycles 5   # neural embeddings (requires GPU)
# or: mix benchmark.run --cycles 5     # fallback trigram (no GPU needed)

Results are written to graphonomous/benchmark_results/:

• ingest.json — corpus ingestion metrics
• retrieval.json — per-query retrieval results
• topology.json — κ detection and impact prediction
• learning.json — outcome, feedback, novelty, interaction
• goals.json — goal lifecycle, coverage, review
• graph_ops.json — query_graph, traverse, stats, retrieval, deliberation
• consolidation.json — decay curves and survival
• attention.json — goal prioritization
• combined.json — all phases + system metadata
• longmemeval.json — LongMemEval competitive benchmark (500 questions, per-question results)

5.2 Individual Phases

mix benchmark.ingest [--purge]
mix benchmark.retrieval
mix benchmark.topology
mix benchmark.learning
mix benchmark.goals
mix benchmark.graph_ops
mix benchmark.consolidation [--cycles N]
mix benchmark.attention
mix benchmark.longmemeval [--split oracle|s] [--limit N] [--neural]

6. Future Work

6.1 Completed

• Neural embeddings via Bumblebee/all-MiniLM-L6-v2 + EXLA (F1=0.415 graph-expanded)
• File-path-based domain extraction for ground truth
• Automated edge extraction via EdgeExtractor (Elixir imports/aliases, JS/TS imports, Markdown cross-references)
• Deliberation benchmark validates converged + conclusions return shape
• Outcome history pre-seeding for attention prioritization
• Batch embedding (batch_size=8) via Nx.Serving with embed_many_binary/2 API
• Graph-expanded vs flat baseline ablation with per-query delta reporting
• Domain-aware re-ranking (0.95 decay per duplicate domain)

6.2 Performance (OS-E001.2)

• Incremental consolidation (skip unchanged nodes)
• Attention survey caching / precomputation
• Retrieval index optimization for 10K+ node graphs

6.3 Comparative (OS-E001.3)

• Flat RAG baseline on same corpus — integrated into retrieval benchmark
• Single-timescale ablation study
• Compare with Hindsight’s retain/recall/reflect API
• Run LongMemEval benchmark for direct competitive comparison — 90.4% Session Hit Rate (neural, 100 questions)
• Trigram vs neural retrieval ablation — 2.8% (trigram) vs 90.4% (neural) SHR, 32× improvement

6.4 Scale (OS-E001.4)

• Multi-session evaluation (knowledge accumulation over days)
• Federation benchmark (two Graphonomous instances syncing)
• Neural embeddings at 18K-node scale — 87ms/embed, ~41 min total

Appendix: Complete Test Results

System Fingerprint

Engine:       Graphonomous 0.2.0
Elixir:       1.19.4
OTP:          28
Embedder:     Bumblebee/all-MiniLM-L6-v2 + EXLA (CUDA GPU, batch=8) or trigram fallback
Date:         2026-04-02
Corpus:       18,165 files via scan_directory, 14 projects
Edges:        12,880 (12,871 automated + 9 heuristic)
SCCs:         22 (max κ=27)
MCP coverage: 22/22 tools (100%), ~75 tests passed
Retrieval F1: 0.415 (graph-expanded) / 0.391 (flat baseline) [neural embeddings]

Citation

Published under the OpenSentience research protocol series. This is a living document — results will be updated as the benchmark evolves.