Madani Lab · WAB v0.3.4 · open source
Open-source benchmark for workspace agentic architectures. We score 7 reference workspaces across 12 pillars and publish the methodology + 18 papers.
§ 02 · the problem
Pilot failure rates have not improved as models improved. Four data points map where production agents actually break.
95%
failure rateNot because the models are too weak, but because the workspaces around them are too brittle. The 5% that ship designed their infrastructure to be portable, exportable, and re-groundable from day one.
MIT Sloan 2025 · Gartner Q4 2025 · Madani field study (47 EU enterprises, WSB-08)
92%
explained varianceSwapping Claude Sonnet for Opus produces a ~15% lift. Doubling the workspace quality (α = Q × Q) produces an 83% lift. Capital allocated to model selection has diminishing returns; capital allocated to harness engineering compounds.
Madani Lab · 142 production tasks · R² = 0.78 (WSB-04)
7 of 8
win rateReplicating Stanford’s Data Processing Inequality (DPI) finding in production: a well-structured single agent wins 7 of 8 head-to-head comparisons. The lone multi-agent win is a naturally parallel task — exactly where DPI theory predicts.
Madani replication of Tran & Kiela · arXiv:2604.02460 (WSB-05)
38%
failure sharepass@k hides what actually breaks in production. Apply MAST’s 14-failure-mode taxonomy and the picture flips: reliability is a harness discipline (idempotency keys, atomic writes, mid-task re-grounding), not a model property.
Madani · 1,200 production runs · MAST baseline Cleric & Yu 2025 (WSB-07)
§ 03 · methodology
Each pillar is scored on a CMMI-inspired L0–L4 maturity ladder. The composite is a weighted average across the four clusters, normalized 0–100, with letter grades (A ≥75 · B 60–74 · C 45–59 · D 30–44 · F <30).
the mind of the agent
Depth, freshness, and accessibility of the information available to the agent at decision time.
Persistent state across sessions · retrieval discipline · decay-aware compaction.
Single-thread default · evidence-based delegation when DPI conditions are satisfied.
how it executes
Modular, composable, hot-swappable capabilities the agent can invoke without re-engineering.
Pre-task self-assessment · post-task update · cybernetic feedback loop.
Model-agnostic prompts · exportable state · no vendor lock-in.
safety & governance
pass@k + MAST 14-failure taxonomy · idempotency keys · replay harness.
Hard rules · compliance gates · audit trail · human-in-the-loop checkpoints.
Vault op:// reference · zero plaintext secrets · scoped, rotatable tokens.
Structured logging · metrics · trace IDs · token-spend telemetry.
production readiness
Reflexion loops · dreams · skill discovery · capability profile evolution.
Replicable across contexts · documented onboarding · deterministic install.
maturity ladder · scored per pillar
§ 04 · research papers
18 paper-grade · click to expand abstract.
Why model intelligence is no longer the bottleneck — and what is.
Four orthogonal clusters · twelve dimensions · derived from production failure analysis on 142 tasks.
A 60-cell acceptance matrix that operationalizes "AI-ready" beyond marketing certification.
From paper to production — explicit translations from academic primitives to deployable components.
Shannon mutual information applied to the workspace-as-channel abstraction · R² 0.78 across 142 tasks.
arXiv:2604.02460 in Italian SMB context · 7/8 single-thread wins · the multi-agent penalty compounds non-linearly per hop.
Calibration collapses 2.6× from Easy to Hard tasks · cross-agent peer evaluation contributes nearly as much as self-introspection · ECE 0.24 → 0.087 in 4 days.
78.7% of multi-agent failures are NOT model problems · Step Repetition (15.7%) is the #1 single failure mode · Hallucination is deliberately excluded from the taxonomy.
Field study of 47 EU enterprises · portability explains 64% of outcome variance · 23-artifact checklist that separates the 5% from everyone else.
1.2M agent turns · 340M tokens · SNR half-life 340 turns at baseline · three interventions compound multiplicatively to 950-turn half-life.
Cognition steel-man validated · 14-deployment audit · 12 of 14 multi-agent deployments rolled back or abandoned · context dilution dominant in 11 of 14.
Adapting Shinn et al. (NeurIPS 2023, arXiv:2303.11366) from short-horizon benchmarks to multi-month production lifecycles · 17pp task-success lift sustained over 12 months.
The 270s vs 1200s decision · why 5-minute cache windows reshape every autonomous loop architecture · 87% cost reduction with zero accuracy impact.
From benchmark-time RAGAS to continuously-running retrieval QA · why every long-lived agent needs a CI for its memory · the recall drift axis Es et al. did not measure.
Adapting Karpathy autoresearch from individual experimentation to durable workspace skill · 6 months of production runs · 7 counterintuitive findings on composite scoring and adaptive sleep.
How to encode "never do X" rules so that production agents respect them under all conditions including adversarial · 41,302 gate decisions · zero observed violations · 7 counterintuitive takes.
23 services · zero secrets in repo · runtime resolution via 1Password CLI · 12 months production · 7 counterintuitive takes on credentials at scale.
Why "skills" beat "tools" and "abilities" as the unit of agentic capability composition · 42 active skills · power-law usage · 7 counterintuitive takes on skill design.
Why agent memory needs five separate tiers — semantic, episodic, procedural, personalized, environment-dynamics — and why collapsing them is the silent reason production agents fail at week three.
Curator, Dreams, Reflexion produced 50 proposals per run and applied zero. The five-layer engine — PP gates · alpha gates · LLM-behavior gates · snapshot · log — closes the gap by codifying when a machine decision is safer than a human one.
§ 05 · contribute
The benchmark is open · the methodology is open · the audit tooling is open. Three ways to engage.
01 · github
WAB v0.3.4 · 12-pillar architecture · L0-L4 maturity · audit matrix · all in the public repo.
ceomadani/workspace-agentic-benchmark ↗02 · pull request
Audit your workspace against the 60-cell matrix · open a PR to /workspaces/{slug}.md · we review within 7 days.
open a PR ↗03 · email
Paper drafts · cross-replication studies · enterprise procurement · MAST-style audits · we read everything.
lab@madani.agency