AMNOG Modul 1 generation
Updated
- Non-inferiority
- 70%
- Regulaido
- Win rate
- 50%
- Holistic
- —
- Evidence
- 2 runs
- 2 dossiers
ISPOR Europe 2026 · HTA86
HTA-Bench Leaderboard
Evaluating end-to-end regulatory and HTA dossier generation across 8 authority-accepted G-BA AMNOG and ICH CTD dossiers (118,000+ words, 175 statutory tables), calibrated against 180 blinded ratings by 17 HTA experts.
- Benchmark Corpus
- 8 Dossiers
- AMNOG Modules 1–3 & ICH CTD 2.3.S
- Reference Scale
- 118k+ Words
- 175 statutory tables (135 dynamic)
- Human Calibration
- 180 Ratings
- 17 blinded HTA specialists · 2.2–2.6 pp error
- Top System Score
- 75.7 / 100
- Regulaido Engine · Argon Med (77.0 task-optimal)
Agent Harness & Frontier Model Leaderboard
Updated continuously as benchmark coverage, engines, and models advance · Current release: v2.3 / Reference Audit v1.2.1
Agent Harness Comparison (N=8 Dossiers)
Backbone LLM fixed at Gemini 3.8 Flash · Reference Audit (0–100), statutory table population (135 dynamic tables), verified hallucinations per dossier, and mean Standard-Tier API cost
| Harness (Flash) | Score | Tables | Hallu. | Cost |
|---|---|---|---|---|
| Regulaido Engine (Slow) (Flash) | 72.7 | 100% | 5.0 | $3.56 |
| Regulaido Agent (Flash) | 67.2 | 96% | 7.0 | $2.71 |
| Regulaido Engine (Fast) (Flash) | 66.4 | 100% | 6.8 | $0.52 |
| Claude Code (Flash) | 66.0 | 76% | 8.5 | $3.90 |
| OpenHands (Flash) | 62.0 | 72% | 11.8 | $4.35 |
General-purpose coding CLIs skip 24%–28% of required statutory Word tables (97/135 to
103/135) and emit un-linked plain-text [n] markers without page/chunk provenance.
Cost–Quality Pareto Frontiers (N=8 Dossiers)
Agent harnesses compared on Gemini 3.8 Flash (Score 55–85 vs. USD Cost)
With Gemini 4 Argon (Med), Regulaido Engine reaches 75.7 / 100 ($4.50, 100% tables, 3.8 hallucinations). Gemini 3.8 Flash (High) reaches 72.7 / 100 ($3.56), outperforming Claude Opus 5.5 High (71.6 at $6.80) and GPT-6 Astra High (63.2 at $3.32).
Performance by Dossier Module (N=8 Dossiers)
Best-in-class frontier model inside Regulaido Engine compared with single-model Gemini 3.8 Flash baselines
| Dossier Suite | Best Model in Regulaido | Regulaido (Best) | Regulaido (Flash) | Claude Code (Flash) | Tables (Reg. vs. CLI) |
|---|---|---|---|---|---|
AMNOG Module 2 — Indication, Therapeutic Need & MoA 2 dossiers · 10 statutory tables | Gemini 3.8 Flash | 89.9 | 89.4 | 81.0 | 100% vs. 90% |
ICH CTD 2.3.S — CMC Quality Overall Summary 1 dossier · 33 statutory tables · 85 CMC PDFs | Gemini 4 Argon (Med) | 86.6 | 78.1 | 71.6 | 100% vs. 85% |
AMNOG Module 1 — Executive Clinical Synthesis 2 dossiers · 26 statutory tables | Gemini 4 Argon | 78.4 | 74.2 | 69.5 | 100% vs. 85% |
AMNOG Module 3 — Target Population Epidemiology & GKV Costs 3 dossiers · 106 statutory tables | Gemini 4 Argon / Opus 5.5 | 64.4 | 58.6 | 51.9 | 100% vs. 71% |
All 8 Dossiers — Full Benchmark Mean 8 dossiers · 118,000+ words · 175 statutory tables | Argon Med (75.7) / Task-Optimal (77.0) | 77.0 | 72.7 | 66.0 | 100% vs. 76% |
Additional Published Campaign Snapshots
AMNOG Modul 2 generation
Updated
- Non-inferiority
- 70%
- Regulaido
- Win rate
- 50%
- Holistic
- —
- Evidence
- 2 runs
- 2 dossiers
AMNOG Modul 3 generation
Updated
- Non-inferiority
- 70%
- Regulaido
- Win rate
- 50%
- Holistic
- —
- Evidence
- 2 runs
- 2 dossiers
Methodology & Judge Calibration
Key evaluation specifications · For the full methodology narrative, see our research post →
Human Expert vs. Judge Acceptance (N=180 Blinded Ratings)
Blinded acceptance rate (AI win or tie vs. accepted G-BA human dossier) across 17 HTA specialists, our calibrated 3-family judge panel + PPI, and an uncalibrated single LLM judge
17 HTA Experts Calibrated Judge Single LLM Judge
Clinical & Epi Synthesis Substantive prose · n = 103
Experts: 45% Calibrated: 42% Single Judge: 72%
Substantive CSR & epidemiology synthesis (49% human acceptance in Module 1).
Verbatim SmPC Copy 1:1 regulatory quotes · n = 42
Experts: 17% Calibrated: 11% Single Judge: 78%
G-BA convention requires verbatim 1:1 quotes from the Fachinformation; single judges rewarded paraphrased essays.
G-BA Search Logs Statutory checklists · n = 35
Experts: 11% Calibrated: 14% Single Judge: 96%
Two-sentence statutory declaration ("2 Sätze reichen!"); single judges exhibited a +84.1 pp verbosity bias.
- (01) Corpus & Hermetic Inputs
- 8 complete authority-accepted G-BA AMNOG (Modules 1–3) and ICH CTD 2.3.S dossiers (118,000+ words, 175 statutory tables). All systems receive the identical blank .docx template and hermetic source PDFs with web search disabled.
- (02) Stage 1: Section Audit (0–80 pts)
- Evaluates each section against both the accepted human dossier and primary source PDFs across factual & numerical precision (40), regulatory & methodological compliance (25), analytical completeness & table population (20), and G-BA style register (15).
- (03) Stage 2: Whole-Document Audit (0–20 pts)
- Scores cross-section numerical and subpopulation consistency (8), structural/XML/table hygiene (6), and inline citation-to-bibliography resolution (6).
- (04) Calibration & Synthetic Anchors
- Calibrated via a 3-family judge panel (Gemini, GPT, Claude) + Prediction-Powered Inference (PPI) against 180 blinded ratings by 17 HTA experts (within 2.2–2.6 pp). Anchored at 0.0 (empty template), 98.5 (human gold), 94.9 (paraphrased ceiling), and 97.9% recall across 47 planted error traps.