DRIFTGATE
← Back to Landing
SUITE RUN REPORT · ID: a1b2c3d4-0004 CRITICAL REGRESSION DETECTED

Behavioral Regression: account_closure

Baseline (Prompt v1)
meta-llama/Llama-3.1-8B-Instruct

Strict JSON schema requiring fields: status, priority, category, summary.

Candidate (Prompt v2 "Empathetic")
meta-llama/Llama-3.1-8B-Instruct

Conversational update silently renamed required key 'summary' → 'description'.

Regressed Case Pass Delta
100% → 0%
Statistical Significance
p = 1.87e-18
Wilson 95% Confidence
[-1.00, -0.85]
Admitted Verifiers
4/5 Calibrated

Behavioral Cluster Partition Breakdown

Cluster / Behavioral Case Baseline Pass Candidate Pass Effect Size p-Value (Adj) Classification
account_closure
required_field:summary		 40/40 (100%) 0/40 (0%) -1.000 1.872e-18 REGRESSION
billing
json_schema_valid		 40/40 (100%) 40/40 (100%) 0.000 1.000 STABLE
technical_outage
tool_call:lookup_customer		 40/40 (100%) 40/40 (100%) 0.000 1.000 STABLE
password_reset
json_schema_valid		 40/40 (100%) 40/40 (100%) 0.000 1.000 STABLE
incomplete
non_empty_response		 40/40 (100%) 40/40 (100%) 0.000 1.000 STABLE

Calibration Gate Auditing (The Moat)

Verifiers must be tested on ground-truth slices before admission. Tests with False Positive Rate > 0.100 are rejected.

Verifier Type Behavior Descriptor False Positive Rate Recall Gate Status
assertion (required_field) account_closure 0.000 1.000 ADMITTED
assertion (required_field) billing 0.000 1.000 ADMITTED
uncalibrated_llm_judge account_closure (rubber stamp) 1.000 1.000 REJECTED (FPR > 0.10)