Chapter 3.7 AI Testing and Evaluation

Every time you updated a Part II workflow node and re-ran your test contacts, you were testing. The practice was informal: run a few contacts, read the n8n execution log, check that the advisory path was reasonable. That informal practice works at low volume with a small fixture set and a simple prompt.

It fails silently at scale when subtle changes to the prompt or the underlying model shift the output distribution without any individual execution producing an error.

Chapter 3.7 formalizes the informal. It defines what counts as a test, what counts as a failure, how to detect regressions before they reach production, and how human review fits into an automated evaluation pipeline.

Learning Objectives

After completing this chapter, you will be able to:

  • Explain why AI testing requires different approaches from software testing probabilistic outputs, distribution-level correctness, and the distinction between output variance and behavioral regression.
  • Design the 10-fixture baseline set (F001–F010) with known inputs, expected output ranges, and the behavioral properties each fixture is designed to verify.
  • Build the AI test pyramid: contract tests (output schema compliance), behavioral tests (expected advisory path for known inputs), and calibration tests (confidence score reasonableness for boundary inputs).
  • Implement the confidence calibration verification workflow that checks whether the model’s stated confidence correlates with actual accuracy across the fixture population.
  • Configure prompt drift detection using p6_prompt_version to trigger an automated regression run when a prompt update is deployed, comparing the new output distribution to the baseline.
  • Troubleshoot a regression where a prompt update improved market classification accuracy but caused team quality scores to shift downward, using the fixture set to isolate the affected behavioral dimension.

3.7.1 Why AI Testing Differs from Software Testing

Determinism vs. Probabilism in Testing

Software testing assumes determinism: the same input always produces the same output. A unit test that passes once will pass again under identical conditions. Failures are binary the output either matches or it does not.

AI testing operates under probabilism: the same input may produce different outputs on different calls due to model temperature, provider-side model updates, and context window nondeterminism. A test that passes today may produce a marginally different output tomorrow not because the system is broken, but because AI outputs are draws from a probability distribution, not lookups from a function table. A system that passes every individual test but whose output distribution has shifted is still exhibiting a regression.

This distinction has three practical consequences:

Software testing AI testing
Expected output is an exact value Expected output is a range or schema constraint
A test either passes or fails A test passes, fails, or requires calibration review
Failures are deterministic reproducible Failures may be stochastic retrying may pass
Test coverage measures line/branch coverage Test coverage measures input distribution coverage
Regression = the same input now returns a different value Regression = the same input population now returns a shifted distribution

AI testing does not replace software testing the pipeline nodes around the AI call (Normalize, Validate, Route) are deterministic code and should be unit-tested conventionally. AI testing applies to the AI call itself and to the output contract it must satisfy.

TipDesign Practice

Applying software testing intuitions to an AI call leads to brittle tests that fail on natural output variance and miss actual regressions. A test that asserts advisory_path === "standard_review" on a single call will occasionally fail when the AI legitimately returns "expedited_review" for a borderline contact. A test that asserts the advisory path distribution for a known population stays within ±10% of baseline is a regression test that catches actual behavioral shifts.

3.7.2 Output Contracts as Test Anchors

Output Contracts as Test Anchors

The output contract (defined in 3.3.2 Agent Contracts and extended across Chapters 3.3–6.4) is the first and most important test anchor. Every AI call must satisfy its output contract before any behavioral test is applied.

Output contract violations are always failures they are not probabilistic. If the AI returns a response that cannot be parsed as JSON, or that is missing advisory_path, or that has confidence_score outside [0.00, 1.00], the output contract is violated. The validation gate in Practical 3.3 catches contract violations at runtime; the testing framework catches them at evaluation time against the full fixture set.

Contract tests vs. behavioral tests:

Test type What it asserts Pass/fail
Contract test Response parses as JSON; all required fields present; field types match schema; confidence in [0.00, 1.00] Binary always
Behavioral test Advisory path matches expected for this fixture; confidence ≥ minimum threshold Toleranced based on population

Contract tests run first. If any fixture fails a contract test, the prompt version is blocked from deployment regardless of behavioral test results.

3.7.3 Fixture-Based Evaluation

Fixture

A fixture is a known input paired with an expected output specification. For AI advisory systems, the output specification is not an exact value but a contract:

fixture_id:              F001
input:                   { engagement_type, inquiry_text, vap_combined_score, ... }
expected_advisory_path:  "expedited_review"
expected_confidence_min: 0.70
rationale:               High-score, high-engagement contact; should route to expedited
pass_condition:          advisory_path === expected AND confidence >= expected_confidence_min

Fixtures are not random test contacts. They are curated inputs that represent: - Canonical cases: clear high-score and low-score examples that should always produce the expected path - Edge cases: borderline contacts, missing fields, low-confidence inputs, unusual engagement types - Regression anchors: contacts that were previously misclassified and are now correctly handled verifying the fix holds

The fixture set is the only repeatable, authoritative way to evaluate a prompt version. Real contact data changes; fixture data does not.

TipDesign Practice

Store fixtures as Code node constants in a dedicated n8n fixture workflow, not as live HubSpot contacts. Live contacts can be updated, enriched, or deleted fixture inputs must remain stable to serve as regression anchors. Version-control the fixture set alongside the prompt version string.

3.7.4 The 10-Fixture Baseline Set

Ten fixtures covering the key behavioral zones for the Vantage Advisory Partners advisory system. Each fixture is a distinct test case.

ID Scenario Expected path Expected confidence min Test category
F001 High score (28/30), partnership engagement, strong inquiry expedited_review 0.80 Canonical
F002 Low score (8/30), exploratory engagement, generic inquiry decline 0.75 Canonical
F003 Mid score (18/30), advisory engagement, ambiguous inquiry standard_review 0.65 Edge borderline
F004 High score (26/30), but company_size_category: micro expedited_review 0.70 Edge firmographic override test
F005 Valid payload, but AI returns confidence = 0.45 rule_fallback N/A Edge validation gate
F006 Missing inquiry_text field entirely rule_fallback N/A Edge contract violation
F007 Mid score (19/30), inquiry_text contains “not interested at this time” standard_review OR decline 0.65 Edge sentiment signal
F008 High score (25/30), but all 5 peer contacts are decline expedited_review 0.65 Edge peer context anchoring
F009 Agent A classifies engagement_type as exploratory (low certainty) standard_review 0.65 Multi-agent path
F010 Valid mid-score contact, second submission from same email standard_review 0.65 Repeat submission

F005 and F006 test the rule fallback path specifically. For F005, the expected behavior is that Parse Response A detects confidence < 0.65, triggers the validation gate, and routes to the rule fallback the AI advisory path is not reached. For F006, the pipeline’s Normalize stage should catch the missing field and route to an error path before the AI call. A system that handles F005 and F006 correctly has a working fallback path. A system that does not has silent failure modes.

F008 tests peer context anchoring. The peer context from Chapter 3.1 is injected with a calibration instruction: “Do not let peer context override independent analysis of the current submission.” F008 verifies this instruction holds a high-score contact should not be routed to decline merely because its peer group was all declined.

CautionProduction Risk

Do not treat the 10-fixture baseline set as a comprehensive regression suite. Ten fixtures cover key behavioral zones; they do not cover the full input distribution. Large regression suites with 50–100 fixtures, covering the full range of engagement types, score distributions, and firmographic combinations, belong in the capstone. The 10-fixture set is a minimum viable testing baseline for Chapters 3.1–6.6.

3.7.5 Confidence Calibration

Confidence Calibration

A well-calibrated system means stated confidence correlates with observed accuracy; an overconfident system gives governance decisions a false basis.

A well-calibrated AI advisory system means the confidence scores it reports correlate with actual accuracy. If the system reports 0.90 confidence, it should be correct approximately 90% of the time. If it reports 0.90 confidence but is only correct 60% of the time, it is overconfident and the confidence score is not a reliable signal for governance decisions or alert thresholds.

Measuring calibration:

  1. Run all 10 fixtures. Record advisory_path and confidence_score for each.
  2. Group fixtures by confidence band: [0.60–0.69], [0.70–0.79], [0.80–0.89], [0.90+].
  3. For each band, compute the proportion of fixtures where advisory_path === expected_advisory_path (accuracy within that band).
  4. Plot stated confidence (x-axis) against observed accuracy (y-axis). A calibrated system produces points near the diagonal.

With only 10 fixtures, the calibration measurement is approximate. The goal at this stage is to detect gross miscalibration a system claiming 0.90 confidence while being correct only 50% of the time not to produce a statistically reliable calibration curve.

The capstone fixture set (50–100 fixtures) enables more reliable calibration measurement. A system with 10 fixtures can detect that it is badly miscalibrated. A system with 100 fixtures can measure precisely where the miscalibration occurs.

Calibration and the validation gate:

The 0.65 threshold in the Practical 3.3 validation gate is a governance assumption: below 0.65, the AI’s advisory path is not reliable enough to act on. If the calibration measurement shows the system is overconfident (high stated confidence, lower accuracy), the threshold may need to be raised. If the system is underconfident (lower stated confidence, higher accuracy), the threshold may be conservatively too high.

TipDesign Practice

The governance fields from Chapter 3.6 specifically override_authorized_by and the decision to set an override rely on the AI’s confidence score as one signal. If confidence is miscalibrated, governance decisions that partially depend on it are also miscalibrated. Confidence calibration is not an academic metric; it is a governance prerequisite.

3.7.6 Regression Suites and Prompt Version Testing

Regression Suite

A regression suite is the complete fixture set run against a new prompt version to detect behavioral shifts before deployment. The workflow is:

1. Increment prompt version string (e.g., v1.0 → v1.1)
2. Update Build Prompt node with revised prompt
3. Run all fixtures against the new version
4. Compare per-fixture results: advisory_path and confidence_score
5. For each fixture: classify the result
   - PASS: path matches expected; confidence ≥ minimum
   - REGRESSION: path changed from expected
   - CONFIDENCE SHIFT: path correct; confidence shifted by > 0.10
   - CONTRACT VIOLATION: response fails output contract
6. If any REGRESSION or CONTRACT VIOLATION: do not deploy v1.1
7. If CONFIDENCE SHIFTS only: review; decide whether to deploy

A regression is not just any change it is a change from the expected output for a fixture that previously passed. A change that improves accuracy on previously wrong fixtures is not a regression; it is a fix.

The p6_prompt_version property from Chapter 3.5 is what connects the fixture-based regression workflow to the population-level observability workflow. After deploying v1.1, the observability workflow computes the advisory path distribution for contacts processed under v1.1 and compares it to the v1.0 distribution.

If the regression suite showed no regressions but the observability distribution shifts materially, a model update (not the prompt change) may be responsible the prompt drift detection from Prompt Version Correlation and Drift Rate identifies this. A system with fixture tests but no observability can pass every fixture while silently drifting at population level. A system with both catches the drift.

3.7.7 The AI Test Pyramid

The software test pyramid many unit tests, fewer integration tests, fewest end-to-end tests applies to AI systems with AI-specific content at each tier.

Pyramid tier AI equivalent Volume Automation What it catches
Unit Output contract validation Runs on every fixture, every version Fully automated Schema violations, missing fields, out-of-range values
Integration Fixture behavioral tests 10 fixtures (baseline); 50–100 (capstone) Fully automated Prompt regressions, confidence calibration shifts
End-to-end Human review loop 5–10% of real executions, sampled Manual Model drift, edge cases not in fixture set, output quality

The key insight is that the human review loop is not a replacement for automated testing it is a complementary tier that catches what fixtures cannot: novel inputs outside the fixture distribution, subjective quality assessments, and slow model drift that accumulates over months rather than appearing in a single version change.

NoteEngineering Rationale

Design the test pyramid so that the automated tiers (contract + behavioral) catch all failures that are detectable from the output contract. Reserve human review for the residual cases where the output technically passes the contract but something about the advisory recommendation requires judgment a fixture cannot encode.

3.7.8 Human Review Loops

Human review loops integrate judgment into the evaluation cycle at defined points:

Pre-deployment review: Before a new prompt version is deployed, a team member reviews a sample of fixture results, particularly any CONFIDENCE SHIFT cases the automated suite flagged. This is the approval step before updating p6_prompt_version in production.

Ongoing sampling review: A weekly sample of 5–10% of real executions selected by the observability workflow based on confidence score distribution, specifically contacts near the 0.65 threshold is queued for human review. The reviewer assesses whether the AI’s advisory path was appropriate given the contact’s profile. Disagreements are recorded and may become new fixtures.

Alert-triggered review: When the observability workflow (3.5.6 Alerting Design) issues a 🚨 Critical alert, human review of the flagged contacts is mandatory before the next batch execution. The alert identifies that something has changed at population level; human review identifies what.

NoteEngineering Rationale

Human review loops are not a sign of system immaturity they are an architectural component. An AI advisory system without a defined human review mechanism does not have a safety net for the failures that automated tests cannot detect. The review loop is where fixture set gaps are identified and where domain expertise is injected back into the testing cycle.

Reference Diagrams

Figure 3.7.3 Prompt Regression Detection

Figure 39.1 shows a fixture-set comparison between prompt v1.0 and v1.1. Each fixture row shows the expected path, the v1.0 result, and the v1.1 result. Regressions (path changed from expected) are highlighted. This is the output of the regression suite workflow described in 3.7.6 Regression Suites and Prompt Version Testing.

%%{init:{"theme":"base","themeVariables":{"primaryColor":"#eef2ff","primaryTextColor":"#1e1b4b","primaryBorderColor":"#6366f1","lineColor":"#6366f1","clusterBkg":"#f8f9ff","clusterBorder":"#6366f1","titleColor":"#1e1b4b","edgeLabelBackground":"#f6f4ef","fontFamily":"system-ui,sans-serif","fontSize":"13px"}}}%%

flowchart TD
    FS["10 Fixture Suite F001–F010 (known inputs + expected paths)"]:::process

    FS --> V0["Run against v1.0 Result: 10 / 10 PASS"]:::success
    FS --> V1["Run against v1.1 Result: 8 PASS · 2 FAIL"]:::fallback

    V0 --> CMP["Compare Results per Fixture"]:::process
    V1 --> CMP

    CMP --> REG["Regressions detected F004: expedited_review → standard_review F007: standard_review → decline"]:::decision

    REG --> BLOCK["DEPLOYMENT BLOCKED Review prompt changes affecting high-score/micro-company routing (F004) and sentiment-signal routing (F007)"]:::fallback
    classDef trigger  fill:#dcfce7,stroke:#16a34a,color:#14532d,font-weight:600
    classDef process  fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
    classDef decision fill:#fef9c3,stroke:#ca8a04,color:#78350f,font-weight:600
    classDef success  fill:#d1fae5,stroke:#059669,color:#064e3b,font-weight:700
    classDef fallback fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
Figure 39.1: Per-Fixture Version Comparison. Per-fixture comparison of v1.0 vs. v1.1 results: two regressions in v1.1 block deployment, isolating which prompt change broke which behavioral path.
Fixture Expected path v1.0 result v1.1 result
F001 expedited_review expedited_review ✅ expedited_review ✅
F002 decline decline ✅ decline ✅
F003 standard_review standard_review ✅ standard_review ✅
F004 expedited_review expedited_review ✅ standard_review 🔴 REGRESSION
F005 rule_fallback rule_fallback ✅ rule_fallback ✅
F006 rule_fallback rule_fallback ✅ rule_fallback ✅
F007 standard_review standard_review ✅ decline 🔴 REGRESSION
F008 expedited_review expedited_review ✅ expedited_review ✅
F009 standard_review standard_review ✅ standard_review ✅
F010 standard_review standard_review ✅ standard_review ✅

Result: 2 regressions detected in v1.1 (F004, F007).

Deployment decision: 🚨 BLOCKED review prompt changes that affected high-score/micro-company routing (F004) and sentiment-signal routing (F007). Revert or revise v1.1 before deploying.

Note: If v1.0 had regressions on F004/F007 and v1.1 fixed them, re-evaluate whether the new behavior is a fix or a regression fixture rationale determines the correct interpretation.

Practical Exercise 3.7 Fixture-Based Evaluation Workflow

Business Scenario

Vantage Advisory Partners is about to update the advisory prompt from v1.0 to v1.1 to improve market classification accuracy. The team needs to verify the change does not introduce regressions in other behavioral dimensions before deploying it to production contacts.

The Problem

There is no repeatable, authoritative way to verify prompt behavior across known inputs. Individual test contacts in HubSpot can be modified, and the execution log does not record which prompt version produced which output. Without a stable fixture set and automated comparison, a prompt update can silently break edge-case routing.

The Architectural Solution

A manually triggered fixture evaluation workflow that runs 10 curated test inputs through the live AI pipeline, validates each response against its output contract, compares the advisory path to the expected result, and posts a deployment recommendation to Slack.

Workflow

Figure 39.2 shows the manually triggered fixture evaluation workflow: 10 curated inputs loop through the live AI pipeline, each response validated against its output contract and compared to the expected advisory path before a deployment recommendation is posted to Slack.

%%{init:{"theme":"base","themeVariables":{"primaryColor":"#eef2ff","primaryTextColor":"#1e1b4b","primaryBorderColor":"#6366f1","lineColor":"#6366f1","clusterBkg":"#f8f9ff","clusterBorder":"#6366f1","titleColor":"#1e1b4b","edgeLabelBackground":"#f6f4ef","fontFamily":"system-ui,sans-serif","fontSize":"13px"}}}%%

flowchart TD
    A(["Manual Trigger"]):::trigger --> B["Load Fixtures (Code node constants)"]:::process
    B --> C[["Loop Over Items 10 fixtures"]]
    C --> D["Build Prompt + HTTP Request (LLM)"]:::process
    D --> E["Validate Contract (Code node)"]:::process
    E --> F["Compare to Expected (Code node)"]:::process
    F --> G["Slack Post Evaluation Report + Deployment Decision"]:::success
    classDef trigger  fill:#dcfce7,stroke:#16a34a,color:#14532d,font-weight:600
    classDef process  fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
    classDef decision fill:#fef9c3,stroke:#ca8a04,color:#78350f,font-weight:600
    classDef success  fill:#d1fae5,stroke:#059669,color:#064e3b,font-weight:700
    classDef fallback fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
Figure 39.2: Fixture Evaluation Workflow. Fixture evaluation workflow: 10 curated inputs loop through the live AI pipeline, each validated against its output contract, producing a CLEAR or BLOCKED deployment recommendation.

Step 1 Create the Fixture Workflow

Purpose

Establish the workflow container and trigger that runs the fixture evaluation on demand. Unlike the observability workflow (which runs on a schedule), the fixture evaluation workflow is manually triggered it executes before each prompt version deployment to produce a deployment recommendation. A scheduled trigger would defeat the purpose: the fixture evaluation must run against the specific prompt version being tested.

Operation Summary

Property Value
Workflow name VAP Fixture Evaluation Prompt v1.0
Trigger type Manual Trigger (no schedule)
Primary Function On-demand evaluation of all fixtures against the live AI pipeline
Output Evaluation report posted to #vap-ops with CLEAR or BLOCKED recommendation

Engineering Rationale

NoteEngineering Rationale

The fixture evaluation workflow is deliberately separate from Workflow A. Running fixtures through the production workflow would intermingle test traffic with real contacts, pollute the p6_ observability properties on real contacts with fixture-driven values, and risk triggering downstream Workflow B actions on test inputs. The standalone fixture workflow calls the same Build Prompt node and AI endpoint as Workflow A, but it routes all results through evaluation logic rather than into HubSpot Contact records.

Step 2 Define Fixtures as Code Node Constants

Purpose

Store the 10 fixture definitions as stable, version-controlled constants in the workflow rather than as live HubSpot contacts. Real contact data can be updated, enriched, or deleted fixture inputs must remain unchanged across every evaluation run to serve as reliable regression anchors. If the fixture inputs change between runs, a changed result cannot be attributed to a prompt change versus an input change.

Operation Summary

Property Value
Node Type Code node
Node Name Load Fixtures
Primary Function Return the 10 fixture objects as stable, immutable test inputs
Input None constants only
Output Array of 10 fixture objects for the Loop Over Items node

Fixture Object Structure

// Load Fixtures Code node
// Returns all 10 fixtures as a stable array of objects
return [
  {
    fixture_id: "F001",
    input: {
      engagement_type: "partnership",
      inquiry_text: "We are exploring a long-term partnership to co-develop advisory products for the SMB segment.",
      vap_combined_score: 28,
      company_size_category: "mid-market",
      peer_context: "3 of 5 peer contacts: expedited_review"
    },
    expected_advisory_path: "expedited_review",
    expected_confidence_min: 0.80,
    rationale: "High score, partnership engagement, strong inquiry canonical expedited case"
  },
  {
    fixture_id: "F002",
    input: {
      engagement_type: "exploratory",
      inquiry_text: "Just seeing what you offer.",
      vap_combined_score: 8,
      company_size_category: "micro",
      peer_context: "5 of 5 peer contacts: decline"
    },
    expected_advisory_path: "decline",
    expected_confidence_min: 0.75,
    rationale: "Low score, exploratory engagement, generic inquiry canonical decline case"
  }
  // ... F003 through F010 following the same structure from @sec-3-7-4
];

Construct F005 with inquiry_text designed to produce low AI confidence use a minimal, ambiguous, or off-topic submission (e.g., "n/a" or a single word) that the model reliably cannot assess with confidence ≥ 0.65. Adjust if the model produces confidence above the threshold on your first test run.

Output Table

Field Description
fixture_id Unique identifier F001 through F010
input Full contact payload passed to the Build Prompt node
expected_advisory_path The path the AI is expected to return for this fixture
expected_confidence_min Minimum acceptable confidence for a PASS result (N/A for fallback fixtures)
rationale Why this fixture exists and what behavioral property it verifies

Engineering Rationale

NoteEngineering Rationale

Fixture inputs are defined as Code node constants, not database records or HubSpot contacts, because the evaluation framework requires that the same inputs produce comparable results across every run. Storing fixtures in a database introduces the risk of accidental modification. Storing them as Code node constants makes modification an explicit, visible workflow edit which is the correct barrier for a regression anchor.

Step 3 Run Each Fixture Through the AI Pipeline

Purpose

Execute each fixture’s input payload through the same Build Prompt and AI call nodes used in Workflow A, producing a per-fixture AI response that can be validated against the output contract and compared to the expected advisory path. This step is the evaluation engine without it, the fixture definitions exist but produce no results.

Operation Summary

Property Value
Node Types Loop Over Items → Build Prompt (Code node) → HTTP Request (LLM) → Parse Response (Code node)
Primary Function Run each of the 10 fixture inputs through the live AI pipeline and return structured results
Input Array of 10 fixture objects from Step 2
Output Per-fixture object: fixture_id, actual_advisory_path, actual_confidence_score, raw_response

Implementation Logic

// Per-fixture output structure returned from Parse Response Code node
// after each LLM call in the loop
return {
  json: {
    fixture_id:              $input.item.json.fixture_id,
    expected_advisory_path:  $input.item.json.expected_advisory_path,
    expected_confidence_min: $input.item.json.expected_confidence_min,
    actual_advisory_path:    parsedResponse.advisory_path,
    actual_confidence_score: parsedResponse.confidence_score,
    raw_response:            rawLLMOutput,
    parse_success:           true   // false if JSON.parse threw
  }
};

Connect the Loop Over Items node to the fixture array from Step 2. On each iteration, pass $input.item.json.input as the contact payload to the Build Prompt Code node. The LLM HTTP Request and Parse Response logic are the same nodes used in Workflow A reference them directly or copy the node configuration.

Output Table

Field Description
fixture_id Identifier carried through for result matching
actual_advisory_path Advisory path returned by the AI for this fixture input
actual_confidence_score Confidence score returned by the AI
raw_response Full LLM response text used for contract validation
parse_success Whether the response parsed as valid JSON

Engineering Rationale

NoteEngineering Rationale

The fixture evaluation workflow calls the same Build Prompt node and LLM endpoint as Workflow A. This is intentional: evaluating against the actual production prompt configuration ensures that a change to the prompt node in Workflow A is reflected in the next fixture evaluation run. If the fixture workflow used a separate prompt configuration, it could pass all 10 fixtures on a prompt that would fail in production because the evaluation and the production prompt diverged.

Step 4 Contract Validation

Purpose

Verify that each AI response satisfies the output contract before any behavioral comparison is attempted. Contract violations are binary failures a response that cannot be parsed as JSON, is missing required fields, or has a confidence score outside [0.00, 1.00] fails regardless of what advisory path it contains. A prompt version that produces contract violations on any fixture is blocked from deployment even if all other fixtures pass.

Operation Summary

Property Value
Node Type Code node
Node Name Validate Contract
Primary Function Check each AI response against the output contract schema
Input Per-fixture result from Step 3
Output Per-fixture result with contract_valid and contract_violations fields appended

Implementation Logic

// Validate Contract Code node
const result = $json;
const violations = [];

// Check 1: Response parsed as JSON
if (!result.parse_success) {
  violations.push("PARSE_FAILURE: response is not valid JSON");
}

// Check 2: Required fields present
const required = ["advisory_path", "confidence_score", "rationale"];
for (const field of required) {
  if (result.actual_advisory_path === undefined && field === "advisory_path") {
    violations.push(`MISSING_FIELD: ${field}`);
  }
  if (result.actual_confidence_score === undefined && field === "confidence_score") {
    violations.push(`MISSING_FIELD: ${field}`);
  }
}

// Check 3: Confidence score in valid range
const conf = parseFloat(result.actual_confidence_score);
if (!isNaN(conf) && (conf < 0.00 || conf > 1.00)) {
  violations.push(`OUT_OF_RANGE: confidence_score ${conf} not in [0.00, 1.00]`);
}

return {
  json: {
    ...result,
    contract_valid:      violations.length === 0,
    contract_violations: violations
  }
};

Output Table

Field Description
contract_valid true if all three checks pass; false if any violation found
contract_violations Array of violation descriptions empty if contract is satisfied

Engineering Rationale

NoteEngineering Rationale

Contract validation runs before behavioral comparison because a contract-violating response cannot be meaningfully compared to an expected advisory path. If the response did not parse as JSON, actual_advisory_path is undefined comparing it to expected_advisory_path produces a false REGRESSION rather than correctly identifying the root cause as a CONTRACT_VIOLATION. Ordering validation before comparison ensures each fixture is classified by its actual failure mode.

Step 5 Behavioral Comparison

Purpose

Classify each fixture result as PASS, REGRESSION, or CONTRACT_VIOLATION based on whether the actual advisory path matches the expected path and the actual confidence meets the minimum threshold. This classification is the deployment decision input the evaluation report in Step 6 summarizes these per-fixture results into an overall CLEAR or BLOCKED recommendation.

Operation Summary

Property Value
Node Type Code node
Node Name Compare to Expected
Primary Function Produce PASS / REGRESSION / CONTRACT_VIOLATION per fixture
Input Per-fixture result with contract validation from Step 4
Output Per-fixture result with path_match, confidence_pass, and result classification

Implementation Logic

// Compare to Expected Code node
const item = $json;

// Contract violations take precedence
if (!item.contract_valid) {
  return { json: { ...item, path_match: false, confidence_pass: false, result: "CONTRACT_VIOLATION" } };
}

const path_match = item.actual_advisory_path === item.expected_advisory_path;

// Rule-fallback fixtures have no confidence minimum N/A
const isRuleFallback = item.expected_advisory_path === "rule_fallback";
const confidence_pass = isRuleFallback
  ? true
  : parseFloat(item.actual_confidence_score) >= item.expected_confidence_min;

const result = (!path_match) ? "REGRESSION"
             : (!confidence_pass) ? "CONFIDENCE_SHIFT"
             : "PASS";

return { json: { ...item, path_match, confidence_pass, result } };

Output Table

Field Description
path_match true if actual_advisory_path === expected_advisory_path
confidence_pass true if confidence ≥ minimum; true for rule_fallback fixtures (N/A)
result PASS, REGRESSION, CONFIDENCE_SHIFT, or CONTRACT_VIOLATION

Engineering Rationale

NoteEngineering Rationale

CONFIDENCE_SHIFT is classified separately from REGRESSION because these two outcomes require different responses. A REGRESSION means the AI returned a different advisory path than expected the prompt change broke a specific routing behavior and must be investigated before deployment. A CONFIDENCE_SHIFT means the AI returned the correct path but with lower confidence than the minimum this may be acceptable depending on how far the confidence shifted and whether the validation gate threshold needs recalibration. Conflating the two into a single FAIL classification loses this distinction.

Step 6 Output the Evaluation Report

Purpose

Aggregate the 10 per-fixture results into a single structured evaluation report and post it to #vap-ops with an explicit deployment recommendation. The report is the artifact that authorizes or blocks the prompt version deployment it must be scannable in under 60 seconds and contain enough per-fixture detail to identify which fixture caused a blockage and why.

Operation Summary

Property Value
Node Type HTTP Request
Method POST
Endpoint Your Slack incoming webhook URL for #vap-ops
Primary Function Deliver the fixture evaluation report and deployment recommendation
Input Aggregated per-fixture classification results from Step 5
Output Slack message in #vap-ops with CLEAR or BLOCKED status

Request Payload

{
  "text": "VAP FIXTURE EVALUATION {{promptVersion}}\n──────────────────────────────────\nTotal fixtures:      10\nPasses:              {{passCount}}\nRegressions:         {{regressionCount}}\nConfidence shifts:   {{confidenceShiftCount}}\nContract violations: {{contractViolationCount}}\n──────────────────────────────────\nPer-fixture results:\n{{perFixtureTable}}\n──────────────────────────────────\nDeployment recommendation: {{deploymentStatus}}"
}

Construct the per-fixture table and deployment status in a preceding Code node (Build Report) that aggregates all 10 results before the Slack HTTP Request. Compute:

// Build Report Code node (runs after all 10 fixture results are collected)
const results = $input.all().map(i => i.json);
const passCount = results.filter(r => r.result === "PASS").length;
const regressionCount = results.filter(r => r.result === "REGRESSION").length;
const violationCount = results.filter(r => r.result === "CONTRACT_VIOLATION").length;
const shiftCount = results.filter(r => r.result === "CONFIDENCE_SHIFT").length;

const blocked = regressionCount > 0 || violationCount > 0;
const deploymentStatus = blocked ? "🚨 BLOCKED" : "✅ CLEAR TO DEPLOY";

const perFixtureTable = results.map(r =>
  `${r.fixture_id}: expected=${r.expected_advisory_path} actual=${r.actual_advisory_path || "N/A"} → ${r.result}`
).join("\n");

return { json: { passCount, regressionCount, violationCount, shiftCount, deploymentStatus, perFixtureTable } };

Output Table

Output Description
Slack message Evaluation report in #vap-ops with counts, per-fixture table, and deployment recommendation
deploymentStatus ✅ CLEAR TO DEPLOY if no REGRESSION or CONTRACT_VIOLATION; 🚨 BLOCKED otherwise

Engineering Rationale

NoteEngineering Rationale

The deployment recommendation is binary CLEAR or BLOCKED because a human reviewing the report should not have to interpret raw counts to make a deployment decision. The report provides the counts and per-fixture detail for investigation; the recommendation line provides the action. CONFIDENCE_SHIFT results do not trigger a BLOCKED status because they require judgment a shift from 0.81 to 0.78 is acceptable; a shift from 0.82 to 0.63 (below the validation gate) may not be. Regressions and contract violations are never acceptable and trigger BLOCKED automatically.

Step 7 Test Calibration

Purpose

Assess whether the AI’s stated confidence scores correlate with observed accuracy across the passing fixture results. Calibration measurement at 10 fixtures is approximate the goal is to detect gross miscalibration before it propagates into governance decisions that partially rely on confidence as a signal. A prompt version that is overconfident has a different operational risk profile than one that is underconfident.

Operation Summary

Property Value
Node Type Code node (optional can be done manually)
Node Name Compute Calibration
Primary Function Group passing fixtures by confidence band and compute observed accuracy per band
Input Per-fixture results from Step 5
Output Calibration summary: accuracy per confidence band; overall calibration assessment

Implementation Logic

// Compute Calibration Code node
// Groups passing fixtures by confidence band and measures observed accuracy per band
const results = $input.all().map(i => i.json).filter(r => r.result !== "CONTRACT_VIOLATION");

const bands = {
  "0.60-0.69": { total: 0, correct: 0 },
  "0.70-0.79": { total: 0, correct: 0 },
  "0.80-0.89": { total: 0, correct: 0 },
  "0.90+":     { total: 0, correct: 0 }
};

for (const r of results) {
  const conf = parseFloat(r.actual_confidence_score);
  if (isNaN(conf)) continue;
  const band = conf >= 0.90 ? "0.90+"
             : conf >= 0.80 ? "0.80-0.89"
             : conf >= 0.70 ? "0.70-0.79"
             : "0.60-0.69";
  bands[band].total++;
  if (r.path_match) bands[band].correct++;
}

const calibrationSummary = Object.entries(bands)
  .filter(([, b]) => b.total > 0)
  .map(([band, b]) => ({
    band,
    fixtures: b.total,
    accuracy: (b.correct / b.total).toFixed(2),
    assessment: b.correct / b.total >= parseFloat(band.split("-")[0] || "0.90") - 0.10
      ? "Calibrated" : "Miscalibrated"
  }));

return { json: { calibrationSummary } };

Note whether the system appears calibrated (accuracy near stated confidence), overconfident (accuracy below stated confidence), or underconfident (accuracy above stated confidence) for your 10-fixture sample. With 10 fixtures, some bands may contain only 1–2 fixtures the calibration assessment is directional, not statistically precise.

ImportantCritical Requirement

If F005 (low-confidence trigger) does not produce a rule fallback in testing, the AI may be producing higher confidence than expected for the minimal input. Adjust the F005 inquiry_text to be more genuinely ambiguous (a single word or an obviously off-topic submission) until the confidence reliably falls below 0.65.

Deliverable: Fixture evaluation workflow. All 10 fixtures run. Evaluation report posted to Slack. Calibration summary noted. Deployment status: CLEAR or BLOCKED (with rationale).

Estimated time: 3–4 hours.

Production Consideration

Fixture evaluation workflows are run manually before each prompt version deployment. The most common operational failure is running the fixtures against the wrong prompt version because the Build Prompt node in the fixture workflow was updated without updating p6_prompt_version. Always verify that the prompt version constant in the fixture workflow matches the version string being tested, and that it matches the value that will be written to p6_prompt_version in production.

For regression suites with more than 10 fixtures, the Loop Over Items approach may produce n8n execution timeouts if each LLM call takes 2–4 seconds. At 50+ fixtures, batch the loop iterations and collect results in a persistent store (Notion database or Google Sheet) rather than passing all results through the execution chain in a single workflow run.

Discussion Questions

  1. Fixture F008 tests whether peer context anchors the AI incorrectly toward the peer group’s advisory path. If you run F008 ten times and observe the AI routing to expedited_review 7 times and decline 3 times, does this fixture pass, fail, or require a new test design? What would you change about the fixture’s pass condition to handle probabilistic variance?

  2. The regression suite blocks deployment of v1.1 if any fixture produces a REGRESSION. But what if v1.1 fixes a known problem with v1.0 a case where v1.0 was wrong and v1.1 is correct? How would you update the fixture set to distinguish a fix from a regression?

  3. The human review loop in 3.7.8 Human Review Loops selects contacts near the 0.65 confidence threshold for review. If the calibration analysis shows the system is underconfident (actual accuracy higher than stated confidence), should the review sample be shifted toward higher-confidence contacts, lower-confidence contacts, or kept the same? Justify your answer.

Chapter Summary

AI testing differs from software testing in one fundamental way: expected outputs are ranges and distributions, not exact values. This distinction drives everything the design of fixtures, the definition of regression, the interpretation of confidence calibration.

Key Principle

A regression is not a changed value it is a distribution shift. Contract tests catch schema violations; behavioral tests catch path changes; human review catches quality failures that no fixture encodes. All three tiers are required.

The testing framework has three components: output contract validation (fully automated, binary), fixture behavioral tests (the 10-fixture baseline set, fully automated), and human review loops (sampled, manual). Together they form the AI test pyramid.

Confidence calibration connects the testing framework to the governance framework from Chapter 3.6: if the AI is overconfident, the 0.65 validation gate threshold should be raised. The p6_prompt_version property from Chapter 3.5 connects fixture results to population-level observability a regression-free v1.1 that nonetheless shifts the production distribution signals model drift, not prompt regression.

Transition to Chapter 3.8

Chapter 3.7 completes the seven technical capabilities of Part III: multi-context (6.1), pipelines (6.2), multi-agent (6.3), event-driven (6.4), observability (6.5), governance (6.6), testing (6.7). Each chapter added one capability to the Vantage Advisory Partners platform.

Chapter 3.8 steps back from individual capabilities to the complete architecture. It answers the questions that run through all seven chapters: How do the pieces fit together? When should you not add a piece? What are the real tradeoffs? And what does it take to operate this architecture in production? The three Architecture Decision Records ADR-6A, ADR-6B, ADR-6C formalize the decisions behind the architecture. The synthesis map shows how every Part III chapter contributes to the foundation the capstone will build on.

Key Takeaways

  1. AI testing differs from software testing because AI outputs are probabilistic regressions are distribution shifts, not value changes.
  2. Output contracts are test anchors. Contract violations are always binary failures. Behavioral tests are toleranced.
  3. The 10-fixture baseline set covers canonical cases, edge cases, and regression anchors. Comprehensive regression suites belong in the capstone.
  4. Confidence calibration measures whether stated confidence correlates with actual accuracy. Overconfidence signals that the 0.65 validation gate threshold should be raised.
  5. The regression suite workflow: run all fixtures against the new prompt version, compare per-fixture results, block deployment on any REGRESSION or CONTRACT_VIOLATION.
  6. The AI test pyramid has three tiers: output contract validation (base), fixture behavioral tests (middle), human review loop (top). Human review is an architectural component, not a sign of immaturity.
  7. p6_prompt_version connects the fixture-based regression workflow to the population-level observability workflow from Chapter 3.5.

End of Chapter 3.7 AI Testing and Evaluation