Chapter 3.5 AI Observability and Monitoring

In Chapter 2.9, you learned to read the n8n execution log to diagnose specific failures a reactive skill applied when something breaks. Observability is a proactive discipline: designing the system so you can measure whether it is behaving correctly across thousands of executions before something breaks.

The execution log tells you what happened in one execution. An observability layer tells you whether the system’s behavior across all executions is within the expected range and alerts you when it begins to drift. Chapter 3.5 starts where Chapter 2.9 ends.

The four p6_ properties configured in 3.0.6 Observability Bootstrap are already populated from Practicals 6.1 through 6.4. This chapter builds the observability layer on top of them.

Learning Objectives

After completing this chapter, you will be able to:

Describe the three pillars of AI observability (Logs, Metrics, Alerts) and explain why each is necessary and why none is sufficient alone.
Implement the five health metrics (AI execution rate, confidence score distribution, rule fallback rate, agent call count drift, prompt version consistency) with their alert thresholds and escalation conditions.
Build the weekly observability workflow that queries HubSpot for the five metrics, computes current values against the baseline, and sends a structured health report to Slack.
Configure prompt drift monitoring using the p6_prompt_version property to detect when a prompt version change shifts the confidence score distribution beyond the acceptable variance band.
Explain the difference between reactive diagnosis (reading the execution log after a failure) and proactive observability (detecting distribution drift before failures occur), using the confidence distribution shift scenario as a concrete example.
Troubleshoot a platform where the rule fallback rate has risen from 12% to 41% over two weeks, by using the observability metrics to isolate the contributing factor.

3.5.1 From Diagnosis to Measurement

Business Scenario

Vantage Advisory Partners has been running their AI advisory pipeline for several weeks. Individual executions complete without errors but no one can say whether the system’s advisory outputs are consistent over time, or whether the AI’s behavior has quietly shifted.

The Problem

Diagnosis answers what went wrong with this execution but only after a failure occurs. There is no mechanism to detect whether the system’s behavior across all executions is within the expected range before a failure surfaces.

The Architectural Solution

A proactive observability layer that measures AI system behavior across the population of executions, computes aggregate metrics, and alerts on distribution drift before individual failures occur.

Diagnosis

Chapter 2.9 taught diagnosis: reading execution history to determine why a specific run failed. Diagnosis requires a failure to examine. It answers the question: what went wrong with this execution?

Observability

Observability answers a different question: is the system behaving the way it should, across all executions? A system can pass diagnosis indefinitely no individual execution fails while producing systematically degraded advisory outputs because the AI model’s behavior has drifted. Monitoring (is the system up?) does not catch this either. Only measurement across the population of executions reveals population-level drift.

Concept	Question answered	Phase
Diagnosis (Chapter 2.9)	Why did this execution fail?	Reactive requires a failure
Monitoring	Is the system running?	Binary up or down
Observability	Is the system behaving correctly at population level?	Proactive continuous measurement

AI systems require all three, but observability is the one Part II does not provide. An advisory system with a rising rule fallback rate has a measurable signal that something is wrong even if every individual execution completes without a workflow error.

3.5.2 The AI Observability Model

The AI Observability Three-Pillar Model

The observability three-pillar model logs, metrics, traces applies to AI systems with AI-specific content at each pillar.

Logs a per-execution record of what each workflow did: advisory path, confidence score, evaluation source, agent call count, prompt version. In Part III, logs are the p6_ properties written to each HubSpot Contact.

Metrics aggregate statistics computed from logs across the population of executions: confidence score distributions, fallback rate, path distribution, agent call count distribution. Computed by the observability workflow.

Traces the path a specific execution took, queryable per contact from HubSpot: p6_last_execution_source + p6_last_advisory_path + p6_agent_call_count combination.

Pillar	General definition	AI-specific content	Part III source
Logs	Record of what each execution did	Per-execution: advisory path, confidence score, evaluation source, agent call count, prompt version	`p6_` properties written to each HubSpot Contact
Metrics	Aggregate statistics computed from logs	Distribution of confidence scores, fallback rate, path distribution, agent call count distribution	Computed by the observability workflow from `p6_` properties
Traces	Which path a specific execution took	Per-contact: `p6_last_execution_source` + `p6_last_advisory_path` + `p6_agent_call_count` combination	Queryable per-contact from HubSpot

The observability workflow reads from the Logs pillar, computes Metrics, and flags Traces for review when metric thresholds are exceeded.

Design Practice

Without the Metrics pillar, AI system health is invisible. A rising rule fallback rate is not visible in the n8n execution log individual executions complete without error. It is only visible when p6_last_execution_source values are aggregated across executions and the proportion of rule_fallback values is computed against the baseline from 3.0.6 Observability Bootstrap Practical B.

3.5.3 Instrumentation Flow

The instrumentation flow is the path from individual execution to aggregate metric to health decision.

Workflow executes
  → writes p6_ properties to HubSpot Contact
  → Observability Workflow (weekly Schedule Trigger) queries contacts
  → Code node computes distributions
  → Comparison Code node checks against baseline
  → Health report posted to #vap-ops Slack channel

The four p6_ properties are the raw data. They are not metrics they are per-contact records. The observability workflow aggregates them across all contacts processed in the reporting window to produce the five health metrics.

Cadence: Weekly is the appropriate cadence for advisory volume at Vantage Advisory Partners scale. For higher-volume systems (Part III Capstone: Meridian Venture Partners at 200–300 submissions per quarter), daily cadence is appropriate for the fallback rate and confidence distribution metrics.

3.5.4 Health Metrics

Five metrics, each computed from the p6_ properties, each with a defined baseline source and alert condition.

Confidence Score Distribution

Confidence Score Distribution

Source: p6_last_confidence_score across all contacts in the reporting window.

Computed: Mean, standard deviation, and proportion of scores below 0.65 (the validation gate threshold from Practical 3.3).

Baseline: Mean and standard deviation from Practical 3.0 Part B.

Signal: A sustained downward drift in mean confidence not a single low-confidence execution indicates model drift or input data quality degradation. A system whose mean confidence drifts down by 0.10 over two consecutive reports has changed its behavior materially. A system that drifts by 0.02 has not.

Alert condition: Mean confidence < (baseline mean − 0.10) for two consecutive weekly reports.

Rule Fallback Rate

Rule Fallback Rate

Source: p6_last_execution_source across all contacts. Count of rule_fallback divided by total contacts.

Baseline: Fallback rate from Practical 3.0 Part B.

Signal: A rising rule fallback rate means the AI is producing more invalid or low-confidence outputs. Causes include prompt drift, model update, input data format change, or increasing edge-case submission volume. A system with a rising fallback rate is routing more decisions through the rule engine, not the AI. That may be intentional or it may be a failure.

Alert condition: Fallback rate > (baseline rate + 0.20) for any single weekly report.

Advisory Path Distribution

Advisory Path Distribution

Source: p6_last_advisory_path across all contacts. Proportion per path value.

Baseline: Path distribution from Practical 3.0 Part B.

Signal: A shift in the distribution more contacts routing to expedited_review than baseline, or fewer to standard_review indicates scoring calibration drift. Individual executions all complete correctly, but the population-level routing has shifted.

Alert condition: Any path proportion changes by more than 0.20 from baseline for two consecutive weekly reports.

Agent Call Count Distribution

Agent Call Count Distribution

Source: p6_agent_call_count across all contacts. Proportion with value 1 vs. 2+.

Baseline: After Practical 3.3, the expected count is 2 for all contacts processed by Workflow A.

Signal: An unexpected increase in p6_agent_call_count = 1 in a period where all executions should be multi-agent indicates silent multi-agent fallback Agent A is failing and the system is completing via the rule fallback path. This does not surface as an n8n error, because the fallback path is a valid workflow path. A system where 20% of executions are single-agent when the architecture specifies two agents has a failure that individual execution logs will not reveal.

Alert condition: More than 15% of contacts in the reporting window have p6_agent_call_count = 1 when the expected value for the current architecture is 2.

Prompt Version Correlation and Drift Rate

Prompt Version Correlation and Drift Rate

Source: p6_last_advisory_path distribution correlated with prompt version. Prompt version is tracked as a constant in the Build Prompt Code node and written to a fifth audit property p6_prompt_version (add this property alongside the four from 3.0.6 Observability Bootstrap).

Prompt drift rate: The proportion of contacts whose p6_last_advisory_path changed between consecutive periods for the same p6_prompt_version. If the prompt has not changed and the distribution has shifted, the shift is model-driven. A system showing distribution drift with no prompt version change has been affected by a provider-side model update.

Prompt regression: If the prompt was updated (new p6_prompt_version value) and the distribution shifted, compare the distributions before and after the version change using the fixture set from Chapter 3.7.

Alert condition: Distribution shift > 0.15 for any path proportion between consecutive periods with the same p6_prompt_version.

Production Risk

Do not conflate prompt drift (model-driven behavior change for the same prompt) with prompt regression (behavior change caused by a prompt update). Prompt drift requires investigating the AI provider; prompt regression requires investigating the prompt change. The p6_prompt_version property is what makes this distinction traceable.

3.5.5 Dashboards and Reporting

The weekly health report posts to Slack with a structured summary of all five metrics and their comparison to baseline. The format should be scannable in under 30 seconds.

Recommended report structure:

VAP AI HEALTH REPORT Week of YYYY-MM-DD
Contacts assessed this week: N
─────────────────────────────────────────
✅ Mean confidence:    0.76  (baseline: 0.74  | delta: +0.02)
✅ Fallback rate:      12%   (baseline: 17%   | delta: -5%)
⚠️  Path distribution: expedited_review +18% above baseline
✅ Multi-agent rate:   94%   (expected: 100%  | 3 fallbacks)
✅ Prompt version:     v1.0  (no version change this period)
─────────────────────────────────────────
Alert status: 1 WARNING (path distribution drift)

The ✅ / ⚠️ / 🚨 icons are computed by the Comparison Code node against the thresholds from 3.5.4 Health Metrics.

Engineering Rationale

HubSpot does not provide a native analytics view for custom properties across contacts. The observability workflow in n8n is the analytics layer. The weekly report is the output. For higher-volume systems, a Notion database or Google Sheet updated by the observability workflow provides a persistent time-series record that Slack does not.

3.5.6 Alerting Design

Not all metric deviations warrant the same response.

Severity	Condition	Action
🚨 Critical	Fallback rate exceeds threshold; unexpected single-agent rate	Immediate Slack alert; investigate before next execution
⚠️ Warning	Path distribution drift; confidence mean drift	Included in weekly report; investigate within 3 business days
ℹ️ Info	Minor deviations within tolerance; prompt version increments	Logged in weekly report; no action required

Engineering Rationale

Do not alert on the first threshold breach. A single-week deviation may be noise small contact volume or an unusual submission batch. Alert on sustained breaches (two consecutive periods) for slow-signal metrics (confidence mean, path distribution). Alert immediately on structural signals (unexpected single-agent fallback rate), because those indicate a system failure, not behavioral drift.

Reference Diagrams

Figure 3.5.2 Instrumentation Flow

The diagram below shows the path from a single execution through property writes to aggregate metric computation.

%%{init:{"theme":"base","themeVariables":{"primaryColor":"#eef2ff","primaryTextColor":"#1e1b4b","primaryBorderColor":"#6366f1","lineColor":"#6366f1","clusterBkg":"#f8f9ff","clusterBorder":"#6366f1","titleColor":"#1e1b4b","edgeLabelBackground":"#f6f4ef","fontFamily":"system-ui,sans-serif","fontSize":"13px"}}}%%

flowchart TD
    A["Single Execution Workflow A"]:::process --> B["Writes p6_ properties to HubSpot Contact"]:::success
    B --> C["Observability Workflow scheduled reads all contacts"]:::process
    C --> D["Aggregate Metrics avg_confidence, fallback_rate error_count, p95_duration"]:::process
    D --> E["Health Report written"]:::process
    E --> F{"Threshold breached?"}:::decision
    F -->|"yes"| G["Slack Alert sent"]:::success
    F -->|"no"| H["Nominal no alert"]:::process
    classDef trigger  fill:#dcfce7,stroke:#16a34a,color:#14532d,font-weight:600
    classDef process  fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
    classDef decision fill:#fef9c3,stroke:#ca8a04,color:#78350f,font-weight:600
    classDef success  fill:#d1fae5,stroke:#059669,color:#064e3b,font-weight:700
    classDef fallback fill:#fee2e2,stroke:#dc2626,color:#7f1d1d

Figure 37.1: Instrumentation and Observability Flow. Instrumentation flow from single execution through property writes to aggregate metric computation and health report.

Figure 3.5.3 Prompt Drift Monitoring

Figure 37.2 shows how prompt version correlation distinguishes model-driven drift from prompt regression.

%%{init:{"theme":"base","themeVariables":{"primaryColor":"#eef2ff","primaryTextColor":"#1e1b4b","primaryBorderColor":"#6366f1","lineColor":"#6366f1","clusterBkg":"#f8f9ff","clusterBorder":"#6366f1","titleColor":"#1e1b4b","edgeLabelBackground":"#f6f4ef","fontFamily":"system-ui,sans-serif","fontSize":"13px"}}}%%

flowchart LR
    subgraph CASE_A["CASE A Same prompt version"]
        A1["Baseline normal distribution"]:::process --> A2["Shift detected"]:::process
        A2 --> A3["MODEL DRIFT investigate provider"]:::process
    end
    subgraph CASE_B["CASE B New prompt version"]
        B1["Baseline normal distribution"]:::process --> B2["Prompt v1.1 deployed"]:::process
        B2 --> B3["Shift detected"]:::process --> B4["PROMPT REGRESSION revert or fix prompt"]:::process
    end
    PV["prompt_version property"]:::process -.->|"correlating variable"| CASE_A
    PV -.->|"correlating variable"| CASE_B
    classDef trigger  fill:#dcfce7,stroke:#16a34a,color:#14532d,font-weight:600
    classDef process  fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
    classDef decision fill:#fef9c3,stroke:#ca8a04,color:#78350f,font-weight:600
    classDef success  fill:#d1fae5,stroke:#059669,color:#064e3b,font-weight:700
    classDef fallback fill:#fee2e2,stroke:#dc2626,color:#7f1d1d

Figure 37.2: Drift vs. Regression Distinguishing Signal. The same distributional shift indicates model drift when the prompt is unchanged, and regression when a new prompt version was deployed.

Baseline path distribution (Period N-1, p6_prompt_version: v1.0): standard_review 60%, expedited_review 25%, decline 15%.

Period N path distribution (both cases share this shifted distribution): standard_review 42% (⬇️ -18%), expedited_review 38% (⬆️ +13%), decline 20% (⬆️ +5%).

Case	`p6_prompt_version` in Period N	Diagnosis	Action	Alert
A same prompt version	`v1.0` (unchanged)	Model drift AI provider updated the underlying model	Run fixture set to quantify; investigate provider release notes	⚠️ Warning (sustained) or 🚨 Critical
B new prompt version	`v1.1` (changed from v1.0)	Prompt regression candidate prompt change may have altered routing behavior	Run Chapter 3.7 fixture set against v1.0 and v1.1; if delta > tolerance, revert prompt	⚠️ Warning (investigate before next deployment)

Practical Exercise 3.5 Weekly Health Report Workflow

Business Scenario

Vantage Advisory Partners has been running their AI advisory pipeline (Practicals 6.1–6.4) for several weeks. Dozens of contacts have been processed and their results stored in p6_ properties. No individual execution has failed but no one can answer whether the system’s advisory outputs are consistent, whether the AI confidence is holding, or whether the rule fallback rate has changed since launch.

The Problem

The n8n execution log shows individual runs. It does not show whether confidence scores are drifting downward, whether more contacts are hitting the rule fallback path, or whether the advisory path distribution has shifted. Without aggregation across executions, population-level degradation is invisible.

The Architectural Solution

A weekly observability workflow queries all contacts processed in the reporting window, aggregates the five p6_ metrics, compares them to the Practical 3.0 Part B baseline, assigns alert status per metric, and posts a structured health report to Slack.

Step 1 Add `p6_prompt_version` to HubSpot

Purpose

Capture the active prompt version string alongside the four existing p6_ properties so the observability workflow can correlate distribution shifts with prompt changes distinguishing model drift from prompt regression.

Operation Summary

Property	Value
Property name	`p6_prompt_version`
HubSpot type	Single-line text
Property group	Phase 6 Observability (alongside other `p6_`)
Written by	Build Prompt Code node in Workflow A
Example value	`"v1.0"`

Implementation

In HubSpot Property Settings, add p6_prompt_version as a single-line text property in the same Phase 6 Observability group used by the other four p6_ properties. Then update every Build Prompt Code node in your Part III workflows (Workflow A and any variant workflows) to write the current prompt version string:

// In Route + Store add alongside existing p6_ writes
properties: {
  p6_last_execution_source: executionSource,
  p6_last_confidence_score: String(confidenceScore),
  p6_last_advisory_path:   advisoryPath,
  p6_agent_call_count:     String(agentCallCount),
  p6_prompt_version:       "v1.0"   // increment when prompt changes
}

The version string is a constant, not computed every execution under the same prompt writes the same version. Increment it manually whenever the Build Prompt node content changes.

Engineering Rationale

The prompt version is written per-execution so that contacts assessed under different prompt versions are queryable by version. If you update the prompt and observe a distribution shift in the observability workflow, p6_prompt_version tells you whether the shift is version-correlated. Without this property, model drift and prompt regression produce identical observability signals.

Step 2 Create the Observability Workflow

Purpose

Establish the workflow container and recurring schedule that drives the weekly health report execution.

Operation Summary

Property	Value
Workflow name	`VAP Observability Weekly Health Report`
Trigger type	Schedule Trigger
Schedule	Every Monday at 08:00 (local server time)
Primary function	Aggregate `p6_` metrics and post to Slack
Output	Structured health report in `#vap-ops`

Implementation

Create a new n8n workflow. Add a Schedule Trigger node configured for weekly execution on Monday at 08:00. This workflow does not connect to the advisory pipeline it runs independently and reads historical data from HubSpot.

Production Considerations

Production Risk

If the Schedule Trigger is misconfigured or the workflow is deactivated, health drift goes undetected silently. Add an error notification branch to the Schedule Trigger node so that if the workflow fails to complete, a distinct Slack message is sent to #vap-ops indicating the observability run did not complete. A failed observability workflow and a healthy system are indistinguishable without this guard.

Step 3 Query the Contact Population

Purpose

Retrieve the contacts processed during the reporting window the raw data for all five metric computations.

Operation Summary

Property	Value
Method	POST
Endpoint	`/crm/v3/objects/contacts/search`
Primary Function	Filter contacts with observability properties set
Output	Array of contact records with `p6_` properties

Request Payload

{
  "filterGroups": [
    {
      "filters": [
        {
          "propertyName": "p6_last_confidence_score",
          "operator": "HAS_PROPERTY"
        },
        {
          "propertyName": "lastmodifieddate",
          "operator": "GT",
          "value": "{{$now.minus({days: 7}).toMillis()}}"
        }
      ]
    }
  ],
  "properties": [
    "hs_object_id",
    "p6_last_execution_source",
    "p6_last_confidence_score",
    "p6_last_advisory_path",
    "p6_agent_call_count",
    "p6_prompt_version"
  ],
  "sorts": [
    { "propertyName": "lastmodifieddate", "direction": "DESCENDING" }
  ],
  "limit": 100
}

The HAS_PROPERTY filter on p6_last_confidence_score ensures only contacts that have been through the Part III pipeline are included contacts without observability data are excluded. The lastmodifieddate GT 7 days ago filter scopes the population to the current reporting window.

Response Processing

const results = $json.results;
const contacts = results.map(c => c.properties);
// contacts is now an array of property objects for metric computation

results contains the matching contact records. Each element has a properties object containing the five p6_ fields and hs_object_id. The .map(c => c.properties) extraction flattens the structure for the Compute Metrics node.

Output Table

Output	Description
`hs_object_id`	Contact ID (for per-contact trace queries)
`p6_last_execution_source`	`"ai"` or `"rule_fallback"` used for fallback rate
`p6_last_confidence_score`	String; parse to float for distribution computation
`p6_last_advisory_path`	Advisory path value used for path distribution
`p6_agent_call_count`	String; parse to int for agent count distribution
`p6_prompt_version`	Prompt version string used for drift correlation

Production Considerations

Critical Requirement

If the search returns zero results, verify that p6_last_confidence_score was written by earlier practicals. Check the Route + Store node in Workflow A for a HubSpot Update Contact call that includes this property. If results are returned but fewer than expected, the limit: 100 cap may be truncating the population implement pagination for systems with more than 100 contacts per week.

Step 4 Compute Metrics

Purpose

Aggregate the raw p6_ property values across all contacts in the reporting window into the five health metrics used for baseline comparison and alerting.

Operation Summary

Property	Value
Node type	Code node
Node name	`Compute Health Metrics`
Input	Array of contact property objects from Step 3
Output	Metrics object: five computed values with contact count

Implementation Logic

const contacts = $json.results.map(c => c.properties);
const N = contacts.length;

const confidenceScores = contacts
  .map(c => parseFloat(c.p6_last_confidence_score))
  .filter(v => !isNaN(v));

const meanConfidence = confidenceScores.reduce((a, b) => a + b, 0)
  / confidenceScores.length;

const fallbackRate = contacts.filter(
  c => c.p6_last_execution_source === "rule_fallback"
).length / N;

const pathCounts = contacts.reduce((acc, c) => {
  const path = c.p6_last_advisory_path || "unknown";
  acc[path] = (acc[path] || 0) + 1;
  return acc;
}, {});

const singleAgentRate = contacts.filter(
  c => parseInt(c.p6_agent_call_count) === 1
).length / N;

return { json: { N, meanConfidence, fallbackRate, pathCounts, singleAgentRate } };

Each computation maps directly to one of the five health metrics from 3.5.4 Health Metrics. meanConfidence feeds the confidence distribution metric. fallbackRate feeds the rule fallback rate metric. pathCounts feeds the advisory path distribution metric. singleAgentRate feeds the agent call count metric. Prompt version correlation is computed in the Compare to Baseline node using the raw contacts array alongside pathCounts.

Output Table

Output	Description
`N`	Total contacts in the reporting window
`meanConfidence`	Mean of `p6_last_confidence_score` across all contacts
`fallbackRate`	Proportion of contacts with `p6_last_execution_source = "rule_fallback"`
`pathCounts`	Object: count per `p6_last_advisory_path` value
`singleAgentRate`	Proportion of contacts with `p6_agent_call_count = 1`

Engineering Rationale

The !isNaN(v) filter on confidence scores is required because HubSpot stores numeric properties as strings. A contact whose p6_last_confidence_score was written as "N/A" (rule fallback contacts may not have a score) will produce NaN on parseFloat and corrupt the mean calculation if not filtered. Filter before computing, not after.

Step 5 Compare to Baseline

Purpose

Evaluate computed metrics against the Practical 3.0 Part B baseline values, assign alert status per metric, and produce the structured health report object that Step 6 will post to Slack.

Operation Summary

Property	Value
Node type	Code node
Node name	`Compare to Baseline`
Input	Metrics object from Step 4
Output	Health report object with per-metric status and deltas

Implementation Logic

// Baseline constants from Practical 3.0 Part B
const BASELINE = {
  meanConfidence:   0.74,
  fallbackRate:     0.17,
  pathDist: {
    standard_review:  0.60,
    expedited_review: 0.25,
    decline:          0.15
  },
  singleAgentRate: 0.00  // expected: all contacts multi-agent
};

const { N, meanConfidence, fallbackRate, pathCounts, singleAgentRate } = $json;

// Compute path proportions from counts
const pathProportions = {};
for (const [path, count] of Object.entries(pathCounts)) {
  pathProportions[path] = count / N;
}

// Assign status per metric
const confidenceDelta = meanConfidence - BASELINE.meanConfidence;
const confidenceStatus = confidenceDelta < -0.10 ? "🚨" : confidenceDelta < -0.05 ? "⚠️" : "✅";

const fallbackDelta = fallbackRate - BASELINE.fallbackRate;
const fallbackStatus = fallbackDelta > 0.20 ? "🚨" : fallbackDelta > 0.10 ? "⚠️" : "✅";

const pathDeltas = {};
let pathStatus = "✅";
for (const [path, baseProp] of Object.entries(BASELINE.pathDist)) {
  const delta = (pathProportions[path] || 0) - baseProp;
  pathDeltas[path] = delta;
  if (Math.abs(delta) > 0.20) pathStatus = "⚠️";
}

const agentStatus = singleAgentRate > 0.15 ? "🚨" : singleAgentRate > 0.05 ? "⚠️" : "✅";

return { json: {
  N, weekOf: new Date().toISOString().slice(0, 10),
  confidence:  { value: meanConfidence, delta: confidenceDelta, status: confidenceStatus },
  fallback:    { value: fallbackRate,   delta: fallbackDelta,   status: fallbackStatus },
  pathDeltas,  pathStatus,
  agentRate:   { value: singleAgentRate, status: agentStatus }
}};

The baseline constants must match the values recorded during Practical 3.0 Part B. If you did not record a baseline, use the first week of observability data as the baseline and update these constants before the second report is computed.

Output Table

Output	Description
`weekOf`	ISO date string identifying the report period
`confidence`	`{ value, delta, status }` confidence mean vs. baseline
`fallback`	`{ value, delta, status }` fallback rate vs. baseline
`pathDeltas`	Per-path delta from baseline proportion
`pathStatus`	Aggregate path distribution status symbol
`agentRate`	`{ value, status }` single-agent rate vs. 0% expected

Production Considerations

Design Practice

Store baseline constants as a named Code node (Baseline Constants) that feeds into the Compare node, rather than inlining them as literals. When the baseline is updated after a calibration review, only the Baseline Constants node needs to be modified the comparison logic remains unchanged. Mixing constants and logic in the same node makes both harder to maintain.

Step 6 Post the Health Report

Purpose

Deliver the computed health report to the #vap-ops Slack channel in a structured, scannable format that enables a team member to assess system health in under 30 seconds.

Operation Summary

Property	Value
Method	POST
Endpoint	Your Slack incoming webhook URL for `#vap-ops`
Primary Function	Deliver weekly health report to operations channel
Output	Slack message with per-metric status and deltas

Request Payload

{
  "text": "VAP AI HEALTH REPORT Week of {{$json.weekOf}}\nContacts assessed this week: {{$json.N}}\n─────────────────────────────────────────\n{{$json.confidence.status}} Mean confidence: {{$json.confidence.value.toFixed(2)}} (baseline: 0.74 | delta: {{$json.confidence.delta >= 0 ? '+' : ''}}{{$json.confidence.delta.toFixed(2)}})\n{{$json.fallback.status}} Fallback rate: {{($json.fallback.value * 100).toFixed(0)}}% (baseline: 17% | delta: {{$json.fallback.delta >= 0 ? '+' : ''}}{{($json.fallback.delta * 100).toFixed(0)}}%)\n{{$json.pathStatus}} Path distribution: see detail\n{{$json.agentRate.status}} Single-agent rate: {{($json.agentRate.value * 100).toFixed(0)}}% (expected: 0%)\n─────────────────────────────────────────\nAlert status: {{[$json.confidence.status, $json.fallback.status, $json.pathStatus, $json.agentRate.status].includes('🚨') ? '🚨 CRITICAL' : [$json.confidence.status, $json.fallback.status, $json.pathStatus, $json.agentRate.status].includes('⚠️') ? '⚠️ WARNING' : '✅ NOMINAL'}}"
}

The n8n expression syntax interpolates the health report values computed in Step 5. In the actual n8n HTTP Request node, construct the message body in a Set node first if the expression complexity causes parsing issues.

Engineering Rationale

For persistent time-series tracking, add a second output from this step that writes the health report values to a Notion database or Google Sheet row. Slack messages are not searchable or aggregable over time a row in a persistent store allows week-over-week trend analysis that Slack cannot provide. The Slack message handles the immediate notification; the persistent store handles longitudinal trending.

Step 7 Test

Purpose

Verify end-to-end workflow execution before the first scheduled Monday run, confirming that all five metrics are computed, baseline comparison produces correct status symbols, and the Slack message is delivered in the expected format.

Implementation

Trigger the workflow manually using the n8n Test Workflow button. Verify:

HubSpot Search returns at least one contact record (from Practicals 6.1–6.4)
Compute Metrics node output shows non-null values for all five fields check the execution output panel
Compare to Baseline node produces status symbols (✅/⚠️/🚨) for all four metric groups
Slack message appears in #vap-ops with data populated not placeholder expressions

If N = 0, the HubSpot Search filter is not matching processed contacts verify that Route + Store in Workflow A is writing p6_last_confidence_score to the contact. If the Slack message shows raw expression syntax (e.g., {$json.weekOf}), the expressions were not evaluated check that the HTTP Request node is in JSON body mode, not raw text mode.

Critical Requirement

If the HubSpot Search returns zero results, p6_last_confidence_score may not have been written by earlier practicals check that Parse Response writes the property in Route + Store. If results are returned but confidenceScores is empty, the property value may be a string in HubSpot; ensure parseFloat() and the !isNaN(v) filter are both in place.

Deliverable: Weekly observability workflow posting to Slack. All five metrics computed. Baseline comparison implemented. Alert status per metric. p6_prompt_version added to HubSpot and written by all Build Prompt nodes.

Estimated time: 3–4 hours.

Production Consideration

The observability workflow is itself a production component if it fails silently, health drift goes undetected. Add error handling to the HubSpot Search node: if the request fails or returns an HTTP error, route to a Slack notification distinct from the health report (e.g., #vap-ops: OBSERVABILITY WORKFLOW FAILED manual review required). Without this, a broken observability workflow is indistinguishable from a healthy system that produced no alerts.

For higher-volume systems where the contact population exceeds 100, implement pagination in the HubSpot Search node. The limit: 100 setting silently truncates the population and produces misleading metric computations if not paginated.

Discussion Questions

The observability workflow filters contacts by lastmodifieddate > 7 days ago. A contact processed 8 days ago and re-processed 6 days ago appears with the most recent values only. What information is lost by this approach, and how would you redesign the data model to retain the full history?
The rule fallback rate alert fires when the rate exceeds (baseline rate + 0.20). If the baseline rate is 0%, the alert fires at 20% fallback. If the baseline rate is 40%, the alert fires at 60%. Does an absolute delta threshold make sense across different baseline rates? What alternative threshold design would handle high-baseline-rate systems better?
Model drift and prompt regression both produce the same observability signal a distribution shift. What is the minimum metadata you would need to capture at execution time to distinguish one from the other reliably?

Chapter Summary

Observability is proactive measurement across the population of executions not reactive diagnosis of individual failures. The three pillars (logs, metrics, traces) are implemented through the p6_ properties as the logs layer, the weekly observability workflow as the metrics layer, and per-contact HubSpot property queries as the traces layer.

Key Principle

An execution log tells you what happened in one execution. An observability layer tells you whether the system’s behavior across all executions is within the expected range. AI systems require both and only observability detects silent population-level drift.

Five health metrics confidence score distribution, rule fallback rate, advisory path distribution, agent call count distribution, and prompt version correlation are computed weekly and compared to the baseline from Practical 3.0 Part B. Each has a defined alert condition and severity. A system with all five metrics within baseline range is behaving correctly. A system with one or more metrics out of range has a signal that requires investigation before it becomes a failure.

Prompt drift and prompt regression are distinguished by p6_prompt_version: a distribution shift with the same prompt version is model-driven; a shift with a new prompt version is a candidate regression requiring fixture-set comparison (Chapter 3.7).

Transition to Chapter 3.6

The observability layer in Chapter 3.5 measures what the AI system is doing. The governance layer in Chapter 3.6 controls what it is allowed to do and records every control decision.

Chapter 2.4 introduced the boolean manual_override_active flag as the governance mechanism. The boolean records a state but records nothing about why the override was set, by whom, for how long, or under which policy. At Part III scale, those missing fields are operational requirements, not documentation conveniences. Chapter 3.6 extends the boolean into a four-field governance record and adds the override lifecycle state machine that Part II’s binary flag cannot express.

Key Takeaways

Observability answers whether the system is behaving correctly across all executions. Diagnosis answers why a specific execution failed. Both are required; Part II provides diagnosis; Part III adds observability.
The three pillars logs, metrics, traces are implemented through p6_ properties (logs), the weekly observability workflow (metrics), and per-contact HubSpot queries (traces).
Five health metrics: confidence score distribution, rule fallback rate, advisory path distribution, agent call count distribution, and prompt version correlation. Each has a defined baseline source and alert condition.
p6_prompt_version distinguishes model-driven drift (same prompt version, distribution shifted) from prompt regression (new prompt version, distribution shifted).
Alert on sustained breaches for slow-signal metrics (two consecutive periods). Alert immediately on structural signals (unexpected single-agent rate increase).
The observability workflow is the analytics layer HubSpot does not provide natively. Add a persistent time-series store (Notion, Google Sheet) for higher-volume systems.

End of Chapter 3.5 AI Observability and Monitoring

Learning Objectives

3.5.1 From Diagnosis to Measurement

3.5.2 The AI Observability Model

3.5.3 Instrumentation Flow

3.5.4 Health Metrics

Confidence Score Distribution

Rule Fallback Rate

Advisory Path Distribution

Agent Call Count Distribution

Prompt Version Correlation and Drift Rate

3.5.5 Dashboards and Reporting

3.5.6 Alerting Design

Reference Diagrams

Figure 3.5.2 Instrumentation Flow

Figure 3.5.3 Prompt Drift Monitoring

Practical Exercise 3.5 Weekly Health Report Workflow

Business Scenario

The Problem

The Architectural Solution

Step 1 Add p6_prompt_version to HubSpot

Purpose

Operation Summary

Implementation

Engineering Rationale

Step 2 Create the Observability Workflow

Purpose

Operation Summary

Implementation

Production Considerations

Step 3 Query the Contact Population

Purpose

Operation Summary

Request Payload

Response Processing

Output Table

Production Considerations

Step 4 Compute Metrics

Purpose

Operation Summary

Implementation Logic

Output Table

Engineering Rationale

Step 5 Compare to Baseline

Purpose

Operation Summary

Implementation Logic

Output Table

Production Considerations

Step 6 Post the Health Report

Purpose

Operation Summary

Request Payload

Engineering Rationale

Step 7 Test

Purpose

Implementation

Production Consideration

Discussion Questions

Chapter Summary

Transition to Chapter 3.6

Key Takeaways

Step 1 Add `p6_prompt_version` to HubSpot