Chapter 3.4 Event-Driven AI Systems

In the Part II platform, you configured HubSpot workflow actions to send webhook events to n8n a lifecycle stage change in HubSpot triggered Workflow B. You have already used webhook events as a trigger mechanism. What you have not done is design for event-driven architecture: the discipline of deciding when event-driven triggering is more reliable than polling, how to handle events that arrive out of order or more than once, what to do when an event cannot be processed, and when to keep a polling fallback alongside event-driven triggers.

Part II configured webhook triggers. Chapter 3.4 teaches you to design event-driven systems.

Learning Objectives

After completing this chapter, you will be able to:

  • Explain the trade-offs between polling and event-driven triggering delay vs. loss/duplication and apply a decision framework to select the correct trigger pattern for a given workflow.
  • Design an event payload schema with the minimum fields required for reliable processing: object type, object ID, event type, timestamp, changed properties, and a correlation ID.
  • Implement event contract validation that catches payload schema violations, out-of-range field values, and missing required fields before workflow execution begins.
  • Apply the three event reliability patterns (idempotency key, dead-letter routing, polling fallback) to design an event-driven system that degrades gracefully under duplicate delivery and event loss.
  • Write a Mini-ADR documenting the decision to use event-driven triggering for a specific workflow, with rationale that addresses the failure mode trade-offs.
  • Troubleshoot an event-driven workflow where the same deal is being processed twice for a single property change event, by diagnosing the idempotency key logic.

3.4.1 Polling vs. Event-Driven Architecture

Polling vs. Event-Driven Architecture

The polling architecture and the event-driven architecture solve the same problem detecting when something has changed with opposite tradeoffs.

Property Polling Event-Driven
Detection latency N hours (poll interval) Near-zero (fires at state change)
Unnecessary executions Many (polls fire whether or not state changed) None (trigger fires only when event occurs)
Infrastructure dependency Schedule Trigger only; no external configuration Requires webhook registration and a publicly accessible n8n endpoint
Reliability model Poll either succeeds or fails; no event loss Events can be missed (network outage), duplicated (retry), or reordered
Idempotency requirement Low (polling re-reads current state) High (at-least-once delivery requires idempotent handlers)

Neither architecture is universally superior. The choice is a design decision based on latency requirements, event volume, acceptable miss rate, and operational complexity tolerance exactly the trade-off Mini-ADR 6.4-A captures.

The polling limit identified in 3.0.2 Structural Limits of Part II is a latency problem: a Qualified Prospect who needs a callback within two hours cannot wait for the next poll cycle. Event-driven triggering eliminates that latency. It introduces different challenges, covered in this chapter.

TipDesign Practice

The failure mode of polling is delay. The failure mode of event-driven is loss or duplication. Choosing between them means choosing which failure mode you design for. Both have remedies; neither has a free lunch.

3.4.2 Event Payload Design

Event Payload Design

A HubSpot webhook event payload is not a full contact record. It is a change notification: what changed, what the new value is, what the previous value was, when it changed, and on which object. The payload is small and targeted.

Example Contact property change event:

{
  "eventId": "evt_01J4K8X7N2",
  "subscriptionType": "contact.propertyChange",
  "portalId": 12345678,
  "objectId": 987654,
  "propertyName": "lifecyclestage",
  "propertyValue": "salesqualifiedlead",
  "previousPropertyValue": "lead",
  "occurredAt": 1720000000000,
  "attemptNumber": 0
}

The objectId is the HubSpot contact ID. The event does not include vap_combined_score, inquiry_text, or any other contact property only the changed property. To retrieve the full contact for processing, the event handler must call the HubSpot Contacts API using objectId.

Receive the event, then fetch the full contact do not attempt to process the event payload as a contact record.

This two-step pattern receive event → fetch full contact is structurally different from the Part II polling pattern, which reads the full contact record at poll time. The event handler normalizes the event payload to identify which contact changed. The pipeline from Chapter 3.2 then processes the full contact. A system that skips the fetch step will attempt to run the pipeline against a partial event payload and fail at the Normalize stage.

3.4.3 Event Contracts

Event Contracts

An event contract defines the minimum fields that a processable event must contain. Events that fail the contract are rejected before entering the processing pipeline the same discipline as the Normalize stage schema validation from Chapter 3.2, applied at the event boundary.

Minimum processable event contract:

Field Required Validation
eventId Yes Present and non-empty string
subscriptionType Yes Must match expected type(s)
objectId Yes Numeric and > 0
propertyName Yes Must match the subscribed property
propertyValue Yes Must match expected enum for property
occurredAt Yes Numeric timestamp; not in the future

Events that fail the contract are routed to the dead-letter store (see 3.4.5 Dead-Letter Queues) rather than rejected silently. A rejected event that is not logged cannot be replayed when the root cause is resolved.

CautionProduction Risk

Do not validate only the presence of fields validate their values. An event with "objectId": 0 passes a presence check but fails to identify a real contact. Validate objectId > 0 explicitly. An event with a propertyValue outside the expected enum should be routed to review, not processed with an unrecognized value.

3.4.4 Event Reliability

Idempotency

HubSpot delivers webhook events with at-least-once semantics: the same event may be delivered more than once. This happens when HubSpot does not receive a 200 response from n8n within its timeout window and retries delivery. The attemptNumber field increments with each retry.

Idempotency is the property that processing the same event multiple times produces the same result as processing it once. An event handler that is not idempotent will create duplicate records, send duplicate notifications, or apply duplicate state changes on retry. A system with an idempotency check can safely receive the same event twice. A system without one cannot.

The idempotency check is simple: before processing, read the contact’s current state from HubSpot. If the contact is already in the state the event implies it should be in, the event has already been processed exit without action.

// Idempotency check run immediately after event contract validation
const contact = await getHubSpotContact($json.objectId);
if (contact.lifecyclestage === $json.propertyValue) {
  // Already processed exit cleanly
  return { json: { skipped: true, reason: "already_processed", eventId: $json.eventId } };
}
// Proceed with processing
NoteEngineering Rationale

Place the idempotency check as the first node after event contract validation before any HubSpot writes, before any AI calls, and before any Slack notifications. An idempotency check that runs after a partial processing sequence does not prevent duplicates; it only prevents the part of the sequence that runs after the check.

3.4.5 Dead-Letter Queues

Dead-Letter Queue

A dead-letter queue (DLQ) holds events that cannot be processed after exhausting retry attempts. Without a DLQ, failed events are lost they cannot be replayed when the root cause is resolved.

In n8n, a DLQ can be implemented as a HubSpot note on the affected contact, a row in a Notion database, or a Slack message to a dedicated #dlq-events channel. The implementation is less important than the discipline: every failed event must be logged with enough information to replay it.

Minimum DLQ record:

Field Value
event_id From the original event payload
event_payload Full event payload (JSON string)
failure_reason The error message or validation failure description
attempt_count Number of processing attempts
first_attempted_at Timestamp of first delivery
last_attempted_at Timestamp of most recent failure
status pending_review

A DLQ is not a monitoring tool it is a recovery tool. The team must review it regularly and replay events when the failure condition is resolved.

CautionProduction Risk

Do not use the n8n execution log as your DLQ. The execution log shows that an execution failed; it does not retain the event payload in a form that supports replay. The DLQ must store the original event payload in a separate, queryable location.

3.4.6 Replay Patterns

Replaying a dead-letter event means resubmitting the original event payload to the event handler. The handler must process a replayed event identically to the original delivery.

Idempotency covers replay. The idempotency check from 3.4.4 Event Reliability handles replay automatically: if the event was partially processed before failing (for example, the HubSpot Contact was fetched but the AI call failed), the replay triggers a fresh fetch and AI call there are no duplicate side effects from the first partial execution because the HubSpot Contact write had not yet occurred.

The replay workflow: 1. Retrieve the event payload from the DLQ record 2. Submit the payload to the n8n webhook URL that handles live events (same handler, same URL) 3. The handler processes the replay identically to a live event 4. On success, update the DLQ record status to replayed 5. On failure, increment attempt_count and update last_attempted_at

What replay cannot recover: If the contact’s state has changed significantly since the event was first received, replaying the original event may produce a meaningfully different advisory output than it would have at the time of original delivery. Log the replay_lag (time between first_attempted_at and replay) in the DLQ record so the operations team can assess whether the replay result is still valid.

3.4.7 Hybrid Event Systems

Hybrid Event Architecture

Pure event-driven triggering relies on every relevant state change generating a correctly delivered webhook event. Events can be missed: n8n downtime, network interruptions, HubSpot service degradation, or webhook subscription gaps all produce undelivered events that the DLQ cannot capture (because the event was never received).

The hybrid architecture adds a low-frequency polling sweep daily or weekly, not hourly that catches contacts whose events were never delivered. The polling sweep is the safety net; the event-driven trigger is the primary path.

The hybrid architecture has three properties:

  1. Single downstream pipeline. Both the event-driven path and the polling sweep route to the same five-stage pipeline from Chapter 3.2. The idempotency check at the pipeline entry handles contacts that arrive from both paths within the same processing window.

  2. Different latency, same outcome. A contact processed via the event-driven path receives its advisory output within seconds of the state change. A contact processed via the polling sweep receives the same advisory output days later. The outcome is identical; the latency is not.

  3. Polling sweep scope is narrow. The polling sweep does not re-process all contacts. It processes only contacts where the expected follow-up action has not yet been logged. Filter the polling query on the absence of the audit property that the event-driven path writes on success.

TipDesign Practice

The hybrid architecture is not a compromise it is the production-correct design for event-driven systems that cannot tolerate silent event loss. Pure event-driven triggering is appropriate when event loss has a low business consequence. For the Meridian Venture Partners capstone domain where a missed deal event costs a partner relationship the hybrid architecture is the safer default.

Reference Diagrams

Figure 3.4.1 Polling vs. Event-Driven Architecture

Figure 36.1 shows the structural difference between the polling and event-driven trigger architectures. Detection latency is annotated on the polling path; event delivery reliability challenges are annotated on the event-driven path.

%%{init:{"theme":"base","themeVariables":{"primaryColor":"#eef2ff","primaryTextColor":"#1e1b4b","primaryBorderColor":"#6366f1","lineColor":"#6366f1","clusterBkg":"#f8f9ff","clusterBorder":"#6366f1","titleColor":"#1e1b4b","edgeLabelBackground":"#f6f4ef","fontFamily":"system-ui,sans-serif","fontSize":"13px"}}}%%

flowchart LR
    subgraph POLL["Polling Part II (latency = poll interval)"]
        direction TB
        P1["Cron Schedule"]:::process --> P2["HubSpot API call"]:::process
        P2 --> P3{"Changes?"}:::decision
        P3 -->|"yes"| P4["Process contact"]:::process
        P4 --> P5["Wait interval"]:::process
        P5 --> P1
        P3 -->|"no"| P5
    end

    subgraph EVENT["Event-Driven Part III (near-zero latency)"]
        direction TB
        E1["HubSpot Webhook"]:::trigger --> E2["n8n receives event"]:::process
        E2 --> E3["Immediate processing"]:::process
        E3 --> E4["Done"]:::process
        E2 -.->|"reliability challenges"| E5(["duplicate / missed events"]):::process
    end
    classDef trigger  fill:#dcfce7,stroke:#16a34a,color:#14532d,font-weight:600
    classDef process  fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
    classDef decision fill:#fef9c3,stroke:#ca8a04,color:#78350f,font-weight:600
    classDef success  fill:#d1fae5,stroke:#059669,color:#064e3b,font-weight:700
    classDef fallback fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
Figure 36.1: Polling trades detection latency for reliability; event-driven eliminates latency but requires idempotency and DLQ to handle duplicate or missed delivery.

Figure 3.4.2 Event-Driven AI System Architecture

Figure 36.2 shows the event handler happy path: event contract validation → idempotency check → contact fetch → Chapter 3.2 pipeline → p6_ property writes. Each stage routes failures to the dead-letter queue; the full DLQ lifecycle is detailed in Figure 36.3.

%%{init:{"theme":"base","themeVariables":{"primaryColor":"#eef2ff","primaryTextColor":"#1e1b4b","primaryBorderColor":"#6366f1","lineColor":"#6366f1","clusterBkg":"#f8f9ff","clusterBorder":"#6366f1","titleColor":"#1e1b4b","edgeLabelBackground":"#f6f4ef","fontFamily":"system-ui,sans-serif","fontSize":"13px"}}}%%

flowchart TD
    W["HubSpot Webhook n8n Webhook Trigger"]:::trigger --> V["Contract Validation schema + HMAC check"]:::process
    V --> I["Idempotency Check deduplicate by event_id"]:::process
    I --> F["Contact Fetch HubSpot Contacts API"]:::process
    F --> P["Five-Stage AI Pipeline Validate → Enrich → Assess → Score → Output"]:::process
    P --> O["Output"]:::success
    N["Each stage routes failures to DLQ — see Figure 3.4.3"]:::fallback
    classDef trigger  fill:#dcfce7,stroke:#16a34a,color:#14532d,font-weight:600
    classDef process  fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
    classDef decision fill:#fef9c3,stroke:#ca8a04,color:#78350f,font-weight:600
    classDef success  fill:#d1fae5,stroke:#059669,color:#064e3b,font-weight:700
    classDef fallback fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
Figure 36.2: Event-driven pipeline from webhook delivery through five processing stages to AI assessment output.

Figure 3.4.3 Replay and Dead-Letter Pattern

Figure 36.3 shows the lifecycle of a failed event: from initial delivery failure through DLQ storage to manual replay and outcome logging.

%%{init:{"theme":"base","themeVariables":{"primaryColor":"#eef2ff","primaryTextColor":"#1e1b4b","primaryBorderColor":"#6366f1","lineColor":"#6366f1","clusterBkg":"#f8f9ff","clusterBorder":"#6366f1","titleColor":"#1e1b4b","edgeLabelBackground":"#f6f4ef","fontFamily":"system-ui,sans-serif","fontSize":"13px"}}}%%

flowchart TD
    SF["Stage Failure"]:::process --> DLQ[("Write to DLQ event_id, payload, error timestamp, attempt_count")]
    DLQ --> OPS["Operations reviews DLQ records"]:::process
    OPS --> RP["Manual Replay Workflow Read DLQ → Resubmit payload to live handler"]:::fallback
    RP -->|"Success"| RS["Update DLQ: status=REPLAYED replayed_at"]:::fallback
    RP -->|"Failure"| RF["Update DLQ: status=FAILED replay_error"]:::fallback
    classDef trigger  fill:#dcfce7,stroke:#16a34a,color:#14532d,font-weight:600
    classDef process  fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
    classDef decision fill:#fef9c3,stroke:#ca8a04,color:#78350f,font-weight:600
    classDef success  fill:#d1fae5,stroke:#059669,color:#064e3b,font-weight:700
    classDef fallback fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
Figure 36.3: Failed events are stored in the DLQ with full payload; a manual replay workflow resubmits to the live handler and logs the outcome.

Mini-ADR 6.4-A When Should Polling Remain?

Required format:

Context: [The reliability requirement and business consequence that makes this decision necessary for the specific domain]

Decision: [Pure event-driven or hybrid with polling fallback]

Rationale: [Why? Name the trade-off explicitly: what does the chosen approach give up, and why is that acceptable given the domain context?]

Your Mini-ADR 6.4-A task: Write a one-paragraph ADR for the Meridian Venture Partners capstone. The Meridian brief describes 200–300 deal submissions per quarter, with 14 missed deals attributed to slow initial screening.

Consider: What is the business consequence of a missed deal event in a VC firm context not a missed follow-up, but a submission that arrives and is never screened? Does that consequence level justify the added operational complexity of the hybrid architecture? If a deal submission event is lost and not caught by a polling sweep for 24 hours, what is the likely outcome?

This ADR is the direct precursor to Capstone ADR-B (Event-Driven vs. Hybrid Architecture).

Practical Exercise 3.4 Event Handler with Mock Trigger

Business Scenario

A Vantage Advisory Partners Qualified Prospect received a competing proposal on a Tuesday morning. The follow-up workflow was scheduled to run at noon. By the time it fired, the prospect had already signed with a competitor. The polling architecture performed exactly as designed and the business consequence was a lost engagement.

The Problem

Workflow C polls HubSpot every N hours to identify follow-up candidates. Between polls, lifecycle stage changes are invisible to the workflow. Detection latency is a structural property of the polling architecture. No configuration change eliminates it.

The Architectural Solution

Replace the Schedule Trigger with an event-driven handler. HubSpot fires a webhook event when a contact’s lifecyclestage changes to salesqualifiedlead. The n8n event handler validates the event contract, applies an idempotency check, fetches the full contact, and routes it to the Chapter 3.2 pipeline for processing. A DLQ captures any events the handler cannot process.

Mock event payload

The practical simulates a webhook event by sending a manually constructed JSON payload to a new n8n Webhook node. Use this payload structure:

{
  "eventId": "evt_test_001",
  "subscriptionType": "contact.propertyChange",
  "portalId": 12345678,
  "objectId": 987654,
  "propertyName": "lifecyclestage",
  "propertyValue": "salesqualifiedlead",
  "previousPropertyValue": "lead",
  "occurredAt": 1720000000000,
  "attemptNumber": 0
}

Replace objectId with the HubSpot ID of one of your Vantage test contacts.

Step 1 Build the Event Contract Validation Node

Purpose

Every event-driven system receives events it cannot process: malformed payloads, missing required fields, invalid property values, and events from unexpected subscription types. Without a contract validation step, these events enter the pipeline and fail at unpredictable points a missing objectId will cause the contact fetch in Step 3 to error; an invalid propertyValue will cause the Normalize stage to produce an incomplete canonical schema. The Event Contract Validation node enforces a defined boundary: events that do not meet the minimum processable contract are rejected before they touch the pipeline, logged to the DLQ, and made recoverable. Events that pass are guaranteed to have the fields the downstream steps require.

Operation Summary

Property Value
Node Type Code (JavaScript)
Primary Function Validate all six required event fields before pipeline entry
Input Raw webhook event payload from Webhook trigger
Output (valid) { contract_valid: true } + original event fields
Output (invalid) { contract_valid: false, failure_reason: "..." }

Implementation Logic

// Event Contract Validation Code node
const event = $input.first().json;
const failures = [];

// Field 1: eventId present and non-empty
if (!event.eventId || typeof event.eventId !== 'string' || event.eventId.trim() === '') {
  failures.push('eventId: missing or empty');
}

// Field 2: subscriptionType must match expected type
const allowedTypes = ['contact.propertyChange'];
if (!event.subscriptionType || !allowedTypes.includes(event.subscriptionType)) {
  failures.push(`subscriptionType: "${event.subscriptionType}" not in allowed set`);
}

// Field 3: objectId numeric and > 0
if (!event.objectId || typeof event.objectId !== 'number' || event.objectId <= 0) {
  failures.push(`objectId: must be numeric and > 0 (received: ${event.objectId})`);
}

// Field 4: propertyName must match subscribed property
const allowedProperties = ['lifecyclestage'];
if (!event.propertyName || !allowedProperties.includes(event.propertyName)) {
  failures.push(`propertyName: "${event.propertyName}" not in subscribed properties`);
}

// Field 5: propertyValue must match expected enum
const allowedValues = ['salesqualifiedlead', 'opportunity', 'customer'];
if (!event.propertyValue || !allowedValues.includes(event.propertyValue)) {
  failures.push(`propertyValue: "${event.propertyValue}" not in expected enum`);
}

// Field 6: occurredAt numeric timestamp, not in the future
const now = Date.now();
if (!event.occurredAt || typeof event.occurredAt !== 'number' || event.occurredAt > now) {
  failures.push(`occurredAt: must be numeric and not in the future (received: ${event.occurredAt})`);
}

if (failures.length > 0) {
  return [{
    json: {
      contract_valid: false,
      failure_reason: failures.join('; '),
      original_event: event,
      validated_at: new Date().toISOString()
    }
  }];
}

return [{
  json: {
    contract_valid: true,
    ...event,
    validated_at: new Date().toISOString()
  }
}];

Failures are accumulated as an array and joined for the DLQ record this way, a single invalid event produces one DLQ entry listing all contract violations rather than failing on the first and hiding the rest. The original_event is preserved in the invalid output for DLQ storage so the full payload is available for replay after the root cause is resolved.

Validation Field Table

Field Validation Rule
eventId Present, non-empty string
subscriptionType Matches contact.propertyChange
objectId Numeric, greater than 0
propertyName Matches lifecyclestage (or configured subscribed property)
propertyValue Matches expected lifecycle stage enum values
occurredAt Numeric timestamp; not in the future

Output Table

Output Description
contract_valid Boolean true if all six fields pass validation
failure_reason String listing all failures (only when contract_valid: false)
original_event Full event payload (only when contract_valid: false)
validated_at ISO8601 timestamp of validation execution

Engineering Rationale

NoteEngineering Rationale

Event contract validation is the event-driven equivalent of the Normalize Validation Gate from Practical 3.2. Both enforce schema contracts at a pipeline entry boundary. The difference is the domain: the Normalize gate validates a contact record’s internal fields; the Event Contract Validation node validates an event notification’s structural properties. Both produce the same outcome on failure: a logged, recoverable error that does not propagate into the downstream pipeline.

Step 2 Build the Idempotency Check

Purpose

HubSpot delivers webhook events with at-least-once semantics: the same event may arrive more than once if n8n does not return a 200 response within HubSpot’s timeout window and HubSpot retries. Without an idempotency check, a retried event processes the same contact twice producing a duplicate HubSpot property write, a duplicate advisory assessment, and potentially a duplicate Slack notification. The idempotency check reads the contact’s current state from HubSpot before any processing and exits cleanly if the state the event implies has already been applied. A system with this check can safely receive the same event any number of times.

Operation Summary

Property Value
Node Type HTTP Request (read contact state) + IF node (compare states)
Method GET
Endpoint /crm/v3/objects/contacts/{objectId}?properties=lifecyclestage
Primary Function Compare current contact state against event-implied state
Output (match) { skipped: true, reason: "already_processed" } workflow exits
Output (no match) Proceeds to Step 3 Fetch Full Contact

HTTP Request Configuration (Read Current State)

Method: GET
URL: https://api.hubapi.com/crm/v3/objects/contacts/{{ $json.objectId }}
Query parameter: properties=lifecyclestage
Authorization: Bearer <HubSpot Private App Token>

This call reads only the lifecyclestage property the minimum required for the idempotency check. It does not fetch the full contact record. Step 3 fetches the full record separately, after idempotency is confirmed, with the complete property list the pipeline requires.

IF Node Configuration

// IF node condition True branch = already processed (skip); False branch = proceed
{{ $json.properties.lifecyclestage === $('Event Contract Validation').first().json.propertyValue }}

If the contact’s current lifecyclestage matches the event’s propertyValue, the state change has already been applied either by a prior delivery of this event or by a polling sweep. The True branch returns:

return [{
  json: {
    skipped: true,
    reason: "already_processed",
    eventId: $('Event Contract Validation').first().json.eventId,
    contact_id: $('Event Contract Validation').first().json.objectId,
    skipped_at: new Date().toISOString()
  }
}];

The workflow then exits. The False branch proceeds to Step 3.

Output Table

Output (True skip) Description
skipped Boolean true
reason "already_processed"
eventId Original event ID for audit log
skipped_at ISO8601 timestamp
Output (False proceed) Description
Current contact state Passed through; contact has not yet applied this state

Engineering Rationale

NoteEngineering Rationale

The idempotency check must run before any writes including the p6_ property writes and any Slack notifications. An idempotency check that runs after a partial processing sequence does not prevent duplicates for the steps that ran before it. Positioning this check immediately after Event Contract Validation and before all other steps ensures the duplicate detection covers the full workflow. A clean exit on skipped: true is not an error it is the correct and expected behavior when HubSpot retries delivery.

Step 3 Fetch the Full Contact Record

Purpose

HubSpot webhook events are change notifications, not contact records. The event payload contains only the changed property, the previous value, and the object ID. The five-stage pipeline from Practical 3.2 requires a canonical contact schema with inquiry_text, engagement_type, company_name, and other fields that the event payload does not carry. This step fetches the full contact record using objectId and retrieves all properties the pipeline’s Ingest and Normalize stages require. Without this step, the pipeline would attempt to normalize an event payload as a contact record and fail at the Normalize stage when required fields are absent.

Operation Summary

Property Value
Node Type HTTP Request
Method GET
Endpoint /crm/v3/objects/contacts/{objectId}
Primary Function Fetch full contact record with all pipeline-required properties
Input objectId from the validated event payload
Output Full contact record in HubSpot API format

Request Payload

Method: GET
URL: https://api.hubapi.com/crm/v3/objects/contacts/{{ $('Event Contract Validation').first().json.objectId }}
Query parameter: properties=firstname,lastname,email,lifecyclestage,inquiry_text,engagement_type,company_name,source,vap_combined_score,vap_advisory_path,vap_ai_confidence,vap_priority_label,vap_score_timestamp
Authorization: Bearer <HubSpot Private App Token>

The properties query parameter must explicitly list every property the pipeline requires. HubSpot does not return all properties by default only those listed in the request. Include all canonical schema fields from the Normalize stage (inquiry_text, engagement_type, company_name, source) and all peer context fields the Retrieve node in Practical 3.1 depends on (vap_combined_score, vap_advisory_path, vap_ai_confidence, vap_priority_label, vap_score_timestamp).

Request Field Table

Property Requested Pipeline Stage That Requires It
inquiry_text Normalize canonical schema required field
engagement_type Normalize canonical schema required field
company_name Normalize canonical schema required field; Enrich
source Normalize canonical schema required field
lifecyclestage Route + Store confirmation of state change
vap_combined_score Peer context Retrieve node (Practical 3.1)
vap_advisory_path Peer context Retrieve node
vap_ai_confidence Peer context Retrieve node
vap_priority_label Peer context Retrieve node
vap_score_timestamp Peer context sort field

Response Processing

// The HubSpot GET response structure
const contact = $json;
// Properties are nested under contact.properties
// contact.id is the HubSpot contact ID

// Pass to Normalize stage in the format it expects
return [{
  json: {
    contact_id: contact.id,
    inquiry_text: contact.properties.inquiry_text,
    engagement_type: contact.properties.engagement_type,
    company_name: contact.properties.company_name,
    source: contact.properties.source,
    lifecyclestage: contact.properties.lifecyclestage,
    // Peer context properties passed through for the Retrieve stage
    vap_combined_score: contact.properties.vap_combined_score,
    vap_advisory_path: contact.properties.vap_advisory_path,
    // ... etc.
    fetched_at: new Date().toISOString()
  }
}];

Output Table

Output Description
contact_id HubSpot contact ID (from $json.id)
Canonical fields All fields required by the Normalize stage canonical schema
Peer context fields All fields required by the Retrieve node from Practical 3.1
fetched_at ISO8601 timestamp of fetch execution

Engineering Rationale

NoteEngineering Rationale

Two separate HubSpot API calls are made in Steps 2 and 3: one to read only lifecyclestage for the idempotency check, and one to fetch the full contact record. This separation is intentional. The idempotency check is a lightweight guard that should exit cleanly if the event is a duplicate fetching the full record on every delivery, including duplicates, wastes API quota and adds latency to the skip path. Separating the calls means the idempotency check costs one API call when it exits, and two total when it proceeds. Combining them would always cost one full-fetch call even on the duplicate path.

Step 4 Connect to the Chapter 3.2 Pipeline

Purpose

The event-driven handler and the polling-based workflow from Practical 3.2 share the same downstream processing pipeline. This architectural choice a single pipeline serving multiple trigger paths ensures that a contact processed via the event-driven handler and a contact processed via the polling sweep receive identical advisory outputs, identical property writes, and identical Slack notifications. The idempotency check in Step 2 prevents a contact from being processed twice even if both trigger paths fire for the same contact within the same processing window. The pipeline is the single source of truth for what processing means; the trigger path is irrelevant to the outcome.

Operation Summary

Property Value
Node Type Connection (no new node)
Primary Function Route fetched contact record to [NORMALIZE] stage of Practical 3.2 pipeline
Input Full contact record from Step 3
Output All pipeline stages execute identically to Practical 3.2

Configuration

Connect the Step 3 HTTP Request output to the [NORMALIZE] Canonical Schema Code node from Practical 3.2. The Normalize stage receives the contact record in the same structure it would receive from a polling-triggered webhook the fetch in Step 3 was designed to produce exactly that structure.

Confirm the connection is to [NORMALIZE] and not to [INGEST]. The Ingest stage in Practical 3.2 is the Webhook trigger node itself it does not exist in the event handler workflow because the event handler has its own Webhook trigger. The pipeline begins at [NORMALIZE] for the event-driven path.

Output Table

Output Description
Pipeline execution All five stages execute in sequence from [NORMALIZE] onward
p6_ properties Written in [ROUTE + STORE] as configured in Practical 3.2

Engineering Rationale

NoteEngineering Rationale

A single shared pipeline serving multiple trigger paths is the correct design for systems where processing consistency matters more than trigger diversity. If the event-driven path and the polling path had separate pipeline implementations, a change to the advisory prompt in the event-driven pipeline would not apply to contacts caught by the polling sweep producing different advisory outputs for the same contact depending on which trigger fired first. A single pipeline eliminates this divergence by design.

Step 5 Build the DLQ Logging Node

Purpose

Every failed event whether at contract validation, idempotency check, contact fetch, or anywhere in the five-stage pipeline must be stored with enough information to replay it after the failure condition is resolved. Without a DLQ, failed events are lost silently: the contact is never processed, no advisor is alerted, and there is no record that the event was ever received. The DLQ is a recovery tool, not a monitoring tool its purpose is to make failed events replayable, not to serve as a dashboard. Every failure path in this workflow converges on the DLQ Logging node.

Operation Summary

Property Value
Node Type Code (JavaScript) + HTTP Request (Slack notification)
Primary Function Log failed event payload and failure metadata to DLQ store
Input Failed event payload + failure reason from any On Error path
Output Slack message to #vap-ops with DLQ record fields

Implementation Logic

// DLQ Log Failed Event Code node
const failedPayload = $input.first().json;

const dlqRecord = {
  event_id: failedPayload.eventId ?? failedPayload.original_event?.eventId ?? 'unknown',
  event_payload: JSON.stringify(failedPayload.original_event ?? failedPayload),
  failure_reason: failedPayload.failure_reason ?? $execution.error?.message ?? 'Unknown failure',
  attempt_count: (failedPayload.attemptNumber ?? 0) + 1,
  first_attempted_at: failedPayload.occurredAt ?? failedPayload.original_event?.occurredAt ?? Date.now(),
  last_attempted_at: Date.now(),
  status: 'pending_review'
};

// Format for Slack message
const slackMessage = `
*DLQ Event Pending Review*
Event ID: \`${dlqRecord.event_id}\`
Failure: ${dlqRecord.failure_reason}
Attempt: ${dlqRecord.attempt_count}
Occurred at: ${new Date(dlqRecord.first_attempted_at).toISOString()}
Payload: \`\`\`${dlqRecord.event_payload.substring(0, 500)}...\`\`\`
`.trim();

return [{
  json: {
    dlqRecord,
    slackMessage,
    logged_at: new Date().toISOString()
  }
}];

After this Code node, insert an HTTP Request node that posts slackMessage to #vap-ops via the Slack API. The Slack message is the practical’s DLQ store sufficient for testing and for Chapter 3.4 validation. The Production Consideration at the end of this practical addresses the migration to a queryable store.

DLQ Record Field Table

Field Required Description
event_id Yes Original eventId from the event payload
event_payload Yes Full event payload as JSON string required for replay
failure_reason Yes Human-readable description of the failure
attempt_count Yes Number of processing attempts (starts at 1)
first_attempted_at Yes Timestamp of the original event delivery
last_attempted_at Yes Timestamp of the most recent failure
status Yes "pending_review" on initial logging

Output Table

Output Description
dlqRecord Complete DLQ record object with all seven required fields
slackMessage Formatted Slack message string for #vap-ops channel
logged_at ISO8601 timestamp of DLQ logging execution

Engineering Rationale

NoteEngineering Rationale

The DLQ Logging node connects to every On Error path in the event handler contract validation failure, idempotency check error, contact fetch failure, and any pipeline stage failure. This single convergence point ensures consistent DLQ record structure regardless of where in the handler the failure occurred. A system with different DLQ handlers at each failure point produces inconsistent record schemas that are difficult to query and replay systematically. A single DLQ node with consistent output is a queryable, replayable store even when the underlying storage is Slack.

Step 6 Test Idempotency

Purpose

Idempotency testing is the only way to confirm that the check works before it matters in production. At-least-once delivery is a guarantee that the same event will arrive more than once under retry conditions conditions that are difficult to simulate in production but easy to simulate in testing. Submitting the same mock payload twice in quick succession confirms that the idempotency check catches the second delivery and exits cleanly, that the HubSpot contact is updated exactly once, and that exactly one Slack notification fires. A system whose idempotency check has never been tested cannot be trusted to behave correctly when HubSpot retries a delivery.

Operation Summary

Property Value
Primary Function Submit identical mock payload twice; verify second is skipped
Input Same mock event payload submitted twice
Output (first) Full pipeline executes; contact updated; Slack fires
Output (second) Idempotency check exits with skipped: true; no writes

Test Configuration

Submit the mock payload from the workflow setup section twice using an HTTP POST to the n8n Webhook URL:

# First submission should process fully
curl -X POST https://<your-n8n-url>/webhook/<webhook-path> \
  -H "Content-Type: application/json" \
  -d '{
    "eventId": "evt_test_001",
    "subscriptionType": "contact.propertyChange",
    "portalId": 12345678,
    "objectId": 987654,
    "propertyName": "lifecyclestage",
    "propertyValue": "salesqualifiedlead",
    "previousPropertyValue": "lead",
    "occurredAt": 1720000000000,
    "attemptNumber": 0
  }'

# Second submission same payload should skip
curl -X POST https://<your-n8n-url>/webhook/<webhook-path> \
  -H "Content-Type: application/json" \
  -d '{
    "eventId": "evt_test_001",
    "subscriptionType": "contact.propertyChange",
    ...
    "attemptNumber": 1
  }'

After both submissions, verify in HubSpot that: - lifecyclestage updated exactly once - p6_last_execution_source was written exactly once - Exactly one Slack notification was sent to #vap-ops - The second n8n execution shows skipped: true in its output

Implementation Logic

// Confirm second execution exits at idempotency check
// Expected output for second submission:
{
  "skipped": true,
  "reason": "already_processed",
  "eventId": "evt_test_001",
  "contact_id": 987654,
  "skipped_at": "<ISO8601 timestamp>"
}

Output Table (Verification Checklist)

Verification Expected Result
HubSpot lifecyclestage write count Exactly 1
HubSpot p6_last_execution_source writes Exactly 1
Slack #vap-ops notifications Exactly 1
Second n8n execution skipped true
Second n8n execution reason "already_processed"

Engineering Rationale

ImportantCritical Requirement

If the idempotency check always passes (never skips), verify that the HubSpot API call in Step 2 is reading lifecyclestage and not a custom property. The HubSpot Contacts API returns standard properties only if they are explicitly listed in the properties request parameter. Check the GET request URL: if ?properties=lifecyclestage is not present, the response will not include the lifecycle stage field and the IF node comparison will always evaluate against undefined which will never match the event’s propertyValue.

Production Consideration

This practical uses a Slack message as the DLQ store. In production, a Slack-based DLQ is not queryable and cannot support the replay workflow from 3.4.6 Replay Patterns. Before moving to the Meridian capstone, plan to migrate the DLQ store to a Notion database or a HubSpot Note with a consistent schema. Chapter 3.5 (Production Observability) introduces aggregate health metrics DLQ depth (the count of pending_review records) is a key health signal that requires a queryable store, not a Slack channel.

Deliverable: Event handler workflow with mock payload trigger, event contract validation, idempotency check, contact fetch, Chapter 3.2 pipeline connection, and DLQ logging on failure. Idempotency tested with duplicate payload submission. Mini-ADR 6.4-A written.

Estimated time: 2–3 hours.

Discussion Questions

  1. The idempotency check in 3.4.4 Event Reliability reads the contact’s current state from HubSpot before processing. If two identical events arrive within 200ms of each other and both pass the idempotency check before either has written its results, what failure mode occurs? How would you prevent it?

  2. The DLQ in Practical 3.4 uses a Slack message. In a system processing 200 events per day, how would you redesign the DLQ storage to make it queryable and to support the replay workflow in 3.4.6 Replay Patterns?

  3. The hybrid architecture uses a polling sweep to catch contacts whose events were missed. How would you design the polling query so that it does not re-process contacts that were already handled by the event-driven path? Which HubSpot property would you filter on, and what value indicates successful event-driven processing?

Chapter Summary

Event-driven architecture eliminates detection latency by firing when state changes occur rather than checking whether they occurred. The tradeoff: events can arrive more than once, out of order, or not at all. Three patterns make event-driven AI systems reliable in production: event contracts (reject invalid events before they reach the pipeline), idempotency (process the same event multiple times safely), and dead-letter queues (capture and recover from failed events rather than losing them).

Replay is not automatic it is a deliberate operation performed after the failure condition is resolved. Idempotency covers replay: a replayed event passes through the same idempotency check as a live event.

The hybrid architecture event-driven primary path with low-frequency polling fallback is the production-correct design for domains where event loss has high business consequence.

Transition to Chapter 3.5

In Chapter 2.9, you learned to read the n8n execution log to diagnose specific failures a reactive skill applied when something breaks. Observability is a proactive discipline: designing the system so you can measure whether it is behaving correctly across thousands of executions before something breaks. The execution log tells you what happened in one execution. An observability layer tells you whether the system’s behavior across all executions is within the expected range and alerts you when it begins to drift. Chapter 3.5 starts where Chapter 2.9 ends.

The four p6_ bootstrap properties configured in 3.0.6 Observability Bootstrap have now been written across multiple practicals. Chapter 3.5 formalizes the observability architecture built on top of them.

Key Takeaways

  1. Polling and event-driven triggering are both valid architectures with opposite tradeoffs: polling has detection latency; event-driven has delivery reliability challenges. The choice is a design decision, not a quality judgment.
  2. HubSpot webhook events deliver the changed property and the previous value not the full contact record. The event handler must fetch the full contact using objectId.
  3. Event contracts validate event payloads before they enter the processing pipeline. Route contract-invalid events to the DLQ, not to silent discard.
  4. At-least-once delivery requires idempotent event handlers. Place the idempotency check before any writes as the first node after contract validation.
  5. Dead-letter queues hold failed events with full payload and failure metadata. They are recovery tools, not monitoring tools the team must review them and replay events when failures are resolved.
  6. Replay is safe when the event handler is idempotent. The idempotency check handles partial processing from prior attempts automatically.
  7. The hybrid architecture adds a low-frequency polling sweep as a safety net. Both paths share the same downstream pipeline and the same idempotency check.

End of Chapter 3.4 Event-Driven AI Systems