Chapter 2.2 Progressive Problem Expansion

Chapter 2.1 established that a CRM is a state machine and defined the entity model that organizes contacts, companies, and deals. Chapter 2.2 begins the engineering work of defining what problem the brokerage’s CRM automation system must solve, what requirements that problem generates, and what constraints and failure points the system must handle gracefully.

The engineering discipline of progressive problem expansion starts with a simple version of the client’s problem and adds complexity iteratively, surfacing the constraints and failure points that simple problem statements conceal.

A client who says “we need to follow up with leads faster” has stated a symptom. The underlying problem has six distinct failure modes, eight functional requirements, six non-functional requirements, four technical and business constraints, and five potential failure points that a well-designed system must address.

Progressive problem expansion is the process of discovering all of these before writing a single workflow node.

Learning Objectives

After completing this chapter, you will be able to:

Apply progressive problem expansion to transform a client’s symptom statement into a structured specification with failure modes, functional requirements, non-functional requirements, technical constraints, and failure points.
Distinguish between a symptom (“leads don’t get followed up”) and an underlying engineering problem with multiple failure modes that each generate distinct requirements.
Produce a structured requirement specification document that provides the input for the nine-layer architecture mapping in Chapter 2.3.
Explain why discovering constraints after deployment is significantly more expensive than discovering them during problem expansion, using the brokerage scenario as an example.
Identify the failure points in a CRM automation system that are invisible to the client but architecturally significant for the engineer designing the solution.

2.2.1 Client Problem Definition

The brokerage’s lead management problem has six distinct failure modes, each with a different root cause and a different automation response.

Failure 1 Response latency: Inbound leads wait 4 to 24 hours for initial broker contact because the intake process is manual. A broker receives a lead notification by email, reads it when available, and responds at their discretion. For Hot-tier leads with hard lease deadlines, this latency often results in the prospect engaging a competing brokerage before the first contact is made.

Failure 2 Unstructured prioritization: Brokers prioritize leads based on personal judgment, not on consistent qualification criteria. Two brokers receiving the same lead on the same day may assess its urgency differently, producing inconsistent follow-up behavior across the team. High-signal leads that arrive during a busy week may receive the same slow response as low-signal leads.

Failure 3 Follow-up gaps: Approximately 30% of MQL-stage prospects have no recorded contact within 14 days of their initial inquiry. These leads go cold without the operations director knowing they were missed, because there is no system that monitors for overdue follow-up and alerts the team.

Failure 4 Pipeline opacity: The operations director cannot produce an accurate pipeline report on demand. The pipeline data lives in individual broker email inboxes and a shared spreadsheet updated inconsistently. The question “how many SQL leads do we currently have, and which ones are overdue for follow-up?” cannot be answered without manually reviewing every broker’s inbox.

Failure 5 Duplicate and conflicting outreach: When a returning contact submits the intake form a second time, Workflow A may create a new Contact record (if deduplication fails) or trigger the full intake notification sequence on an existing Contact who is already in active correspondence with a broker. In both cases, the broker receives a notification about a contact they are already managing, and the contact may receive outreach from multiple team members.

Failure 6 No audit trail: When a prospect asks “when did you last contact me?”, the answer requires searching individual email inboxes. There is no single record of every automated and manual action taken on a contact. This absence makes compliance reporting impossible and makes it difficult to hand off a contact from one broker to another without losing context.

Diagram 2.2.1 Six Failure Modes and Their Automation Responses

Six failure modes and their automation responses {#fig-2-2-1}
Failure Mode	Root Cause	Automation Response
Response latency	Manual intake process	Workflow A: auto-score and notify within seconds of submission
Unstructured prioritization	No scoring criteria	Hybrid scoring model: 5 rule signals + AI intent → `priority_label`
Follow-up gaps	No monitoring system	Workflow C: scheduled polling every 30 min, overdue query + alert
Pipeline opacity	No CRM data model	HubSpot as system of record; Deal creation on SQL transition
Duplicate/conflicting outreach	No deduplication or governance	Batch upsert by email; governance gate on resubmissions
No audit trail	No logging discipline	Audit Note on every automated action (all 4 workflows)

2.2.2 System Requirements

Functional Requirements

Functional requirements describe what the system must do.

The eight functional requirements for the brokerage’s CRM automation system are:

FR-01 Intake and Deduplication: The system must receive inbound lead submissions from the brokerage’s website contact form, normalize the submitted field values, and create or update HubSpot Contact records idempotently by email address. A returning contact submitting a second form must update the existing record, not create a duplicate.

FR-02 Lead Scoring: The system must compute a combined_score for every inbound lead within 5 seconds of form submission. The score must use at least five rule-based qualification signals and an AI-assisted intent assessment, combined according to a defined formula with documented weights and constraints.

FR-03 Lifecycle Qualification: The system must assign a lifecyclestage to every contact based on the combined score and a set of SQL entry criteria (presence of phone number, company name, and qualifying property size or timeline). The lifecycle stage must be one of the defined values in the PERMITTED_TRANSITIONS state machine.

FR-04 Follow-up Task Creation: The system must create a HubSpot Task for each qualifying lead, with a due date computed as a business-hours-adjusted offset from the submission time, calibrated to the contact’s priority tier (Hot, Warm, Cool, or Cold).

FR-05 Follow-up Monitoring: The system must monitor for overdue follow-up Tasks on a schedule (every 30 minutes during business hours) and send reminder notifications to the assigned broker when TP1, TP2, or escalation thresholds are exceeded.

FR-06 Lifecycle Transition Validation: The system must validate all lifecycle stage changes against the PERMITTED_TRANSITIONS state machine. Invalid transitions must be detected and rejected, with an alert sent to the operations team.

FR-07 External Data Synchronization: The system must receive and process survey responses from the brokerage’s property requirements questionnaire (Typeform), enriching the matching Contact record with survey data in a governed, idempotent manner.

FR-08 Audit Logging: The system must write a structured audit Note to every Contact record for every significant automated action, including intake scoring, lifecycle transitions, follow-up reminders, and governance holds.

Non-Functional Requirements

Non-functional requirements describe how the system must perform.

The six non-functional requirements are:

NFR-01 Response Time: Workflow A must complete end-to-end processing within 5 seconds of receiving a form submission, including the OpenAI API call.

NFR-02 Reliability: All workflows must handle API errors gracefully, logging failures to the Contact record and to a designated Slack error channel without crashing the workflow execution.

NFR-03 Idempotency: All write operations must be idempotent. Receiving the same webhook event twice must produce the same final state in HubSpot, not a duplicate record or a double-scored contact.

NFR-04 Observability: The system must produce enough audit information (HubSpot Notes, n8n execution logs, Slack alerts) that an engineer can trace any contact’s complete automated history without access to raw API logs.

NFR-05 Maintainability: All scoring weights, follow-up cadence intervals, Slack channel names, and API endpoints must be stored as n8n environment variables. Changes to these values must not require workflow node edits or redeployment.

NFR-06 Governance: The system must enforce a communication governance gate on all outbound notification actions. Contacts under manual override, opted-out status, or active cooldown must not receive automated notifications until the governing condition is cleared.

Diagram 2.2.2 Functional Requirements Mapping to Workflow Architecture {#fig-2-2-2}

Functional requirements mapped to workflow responsibility {#fig-2-2-2}
FR ID	Requirement	Workflow
FR-01	Intake / Dedup	Workflow A: batch upsert by email
FR-02	Lead Scoring	Workflow A: rule score + AI score
FR-03	Lifecycle Qualification	Workflow A: Qualification Switch
FR-04	Task Creation	Workflow A: Task + timing metadata
FR-05	Follow-up Monitoring	Workflow C: scheduled polling + alert
FR-06	Transition Validation	Workflow B: `PERMITTED_TRANSITIONS`
FR-07	External Sync	Workflow D: Typeform integration
FR-08	Audit Logging	All workflows: structured Notes

Non-functional requirement enforcement mapping
NFR ID	Requirement	Enforcement
NFR-01	Response Time	Async processing after 200 response
NFR-02	Reliability	Error branches on all HTTP nodes
NFR-03	Idempotency	Batch upsert; `response_id` check (WF-D)
NFR-04	Observability	Audit Notes + n8n execution log
NFR-05	Maintainability	All config in environment variables
NFR-06	Governance	Governance gate (all 4 workflows)

2.2.3 Constraints and Failure Points

Technical and Business Constraints

Constraints bound the system’s design in ways that must be accommodated rather than eliminated.

Four constraints apply to the brokerage’s system:

TC-01 HubSpot API Rate Limits: HubSpot’s API enforces rate limits per private app. Batch operations (batch upsert, batch note creation) must be used wherever available to stay within rate limits at volume. Workflow C’s per-Contact loop must be designed to process a bounded number of contacts per execution cycle (up to 50) rather than attempting to process the entire overdue queue in a single execution.

TC-02 OpenAI API Latency: The OpenAI Chat Completions API adds 1 to 3 seconds to Workflow A’s processing time. The AI scoring layer must implement graceful degradation: when the API is unavailable or the inquiry description is absent, the system falls back to rule-only scoring without failing the workflow execution.

TC-03 HubSpot Webhook Delivery: HubSpot’s lifecycle change webhook may deliver events with delay or may retry delivery on transient failures. Workflow B’s transition validation must handle duplicate webhook deliveries idempotently. Workflow D’s idempotency check (via last_typeform_response_id) must prevent duplicate survey processing under Typeform webhook retries.

TC-04 n8n Scheduler Reliability: The n8n scheduler does not replay missed executions after a maintenance window or restart. Contacts whose follow-up thresholds pass during a scheduler outage will be detected on the next successful execution, but the timing offset must be accounted for in operational monitoring.

BC-01 Business Hours Constraint: The brokerage’s operational hours are 8am to 8pm in the brokerage’s local timezone, Monday through Friday. All follow-up due timestamps and reminder notifications must be constrained to this window. A timestamp that falls outside business hours must be advanced to the next valid business hour.

BC-02 AI Content Policy: The OpenAI API prompt must not include personally identifiable information beyond what is necessary for intent classification. The inquiry_description field is the only user-submitted content that should be sent to the API. Structured contact fields (name, email, phone) must not be included in the prompt.

BC-03 Opt-Out Compliance: When a contact opts out of automated communication, the opt-out must be recorded immediately in a durable HubSpot property (communication_governance_status) that all workflows check before any outbound action. The opt-out must take effect on the next workflow execution after the property is written.

Failure Points and Defensive Designs

A failure point is an architectural location where a predictable failure mode requires an explicit defensive design response.

Five failure points in the brokerage’s architecture require explicit defensive design.

FP-01 OpenAI API Failure: The AI scoring HTTP Request node times out or returns an error. Defensive design: the Parse Response Code node wraps all API processing in a try/catch block that sets ai_score_valid = false on any failure, routing the workflow to the rule-only combination path. The intake audit Note records ai_fallback = true so the operations team can identify contacts scored without AI assistance.

FP-02 HubSpot API Failure: A batch upsert or property PATCH call returns a non-2xx response. Defensive design: all HTTP Request nodes route errors to a dedicated error branch that sends a Slack alert to #crm-ops-errors with the Contact ID, error code, and payload summary. The error branch does not crash the workflow it logs the failure and exits cleanly, allowing the operations team to manually retry if needed.

FP-03 Missing Required Properties: A Contact update attempts to write to a HubSpot property that does not exist in the account. Defensive design: a pre-deployment property audit verifies that all required properties exist before any workflow is activated. Runtime detection: audit Notes that are missing expected fields indicate a property write failure.

FP-04 Concurrent Workflow Writes: Workflow A and Workflow C both write to the same Contact within a short window. Defensive design: the property ownership model assigns each property to exactly one owner workflow. Workflows A and C write to non-overlapping property namespaces (Workflow A owns timing metadata writes; Workflow C owns sent-at state writes), preventing write conflicts.

FP-05 Invalid Lifecycle Regression: An external integration or manual operator writes a lifecyclestage value that violates the state machine (e.g., advancing a Customer back to Lead). Defensive design: Workflow B’s transition validation detects the invalid transition, sends an alert to #crm-ops-escalations, writes a Note to the Contact, and exits without executing any valid-transition side effects. The Contact remains in the invalid state until the operations team manually corrects it automated reversion is a destructive action that requires human authorization.

Diagram 2.2.3 Failure Points and Defensive Designs {#fig-2-2-3}

Failure points and defensive designs {#fig-2-2-3}
ID	Detection	Response	Logging	Recovery
FP-01 OpenAI API Failure	HTTP Request error branch	`ai_score_valid = false` → rule-only fallback	Audit Note: `ai_fallback = true`	No action needed; rule-only score is valid
FP-02 HubSpot API Failure	HTTP Request error branch (non-2xx)	Slack alert to `#crm-ops-errors` (Contact ID, error code, payload summary)	n8n execution error log	Manual retry by operations team
FP-03 Missing Required Properties	Property audit (pre-deployment); missing fields in audit Notes (runtime)	Block workflow activation until resolved	Property audit report	Create missing properties in HubSpot
FP-04 Concurrent Workflow Writes	HubSpot property history (competing writes)	Property ownership model prevents overlap (A owns timing metadata; C owns sent-at state)	Property history by workflow source	Property ownership model prevents conflict
FP-05 Invalid Lifecycle Regression	Workflow B `PERMITTED_TRANSITIONS` check	Slack alert + Note; halt without side effects; Contact remains in invalid state	Contact Note: “INVALID TRANSITION detected”	Manual correction by operations team

Practical Exercise 2.2 Workflow A Extension

Chapter 2.2’s implementation extends Workflow A from Chapter 2.1 with company name normalization and an extended validation sequence that addresses the edge cases surfaced by the progressive problem expansion.

Business Scenario

The brokerage’s form platform allows free-text entry in the company name field. Submissions arrive with company names in inconsistent formats “Meridian Properties LLC”, “Meridian Properties”, “meridian properties llc” that represent the same organization. The HubSpot Company search uses an exact-match filter, so without normalization each variant creates a separate Company record in HubSpot.

The Problem

Company record fragmentation breaks two downstream systems simultaneously. Pipeline reports that aggregate deal value by company undercount revenue attributed to companies with multiple records. The contact-to-company associations built during intake point to different Company records for the same firm, making it impossible to query all contacts from a single organization.

Additionally, the validation gate from Chapter 2.1 checks only for the presence of required fields. It does not verify email format, does not handle unrecognized lead_source_channel values, and does not truncate oversized inquiry_description payloads all of which will produce malformed data in downstream scoring and CRM writes.

The Architectural Solution

A Company Name Normalization Code node inserted before the Company search step strips trailing business-entity suffixes (LLC, Inc, Ltd, Corp) from a search-only copy of the company name, collapses internal whitespace, and trims leading/trailing spaces. The normalized value is used for the HubSpot search; the original trimmed value is stored in the Company record. An Extended Validation Code node added after the required-fields check validates email format, normalizes unrecognized lead_source_channel values to "unknown", and truncates inquiry_description to 2000 characters.

The brokerage’s form platform allows free-text entry in the company name field. Without normalization, “Meridian Properties LLC” and “Meridian Properties” create separate Company records in HubSpot, fragmenting deal associations across duplicates.

Step 6a Company Name Normalization Code

Purpose

The company name normalization step produces a search-optimized variant of the company name that reduces false non-matches caused by business-entity suffixes and whitespace inconsistencies. Without normalization, “Meridian Properties LLC” and “Meridian Properties” produce separate Company records in HubSpot. This step separates the search key from the stored value the normalized name is used for matching; the original trimmed name is stored.

Operation Summary

Property	Value
Node Type	Code
Position	Before Company Search (Step 6)
Primary Function	Produce `company_normalized` for search; preserve `company` for storage
Output	`company_normalized` field in execution context

Response Processing

let normalized = (items[0].json.company || '').trim();
normalized = normalized.replace(/\s+/g, ' ');
normalized = normalized.replace(/\s*(LLC|Inc\.?|Ltd\.?|Corp\.?|Co\.?)$/i, '').trim();
return [{ json: { ...items[0].json, company_normalized: normalized } }];

The suffix strip is applied only to the search-key copy. The company field retains its original trimmed value and is used as the stored name in the HubSpot Company record.

Output Table

Output	Description
`company_normalized`	Suffix-stripped, whitespace-collapsed value for HubSpot search
`company`	Original trimmed value used for Company record `name` property

Engineering Rationale

The normalization reduces but does not eliminate false non-matches. Companies with variant abbreviations (“Meridian Props” vs. “Meridian Properties”), non-English entity suffixes, or typographical differences will still miss. The current implementation associates the first search result; full disambiguation with a human-review path is deferred to a later section.

Step 4a Extended Validation Code

Purpose

The extended validation step catches malformed values that passed the required-fields gate but would produce bad data in downstream scoring and CRM writes. An email that passes the presence check but contains no @ symbol will fail HubSpot’s property validation silently. An unrecognized lead_source_channel value will score 0 without any indication to the operations team. This step enforces structural quality before any API call is made.

Operation Summary

Property	Value
Node Type	Code
Position	After Required Fields IF (Step 4), before Company Normalization
Primary Function	Validate email format; normalize unknown channel values; cap description length
Output	Validated/normalized fields; `validation_passed` boolean

Response Processing

const email = items[0].json.email || '';
const emailValid = /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email);

const KNOWN_CHANNELS = ['referral', 'organic_property_inquiry', 'paid_search', 'direct', 'general_inquiry'];
let lead_source_channel = items[0].json.lead_source_channel || '';
if (!KNOWN_CHANNELS.includes(lead_source_channel)) {
  lead_source_channel = 'unknown';
}

const inquiry_description = (items[0].json.inquiry_description || '').substring(0, 2000);

Unknown lead_source_channel values are normalized to "unknown" rather than rejected the scoring model assigns 0 points to unknown channels, which is the correct treatment for an unrecognized source. Rejection would silently drop valid submissions where the form platform sends an unexpected channel value.

Output Table

Output	Description
`email`	Must match `[^\s@]+@[^\s@]+\.[^\s@]+` pattern
`lead_source_channel`	Unknown values normalized to `"unknown"` (not rejected)
`inquiry_description`	Truncated to 2000 characters if present
`validation_passed`	Set to `false` if email format check fails; routes to Error Exit

Production Considerations

Design Practice

Normalizing unknown channels to "unknown" rather than rejecting them prevents the workflow from discarding submissions from new or unexpected form configurations. When a new intake channel is added to the brokerage’s website, its submissions will score 0 (unknown channel) rather than disappear giving the operations team an audit trail to detect the unconfigured channel and add it to SCORING_CONFIG.

Diagram 2.2.4 Workflow A Extended Architecture (Sections 5.1–5.2)

%%{init:{"theme":"base","themeVariables":{"primaryColor":"#eef2ff","primaryTextColor":"#1e1b4b","primaryBorderColor":"#6366f1","lineColor":"#6366f1","clusterBkg":"#f8f9ff","clusterBorder":"#6366f1","titleColor":"#1e1b4b","edgeLabelBackground":"#f6f4ef","fontFamily":"system-ui,sans-serif","fontSize":"13px"}}}%%

flowchart TD
    subgraph Intake["Intake"]
        A[Webhook Trigger] --> B[Header Validation]
        B -->|invalid| B1[Error Exit]
        B -->|valid| C["Field Normalization Code email, phone, company, firstname, lastname, lead_source_channel"]:::process
        C --> D[Required Fields IF]
        D -->|missing email/name| D1[Error Exit]
    end

    subgraph Validation["Validation and Normalization"]
        D -->|present| E["Extended Validation Code email pattern, channel normalization, description truncation"]:::process
        E --> F["Company Name Normalization Code strip suffixes, collapse whitespace"]:::process
    end

    subgraph CRM["CRM Writes"]
        F --> G["HTTP Request: Contact Batch Upsert POST /crm/v3/objects/contacts/batch/upsert idProperty: email extracts contactId, contact_action"]:::process
        G --> H["HTTP Request: Company Search POST /crm/v3/objects/companies/search (by normalized name)"]:::process
        H -->|match| I[companyId extracted]
        H -->|no match| J["HTTP Request: Company Create POST /crm/v3/objects/companies"]:::process
        I --> K
        J --> K["HTTP Request: Contact-Company Association PUT /crm/v4/objects/contacts/{contactId}/ associations/companies/{companyId}/contact_to_company"]:::process
    end

    subgraph Output["Output"]
        K --> L["Continue to Rule-Based Enrichment → Chapter 2.3"]:::fallback
    end
    classDef trigger  fill:#dcfce7,stroke:#16a34a,color:#14532d,font-weight:600
    classDef process  fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
    classDef decision fill:#fef9c3,stroke:#ca8a04,color:#78350f,font-weight:600
    classDef success  fill:#d1fae5,stroke:#059669,color:#064e3b,font-weight:700
    classDef fallback fill:#fee2e2,stroke:#dc2626,color:#7f1d1d

Figure 19.1: Workflow A Extended Architecture. Workflow A Extended Architecture (Sections 5.1–5.2)

Discussion Questions

1. Why does workflow extension require re-examining validation logic, not just adding new fields?

2. Your normalization step converts ‘llc’ and ‘LLC’ to ‘LLC’ correctly, but silently accepts ‘L L C’ without normalization. What validation layer would you add and at which stage?

3. A new intake field preferred_contact_time is added to the form. Describe which validation rules apply to it, whether it belongs in the normalization step or a later step, and how you would test that it survives the full workflow without being overwritten.

Chapter Summary

Chapter 2.2 translated the brokerage’s operational pain into engineering requirements. The six failure modes identify what the automation system must fix. The eight functional requirements define what it must do. The six non-functional requirements define how it must perform. The four constraints define the bounds within which it must operate. The five failure points define what must not be allowed to crash the system silently. Progressive problem expansion is the discipline that produces this structured view before any workflow node is configured the discipline that allows an automation engineer to design for failure modes that have not yet occurred rather than discovering them in production.

Transition to Chapter 2.3

Chapter 2.3 builds directly on the system constructed in this chapter.

Key Takeaways

Progressive problem expansion transforms symptom descriptions (slow response time, missed follow-up) into structured engineering requirements by asking why each symptom occurs and what the system must do to prevent it.
The six failure modes (response latency, unstructured prioritization, follow-up gaps, pipeline opacity, duplicate outreach, no audit trail) each require a distinct automation response in the workflow architecture.
Functional requirements describe what the system must do. Non-functional requirements describe how it must perform. Both categories are required for a complete system specification.
Technical constraints (API rate limits, OpenAI latency, HubSpot webhook delivery, scheduler reliability) and business constraints (business hours, AI content policy, opt-out compliance) must be accommodated in the design, not discovered during implementation.
The five failure points (AI failure, HubSpot API failure, missing properties, concurrent writes, invalid lifecycle regression) each require explicit defensive design. A system designed only for the success path will encounter all five failure modes in production.
Company name normalization is a prerequisite for reliable Company search-before-create deduplication. Without normalization, companies with slight naming variations create duplicate Company records.

End of Chapter 2.2 Progressive Problem Expansion