Prediction Can Reveal More Than Purchase Intent
Customer models are built to find patterns that people cannot see quickly: who may buy, respond, churn, spend more, or return. The same data can reveal financial pressure, compulsive behavior, grief, confusion, illness, isolation, or reduced ability to evaluate an offer. A commercially useful signal can also be a vulnerability signal.
The central governance question is not whether prediction is accurate. It is what the business is allowed to do after it predicts that a person is unusually susceptible. A vulnerability boundary separates ordinary personalization from actions that exploit diminished choice, increase foreseeable harm, or turn protective knowledge into targeting advantage.
Optimization Ignores Meaning Unless People Add It
A model trained to increase response will learn correlations with the defined outcome. It does not inherently distinguish healthy enthusiasm from desperation, informed preference from confusion, or sustainable spending from loss chasing. If those patterns improve conversion, the optimization process may reward them unless policy, data design, and review explicitly intervene.
Organizational separation makes the problem worse. Marketing owns promotions, risk owns harm detection, compliance owns rules, and customer service hears distress. When signals cannot cross those boundaries for protection but can cross them for sales, the company has created an information asymmetry against its own customers.
Short-Term Revenue Can Create Long-Term Liability
Targeting a susceptible customer may improve a campaign metric while increasing complaints, refunds, regulatory action, chargebacks, account closures, support cost, and reputational damage. The financial model should include expected harm and remediation, not only incremental response. A profitable campaign at the event level can destroy customer lifetime value and public trust.
Use a harm-adjusted value calculation: incremental contribution minus expected loss, intervention cost, complaint handling, regulatory exposure, and long-term attrition. The purpose is not to assign a price that makes exploitation acceptable. It exposes cases where a narrow revenue metric hides costs that other teams, customers, or society will bear.
Diagnose Where Vulnerability Enters The Model
Inventory behavioral, financial, demographic, service, and inferred features used for targeting. Identify which could indicate temporary or persistent vulnerability, including sudden spending changes, repeated failed payments, unusually long sessions, repeated cancellations, distressed language, bereavement, job loss, or requests for limits. Context matters; one signal should not become a diagnosis.
Trace each feature through training, scoring, segmentation, offer selection, channel timing, human review, and measurement. Warning signs include protection signals reused for marketing, campaign teams unable to explain exclusions, vulnerable groups receiving stronger pressure, and success metrics that ignore harm outcomes. If the same signal drives protection and promotion, the conflict must be explicit.
Choose Protective Uses Before Commercial Uses
A company can exclude a vulnerable segment from certain offers, reduce message frequency, require human review, provide clearer information, offer limits, delay an irreversible transaction, or end the commercial interaction. In lower-risk settings, the correct response may be a gentle choice architecture rather than restriction. The action should match evidence and consequence.
Businesses should avoid pretending they can clinically diagnose customers from ordinary commercial data. The boundary is operational: when available information indicates elevated susceptibility or foreseeable harm, certain targeting actions become prohibited and protective handling becomes available. Human review is valuable when reviewers have authority, training, context, and a documented decision standard.
Build The Vulnerability Boundary
Define four elements: protected indicators, prohibited actions, required interventions, and evidence of outcome. Protected indicators are signals that may reveal vulnerability. Prohibited actions include escalating pressure, urgency, credit, incentives, or frequency because of those signals. Required interventions specify review, exclusion, limits, support, or clearer consent.
Keep the boundary enforceable in data and systems. Tag sensitive features, restrict their use by purpose, separate protection models from promotion models, log every override, and test whether proxy variables recreate the prohibited targeting. Governance language is not enough if the campaign engine can still optimize toward the same susceptible population through correlated behavior.
A Subscription Retention Example
Consider an illustrative subscription service that predicts cancellation. One segment repeatedly attempts to cancel, contacts support in distressed language, and responds strongly to urgent discount messages. A revenue-only model recommends more pressure because it lowers immediate churn. The vulnerability boundary classifies repeated distress and cancellation friction as protected signals.
The system stops promotional escalation, simplifies cancellation, offers a neutral pause option, and routes only high-consequence cases to trained staff. Measurement includes completed cancellations, complaints, later voluntary return, and customer understanding. The company may lose a short-term retention event while preserving lawful, informed choice and a more trustworthy long-term relationship.
Measure Protection And Commercial Performance Together
Track offer exposure by risk segment, opt-out and cancellation completion, complaint rate, repeated contacts, financial distress indicators, intervention timing, override rate, and outcomes after protective action. Compare model lift with harm-adjusted customer value. Aggregate measures should be supplemented with case review because severe harm can disappear inside averages.
Test for proxies and feedback loops. If a protected indicator is removed, location, device, timing, or spending behavior may reproduce it. Review whether interventions reduce harm or merely move customers into a different channel. A model should not be declared responsible because it sends a warning while a separate system continues the same pressure.
Audit One High-Response Segment
Select the customer segment with the largest campaign lift and examine why it responds. Review feature importance, message timing, complaints, support contacts, cancellations, and financial signals. Ask whether the campaign succeeds because it offers genuine relevance or because the segment has less practical ability to resist pressure.
Write one enforceable exclusion and one protective action. Then test both in the production decision path and the analytics pipeline. The most valuable discovery may be that a profitable feature should not be used for targeting. Responsible prediction is partly the discipline to refuse an available optimization.
Sources, Method, And Limits
This article was prompted by The New York Times investigation of DraftKings' use of customer modeling. Regulatory context includes the UK Gambling Commission guidance on identifying customer vulnerability and its guidance on interaction and outcome evaluation.
The boundary is a SynHy governance framework that can apply beyond gambling, but legal duties vary by product, jurisdiction, and data type. NIST's AI Risk Management Framework supports structured risk governance but does not replace sector law. Businesses should involve legal, compliance, customer-protection, data, and domain specialists before defining protected indicators or interventions.