Solving Identity Graph Collapse in AEP P0 Incident

When our healthcare client's Real-Time CDP instance suddenly showed 2.3 million patient profiles collapsing into just 47,000 "super-profiles," we had 72 hours to diagnose the root cause, implement a fix, and restore profile integrity—all while maintaining HIPAA compliance. This is the technical deep-dive of how we did it.

The Problem: Identity Graph Collapse

Identity graph collapse occurs when AEP's identity service incorrectly links profiles that should remain separate. In healthcare, this is catastrophic—imagine Patient A receiving Patient B's medication reminders or test results being associated with the wrong individual.

Initial Symptoms

The client's data engineering team noticed the anomaly during routine monitoring:

• Profile count dropped 98% overnight (2.3M → 47K)
• Individual profiles showed 50+ linked identities (normal: 2-4)
• Segment populations spiked then crashed
• Audience activation to their CRM failed validation

Root Cause Analysis

Step 1: Audit the Identity Graph

We started by examining the identity graph structure using AEP's Identity Service API:

// Query to examine identity graph for a collapsed profile
const response = await fetch(
  'https://platform.adobe.io/data/core/identity/cluster/members',
  {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${accessToken}`,
      'x-api-key': apiKey,
      'x-gw-ims-org-id': orgId,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      "xid": "collapsed-profile-xid",
      "graph-type": "private"
    })
  }
);

// Response showed 12,847 linked identities - RED FLAG
const graph = await response.json();
console.log(`Linked identities: ${graph.members.length}`);

Step 2: Trace the Linking Chain

Identity collapse typically happens through "bridge identities"—shared values that incorrectly link unrelated profiles. We traced the linking chain:

Patient A ──[email]──► [email protected] ◄──[email]── Patient B
                              │
                              ▼
                    [Incorrect Merge]
                              │
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
   Patient A              Patient B            Patient C
   (merged)               (merged)             (merged)
        │                     │                     │
        └─────────────────────┴─────────────────────┘
                              │
                              ▼
                    SUPER-PROFILE (47K)

Step 3: Identify the Poison Data

We discovered the root cause: a batch ingestion job had loaded appointment scheduling data with a critical flaw. The scheduling system used a default placeholder email when patients didn't provide one:

// Problematic ingestion payload
{
  "identityMap": {
    "Email": [{
      "id": "[email protected]",  // ← SHARED ACROSS 47K PATIENTS
      "authenticatedState": "authenticated",
      "primary": false
    }],
    "CRM_ID": [{
      "id": "PAT-2847291",
      "authenticatedState": "authenticated", 
      "primary": true
    }]
  },
  "person": {
    "name": { "firstName": "John", "lastName": "Doe" }
  },
  "appointmentDetails": { ... }
}

The Fix: Multi-Phase Recovery

Phase 1: Stop the Bleeding (Hour 0-4)

First priority: prevent additional collapse while we investigated.

# Disable the problematic dataflow immediately
curl -X PATCH \
  'https://platform.adobe.io/data/foundation/flowservice/flows/{flowId}' \
  -H 'Authorization: Bearer {accessToken}' \
  -H 'x-api-key: {apiKey}' \
  -H 'Content-Type: application/json' \
  -d '{"state": "disabled"}'

Phase 2: Implement Identity Filter (Hour 4-12)

We created a pre-ingestion filter using AEP Data Prep to block poison identities:

// Data Prep mapping with identity validation
{
  "mappings": [
    {
      "sourceType": "EXPRESSION",
      "source": "iif(
        Email != '[email protected]' 
        && Email != '[email protected]'
        && !contains(Email, 'placeholder'),
        Email,
        null
      )",
      "destination": "identityMap.Email[0].id"
    },
    {
      "sourceType": "ATTRIBUTE",
      "source": "CRM_ID",
      "destination": "identityMap.CRM_ID[0].id"
    }
  ]
}

Phase 3: Delete Collapsed Profile Data (Hour 12-36)

This was the most delicate operation. We needed to surgically remove the corrupted profiles without affecting legitimate data. AEP doesn't support direct identity unlinking, so we used the Profile Delete API:

// Identify all profiles linked to the poison email
async function getCollapsedProfiles(poisonEmail) {
  const profiles = [];
  
  // Get all XIDs linked to the poison identity
  const clusterResponse = await fetch(
    'https://platform.adobe.io/data/core/identity/cluster/members',
    {
      method: 'POST',
      headers: authHeaders,
      body: JSON.stringify({
        "nsid": 6,  // Email namespace
        "id": poisonEmail,
        "graph-type": "private"
      })
    }
  );
  
  const cluster = await clusterResponse.json();
  return cluster.members.map(m => m.xid);
}

// Delete the corrupted super-profile
async function deleteCorruptedProfile(xid) {
  const response = await fetch(
    `https://platform.adobe.io/data/core/ups/delete`,
    {
      method: 'POST',
      headers: authHeaders,
      body: JSON.stringify({
        "profileId": xid,
        "targetDatasets": [
          "appointment_scheduling_dataset",
          // Only delete from affected datasets
        ]
      })
    }
  );
  
  return response.json();
}

Phase 4: Re-ingest Clean Data (Hour 36-60)

After profile deletion propagated (24-48 hours in AEP), we re-ingested the appointment data with proper identity handling:

// Transform function to handle missing emails properly
function transformAppointmentRecord(record) {
  const identityMap = {
    "CRM_ID": [{
      "id": record.patientId,
      "authenticatedState": "authenticated",
      "primary": true
    }]
  };
  
  // ONLY include email if it's a real patient email
  if (record.email && 
      !isPlaceholderEmail(record.email) &&
      isValidEmailFormat(record.email)) {
    identityMap["Email"] = [{
      "id": record.email.toLowerCase().trim(),
      "authenticatedState": "authenticated",
      "primary": false
    }];
  }
  
  return {
    ...record,
    identityMap
  };
}

function isPlaceholderEmail(email) {
  const blocklist = [
    'noreply@',
    'unknown@',
    'placeholder@',
    'donotreply@',
    'no-reply@',
    'test@'
  ];
  return blocklist.some(p => email.toLowerCase().includes(p));
}

Phase 5: Implement Identity Guardrails (Hour 60-72)

To prevent future incidents, we implemented multiple layers of protection:

1. Identity Namespace Limits

// Custom merge policy with identity limits
{
  "name": "Healthcare_Strict_Merge",
  "identityGraph": {
    "type": "pdg",
    "options": {
      "maxClusterSize": 100,  // Alert if exceeded
      "strictValidation": true
    }
  },
  "attributeMerge": {
    "type": "dataSetPrecedence",
    "order": ["EHR_Dataset", "PatientPortal_Dataset", "Scheduling_Dataset"]
  }
}

2. Automated Anomaly Detection

# Daily identity health check script
import requests
from datetime import datetime

def check_identity_health():
    # Query for profiles with abnormal identity counts
    query = """
    SELECT 
        COUNT(DISTINCT _id) as profile_count,
        AVG(cardinality(identityMap.Email)) as avg_emails,
        MAX(cardinality(identityMap.Email)) as max_emails
    FROM profile_snapshot
    WHERE _acp_ingestionTimestamp > CURRENT_DATE - INTERVAL '1' DAY
    """
    
    result = execute_query(query)
    
    # Alert thresholds
    if result['max_emails'] > 10:
        send_alert(
            severity='HIGH',
            message=f"Profile with {result['max_emails']} emails detected",
            timestamp=datetime.now()
        )
    
    if result['avg_emails'] > 2:
        send_alert(
            severity='MEDIUM', 
            message=f"Average emails per profile: {result['avg_emails']:.2f}",
            timestamp=datetime.now()
        )

3. Pre-Ingestion Validation

# Dataflow validation rules (pseudocode)
validation_rules:
  - name: block_shared_emails
    type: identity_blocklist
    namespace: Email
    blocklist:
      - "[email protected]"
      - "[email protected]"
      - "*@noreply.*"
    action: drop_identity
    
  - name: identity_count_limit
    type: max_identities_per_namespace
    namespace: Email
    max_count: 5
    action: reject_record
    
  - name: email_format_validation
    type: regex
    namespace: Email
    pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
    action: drop_identity

Timeline Summary

Lessons Learned

1. Identity is Everything in CDP

Your identity strategy is the foundation of your CDP. A single misconfigured identity can collapse millions of profiles. Treat identity configuration as a P0 architectural decision.

2. Placeholder Values Are Poison

Any shared value—placeholder emails, test phone numbers, default IDs—can become a "bridge" that incorrectly links profiles. Always filter these at ingestion.

3. Monitor Identity Health Continuously

Don't wait for downstream failures. Monitor identity graph metrics daily:

• Average identities per profile by namespace
• Maximum identities per profile
• Profile count trends
• Identity linking velocity

4. Have a Recovery Playbook

We resolved this in 72 hours because we had a documented playbook for identity incidents. Without it, this could have been a multi-week recovery.