AEP Data Ingestion Failures: Common Causes and Fixes

Understanding AEP Data Ingestion

Adobe Experience Platform accepts data through two main channels: batch ingestion (files uploaded on schedule) and streaming ingestion (real-time events via API or SDK). Both methods require data to conform to XDM schemas—and that's where most failures occur.

When ingestion fails, AEP provides error codes and partial batches. The key is understanding what these errors mean and how to prevent them at scale.

Common Batch Ingestion Failures

1. XDM Schema Validation Errors

The most frequent failure. Your source data doesn't match the expected XDM schema structure.

Root Causes:

• Data type mismatches (string vs. number vs. object)
• Missing required fields defined in the schema
• Array fields receiving single values (or vice versa)
• Nested object structure doesn't match field group hierarchy

How to Fix:

1. Export your XDM schema as JSON from the Schema Registry
2. Validate sample source data against the schema using a JSON Schema validator
3. Check field-by-field mapping in your data prep workflows
4. Use AEP's sample data preview before full ingestion

2. Identity Namespace Errors

Profile data requires at least one identity field. Failures occur when identity configuration is wrong.

Root Causes:

• No identity field marked as "primary" in the schema
• Identity namespace doesn't exist in your sandbox
• Identity value is null or empty string
• Non-standard namespace without proper configuration

How to Fix:

// Ensure identity map structure is correct
{
  "identityMap": {
    "Email": [{
      "id": "[email protected]",
      "primary": true
    }],
    "ECID": [{
      "id": "12345678901234567890",
      "primary": false
    }]
  }
}

3. Timestamp Format Errors

AEP expects ISO 8601 timestamps. Common failures occur with non-standard date formats.

Invalid Format	Valid Format
2025-01-10	2025-01-10T00:00:00Z
01/10/2025 14:30	2025-01-10T14:30:00.000Z
1704844800 (epoch)	2025-01-10T00:00:00Z

4. File Size and Format Issues

Batch ingestion has limits and format requirements that cause silent failures.

Common Issues:

• File too large: Single files over 512 MB may timeout. Split into smaller batches.
• Wrong compression: AEP expects GZIP for compressed JSON. Other formats fail.
• Encoding issues: Non-UTF-8 encoded files cause character corruption or failure.
• Malformed JSON: Trailing commas, unquoted keys, or single quotes instead of double.

Common Streaming Ingestion Failures

1. Rate Limiting (HTTP 429)

AEP enforces rate limits on streaming endpoints. Bursts of traffic cause temporary failures.

How to Handle:

• Implement exponential backoff with jitter in your client
• Use batch mode within streaming (bundle multiple events per request)
• Distribute load across multiple inlet connections
• Request limit increases through Adobe support for high-volume use cases

2. Authentication Failures (HTTP 401/403)

Streaming requires valid JWT tokens that expire periodically.

Checklist:

• JWT token not expired (default 24-hour expiry)
• Service account has "Data Ingestion" permission in Admin Console
• Correct IMS org ID in the request headers
• API key (client ID) matches the integration

3. Synchronous Validation Failures

Streaming with synchronous validation returns immediate errors but adds latency.

// Request with sync validation
POST /collection/{CONNECTION_ID}?synchronousValidation=true

// Error response
{
  "type": "https://ns.adobe.com/aep/errors/INGEST-1227-400",
  "title": "XDM validation failed",
  "detail": "Field 'eventType' is required but missing"
}

Diagnosing Ingestion Issues

Using the Monitoring Dashboard

AEP's Monitoring section shows batch and streaming health:

1. Navigate to Monitoring → Batch/Streaming
2. Filter by dataset, date range, or status (failed/partial)
3. Click on failed batches to see error details
4. Download failed records for local analysis

Checking Dataset Activity

At the dataset level:

1. Go to Datasets and select your dataset
2. Check the Batches tab for ingestion history
3. Look at Records ingested vs Records failed
4. Export failed records for debugging

API-Based Monitoring

// Get batch status via API
GET /batches/{BATCH_ID}

// Response includes:
{
  "id": "batch123",
  "status": "failed",
  "errors": [{
    "code": "INGEST-1227-400",
    "description": "XDM validation error at row 1547"
  }],
  "metrics": {
    "inputRecordCount": 10000,
    "failedRecordCount": 23
  }
}

Prevention Strategies

1. Pre-Ingestion Validation

Catch errors before they reach AEP:

• Validate JSON structure against XDM schema in your ETL
• Check required fields are present and non-null
• Verify data types match schema expectations
• Test with sample data in a development sandbox first

2. Data Prep Transformations

Use AEP's Data Prep to transform source data during ingestion:

• Map source fields to XDM paths
• Apply functions to convert data types
• Set default values for optional fields
• Filter out invalid records before ingestion

3. Alerting and Monitoring

Set up alerts for ingestion failures:

• Create alerts in AEP for batch failure rates > threshold
• Monitor streaming error rates via API
• Integrate with your observability platform (Datadog, New Relic, etc.)
• Set up PagerDuty/Slack notifications for critical failures

Quick Reference: Error Codes

Error Code	Meaning	Fix
INGEST-1227-400	XDM validation failed	Check field types and structure
INGEST-1510-400	Identity descriptor missing	Add primary identity to schema
INGEST-1232-400	Required field missing	Ensure all required fields have values
INGEST-1020-400	Timestamp parse error	Use ISO 8601 format with timezone
HTTP 429	Rate limit exceeded	Implement backoff, reduce request rate