Snowplow Post

Capabilities

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Connection Options

BDP Console Connection

Iglu Connection

Note: Iglu-only mode uses automatic schema discovery via the /api/schemas endpoint (requires Iglu Server 0.6+). All schemas in the registry will be automatically discovered.

Feature Options

Schema Extraction Options

Warehouse Lineage Options (Advanced)

⚠️ Note: Disabled by default. Prefer warehouse connectors (Snowflake, BigQuery) for column-level lineage.

Field Tagging Options

Performance Options

Filtering Options

Stateful Ingestion

Quick Start

1. BDP Console (Managed Snowplow)

Create a recipe file snowplow_recipe.yml:

source:
  type: snowplow
  config:
    bdp_connection:
      organization_id: "<ORG_UUID>"
      api_key_id: "${SNOWPLOW_API_KEY_ID}"
      api_key: "${SNOWPLOW_API_KEY}"

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"

Run ingestion:

datahub ingest -c snowplow_recipe.yml

2. Open-Source Snowplow (Iglu-Only Mode)

For self-hosted Snowplow with Iglu registry (without BDP Console API):

source:
  type: snowplow
  config:
    iglu_connection:
      iglu_server_url: "https://iglu.example.com"
      api_key: "${IGLU_API_KEY}" # Optional for private registries

    schema_types_to_extract:
      - "event"
      - "entity"

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"

Important notes for Iglu-only mode:

• ✅ Supported: Event and entity schemas with full JSON Schema definitions
• ✅ Supported: Automatic schema discovery via /api/schemas endpoint (requires Iglu Server 0.6+)
• ⚠️ Not supported: Event specifications (requires BDP API)
• ⚠️ Not supported: Tracking scenarios (requires BDP API)
• ⚠️ Not supported: Field tagging/PII detection (requires BDP deployment data)

For complete configuration options, see snowplow_iglu.yml.

3. With Warehouse Lineage (BDP Only - Advanced)

⚠️ Note: This feature is disabled by default and should only be enabled in specific scenarios (see below).

Start from the baseline BDP recipe in 1. BDP Console (Managed Snowplow), then add:

source:
  type: snowplow
  config:
    # Enable warehouse lineage via Data Models API
    warehouse_lineage:
      enabled: true
      platform_instance: "prod_snowflake" # Optional
      env: "PROD" # Optional
      validate_urns: true # Optional

What this creates:

• Table-level lineage: atomic.events → derived.sessions (or other derived tables)
• No direct warehouse credentials needed (uses BDP API)

Supported warehouses: Snowflake, BigQuery, Redshift, Databricks

When to Enable This Feature

✅ Enable warehouse lineage if:

• You want quick table-level lineage without configuring warehouse connector
• You don't have access to warehouse query logs
• You want to document Data Models API metadata specifically

❌ Don't enable if you're using warehouse connectors:

• Snowflake connector provides:
  • Column-level lineage by parsing SQL queries
  • Transformation logic from query history
  • Complete dependency graphs
• BigQuery, Redshift, Databricks connectors similarly provide richer lineage

Best practice: Use warehouse connector for detailed lineage. Only enable this for quick documentation of Data Models metadata.

Requirements: Data Models must be configured in your BDP organization.

Schema Versioning

Snowplow uses SchemaVer (semantic versioning for schemas) with the format MODEL-REVISION-ADDITION:

• MODEL (first digit): Breaking changes - incompatible with previous versions
• REVISION (second digit): Non-breaking changes - additions that are backward compatible
• ADDITION (third digit): Adding optional fields without breaking changes

Example: 1-0-2

• Model: 1 (major version)
• Revision: 0 (no revisions)
• Addition: 2 (two optional field additions)

In DataHub, schemas are represented as:

• Dataset name: {vendor}.{name}.{version} (e.g., com.example.page_view.1-0-0)
• Schema version: Tracked in dataset properties

Entity Mapping: Snowplow → DataHub

This section explains how Snowplow concepts are modeled as DataHub entities.

Entity Type Mapping

Pipeline Architecture in DataHub

Snowplow pipelines are modeled as DataFlow entities with DataJob children representing each processing stage:

Tracker SDKs (Web, Mobile, Server)
            │
            ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                    Pipeline (DataFlow)                                   │
│  urn:li:dataFlow:(snowplow,pipeline-id,PROD)                            │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│   ┌─────────────────┐                                                   │
│   │    Collector    │  ◄── Receives HTTP tracking events                │
│   │    (DataJob)    │                                                   │
│   └────────┬────────┘                                                   │
│            │                                                            │
│            ▼                                                            │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────────────┐│
│   │   IP Lookup     │  │   UA Parser     │  │  PII Pseudonymization   ││
│   │   (DataJob)     │  │   (DataJob)     │  │       (DataJob)         ││
│   │                 │  │                 │  │                         ││
│   │ user_ipaddress  │  │ useragent       │  │ user_id, email          ││
│   │  → geo_*, ip_*  │  │  → br_*, os_*   │  │  → (hashed values)      ││
│   └────────┬────────┘  └────────┬────────┘  └────────────┬────────────┘│
│            │                    │                        │              │
│            └────────────────────┼────────────────────────┘              │
│                                 ▼                                       │
└─────────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
                    ┌─────────────────────────┐
                    │  Atomic Events (Dataset)│
                    │  Enriched event stream  │
                    └────────────┬────────────┘
                                 │
                                 ▼
                    ┌─────────────────────────┐
                    │   Warehouse Tables      │
                    │ (Snowflake, BigQuery)   │
                    └─────────────────────────┘

Lineage Relationships

The connector creates the following lineage relationships:

1. Schema → Event Specification Lineage

Event specifications reference the schemas they require:

┌──────────────────────────────┐
│ Event Schema                 │
│ (vendor.event_name.1-0-0)    │────┐
└──────────────────────────────┘    │     ┌─────────────────────────┐
                                    ├────▶│   Event Specification   │
┌──────────────────────────────┐    │     │  (Tracking Requirement) │
│ Entity Schema                │────┘     └─────────────────────────┘
│ (vendor.context.1-0-0)       │
└──────────────────────────────┘

2. Enrichment Column-Level Lineage

Enrichments transform specific fields. Example for IP Lookup:

┌─────────────────────────────────────────────────────────────────────┐
│                        IP Lookup Enrichment                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   Input                          Output                             │
│   ─────                          ──────                             │
│                                  ┌─────────────────┐                │
│                              ┌──▶│ geo_country     │                │
│                              │   ├─────────────────┤                │
│   ┌─────────────────┐        │   │ geo_city        │                │
│   │ user_ipaddress  │────────┼──▶├─────────────────┤                │
│   └─────────────────┘        │   │ geo_region      │                │
│                              │   ├─────────────────┤                │
│                              │   │ geo_latitude    │                │
│                              ├──▶├─────────────────┤                │
│                              │   │ geo_longitude   │                │
│                              │   ├─────────────────┤                │
│                              └──▶│ ip_isp          │                │
│                                  ├─────────────────┤                │
│                                  │ ip_organization │                │
│                                  └─────────────────┘                │
└─────────────────────────────────────────────────────────────────────┘

Supported enrichments with column-level lineage:

• IP Lookup: user_ipaddress → geo_*, ip_* fields
• UA Parser: useragent → br_*, os_* fields
• YAUAA: useragent → browser, OS, device fields
• Referer Parser: page_referrer → refr_* fields
• Campaign Attribution: page_urlquery → mkt_* fields
• PII Pseudonymization: configured fields → same fields (hashed)
• Currency Conversion: currency fields → converted fields
• Event Fingerprint: event fields → event_fingerprint
• IAB Spiders/Robots: useragent → iab_* classification fields

3. Warehouse Lineage (Optional)

When warehouse_lineage.enabled: true:

┌─────────────────────────┐                    ┌─────────────────────────┐
│     Atomic Events       │   Data Models API  │     Derived Table       │
│ (snowplow.atomic.events)│───────────────────▶│ (warehouse.schema.table)│
└─────────────────────────┘                    └─────────────────────────┘

Container Hierarchy

Organization (Container: DATABASE)
│
├── Event Schema: com.example.page_view.1-0-0 (Dataset)
├── Event Schema: com.example.checkout.1-0-0 (Dataset)
├── Entity Schema: com.example.user_context.1-0-0 (Dataset)
├── Event Specification: "Page View Tracking" (Dataset)
│
├── Tracking Scenario: "Checkout Flow" (Container)
│   ├── Event Specification: "Add to Cart" (Dataset)
│   └── Event Specification: "Purchase Complete" (Dataset)
│
└── Tracking Plan: "Web Analytics" (Container)
    ├── Event Specification (linked)
    └── Schema (linked)

URN Formats

Custom Properties

Each entity type includes relevant custom properties:

Event/Entity Schemas:

• vendor, name, version (SchemaVer format)
• schema_type (event/entity)
• json_schema (full JSON Schema definition)
• deployed_environments (PROD, DEV, etc.)

Event Specifications:

• status (draft, active, deprecated)
• trigger_conditions
• referenced_schemas

Enrichments:

• enrichment_type
• input_fields, output_fields
• configuration details

Custom Platform Instance

Group schemas by environment:

source:
  type: snowplow
  config:
    bdp_connection:
      organization_id: "<ORG_UUID>"
      api_key_id: "${SNOWPLOW_API_KEY_ID}"
      api_key: "${SNOWPLOW_API_KEY}"

    platform_instance: "production"
    env: "PROD"

Schema Filtering

Extract only specific vendor schemas:

source:
  type: snowplow
  config:
    # ... connection config ...

    schema_pattern:
      allow:
        - "com\\.example\\..*" # Allow com.example schemas
        - "com\\.acme\\.events\\..*" # Allow com.acme.events schemas
      deny:
        - ".*\\.test$" # Deny test schemas

Testing the Connection

Use DataHub's built-in test-connection command:

datahub check source-connection snowplow \
  --config snowplow_recipe.yml

This will:

• Test BDP Console API authentication
• Test Iglu registry connectivity (if configured)
• Verify required permissions
• Report capability availability

Limitations

1. BDP-specific features:

   • Event specifications only available via BDP Console API
   • Tracking scenarios only available via BDP Console API
   • Tracking plans only available via BDP Console API
   • Open-source Iglu users won't have these features

2. Iglu Server requirements:

   • Automatic schema discovery requires Iglu Server 0.6+ with /api/schemas endpoint
   • Older Iglu implementations may not support the list schemas API

3. Field tagging in Iglu-only mode:

   • PII/sensitive field detection requires BDP deployment metadata
   • Not available when using Iglu-only mode

Troubleshooting

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

Authentication Errors

Error: Authentication failed: Invalid API credentials

Solution:

1. Verify api_key_id and api_key are correct
2. Check credentials are for the correct organization
3. Ensure credentials haven't expired
4. Generate new credentials in BDP Console if needed

Error: Authentication failed: Forbidden

Solution:

• Check organization_id matches your credentials
• Verify API key has required permissions
• Contact Snowplow support if permissions are unclear

Permission Errors

Error: Permission denied for /data-structures

Solution:

• API key missing read:data-structures permission
• Generate new credentials with correct permissions in BDP Console → Settings → API Credentials

Error: Permission denied for /event-specs

Solution:

• Set extract_event_specifications: false in config, or
• Request read:event-specs permission for your API key

Connection Errors

Error: Request timeout: https://console.snowplowanalytics.com

Solution:

• Check network connectivity to Snowplow Console
• Increase timeout_seconds in configuration
• Verify Console URL is correct

Error: Iglu connection failed

Solution:

• Verify iglu_server_url is correct and accessible
• For private registries, check api_key is valid
• Test connectivity: curl https://iglu.example.com/api/schemas

No Schemas Found

Issue: Ingestion completes but no schemas extracted

Solutions:

1. Check filtering patterns:

   yamlCopy

   schema_pattern:
     allow: [".*"] # Allow all schemas
   

2. Check schema types:

   yamlCopy

   schema_types_to_extract: ["event", "entity"]
   

3. Include hidden schemas:

   yamlCopy

   include_hidden_schemas: true
   

4. Verify schemas exist in BDP Console or Iglu registry

Rate Limiting

Error: HTTP 429: Rate limit exceeded

Solution:

• Connector implements automatic retry with exponential backoff
• Rate limits should be handled automatically
• If issues persist, contact Snowplow support to increase limits