Data Pipelines

Data engineering and analytics architecture diagrams.

Real-Time Streaming

Kafka-based streaming pipeline.

@arch
#!ext icons

[styles]
  stream: color:#F59E0B|width:2
  process: color:#3B82F6|width:2
  store: color:#10B981

[nodes]
  [Sources]
    web:Web Events|icon:cloud
    mobile:Mobile Events|icon:cloud
    iot:IoT Sensors|icon:server

  [Collection]
    collector:Event Collector|icon:server

  [Streaming Platform]
    kafka:Kafka Cluster|icon:aws-amazon-managed-streaming-for-apache-kafka

    [Topics]
      raw:raw-events|cyl
      enriched:enriched-events|cyl
      alerts:alerts|cyl

  [Stream Processing]
    flink:Apache Flink|icon:server
    ksql:ksqlDB|icon:server

  [Storage]
    s3:Data Lake (S3)|icon:aws-s3
    elastic:Elasticsearch|icon:database
    redis:Redis|icon:database

  [Consumers]
    dashboard:Real-time Dashboard|icon:cloud
    alerting:Alert System|icon:server
    ml:ML Pipeline|icon:server

[edges]
  web->collector|style:stream
  mobile->collector|style:stream
  iot->collector|style:stream
  collector->raw|style:stream
  raw->flink|style:process
  flink->enriched|style:stream
  flink->alerts|style:stream
  enriched->ksql|style:process
  ksql->s3|style:store
  ksql->elastic|style:store
  enriched->redis|style:store
  redis->dashboard|style:stream
  alerts->alerting|style:stream
  s3->ml|style:store

Rendered diagram

ETL Pipeline

Classic batch ETL with Airflow orchestration.

@arch
#!ext icons

[styles]
  extract: color:#EF4444
  transform: color:#F59E0B
  load: color:#10B981

[nodes]
  [Sources]
    mysql:MySQL|icon:database
    postgres:PostgreSQL|icon:database
    api:REST API|icon:cloud
    sftp:SFTP Server|icon:server

  [Orchestration]
    airflow:Apache Airflow|icon:server

  [Extract]
    e_mysql:MySQL Extractor|icon:server
    e_pg:Postgres Extractor|icon:server
    e_api:API Extractor|icon:server
    e_sftp:SFTP Extractor|icon:server

  [Staging]
    staging_area:Staging Area (S3)|icon:aws-s3

  [Transform]
    spark:Apache Spark|icon:server
    dbt:dbt|icon:server

  [Data Warehouse]
    dwh:Data Warehouse|icon:aws-amazon-redshift

  [BI]
    metabase:Metabase|icon:cloud
    looker:Looker|icon:cloud

[edges]
  airflow->e_mysql:schedule
  airflow->e_pg:schedule
  airflow->e_api:schedule
  airflow->e_sftp:schedule
  mysql->e_mysql|style:extract
  postgres->e_pg|style:extract
  api->e_api|style:extract
  sftp->e_sftp|style:extract
  e_mysql->staging_area|style:extract
  e_pg->staging_area|style:extract
  e_api->staging_area|style:extract
  e_sftp->staging_area|style:extract
  staging_area->spark|style:transform
  spark->dbt|style:transform
  dbt->dwh|style:load
  dwh->metabase
  dwh->looker

Rendered diagram

ML Feature Pipeline

Feature engineering for machine learning.

@arch
#!ext icons

[nodes]
  [Data Sources]
    events:Event Stream|icon:server
    db:Operational DB|icon:database
    logs:Log Data|icon:doc

  [Feature Engineering]
    stream_proc:Stream Processor|icon:server
    batch_proc:Batch Processor|icon:server

  [Feature Store]
    online:Online Store (Redis)|icon:database
    offline:Offline Store (S3)|icon:aws-s3
    registry:Feature Registry|icon:doc

  [ML Platform]
    training:Training Pipeline|icon:server
    serving:Model Serving|icon:server

  [Applications]
    app:Application|icon:cloud
    monitoring:Model Monitoring|icon:server

[edges]
  events->stream_proc
  db->batch_proc
  logs->batch_proc
  stream_proc->online:real-time features
  batch_proc->offline:batch features
  stream_proc->registry
  batch_proc->registry
  offline->training:train
  online->serving:inference
  training->serving:deploy model
  app->serving:predict
  serving->monitoring:metrics
  monitoring->training:retrain trigger

Rendered diagram

CDC Pipeline

Change Data Capture with Debezium.

@arch
#!ext icons

[styles]
  cdc: color:#8B5CF6|width:2
  stream: color:#F59E0B

[nodes]
  [Source Systems]
    mysql:MySQL|icon:database
    postgres:PostgreSQL|icon:database
    mongo:MongoDB|icon:database

  [CDC]
    debezium:Debezium|icon:server

  [Streaming]
    kafka:Kafka|icon:aws-amazon-managed-streaming-for-apache-kafka

    [Topics]
      mysql_cdc:mysql.changes|cyl
      pg_cdc:postgres.changes|cyl
      mongo_cdc:mongo.changes|cyl

  [Consumers]
    elastic:Elasticsearch|icon:database
    cache:Redis Cache|icon:database
    dwh:Data Warehouse|icon:aws-amazon-redshift
    lake:Data Lake|icon:aws-s3

[edges]
  mysql->debezium|style:cdc
  postgres->debezium|style:cdc
  mongo->debezium|style:cdc
  debezium->mysql_cdc|style:stream
  debezium->pg_cdc|style:stream
  debezium->mongo_cdc|style:stream
  mysql_cdc->elastic|style:stream
  mysql_cdc->cache|style:stream
  pg_cdc->dwh|style:stream
  mongo_cdc->lake|style:stream

Rendered diagram

More Data Pipeline Examples

See the example files in the repository:

• data-platform.tdl - Complete analytics platform