1: Data Ingestion and Transformation

Learning Objectives

By the end of this chapter, you will be able to:

• Perform data ingestion across heterogeneous sources with allowlists, multi-source integration, and the right programming-language and algorithmic choices.
• Read data from streaming sources such as Amazon Kinesis, Amazon MSK, Amazon DynamoDB Streams, AWS DMS, AWS Glue streaming jobs, and Amazon Redshift Streaming Ingestion while handling throttling, cost, orchestration, and software-engineering hygiene.
• Read data from batch sources including Amazon S3, AWS Glue, Amazon EMR, AWS DMS, Amazon Redshift, AWS Lambda, and Amazon AppFlow with fan-in and fan-out patterns and Infrastructure as Code.
• Configure batch ingestion for replayability and format conversion (CSV to Apache Parquet), and package serverless workflows with AWS SAM.
• Consume data APIs with explicit stateful versus stateless semantics, troubleshoot transformation failures, and integrate notification services along with Lambda ephemeral and Amazon EFS storage.
• Set up schedulers with Amazon EventBridge, Apache Airflow, and cron, and build data APIs that publish processed data to downstream systems using AWS CloudFormation or AWS CDK for repeatable deployment.
• Set up event triggers (Amazon S3 Event Notifications, Amazon EventBridge) and right-size container workloads on Amazon EKS and Amazon ECS.
• Call an AWS Lambda function from Amazon Kinesis with the correct concurrency model, JDBC/ODBC connectivity patterns, and large language model integration for data processing.

Executive Summary

• Streaming and batch are two distinct ingestion contracts on AWS; latency, replay semantics, ordering guarantees, and throughput knobs differ across Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, Amazon MSK, AWS DMS CDC tasks, and Amazon Redshift Streaming Ingestion.
• Configuration choices for batch ingestion (file format, partitioning, compression, error tables, COPY options) decide whether a job costs cents or hundreds of dollars and whether it can be replayed safely after a failure.
• Orchestration responsibility splits cleanly: Amazon EventBridge triggers and routes events, AWS Step Functions sequences state and error handling, Amazon MWAA expresses dependency graphs in Python, AWS Glue workflows tie Glue jobs and crawlers together, and AWS Lambda runs the small reactive units in between.
• Lambda from Kinesis is a polling consumer with a concurrency model based on shards (Kinesis Data Streams) or partitions (Amazon MSK); the trade-offs around batch size, parallelization factor, retries, and on-failure destinations decide whether a streaming pipeline survives a poison record.

Assumptions

• Single AWS account with appropriate IAM permissions; regional context is omitted unless a constraint is region-specific.
• Examples use placeholder account IDs, ARNs, bucket names, and stream names that are fictional and consistent across sections.
• Service names follow AWS conventions: Amazon X for managed data services and AWS X for management and governance services.
• No prior chapter context exists; terms introduced here are defined on first use.