Performing Data Ingestion

A data engineering pipeline begins with one decision that constrains every later one: what is the contract between this pipeline and the upstream system that owns the data. Upstream systems publish in three shapes: a transactional store that you must replicate without disturbing (Amazon RDS, Amazon Aurora, on-premises Oracle, Salesforce), a continuous event stream you must consume in order (Amazon Kinesis Data Streams, Amazon MSK, Amazon DynamoDB Streams, Apache Kafka on-premises), and a file drop that lands on a schedule (an Amazon S3 prefix, an SFTP server, a vendor data feed). The ingestion task is to pull each shape into the AWS data platform without losing records, without flooding the source, and without paying for capacity you do not use. The wrong shape choice is hard to reverse: a pipeline that treats a transactional source as a nightly file dump cannot deliver sub-minute freshness no matter how much capacity you add at the destination.

The AWS ingestion landscape spans direct connector services (AWS Database Migration Service for relational sources, Amazon AppFlow for SaaS sources, AWS Transfer Family for SFTP), purpose-built streaming services (Amazon Kinesis Data Streams, Amazon Data Firehose, Amazon MSK), object-store landing zones (Amazon S3), and code-first paths (an AWS Lambda function calling a REST API on a schedule, an AWS Glue Python shell job invoking a JDBC driver). The skill is picking the service whose abstraction matches the upstream contract while keeping the destination open to several downstream consumers.

Allowlists for IP addresses to allow connections to data sources

Many upstream systems require the ingestion process to come from a known IP range before they will accept a connection. The AWS pattern is to deploy the ingestion compute (AWS Glue job, AWS Lambda function, AWS DMS replication instance, Amazon EMR cluster) inside a VPC, attach it to a private subnet, and route outbound traffic through a NAT gateway whose Elastic IP address you publish to the source owner for allowlisting. The Elastic IP address is stable across NAT gateway restarts; the private subnet keeps the compute unreachable from the internet. AWS Glue connections have an explicit NetworkConfiguration block that selects the subnets and security groups for the job. AWS Lambda functions are attached to a VPC via the function configuration; once attached, the function loses default internet access and a NAT gateway becomes mandatory if the function needs to reach a public endpoint.

aws ec2 allocate-address --domain vpc
aws ec2 create-nat-gateway \
 --subnet-id subnet-0ab12345 \
 --allocation-id eipalloc-09f8e7d6c5b4a3210
aws lambda update-function-configuration \
 --function-name ingest-api-poller \
 --vpc-config SubnetIds=subnet-0aa11111,subnet-0bb22222,SecurityGroupIds=sg-0cc33333

Integrate data from multiple sources

A typical pipeline blends two or more shapes: the transactional snapshot from Amazon RDS via AWS DMS, the change-data-capture stream from the same source, the click stream from a web tier landing in Amazon Kinesis Data Streams, and a third-party feed delivered as nightly files on Amazon S3. The integration pattern is to convert every shape to the same target form on Amazon S3 in a single open format (Apache Parquet, ideally with Apache Iceberg metadata) and then express the downstream join in SQL against the AWS Glue Data Catalog through Amazon Athena or Amazon Redshift Spectrum. The discipline is to keep the raw landing zone immutable; transformations always write to a separate curated prefix.

Orchestrate data pipelines

Orchestration is the layer that decides when each ingestion step runs, what its dependencies are, and what happens on failure. AWS offers four primary tools: Amazon EventBridge for event-driven triggers and cron schedules, AWS Step Functions for state-machine sequencing with explicit retry and catch behavior, Amazon MWAA (Managed Workflows for Apache Airflow) for Python DAGs with rich dependency graphs, and AWS Glue workflows for orchestrating Glue jobs and crawlers as a single unit. A common error is to pick one tool for every situation; the right pattern is to pick by shape: EventBridge for fan-in routing, Step Functions for transactional retry logic, MWAA for cross-system DAGs that span Spark and Redshift and external APIs, Glue workflows when the entire pipeline is Glue jobs and crawlers.

Programming languages and frameworks for data engineering

Python is the default language for AWS Glue, AWS Lambda, and Amazon MWAA. SQL is the default for transformations against Amazon Redshift, Amazon Athena, and the AWS Glue Data Catalog. Scala remains a valid choice for AWS Glue and Amazon EMR Spark jobs when performance-sensitive distributed code is required. Java appears in Amazon MSK consumer applications and in Amazon Kinesis Client Library (KCL) consumers. Bash and PowerShell are the right tools for orchestrating CLI calls in CI pipelines. R is used for analytics in Amazon SageMaker notebooks but is rare in pure ingestion paths. The skill is matching the language to the runtime: writing a Spark transform in R fights the runtime; writing a one-time data copy in Scala wastes effort that Bash and the AWS CLI would complete in ten lines.

Data structures and algorithms

Ingestion pipelines depend on a small set of data structures. A queue (Amazon SQS, an internal Kinesis shard buffer) decouples producer and consumer rates. A hash map indexes records by partition key for fast lookup during deduplication. A tree, often a B-tree, backs the partition keys and sort keys of Amazon DynamoDB. Graph data structures appear in two places: dependency graphs in Amazon MWAA DAGs and AWS Step Functions, and identity graphs in entity resolution. The relevant algorithms are sort (for merging late-arriving records), hash (for partitioning), and breadth-first traversal (for DAG scheduling). At ingestion scale, picking the wrong algorithm for deduplication, for example, an O(n^2) nested loop over a million records, will exhaust the Lambda timeout before a hash-based pass would finish.

def deduplicate_records(records):
 seen = set()
 unique = []
 for record in records:
 key = (record["source_system"], record["primary_key"])
 if key in seen:
 continue
 seen.add(key)
 unique.append(record)
 return unique

Decision Anchor

Choose direct connector services (AWS DMS, Amazon AppFlow, AWS Transfer Family) when the upstream owner publishes through a supported protocol and you want a managed source-to-target path with built-in CDC. Choose code-first ingestion (AWS Lambda, AWS Glue Python shell, Amazon ECS task) when the source is custom (a REST API, a vendor-specific protocol) and no native connector exists.

💡 Exam Trap: A Lambda function attached to a VPC loses its default internet path. If the upstream API is on the public internet, the function needs a NAT gateway in a public subnet, an Elastic IP address allocated to that gateway, and a route table on the function's private subnet that sends 0.0.0.0/0 to the NAT gateway. Without that wiring, the function appears to hang and then times out after the configured duration.

💡 Exam Trap: An Elastic IP attached to a NAT gateway is the address the upstream owner allowlists. Replacing the NAT gateway without reusing the EIP changes the outbound address and breaks the source connection silently.

⚠️ Anti-Pattern: Treating an immutable raw landing zone as a working area. If transformations write back to the same Amazon S3 prefix as the source files, replayability disappears, and an Amazon Athena query against the prefix returns inconsistent results during the rewrite window.

📊 Limit: A NAT gateway supports up to 55,000 simultaneous connections to each unique destination (IP, port, protocol). High-fan-out ingestion to a single upstream API may need multiple NAT gateways or an interface VPC endpoint when AWS PrivateLink is available.

🎯 Scenario: A pipeline must pull from an Amazon RDS for PostgreSQL primary in another account, an Amazon Kinesis Data Stream owned by the marketing team, and a vendor SFTP server that only accepts allowlisted IPs. The engineer deploys an AWS DMS task across accounts using a VPC peering connection for the RDS source, an Amazon Kinesis consumer Lambda for the stream, and AWS Transfer Family with a static Elastic IP on the SFTP path. All three write Apache Parquet into the same Amazon S3 raw prefix with source_system as the leading partition key so downstream Amazon Athena queries can union the three feeds in one SQL statement.