1: Designing data processing systems

Learning Objectives

By the end of this chapter, you will be able to:

• Design for security and compliance using Cloud IAM, organization policies, encryption with CMEK and CSEK, Cloud DLP for PII, regional data residency, and project, dataset, and table architecture across development and production environments.
• Design for reliability and fidelity using Dataform, Dataflow, Cloud Data Fusion, LLM-assisted SQL generation, Cloud Composer orchestration, ACID trade-offs across BigQuery, Spanner, and Cloud SQL, and validation patterns.
• Design for flexibility and portability by mapping current and future business requirements, planning for multi-cloud and residency constraints, and using Dataplex and Dataplex Catalog for staging, profiling, and discovery.
• Design data migrations by analyzing stakeholders, processes, and source technologies, and selecting between BigQuery Data Transfer Service, Database Migration Service, Transfer Appliance, Datastream, and Google Cloud networking options.

Executive Summary

• Design decisions in Section 1 are evaluated as scenarios: the exam describes a business constraint and asks which architecture honors it at lowest cost without violating residency, identity, or fidelity rules.
• The principal services to internalize are Cloud IAM with predefined roles, Organization Policy Service, Cloud KMS with CMEK, Cloud DLP, BigQuery dataset locations, Dataform for SQL workflows, Dataflow for streaming and batch, Cloud Data Fusion for visual ETL, Cloud Composer for orchestration, Datastream for change data capture, Database Migration Service for managed lift, Transfer Appliance for offline bulk transfer, and Dataplex for the catalog and governance layer.
• ACID guarantees, recovery point and recovery time objectives, and dataset region selection are the most common distractor axes in Section 1 questions.
• A defensible Section 1 architecture pre-commits to: project topology per environment, IAM inheritance, residency at the dataset and bucket level, CMEK assignment per regulated dataset, DLP redaction in the ingestion path, and a migration plan with measurable cutover criteria.

Assumptions

• Region examples refer to Google Cloud locations such as us-central1, europe-west4, and the multi-region locations US and EU for BigQuery and Cloud Storage. Service availability varies and must be verified against the official product documentation per region.
• Service names follow the Google Cloud product branding update in progress for Section 1 era: products such as BigQuery, Bigtable, Spanner, Dataform, Dataflow, Dataproc, Datastream, Dataplex, Cloud Data Fusion, Cloud Composer, Cloud SQL, AlloyDB for PostgreSQL, Cloud Storage, Cloud KMS, and Cloud DLP are referenced with their stated names.
• Code examples use illustrative names such as acme-prod-data, acme-dev-data, and dataset_finance_eu. These are not real projects.
• Cost estimates are not absolute. Where the chapter mentions cost, the intent is relative selection guidance against documented pricing axes (storage class, location, processing model).