Designing for security and compliance. Considerations include

Security and compliance in a Google Cloud data architecture is the layer that decides who can read which row, where bytes are allowed to reside, which keys protect those bytes, how personally identifiable information is masked before it reaches an analyst, and how a development environment is kept isolated from production. The Section 1 exam questions in this area are almost never about defining what Cloud IAM is. They are about deciding whether a role belongs at the organization, folder, project, dataset, or table level, and about choosing between an organization policy constraint and a per-resource configuration to enforce a residency rule. The decision boundary is the heart of the answer.

The reason design precedes implementation in this area is that several controls are not retroactive. A BigQuery dataset created in the US multi-region cannot be relocated by changing a setting; it must be copied via the BigQuery dataset copy feature, with table-level recreation in a target dataset. A Cloud Storage bucket location is permanent. CMEK assignment cannot rewrap data that was already written without rewriting the data. Organization policy constraints applied late catch only new resources, leaving inherited ones non-compliant. A data engineer who postpones these decisions to implementation is forced into one-way migration projects later.

Identity and Access Management (Cloud IAM and organization policies)

Cloud IAM on Google Cloud follows a strict resource hierarchy: organization, folder, project, and resource. Policy applied at a higher level is inherited at every lower level and is additive only. A user granted roles/bigquery.dataViewer at the organization node sees every dataset in every project below, regardless of any project-level configuration. Removing that visibility requires removing the binding at the level where it was granted, because IAM has no deny-style override on its own. The Organization Policy Service is a separate construct that provides deny-style guardrails through constraints such as constraints/gcp.resourceLocations and constraints/iam.disableServiceAccountKeyCreation. Cloud IAM Deny policies provide explicit deny statements that block access regardless of allow grants and are evaluated before allow policies.

Predefined roles follow the roles/service.role_name format. For BigQuery the practical roles are roles/bigquery.dataViewer, roles/bigquery.dataEditor, roles/bigquery.dataOwner, roles/bigquery.jobUser, roles/bigquery.user, and roles/bigquery.admin. The split between data roles and job roles is the standard distractor: a user with roles/bigquery.dataViewer on a dataset can read tables but cannot run a query unless they also hold roles/bigquery.jobUser at the project level, because query jobs are billed against the project, not the dataset. Custom roles are appropriate when no predefined role matches the required permission set, and they are scoped at organization or project level.

Service accounts are first-class identities. The default Compute Engine service account is granted the roles/editor role on the project at creation, which is the most common over-privilege pattern in audited deployments. The corrective design is to disable automatic role grants on default service accounts via the organization policy iam.automaticIamGrantsForDefaultServiceAccounts and to create per-workload user-managed service accounts that hold only the minimum roles required. Service account impersonation through roles/iam.serviceAccountTokenCreator replaces the need for downloadable service account keys, and key creation can be blocked at the organization level by constraints/iam.disableServiceAccountKeyCreation.

# Block service account key creation across the organization
gcloud resource-manager org-policies enable-enforce \
 --organization=123456789012 \
 iam.disableServiceAccountKeyCreation

# Grant a Dataflow worker SA only the roles it requires for one project
gcloud projects add-iam-policy-binding acme-prod-data \
 --member="serviceAccount:dataflow-prod@acme-prod-data.iam.gserviceaccount.com" \
 --role="roles/dataflow.worker"

gcloud projects add-iam-policy-binding acme-prod-data \
 --member="serviceAccount:dataflow-prod@acme-prod-data.iam.gserviceaccount.com" \
 --role="roles/bigquery.jobUser"

💡 Exam Trap: A scenario states a user can see a BigQuery dataset but cannot execute a query. The most common wrong answer is to grant roles/bigquery.dataEditor at the dataset level. The correct answer is to grant roles/bigquery.jobUser at the project level because query jobs require billing context at the project tier.

💡 Exam Trap: Organization policy constraints are not Cloud IAM. A question that asks how to prevent a user from creating a Cloud Storage bucket in a non-EU region is solved by constraints/gcp.resourceLocations, not by removing roles/storage.admin. Removing the role blocks all bucket creation; the constraint blocks only the disallowed locations.

Data security (encryption and key management)

Data on Google Cloud is encrypted at rest by default with Google-managed encryption keys (GMEK). The two design alternatives are customer-managed encryption keys (CMEK) backed by Cloud KMS and customer-supplied encryption keys (CSEK), available only for Compute Engine persistent disks and Cloud Storage. CMEK is the architecture you select when an audit requirement demands key custody, key rotation control, or revocation capability. Cloud KMS keys are regional, and a CMEK-protected BigQuery dataset must reside in a location compatible with the key location. Cloud External Key Manager (EKM) extends CMEK to keys held by an external partner key management system, which is the answer when the regulatory requirement is for keys to never reside inside Google Cloud at all.

Encryption in transit is provided by default between Google services through TLS, and Application Layer Transport Security (ALTS) protects traffic between Google services internally. For traffic from on-premises systems, Cloud Interconnect or HA VPN with IPsec is the answer when the workload requires private routing of encrypted traffic, while public internet egress is sufficient for TLS-protected APIs.

CMEK assignment is service-specific. For BigQuery, CMEK can be set at the table level, dataset level (default key for all new tables), or model level. For Cloud Storage, CMEK is set at the bucket level via the default key. For Cloud SQL, CMEK is set at instance creation. None of these support post-creation key swap that re-encrypts existing data without rewriting it: the new key applies only to new writes, and existing rows must be re-encrypted by rewriting the table or bucket object.

# Create a CMEK-protected BigQuery dataset
bq --location=EU mk \
 --dataset \
 --default_kms_key=projects/acme-prod-data/locations/europe/keyRings/finance-kr/cryptoKeys/finance-cmek \
 acme-prod-data:dataset_finance_eu

# Verify a Cloud Storage bucket has the expected default CMEK
gcloud storage buckets describe gs://acme-finance-eu \
 --format="value(encryption.defaultKmsKeyName)"

📊 Limit: Cloud KMS automatic key rotation has a minimum rotation period of 1 day and a default of 90 days. The maximum is documented in the official Cloud KMS limits; verify the exact value against the official documentation.

⚠️ Anti-Pattern: Storing service account keys in Cloud Storage with bucket ACL controls instead of using Workload Identity Federation. The key file is a long-lived credential and a single misconfigured ACL is a credential leak. Workload Identity Federation, or service account impersonation, removes the long-lived key entirely.

Privacy strategies for personally identifiable information

Cloud DLP, branded as Sensitive Data Protection, is the primary service for PII discovery and transformation. It provides infoType detectors for categories such as EMAIL_ADDRESS, CREDIT_CARD_NUMBER, US_SOCIAL_SECURITY_NUMBER, PHONE_NUMBER, and PERSON_NAME. The two transformation modes most relevant in pipeline design are de-identification, which transforms the value in place (redaction, masking, hashing, format-preserving encryption, or tokenization with KMS-wrapped keys), and inspection, which scans without modifying. De-identification is run in the ingestion path, typically inside a Dataflow template, so that downstream BigQuery tables never receive raw PII.

For BigQuery, two structural privacy controls complement Cloud DLP. Column-level security uses policy tags from Dataplex Catalog (formerly Data Catalog policy tags) to restrict column access to principals that hold the roles/datacatalog.categoryFineGrainedReader role on the tag. Dynamic data masking applies a masking rule (HASH, NULLIFY, DEFAULT) to a column based on the caller identity, returning masked values for users without the unmasked policy tag binding while leaving the underlying data unchanged. Row-level security uses row access policies, which are SQL filters evaluated per query.

💡 Exam Trap: Cloud DLP de-identification with format-preserving encryption (FPE) using FFX is reversible only when the wrapped key is available. A question that asks how to allow analysts to see hashed values that can still be joined across datasets is solved by deterministic encryption or cryptographic hashing with a shared secret, not by redaction. Redaction destroys join keys.

💡 Exam Trap: Column-level security and row-level security are evaluated together; both must pass. A user blocked from a sensitive column cannot bypass the column rule by using a row-level filter, and the masking applied is whatever the dynamic data masking rule dictates, not an error.

Regional considerations for data sovereignty

Data residency is enforced at three layers: resource location, organization policy, and customer key location. For BigQuery, the dataset location is set at creation and is one of: a single region such as europe-west4, a dual-region such as eur4 (Netherlands plus Finland), or the multi-region EU or US. Cross-region table copies are supported only via the BigQuery dataset copy feature or by exporting and reimporting through Cloud Storage. Queries cannot federate across dataset locations: a join between a US dataset and an EU dataset is not allowed in a single query, the data must first be moved.

For Cloud Storage, the bucket location is permanent and selectable from the same regional, dual-region, and multi-region options. For Spanner, the instance configuration declares the location at creation and supports regional and multi-region (such as nam-eur-asia1) configurations, with each multi-region configuration providing a documented leader placement.

The Organization Policy Service enforces residency across all resources through constraints/gcp.resourceLocations with allowed locations such as in:eu-locations. This is the design answer when the regulatory framework requires that no employee in the organization can create a resource outside an allowed jurisdiction. The constraint is evaluated at resource creation and rejects API calls that would create non-compliant resources.

🎯 Scenario: A European bank ingests credit card transactions in 8 EU markets and must guarantee that no transaction record leaves the EU at any point of processing or storage. The design pre-commits to: BigQuery datasets in the EU multi-region with CMEK from Cloud KMS keys in europe, Cloud Storage staging buckets in dual-region eur4, an organization policy constraints/gcp.resourceLocations set to in:eu-locations, and Dataflow jobs configured with the --region=europe-west4 and worker subnetwork in an EU subnet. A Cloud Logging sink to a US bucket would silently violate the rule; the design therefore also pins logging buckets to EU locations through gcloud logging buckets update.

Legal and regulatory compliance

Compliance design on Google Cloud combines the resource controls already described with audit log retention, customer responsibility for in-scope configuration, and the Compliance Reports Manager listing for frameworks such as ISO 27001, SOC 1, SOC 2, SOC 3, PCI DSS, HIPAA, FedRAMP, and ISO 27017 and 27018. Several frameworks impose specific configuration: HIPAA workloads on Google Cloud require a signed Business Associate Agreement and that the services in scope match the HIPAA-included list; PCI DSS imposes scope reduction through VPC Service Controls perimeters; GDPR imposes Article 28 data processor terms that are addressed by the Data Processing and Security Terms.

Audit logging is split into Admin Activity (always on, no charge, 400-day retention by default), Data Access (off by default for most services, configurable per service and per log type READ, WRITE, ADMIN_READ, ADMIN_WRITE), System Event, and Policy Denied. Enabling Data Access logs for BigQuery is the standard answer when an audit question requires per-query attribution including the SQL text and the principal that ran it.

📊 Limit: Admin Activity audit logs are retained for 400 days in the _Required log bucket and cannot be disabled or shortened. Data Access logs are retained for 30 days in the _Default bucket; configure a longer custom log bucket retention to satisfy longer compliance windows.

Project, dataset, and table architecture for data governance

A defensible BigQuery topology aligns project boundaries with billing and IAM boundaries. The pattern recommended in Google Cloud reference architectures is a separate project per environment per data domain: acme-prod-finance-data, acme-dev-finance-data, acme-prod-marketing-data, and so on. Datasets live within projects and inherit location and default CMEK. Tables live within datasets. IAM bindings at the project level apply to every dataset; bindings at the dataset level constrain to specific datasets; row access policies and column-level policy tags constrain inside tables.

A common alternative is to keep all production data in one project and use datasets to separate domains. The trade-off is simpler billing and centralized administration against weaker blast radius isolation: a service account compromised in finance can read marketing tables if both reside in the same project unless every binding is dataset-level. The per-domain project pattern is the more defensible default for regulated workloads.

acme-organization
|-- folder: shared-services
| |-- project: acme-prod-logging
| |-- project: acme-prod-kms-eu
|-- folder: production
| |-- project: acme-prod-finance-data
| | |-- dataset: dataset_finance_eu (location EU, default CMEK)
| |-- project: acme-prod-marketing-data
|-- folder: development
 |-- project: acme-dev-finance-data

⚠️ Anti-Pattern: Granting roles/bigquery.admin at the project level to a data analyst who only needs to query a single dataset. The admin role grants table deletion, dataset deletion, and the ability to modify any access policy in the project. The correct binding is roles/bigquery.dataViewer at the dataset level plus roles/bigquery.jobUser at the project level.

Multi-environment use cases for development and production

A multi-environment design separates development, staging or pre-production, and production into different projects with different service accounts, different Cloud KMS key rings, and different networks. Code that promotes from development to production passes through a CI/CD pipeline that re-resolves resource references and never carries development service account credentials into production runs. Dataform supports this directly through workspace and release configurations: a development workspace executes against dataset_finance_eu_dev and a release configuration deploys to dataset_finance_eu in production.

VPC Service Controls perimeters are a complementary boundary: a production perimeter wraps the production data projects, and an ingress rule on the perimeter allows only listed identities, often the production CI/CD service account, to read from production buckets and write to production datasets. A developer in the same organization with project-level read access still cannot read production data from outside the perimeter, because VPC Service Controls evaluates the API call origin against the perimeter, not only the IAM binding.

💡 Exam Trap: A user has roles/bigquery.dataViewer on a production dataset but receives VPC_SERVICE_CONTROLS_VIOLATION when running a query from Cloud Shell. The fix is not to add IAM. The fix is to add the user identity to an ingress rule on the perimeter or to access the data from a workstation inside the perimeter. Cloud Shell is outside the perimeter by default.

Decision Anchor: Choose project-per-domain-per-environment topology when blast radius isolation, per-environment CMEK keys, and per-domain billing are required. Choose single-project-with-dataset-separation when the organization is small, the data domains share a single audit boundary, and operational simplicity outweighs isolation.