Learning Objectives
By the end of this chapter, you will be able to:
- Understand the core components of the Databricks Data Intelligence Platform, including its architecture, Delta Lake, and Unity Catalog.
- Understand the platform's compute services (characteristics, limitations, and cost models) and select the most suitable compute option for each workload use case.
Executive Summary
- The platform splits responsibilities between a Databricks-managed control plane (UI, REST APIs, job scheduler, metadata) and a customer-owned data plane (clusters and storage in your cloud account), so data never leaves your subscription, project, or account while orchestration is centralized.
- Delta Lake is the default table format: Parquet data files plus an ordered JSON transaction log give ACID transactions, schema enforcement and evolution, time travel, MERGE, and table optimizations like OPTIMIZE, Z-ORDER, and Liquid Clustering on open storage.
- Unity Catalog is the single governance layer above all workspaces in an account: a three-level namespace (
catalog.schema.object) governs tables, views, volumes, functions, and models with one identity model (users, groups, service principals) and one privilege model (GRANT/REVOKE/DENY). - Compute on Databricks is not one product. All-purpose compute serves interactive notebook work, job compute serves scheduled runs, SQL warehouses serve BI and ad-hoc SQL (Classic, Pro, Serverless), declarative pipeline compute serves Lakeflow Spark Declarative Pipelines, and Serverless variants externalize the data plane to a Databricks-managed pool to remove startup time and node-management overhead. Picking correctly drives both cost (DBUs) and runtime characteristics.
Assumptions
- The reader has general IT literacy, working SQL knowledge, and entry-level familiarity with Spark or pandas.
- Examples use fictional names (catalog
acme, schemas likebronze,silver,gold, tableorders) and a single cloud abstraction (object storage referred to as cloud object storage; specifics around S3, ADLS Gen2, or GCS are called out only where behavior differs). - All product naming follows the platform's latest stable convention: Lakeflow Jobs (the renaming of Workflows), Lakeflow Spark Declarative Pipelines (the renaming of Delta Live Tables), Databricks Git Folders (the renaming of Repos), and Declarative Automation Bundles (the renaming of Databricks Asset Bundles). Older names appear once at first mention so transitional documentation remains recognizable.
- Region-specific feature gating (for example, Serverless availability in a particular cloud region) is noted but should be verified per workspace before adoption.