Spark workloads in Microsoft Fabric run on either a tenant-managed starter pool or a workspace-defined custom Spark pool. The Spark workspace settings page governs which pool a notebook or Spark Job Definition lands on, which Spark runtime executes the code, which environment provides libraries, and whether Fabric attaches new sessions to a high-concurrency session that is already warm. The same page controls session expiration, autotune, and a feature called workspace-level admin overrides that lets a workspace admin force compute defaults on every item in the workspace. Misconfiguring this surface produces session start times that drift from a few seconds (a hot starter pool) to several minutes (a cold custom pool), and produces capacity bills that climb quickly when sessions are not consolidated.
The starter pool is a pre-warmed cluster shared across the tenant. Its node family, node size, autoscale range, and dynamic allocation behavior are fixed by Fabric. Customization is limited to changing the maximum number of nodes within the limits the F SKU allows. A custom Spark pool is owned by the workspace. It has a chosen node family (Memory Optimized at the time of writing), a node size (Small, Medium, Large, X-Large, XX-Large), an autoscale range, and dynamic executor allocation toggled on or off. Starter pools start a session in roughly five to ten seconds. Custom pools provision nodes on demand and start sessions in roughly three minutes. The trade is responsiveness for control over node size, runtime, and library cost.
The Spark runtime is the bundled version of Apache Spark, Delta Lake, Java, Scala, Python, and R that the pool executes. Each runtime maps to specific component versions. A runtime is tied to a Spark pool through the workspace setting or to a specific Spark item through an environment. When a workspace admin upgrades the default runtime, existing notebooks pick up the new runtime on their next session unless they are pinned to an environment that targets the old runtime. Runtime changes are not transparent: a Delta version jump can change protocol levels on existing tables, and a Python version jump can break pip-installed libraries that lack wheels for the new interpreter.
An environment is a Fabric item that bundles a Spark runtime, a set of Python or Conda libraries, Spark configuration properties, and a chosen Spark pool. Environments are how workspaces standardize their data engineering compute. A workspace setting can elect one environment as the workspace default so every new notebook and Spark Job Definition inherits the same libraries and pool. Individual items can override that default by attaching to a different environment. Library installation through an environment is a one-time publishing cost; once published, the libraries are pre-installed on session startup, replacing slow %pip install cells that re-run on every cold start.
High concurrency mode lets several notebooks share a single Spark application. The first notebook starts a session; subsequent notebooks with compatible language, runtime, and environment attach to the same JVM. Session start time for the second through Nth notebook drops to a few seconds because no new cluster spins up. The configuration switch lives in the workspace settings under Data Engineering / Science. High concurrency for pipelines is a related setting that lets notebook activities inside the same pipeline run share a session. Sessions stay alive until the configured session expiration timeout (default twenty minutes of inactivity) so a warm session can be reused across runs.
Autotune is a feature inside the Spark runtime that adjusts shuffle partitions, broadcast join thresholds, and the number of executors based on observed workload behavior. It is off by default and is enabled per workspace through the Spark settings page. Autotune writes its decisions to the Spark configuration of the session and learns over repeated runs of the same job. The workspace admin override toggle pins the workspace defaults so item-level overrides on pool, runtime, and environment are blocked. Pinning is a defensive control for production workspaces where the analytics platform team wants to prevent a notebook author from accidentally attaching to a custom pool that consumes excess capacity units.
📊 Limit: A starter pool session typically starts in five to ten seconds. A custom Spark pool session typically takes around three minutes to provision because it must allocate VMs.
📊 Limit: The default Spark session expiration is twenty minutes of inactivity. The workspace setting accepts values up to several hours subject to the F SKU's session limits.
💡 Exam Trap: A question that describes a notebook author complaining about slow session starts on the starter pool is almost always pointing at a custom pool change or at high-concurrency mode being turned off. The starter pool itself is fast; if it is slow, something attached the notebook to a custom pool by default through the workspace settings or an environment.
💡 Exam Trap: Environments are not pools. An environment can point at a pool, but installing a Python library at the workspace setting level alone has no path; library installation requires an environment item that the workspace setting elects as the default.
Implement item-level access controls
Item-level access controls are how a workspace admin grants narrower access to a single Fabric item without elevating the principal to a workspace role. Sharing a lakehouse, a notebook, a warehouse, or a semantic model issues a per-item permission to a user, group, or service principal. The permission options vary by item type. A lakehouse share offers Read, ReadAll (which extends to underlying OneLake data), ReadData (the SQL analytics endpoint), and Build (semantic model creation). A notebook share offers Read, Edit, and Share. A warehouse share offers Read, ReadData, and ReadAll. A semantic model share offers Read, Build, and Reshare.
Item permissions live alongside workspace roles, not under them. A user with no workspace role but a direct Read permission on a lakehouse can open that lakehouse and query its SQL analytics endpoint, but cannot browse the workspace's other items. This is the recommended pattern for analyst access to a single dataset. Group-based sharing is preferred because it scales: a Microsoft Entra ID security group is granted Read on the lakehouse once, and group membership administered by the identity team determines who actually sees the data.
A common configuration is to combine a workspace role with item permissions to express least privilege. Engineers receive Contributor on the workspace so they can build and edit items. Analysts receive nothing on the workspace and Read on the specific semantic models they consume. Service principals used by orchestration receive Member on the workspace and own the pipelines and notebooks. The configuration is administered through the Manage permissions dialog on each item or through the Fabric REST API for bulk operations.
# Grant Read on a specific lakehouse to a security group via the Fabric REST API
curl -X POST "https://api.fabric.microsoft.com/v1/workspaces/$WS_ID/items/$LH_ID/permissions" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"principal": { "id": "11111111-2222-3333-4444-555555555555", "type": "Group" },
"permissions": { "read": true, "readAll": false, "reshare": false }
}'
Design and implement schedules and event-based triggers
Schedules and event-based triggers are how a Fabric item moves from manual to autonomous. Notebooks, pipelines, Dataflows Gen2, and semantic models each carry their own schedule editor; the same UI pattern repeats with item-specific options. A schedule accepts a frequency (by the minute, hourly, daily, weekly), a start and end window, and an owner. The owner is the principal whose identity executes the scheduled run. If the owner loses access to a downstream resource, the schedule fails until the owner is updated, so schedules that need long lifecycles are best owned by a service principal rather than a user account.
Event-based triggers in Fabric are implemented through Activator. An Eventstream, a OneLake event source, or a workspace event source feeds events into a Reflex item. A rule on that Reflex item watches for a condition, like a file arriving on a specific OneLake path or a measure crossing a threshold, and fires an action. The action can start a pipeline, start a Power Automate flow, or send a Microsoft Teams message. The OneLake events source emits file created, file deleted, and file renamed events for shortcut targets and native files, which is how a Fabric pipeline can run when a partner drops a CSV into a landing zone.
{
"name": "trigger_when_orders_csv_lands",
"rule": {
"source": "onelake_events",
"filter": "path startswith 'lh_sales/Files/raw/orders/' and event = 'BlobCreated'",
"action": {
"type": "RunFabricItem",
"workspaceId": "ws-finance-prod",
"itemId": "pl_ingest_orders"
}
}
}
⚠️ Anti-Pattern: Granting a workspace role to an analyst because Read on a specific item feels too narrow. The cost of the wider role is unintended write access on every other item in the workspace; the right answer is the per-item permission.
🎯 Scenario: A pipeline must run within thirty seconds of a partner uploading a CSV to a OneLake landing folder. Polling on a one-minute schedule averages thirty seconds of lag and burns capacity. The right design is a OneLake events source feeding an Activator rule on a BlobCreated event with a path filter on the landing folder.
Decision Anchor: Choose the starter pool when notebook authors expect interactive feedback and session start under fifteen seconds matters more than node size control. Choose a custom Spark pool when the workload requires a specific node family, node size, or autoscale ceiling, or when an environment with custom libraries should be pre-installed before the session starts.