1: Data Movement

Learning Objectives

By the end of this chapter, you will be able to:

• Load data into Snowflake, outlining loading considerations and the impact of related features, including connector selection and configuration.
• Ingest structured, semi-structured, and unstructured data using stages, file formats, and INFER_SCHEMA, and choose between shares and clones for cross-account distribution.
• Manage storage integrations, encryption, compression, and parsing, and govern access through views, row-level filtering, the Snowflake Marketplace, and private listings.
• Extract and use metadata from staged files, and surface that metadata in Streamlit in Snowflake applications.
• Troubleshoot ingestion errors across external tables, Apache Iceberg tables, hybrid tables, and schema evolution scenarios.
• Design continuous pipelines using Stages, Tasks, Streams, Dynamic Tables, Materialized Views, Snowpipe, and Snowpipe Streaming, federate through the Snowflake Horizon Catalog, and unload data from Snowflake.
• Create user-defined functions and use the Snowflake SQL API, Openflow, and Snowflake Notebooks for orchestration and exploration.
• Use Snowflake Scripting to automate pipeline logic with control flow, cursors, and exception handling.

Executive Summary

• Bulk loads through COPY INTO scale with file count and warehouse size, while Snowpipe and Snowpipe Streaming target low-latency continuous workloads with separate cost models and ordering guarantees.
• Stage type, file format object, and storage integration form the three pillars that decide how data physically enters Snowflake and which encryption, parsing, and metadata behaviors apply.
• Streams, Tasks, Dynamic Tables, and Materialized Views compose into declarative or imperative pipelines whose correctness depends on offset tracking, target lag, and refresh modes rather than on raw SQL alone.
• Sharing, cloning, Marketplace listings, and Iceberg interoperability give the engineer multiple ways to expose data without copying bytes, each with distinct governance and storage-cost implications.

Assumptions

• The reader holds an active SnowPro Core credential and has worked on production Snowflake deployments.
• Examples use a fictional account named ACME_ANALYTICS, database RAW_DB, schema LANDING, and virtual warehouse INGEST_WH.
• Cloud-provider examples default to AWS S3; equivalent constructs exist on Azure Blob Storage and Google Cloud Storage and are called out where the behavior diverges.
• All feature limits cited reflect the latest stable Snowflake documentation revisions and apply to Enterprise Edition unless a higher edition tier is specifically named.