Medallion Architecture Explained — Bronze, Silver, and Gold Layers

The Medallion Architecture is the most widely adopted pattern for organizing data in a lakehouse. It divides your data into three layers — Bronze, Silver, and Gold — each with a clear purpose and quality level.

If you’ve worked with ETL pipelines, you’ve probably built something like this already. The Medallion Architecture just gives it a name and a set of rules that make it scalable.

Why Medallion Architecture?

Without a clear layering strategy, data lakes turn into data swamps — a mess of files with no lineage, no quality guarantees, and no one knows what’s trustworthy.

The Medallion pattern solves this by enforcing a simple contract:

• Bronze — raw, untouched data (the “just land it” layer)
• Silver — cleaned, validated, deduplicated data (the “trust it” layer)
• Gold — aggregated, business-ready data (the “use it” layer)

Each layer has progressively higher data quality, and each serves a different audience.

The Three Layers in Detail

Bronze Layer — Raw Ingestion

The bronze layer is your landing zone. Data arrives here exactly as it was produced — no transformations, no filtering, no schema changes.

Rules for the Bronze layer:

• Store data in its original format (or convert to Delta/Parquet for efficiency)
• Add ingestion metadata: timestamp, source system, batch ID
• Never delete or modify data — append only
• Keep all records, even duplicates and errors

from pyspark.sql.functions import current_timestamp, lit, input_file_name

def land_in_bronze(spark, source_path, bronze_path, source_system):
    raw_df = (
        spark.read
        .format("json")
        .option("multiLine", True)
        .load(source_path)
    )

    bronze_df = (
        raw_df
        .withColumn("_bronze_loaded_at", current_timestamp())
        .withColumn("_source_system", lit(source_system))
        .withColumn("_source_file", input_file_name())
    )

    (
        bronze_df.write
        .format("delta")
        .mode("append")
        .option("mergeSchema", "true")
        .save(bronze_path)
    )

# Land data from multiple sources
land_in_bronze(spark, "s3://raw/crm/*.json", "s3://lakehouse/bronze/crm_events", "salesforce")
land_in_bronze(spark, "s3://raw/web/*.json", "s3://lakehouse/bronze/clickstream", "segment")

Who uses Bronze? Data engineers for debugging and reprocessing. If a Silver transformation has a bug, you rerun it from Bronze — the raw data is always there.

Silver Layer — Cleaned and Conformed

The silver layer is where data becomes trustworthy. You apply cleaning, validation, deduplication, and schema standardization here.

Rules for the Silver layer:

• Deduplicate records
• Enforce data types and schemas
• Handle nulls, invalid values, and outliers
• Standardize naming conventions (snake_case, consistent date formats)
• Join related datasets where it makes sense (e.g., enrich events with user data)

from pyspark.sql.functions import col, when, trim, to_date, row_number
from pyspark.sql.window import Window

def bronze_to_silver_crm(spark):
    bronze = spark.read.format("delta").load("s3://lakehouse/bronze/crm_events")

    # Deduplicate: keep the latest record per event_id
    dedup_window = Window.partitionBy("event_id").orderBy(col("_bronze_loaded_at").desc())

    silver = (
        bronze
        .withColumn("_row_num", row_number().over(dedup_window))
        .filter(col("_row_num") == 1)
        .drop("_row_num")

        # Clean fields
        .withColumn("email", trim(col("email")))
        .withColumn("event_date", to_date(col("event_date"), "yyyy-MM-dd"))

        # Reject invalid records
        .filter(col("event_id").isNotNull())
        .filter(col("email").contains("@"))

        # Standardize status values
        .withColumn("status", when(col("status") == "active", "active")
                              .when(col("status").isin("inactive", "disabled"), "inactive")
                              .otherwise("unknown"))

        # Add processing metadata
        .withColumn("_silver_processed_at", current_timestamp())
    )

    (
        silver.write
        .format("delta")
        .mode("overwrite")
        .option("overwriteSchema", "true")
        .save("s3://lakehouse/silver/crm_events")
    )

bronze_to_silver_crm(spark)

Who uses Silver? Data analysts, data scientists, and ML engineers. This is the layer where most ad-hoc queries happen because the data is clean and reliable.

Gold Layer — Business-Ready Aggregations

The gold layer contains pre-computed, business-aligned datasets. These are optimized for specific use cases: dashboards, reports, ML feature stores, or API responses.

Rules for the Gold layer:

• Aggregate and summarize data for specific business questions
• Use business terminology in column names (not technical field names)
• Optimize for read performance (partitioning, Z-ordering)
• One gold table per business use case or KPI

from pyspark.sql.functions import countDistinct, sum, avg, datediff, current_date

def build_gold_customer_360(spark):
    crm = spark.read.format("delta").load("s3://lakehouse/silver/crm_events")
    orders = spark.read.format("delta").load("s3://lakehouse/silver/orders")

    customer_360 = (
        orders
        .groupBy("customer_id")
        .agg(
            countDistinct("order_id").alias("lifetime_orders"),
            sum("order_total").alias("lifetime_revenue"),
            avg("order_total").alias("avg_order_value"),
            min("order_date").alias("first_order_date"),
            max("order_date").alias("last_order_date"),
        )
        .withColumn("days_since_last_order",
                     datediff(current_date(), col("last_order_date")))
        .withColumn("customer_segment",
                     when(col("lifetime_revenue") > 10000, "enterprise")
                     .when(col("lifetime_revenue") > 1000, "mid_market")
                     .otherwise("starter"))
    )

    (
        customer_360.write
        .format("delta")
        .mode("overwrite")
        .partitionBy("customer_segment")
        .save("s3://lakehouse/gold/customer_360")
    )

build_gold_customer_360(spark)

Who uses Gold? Business stakeholders, executives, and dashboards. Gold tables are what powers your BI tools — Tableau, Power BI, Looker, or Metabase.

How Data Flows Through the Layers

Source Systems          Bronze              Silver               Gold
─────────────          ───────             ───────              ─────
Databases        ──▶   Raw JSON/CSV   ──▶  Cleaned rows    ──▶  KPI tables
APIs             ──▶   Append-only    ──▶  Deduplicated    ──▶  Dashboards
Event streams    ──▶   + metadata     ──▶  Schema-enforced ──▶  ML features
Flat files       ──▶   No transforms  ──▶  Joined/enriched ──▶  API responses

Quality:  LOW ──────────────────────────────────────────────────▶ HIGH
Latency:  LOW ──────────────────────────────────────────────────▶ HIGHER
Audience: Engineers ──────▶ Analysts ──────────▶ Business/Execs

Common Mistakes to Avoid

1. Skipping the Bronze Layer

Some teams try to clean data on ingestion. This seems efficient but is dangerous — if your cleaning logic has a bug, you’ve lost the original data. Always land raw data first.

2. Making Silver Too Complex

Silver should be a general-purpose clean layer, not tailored to one specific dashboard. If you’re building Silver tables for one use case, that belongs in Gold.

3. Too Many Gold Tables

Every Gold table is a maintenance burden. Before creating one, ask: can this query run directly on Silver with acceptable performance? If yes, skip the Gold table.

4. No Data Quality Checks Between Layers

Add assertions between each layer transition:

from great_expectations.dataset import SparkDFDataset

def validate_silver(df):
    ge_df = SparkDFDataset(df)

    # Row count should not drop by more than 10% from bronze
    assert ge_df.expect_table_row_count_to_be_between(
        min_value=bronze_count * 0.9
    ).success

    # No nulls in key columns
    assert ge_df.expect_column_values_to_not_be_null("event_id").success
    assert ge_df.expect_column_values_to_not_be_null("email").success

    # Email format validation
    assert ge_df.expect_column_values_to_match_regex(
        "email", r"^[^@]+@[^@]+\.[^@]+$"
    ).success

    print("Silver validation passed!")

Medallion Architecture vs. Other Patterns

For most data engineering teams, the Medallion Architecture is the best starting point. It’s simple enough to implement in a weekend and scales to petabytes.

When to Add More Layers

Some teams extend the pattern:

• Raw → Bronze: if you need a true immutable archive before even the Bronze landing zone
• Silver → Platinum: for ML feature stores that need different optimization than Gold
• Gold → Diamond: for executive dashboards with extreme pre-computation

Only add layers when you have a clear reason. Three layers handle 90% of use cases.

Conclusion

The Medallion Architecture gives your data lake the structure it needs to be useful:

• Bronze preserves raw data for reprocessing and auditing
• Silver provides clean, reliable data for analysts and data scientists
• Gold delivers pre-computed answers for dashboards and business stakeholders

Start with these three layers. Add data quality checks between them. Resist the urge to over-engineer — the power of this pattern is its simplicity.

In the next post, we’ll compare Airflow vs Prefect vs Dagster — the tools you’d use to orchestrate data flowing through these layers.