What happens to SCD2 tables if the source pipeline performs a Full Refresh?
Reddit r/databricks1w4 min read
Hi everyone, I'm looking for clarification on how Lakeflow/DLT behaves in the following scenario. We have two independent pipelines: Pipeline 1 (Ingestion) Ingests Salesforce objects into Delta tables (e.g. sfdc_ingestion.contact ). These tables represent the current snapshot of Salesforce. Once a week, we plan to run this pipeline with Full Refresh ( full_refresh: true ) because records and columns are sometimes deleted in Salesforce, and incremental ingestion doesn't always reflect those deletions. Pipeline 2 (SCD2) For each ingested table, we have a separate pipeline that creates an SCD2 ta
