Practical data engineering knowledge
In-depth guides on modeling, ETL, Azure, Databricks and data architecture — written from real-world projects.
Incremental loads in Azure Data Factory: the watermark pattern step by step
How to do incremental loads in Azure Data Factory using the watermark pattern: Lookup the last value, Copy Data only for the new window, and a Stored Procedure that updates the control table. A practical guide.
Read articleSSIS Data Flow up to 3× faster: tuning the buffer and Fast Load
How to use AutoAdjustBufferSize, DefaultBufferMaxRows and Fast Load to speed up large loads in the SSIS Data Flow. A practical guide with real numbers.
Read articleCheckpoints in SSIS: resume a package from the exact point of failure
How to use SSIS Checkpoints to resume a long package from the exact point of failure — the 3 configuration properties, the pitfalls with Data Flow and loops, and when (or when not) to use them in 2026.
Read articlePolars streaming: process data larger than RAM — without Spark
How the Polars streaming engine processes datasets that don't fit in memory using scan_parquet + sink_parquet, keeping RAM usage constant and doing away with a cluster.
Read articleMetric Views in Unity Catalog: define the KPI once, use it everywhere
How Databricks Unity Catalog Metric Views turn business KPIs into governed, reusable objects — with the YAML walkthrough, the MEASURE() function, the query pattern, and when (or when not) to use them in 2026.
Read articleCDC in SSIS: incremental loads without scanning the whole table
How to use Change Data Capture with the CDC Control Task to turn full loads into incremental loads in SSIS — with code, the LSN state pattern, and when (or when not) to use it in 2026.
Read articleuv: the Python manager every Data Engineer should know
How uv, from Astral, replaced pip, venv and pyenv in our Databricks and Azure projects — with a 10–100× speed boost.
Read articleIncremental ingestion: stop reloading everything every night
Watermarking, change data capture and the patterns that cut cost and processing windows in ETL pipelines.
Read articleSlowly Changing Dimensions Type 2, without the headache
The essential pattern for tracking history in dimensions — explained with a concrete example and the most common mistakes.
Read articleMedallion Architecture: the pattern that organizes your Lakehouse
How the Bronze, Silver and Gold layers turn a chaotic data lake into a reliable, auditable platform.
Read article