#061: How to Build Incremental Models w/ dbt
Apr 17, 2024You don't need to process every record in a table, every time.
Fortunately, dbt has a great solution for this scenario with their "incremental" materialization option.
When setup properly, they can help you significantly cut costs & processing time.
This is because incremental models only process new data vs rebuilding an entire table (the default setting).
But setting up incremental models isn't always straight forward.
It requires a few steps, an understanding of underlying functionality and some customization.
All that to say, I've noticed this process trips people up and therefore they put off implementing it.
So in today's video I'd like to help you out by covering:
- What incremental models in dbt are all about
- Step by step how to build one
- The process to add/update new data
What will you learn?
- The purpose & functionality of Incremental models in dbt
- How to use the is_incremental() macro
- How to leverage different incremental strategies
Enjoy!
PS - Notice how this video uses very simple data as an example. Just a friendly reminder that you don't need complex or big data to practice and learn effectively.
When you're ready, here are 3 ways I can help you further:
1. The Modern Engineer Playbook (Online Course): Master the art of building modern data architectures. Get an end-to-end blueprint for building simple, structured & scalable data pipelines with modern tools.Â
2. The Playbook for dbt™ (Online Course): Learn to build, automate & scale dbt™ with confidence. Discover common strategies teams follow, avoid costly rookie mistakes & get actionable tips based on real-world experience, not a textbook.
3. Consulting Services:Â Request to work 1:1 with Michael Kahan to help you and your team implement a modern data architecture with a simple, scalable structure.