#71: The Missing Piece In Many Data Pipelines
Jul 03, 2024All data teams (large & small) have at least one thing in common.
Source data.
But not everyone handles it the same way in their pipelines.
For some, they'll reference raw source tables directly in many queries.
For others, they'll create ad-hoc custom tables to address subtle formatting changes.
But without any real over arching strategy or consistent naming behind it.
While a more popular topic is data modeling (ex. kimball, one big table, etc.)
I believe an equally more important area to consider is what you do BEFORE you start creating those core data models.
For many, this "before" layer doesn't exist at all.
In previous videos I've talked about a 3-Layered Data Model.
And today I want to focus solely on Layer 1, which addresses this concept.
It's called a "Staging" layer.
When done right, it can help you establish reliable pipelines from the very start.
The main things we'll cover include:
- The purpose of a Staging layer
- A simple naming convention to follow
- How it can make your data pipelines more clear & consistent
Enjoy!
Michael
Set Your New Data Stack Up for Success with a Free Strategy Call
You’ve modernized your data architecture - now let’s ensure it’s setup for long-term success.
In this Free 30-Minute Strategy Call, we’ll:
- Review your current setup and progress
- Identify potential gaps or risks
- Provide actionable next steps to optimize your process
Whether you’re managing this effort solo or with a small team, you’ll leave with a clear roadmap to confidently build a structured, scalable, and maintainable data architecture.
For best results, please provide as much detail as possible.