#72: Managing the "End" of a Data Pipeline
Jul 10, 2024If there is such thing as the "end" of a data pipeline, it's typically a report.
Or something getting into the hands of our stakeholders.
But if you've ever built one of these reports you know that it's never just a simple handoff.
There's almost always some sort of back and forth.
Or future requests for adjustments.
So what I want to talk about in this video is about where you decide to actually make those changes.
And the secondary impacts of that decision.
The two common schools of thought I've seen are:
- Making changes directly inside the report
- Keeping changes in the database (transformation layer code)
In this video, I want to make a case for why I personally think you're better off going with option 2.
But whether or not you agree with me...
Hopefully it'll encourage you to consider what's best for your team (& company) going forward.
The main things we'll cover include:
- The tradeoffs between storing logic in reports vs database code
- The impacts on logic visibility & consistency
- Controlling the human/management element
Enjoy!
Michael
Set Your New Data Stack Up for Success with a Free Strategy Call
You’ve modernized your data architecture - now let’s ensure it’s setup for long-term success.
In this Free 30-Minute Strategy Call, we’ll:
- Review your current setup and progress
- Identify potential gaps or risks
- Provide actionable next steps to optimize your process
Whether you’re managing this effort solo or with a small team, you’ll leave with a clear roadmap to confidently build a structured, scalable, and maintainable data architecture.
For best results, please provide as much detail as possible.