#72: Managing the "End" of a Data Pipeline

Jul 10, 2024

If there is such thing as the "end" of a data pipeline, it's typically a report.

Or something getting into the hands of our stakeholders.

 

But if you've ever built one of these reports you know that it's never just a simple handoff.

There's almost always some sort of back and forth.

Or future requests for adjustments.

 

So what I want to talk about in this video is about where you decide to actually make those changes.

And the secondary impacts of that decision.

 

The two common schools of thought I've seen are:

  1. Making changes directly inside the report 
  2. Keeping changes in the database (transformation layer code)

 

In this video, I want to make a case for why I personally think you're better off going with option 2.

But whether or not you agree with me...

Hopefully it'll encourage you to consider what's best for your team (& company) going forward.

 

 

The main things we'll cover include: 

  1. The tradeoffs between storing logic in reports vs database code
  2. The impacts on logic visibility & consistency
  3. Controlling the human/management element

 

Enjoy!

Michael

Access More Free Training

There's only so much that can be covered in a single article or video.

But if you'd like to go deeper, come check out The Modern Data Community (it's free).

Inside, you'll get access to free training such as Modern Data Essentials, The Starter Guide for Modern Data and more.

Plus connect directly with Michael Kahan and other Engineers around the world building modern data architectures.

Click the button below to learn more.

Access Free Trainings