#017: The 3 Pillars of Data Engineering
Oct 29, 2022Since 2016 I’ve worked on 7 data platforms in all sizes.
Nowadays, I can grasp an architecture within a few conversations.
But in the early days I often felt lost and overwhelmed.
Today, I’ll share the 3 pillars I focus on to quickly understand any data architecture:
- Sources
- Data warehouse
- Insights
Source systems reflect the business
No company operates using only one tool.
And an architecture isn’t effective if key data points are missing.
Ultimately, everything starts from source integrations.
Which is why the first pillar is identifying sources and how data gets extracted.
Example: 20 sources synced via Fivetran & custom Python scripts.
The data warehouse is the hub
Accessing data is one thing, organizing it is another.
It can be exhausting hearing about tool selection and approaches.
Instead, find the data warehouse and notice how everything moves around it.
As an engineer, the second pillar is honing in on this central hub.
Example: Snowflake is the Airbyte destination, dbt environment and Tableau data source.
Insights are the goal
Businesses don’t invest in data just for fun.
It adds complexity that has real costs.
The value comes from insights and improved decision-making.
Learn how insights are presented and you’ve uncovered the third pillar.
Example: Power BI dashboards accessed by stakeholders throughout the business.
99% of platforms revolve around sources, a warehouse and insights.
Focus on these pillars and you’ll quickly see the big picture.
When you're ready, here are 3 ways I can help you further:
1. The Modern Engineer Playbook (Online Course): Master the art of building modern data architectures. Get an end-to-end blueprint for building simple, structured & scalable data pipelines with modern tools.
2. The Playbook for dbt™ (Online Course): Learn to build, automate & scale dbt™ with confidence. Discover common strategies teams follow, avoid costly rookie mistakes & get actionable tips based on real-world experience, not a textbook.
3. Consulting Services: Request to work 1:1 with Michael Kahan to help you and your team implement a modern data architecture with a simple, scalable structure.