#017: The 3 Pillars of Data Engineering
Oct 29, 2022Since 2016 I’ve worked on 7 data platforms in all sizes.
Nowadays, I can grasp an architecture within a few conversations.
But in the early days I often felt lost and overwhelmed.
Today, I’ll share the 3 pillars I focus on to quickly understand any data architecture:
- Sources
- Data warehouse
- Insights
Source systems reflect the business
No company operates using only one tool.
And an architecture isn’t effective if key data points are missing.
Ultimately, everything starts from source integrations.
Which is why the first pillar is identifying sources and how data gets extracted.
Example: 20 sources synced via Fivetran & custom Python scripts.
The data warehouse is the hub
Accessing data is one thing, organizing it is another.
It can be exhausting hearing about tool selection and approaches.
Instead, find the data warehouse and notice how everything moves around it.
As an engineer, the second pillar is honing in on this central hub.
Example: Snowflake is the Airbyte destination, dbt environment and Tableau data source.
Insights are the goal
Businesses don’t invest in data just for fun.
It adds complexity that has real costs.
The value comes from insights and improved decision-making.
Learn how insights are presented and you’ve uncovered the third pillar.
Example: Power BI dashboards accessed by stakeholders throughout the business.
99% of platforms revolve around sources, a warehouse and insights.
Focus on these pillars and you’ll quickly see the big picture.
When you're ready, here are 2 other ways I can help you:
1. Simple Stack Academy: Level-up your data career by learning more than just SQL. Design, build & automate an end-to-end data architecture using modern tools like dbt™ & Github. (Free 7 Day Preview)
2. Consulting Services: Get personalized support to properly implement a well-structured, scalable, and maintainable data architecture at your company.