#017: The 3 Pillars of Data Engineering

Since 2016 I’ve worked on 7 data platforms in all sizes.

Nowadays, I can grasp an architecture within a few conversations.

But in the early days I often felt lost and overwhelmed.

Today, I’ll share the 3 pillars I focus on to quickly understand any data architecture:

1. Sources
2. Data warehouse
3. Insights

Source systems reflect the business

No company operates using only one tool.

And an architecture isn’t effective if key data points are missing.

Ultimately, everything starts from source integrations.

Which is why the first pillar is identifying sources and how data gets extracted.

Example: 20 sources synced via Fivetran & custom Python scripts.

The data warehouse is the hub

Accessing data is one thing, organizing it is another.

It can be exhausting hearing about tool selection and approaches.

Instead, find the data warehouse and notice how everything moves around it.

As an engineer, the second pillar is honing in on this central hub.

Example: Snowflake is the Airbyte destination, dbt environment and Tableau data source.

Insights are the goal

Businesses don’t invest in data just for fun.

It adds complexity that has real costs.

The value comes from insights and improved decision-making.

Learn how insights are presented and you’ve uncovered the third pillar.

Example: Power BI dashboards accessed by stakeholders throughout the business.

99% of platforms revolve around sources, a warehouse and insights.

Focus on these pillars and you’ll quickly see the big picture.