The Data Engineering Lifecycle and Undercurrents, 4 Years Later
Summary
Joe Reis marks the 4th anniversary of 'Fundamentals of Data Engineering' by reflecting on whether he'd write it the same way today — and largely concludes yes, with two caveats: deeper data modeling coverage and possibly FinOps as a seventh undercurrent. He also wrestles openly with whether AI/ML deserves its own undercurrent, landing on genuine uncertainty rather than a tidy answer.
Key Insight
The durability of the data engineering lifecycle framework stems from its deliberate abstraction away from specific tools — a rare and harder form of technical writing that produces mental models people reach for even through an AI revolution.
Spicy Quotes (click to share)
- 6
The actual test of whether a mental model works is whether people are still reaching for it four years and one AI boom later.
- 4
It doesn't live in one stage. It runs through all of them, all the time. That's the entire idea of an undercurrent, and I hadn't seen anyone lay it out that way before.
- 6
AI amplifies the existing undercurrents more than it's becoming its own.
- 5
Drawing one simple enough to operate from is a fundamentally different kind of work than a tools guide.
- 4
We didn't want to write a technology-centric book, as that goes stale quickly.
Tone
reflective
