The Data Engineering Lifecycle and Undercurrents, 4 Years Later

Data EngineeringIndustry CommentaryAI & LLMsTools & ProductsCareer

Joe Reis marks the 4th anniversary of 'Fundamentals of Data Engineering' by reflecting on whether he'd write it the same way today — and largely concludes yes, with two caveats: deeper data modeling coverage and possibly FinOps as a seventh undercurrent. He also wrestles openly with whether AI/ML deserves its own undercurrent, landing on genuine uncertainty rather than a tidy answer.

The durability of the data engineering lifecycle framework stems from its deliberate abstraction away from specific tools — a rare and harder form of technical writing that produces mental models people reach for even through an AI revolution.
  • 6

    The actual test of whether a mental model works is whether people are still reaching for it four years and one AI boom later.

  • 4

    It doesn't live in one stage. It runs through all of them, all the time. That's the entire idea of an undercurrent, and I hadn't seen anyone lay it out that way before.

  • 6

    AI amplifies the existing undercurrents more than it's becoming its own.

  • 5

    Drawing one simple enough to operate from is a fundamentally different kind of work than a tools guide.

  • 4

    We didn't want to write a technology-centric book, as that goes stale quickly.

reflective