As mentioned in Introduction to Data Engineering – Lake Data Insights, I teach a class, Fundamentals of Data Engineering, that is a senior level / graduate level elective at NDSU. In this post, I want to elaborate on a base concept in the class – the data engineering lifecycle.
The data engineering lifecycle is a foundation concept in the book I use for the class,
Amazon.com: Fundamentals of Data Engineering: Plan and Build Robust Data Systems: 9781098108304: Reis, Joe, Housley, Matt: Books. As the screenshot from the book below shows, the lifecycle depicts the core steps in data engineering along with several “undercurrents” that are important considerations for a data engineer.
Beyond the classroom, I’ve found the lifecycle to be a very useful framework for developing a data architectures in my consulting practice. Frameworks like this need to walk a fine line between being high level enough to be applicable to a variety of problems yet detailed enough to actually provide value. This lifecycle succeeds in doing this.
Future posts, specifically some focused on my work in the electrical utility industry, will reference the lifecycle using the simple template shown below. The lifecycle serves as an excellent starting point for addressing a data engineering challenge and a good tool for documenting the architectural approach.
