PhD defense: Synergizing Data Management, DataOps, and Data Pipelines for AI Enhanced Embedded Systems

Candidate: Aiswarya Raj Munappy Opponent: Tenured associate professor Xiaofeng Wang, University of Bolzano, Italy Committee:  Professor Maria Paasivaara, LUT University Finland Professor Daniel Varro, Linköping University Professor Stefan Wagner, Technical University Munich

Abstract:

Context: Data management is a critical aspect of any artificial intelligence (AI) initiative, playing a pivotal role in the development, training, and deployment of AI models. A well-structured approach to data management ensures that AI models are trained on reliable data, comply with ethical standards, and contribute positively to decision-making processes in embedded systems.

Objectives: This thesis is structured around three primary objectives. The first objective is to comprehensively understand and address the data management challenges associated with embedded systems. Building upon this understanding, the second objective is to explore the data management practices that can help alleviate the data management challenges. Finally, the third objective aims to develop and validate the implementation approaches for enhanced data management.

Method: To achieve the objectives, we conducted research in close collaboration with industry and used a combination of different empirical research methods like interpretive case studies, literature reviews, and action research.

Results: This thesis presents six main results. First, it identifies and categorizes data management challenges, solutions, and limitations. Second, it presents a stairway model delineating the stages of the evolution towards DataOps. Third, it proposes a model for evaluating the maturity of data pipelines and identifies determinants to assess the impact of machine learning (ML) on data pipelines. Fourth, it identifies the differences between unidirectional and bidirectional data pipelines and the significance, benefits, and challenges of bidirectional data pipelines. The thesis also provides a roadmap for the smooth migration from unidirectional to bidirectional data pipelines. Fifth, it presents and validates the conceptual model of an end-to-end data pipeline for ML/DL models. It also discusses how to balance the need for robustness with the complexity of the pipeline. Finally, it presents and validates fault-tolerant data pipelines and an AI-powered 4-stage model for automated fault recovery in data pipelines.

Conclusion: In essence, this research contributes insights and practical guidance for addressing data management challenges in AI-enhanced embedded systems. The identified challenges, solutions, and proposed models pave the way for future research and industry practices, aiming to streamline data operations, enhance the reliability of DL models, and promote efficient data management in evolving technological landscapes of AI-enhanced embedded systems.