Quality assurance for machine-learning programs and notebooks

We are pleased to invite you to an open workshop presenting our ongoing research project, conducted in collaboration with WASP, on quality assurance for machine-learning (ML) programs and notebooks.

Machine-learning software differs fundamentally from traditional software in development practices, organizational roles, and strong dependency on training data. However, software quality assurance (QA) aiming to ensure robustness, maintainability, and functional correctness is much less established than those for engineering traditional software.

In this project, we explore how static analysis can be enhanced by (i) leveraging run-time information available in notebooks, and (ii) exploring and evaluating algorithmic choices and data-processing steps in ML pipelines with (aggregated) properties of the datasets used for training and evaluation.

We will showcase (1) a public benchmark of typical ML-notebook failures with verified fixes, (2) crash predictor for ML notebooks using large language models (LLMs), and (3) a data-aware static analyzer tool for detecting silent bugs caused by mismatches between datasets and pipeline operations.

The workshop will include:

– Presentation of project goals and key findings

– Demonstration of the prototype tools in action

– Discussion of future research directions and industrial challenges and needs

This is an open workshop; the Software Center NDA does not apply.