Towards Automated Validation and Inspection of Machine Learning Pipelines

ABSTRACT:

Machine Learning (ML) is increasingly used to automate impactful decisions, and the risks arising from this wide-spread use are garnering attention from policy makers, scientists, and the media. ML applications are often very brittle with respect to their input data, which leads to concerns about their reliability, accountability, and fairness. I will introduce some of the practical problems in this area and give an overview over two recent approaches on tackling such issues.
Deequ is a library for automating the verification of data quality at scale. It provides a declarative API, which combines common quality constraints with user-defined validation code, and thereby enables ‘unit tests’ for data. Deequ efficiently executes the resulting constraint validation workload by translating it to aggregation queries on Apache Spark, and also supports the incremental validation of data quality on growing datasets.
mlinspect is a library that enables the lightweight lineage-based inspection of ML preprocessing pipelines. The key idea is to extract a directed acyclic graph representation of the dataflow from ML preprocessing pipelines in Python, and to use this representation to automatically instrument the code with predefined inspections based on a lightweight annotation propagation approach. In contrast to existing work, mlinspect operates on declarative abstractions of popular data science libraries like estimator/transformer pipelines and does not require manual code instrumentation.