Release Engineering for Machine Learning
📣 Announcement: 📣 The link to the the Zoom meeting will be shared via Brightspace before the class.
In this course, we will go on a journey that starts at continuous integration and then moves on to continuous delivery, continuous deployment, and continuous experimentation. We will discuss the theory and the current research on various related subjects like containerization, testing, or monitoring and will put the learned theory into practice. As a running example, we will build a pipeline for a machine learning application, which – compared to traditional release engineering – poses additional challenges, like data versioning or model deployment.
Learning Objectives
After following this course, you will be able to:
- Apply standard techniques of release engineering;
- Apply version control techniques to machine learning artefacts, like data or models Design a deployment pipeline for a machine learning application;
- Implement quality control techniques in a machine learning pipeline;
- Analyze and improve existing deployment pipelines;
- Evaluate and document design decisions in deployment pipelines.
Organization
Course code | CS4295 |
Instructors | Sebastian Proksch, LuÃs Cruz |
Schedule | Mondays, 13:45–15:30 Fridays, 13:45–15:30 |
ECTS | 5.0 |
Quarter | Q4 |
Communication | Mattermost: join the REMLA 2022 team. |
Examination type | Software project (35%); Essay (60%); Presentation (5%) |
Target audience | Students of the M.Sc. programme in Computer Science |
Requirements | - Intermediate understanding of OOP languages; - Practical experience with continuous integration; - Basic understanding of software testing principles; - Basic knowledge of machine learning techniques. |
Outline
Please note: The program below is tentative and is subject to change.
Day | Week | Type | Summary |
---|---|---|---|
Apr 22 | 1 | Lecture | Intro: Organization and Course Schedule; Tutorials and Final Project (Slides: Intro/Project, Recording |
Apr 25 | 1 | Lecture | Continuous {Integration, Delivery, Deployment} (Slides, Recording ) |
Apr 28 | 2 | Video Lecture | ML Testing (Rec. reading: [1], [2], [3], Slides, Recording) |
Apr 29 | 2 | Lecture | Containerization (Slides, Recording) |
May 2 | 3 | Tutorial | Cont. Delivery with GitHub: Versioning and Registries (No Slides, Recordings: SMS App, MyWeb, MyLib) |
May 6 | 3 | Lecture | ML Pipelines & Code Quality (Slides, Recording) |
May 9 | 4 | Tutorial | ML Configuration Management (Instructions, Recording) |
May 11 | 4 | Video Tutorial | Kubernetes Introduction (Slides, Recording) |
May 13 | 4 | Lecture | Continuous Experimentation (Slides, Recording) |
May 16 | 5 | Tutorial | Automating data validation with TFDV by Arumoy Shome (Instructions, Slides (TODO), Recording) |
– | 5 | Feedback | Focus: Review Current State of Pipeline Re-implementation, Pipeline Extension Proposal (Checklist for Requirements) |
– | 6 | Tutorial | Monitoring in Kubernetes (No slides, Recording) |
– | 6 | Instructions | How to write an academic paper? (Slides, Start with why, Recording) |
– | 6 | Feedback | Focus: Review First Draft of ToC + Intro |
– | 7 | Feedback | Individual Steering Meetings |
– | 8 | Instructions | Presentation tips (Slides, Recording) |
– | 8 | Feedback | Individual Steering Meetings |
Jun 13 | 9 | Examination | Presentation |
Recommended Reading
- [1] - Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, D. Sculley (2016). What’s your ML test score? A rubric for ML production systems. Reliable Machine Learning in the Wild - NIPS 2016 Workshop (2016) Preprint.
- [2] - Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2020). Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering. Preprint.
- [3] - Sun, Z., Zhang, J. M., Harman, M., Papadakis, M., & Zhang, L. (2020, June). Automatic testing and improvement of machine translation. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (pp. 974-985). Preprint.
- [4] - Haakman, M., Cruz, L., Huijgens, H., & van Deursen, A. (2020). AI Lifecycle Models Need To Be Revised. An Exploratory Study in Fintech. Preprint.
- [5] - van Oort, B., Cruz, L., Aniche, M., & van Deursen, A. (2021). The Prevalence of Code Smells in Machine Learning projects. Preprint.
- [6] - Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020, October). Adoption and effects of software engineering best practices in machine learning. In Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). Preprint.