MLOps 2022 – Lina Weichbrodt, a Lead Machine Learning Engineer at DKB Bank in Germany, presented on  Machine Learning Monitoring in Production: Lessons Learned on June 7th, 2022.

Key points:

• Actionable monitoring is essential for MLOps in Production.
• Ask the machine learning modelers about their fears and concerns and set up appropriate monitoring alerts to notify the appropriate parties to take action.
• An alert with monitoring should provide insight into taking action to remediate the issue being experienced.  Otherwise, the alert is just a notification of an event.  Therefore the recipient of the alert should be trained and aware of how to act to address the alert.
• Create use-cases specific to human-understandable quality indicators, for example, using heuristics.
• Monitoring Evaluation results and Output from a machine learning model are the highest priority compared to the Input of Feature Data distribution.
• Add metrics to your Inference code.
• You cannot monitor the prediction if you do not have the crown jewels data.
• Feature monitoring does not offer actionable abilities.