So you just completed a machine learning model and got great results! Mission accomplished? Not quite. Rather, you’re probably still early in your journey because …
“No machine learning model is valuable, unless it’s deployed to production.”– Luigi Patruno
…And even after you deployed your model, you’re still not done. Rather, you probably want to sustain that model over time.
Now some models might just be one-offs and don’t need a productized system. However, more and more often the target goal to provide sustainable value which requires machine learning operations (also known as ML Ops or data science operations).
To help us navigate through this world, I sat down with Luigi from MLinProduction.com.
So, with some pointers from Luigi, let’s dive into:
- What is machine learning operations?
- Why is maintaining a model important?
- What else do you need to maintain a machine learning system?
- Why is this a hot topic today?
- When should you start planning operations?
- How is machine learning operations evolving?
- Where can you learn more?
What is machine learning operations?
Machine learning operations (ML Ops) is an emerging field that rests at the intersection of development, IT operations, and machine learning. It aims to facilitate cross-functional collaboration by breaking down otherwise siloed teams.
Machine learning operations is more than just a single tool. Rather, it spans a wide set of practices, systems, and responsibilities that data scientists, data engineers, cloud engineers, IT operations, and business stakeholders use to develop, scale, deploy, and maintain machine learning solutions.
More specifically, Luigi thinks of machine learning operations “as everything from planning and building your original pipelines and delivery mechanisms for machine models all the way through seeing those through working in production, QA-ing any issues, and finding any issues proactively before they spoil your system and integrating any changes for the future development of the machine learning pipeline.”
Why is maintaining a model important?
What does this quote from an Ancient Greek philosopher have to do with maintaining machine models? Well, as Luigi explains: “The most predictive a model will be is when you get to production […because…] the world will change and the relationship between the input and the output will deteriorate over time.”
For example, imagine you are predicting the number of people who will buy a ticket for a cruise ship. If you developed your model in early 2020 based on data from 2019 … well, the model probably isn’t very effective in 2021.
Even less dramatic use cases (think a movie recommendation engine or the price to charge a ride-share user) require regular inspection of the input data and the model’s performance as consumer preferences and market conditions constantly evolve.
What else do you need to maintain a machine learning system?
A lot, actually.
The core model maintenance rests on properly monitoring and maintaining the input data and retraining the model when needed. Knowing when and how to execute this is in of itself a significant task and is the most unique piece to maintaining machine learning systems.
However, this is only part of the picture. In fact, per a 2015 paper from Google, the machine learning code is only a small portion of the overall infrastructure needed to maintain a machine learning system.
As such, much of what is already established in the more mature field of software operations applies. Afterall, “Machine learning systems at the end of the day are software systems. So a lot of the operational practices that people are trying to implement in machine learning today are really derived in some way on good software operations practices.” (Luigi interview).
Don’t Miss out on the Latest
Sign up for the Data Science Project Manager’s Tips to learn 4 differentiating factors to better manage data science projects. Plus, you’ll get monthly updates on the latest articles, research, and offers.
Why is the field a hot topic today?
Well, it depends on your perspective. Luigi points out that companies like Google or Facebook have understood the importance of maintaining production-based machine learning system for years. For them, ML Ops isn’t really new.
However, as machine learning has gone mainstream, it has caught the eye of more organizations. And the rising focus on machine learning operations is now following this same path. Luigi clarifies that “It was first how do we even create a model, then it was how do we get the model into the real world … and now it’s how we make sure the model continues to operate well…it’s a natural progression of the development life cycle of the model and most companies just weren’t at this place a few years ago… but now there are more companies at this stage of the game making sure the models operate post-deployment.”
When should you start planning operations?
Data science projects fail at alarming rates. One of the leading causes of failure rests from a hard handoff from the data scientists to another team that needs to maintain the model and system.
By bringing the operational considerations earlier in the project life cycle, you are more likely to avoid this pitfall. Luigi agrees. He puts it more bluntly: “Thinking about the deployment at the end of the development of the model is a terrible idea…you need to think about it at the very beginning of the project.”
How is the field of machine learning operations evolving?
Luigi again points out the similarities with software. “I don’t think you’d be very wide of the mark by looking at how software operations has developed, and then figuring that you’d sort of be in the ballpark with machine learning operations”. Some specific items that he points out as evolving in the field include:
- How do you better handle model change management?
- How do you automate the monitoring of the model performance?
I’ll add two additional observations into ML Ops evolving trends:
- Machine learning systems are more frequently being recognized and treated as full-fledged products. This product life cycle encompasses the project life cycle plus the operations and looks to maximize value throughout the entire system (as opposed to delivering a model, closing the project, and leaving it be). Learn more with this post about Data Science Product Managers.
- Society is demanding more accountability from artificial intelligence. Thus, ethical protocols and legal regulatory compliance are becoming increasingly more important to operating a model. Ask yourself 10 Ethics Questions in Data Science.
Where can you learn more?
- MLinProduction.com – Luigi’s blog provides several best practices and tactical advice on how to build and maintain real-world machine learning systems.
- ml-ops.org – This intuitive site explores a lot of the conceptual and high-level aspects of machine learning operations.
- Why Machine Learning Models Hate “Change” – This Towards Data Science post provides a short and simple overview of model drift, one of the key elements to look out for in operational models.
- Practitioner’s Guide to ML Ops – Google’s detailed 37-page whitepaper outlines the machine learning operations life cycle and its core capabilities.
- What is the Data Science Life Cycle? – Our post provides a view into steps into the development and deployment of a data science model.
- Data Science vs Software Engineering – Another one of our posts. It helps clarifies the differences and similarities between these fields.
- The Data Science Team Lead course – This will help you better understand how to deliver value from your data science projects.
Deliver Data Science Outcomes
Machine learning operations is just one piece to the overall puzzle of delivering sustainable, actionable outcomes from your data science investments.
Empower yourself with the data science-specific project management tools to deliver these outcomes.