MLOps: An “Ops” Just for Machine Learning

If you’ve been around software development for a while, you’ve undoubtedly come across numerous terms appended with the word “Ops”, such as “DevOps”, “TestOps”, or “DataOps”. Of these, “DevOps” (short for “development and information-technology operations”) is probably the most well-known. It refers to a set of software development practices that promote automation and cross collaboration between teams of different disciplines, to reduce software delivery times while achieving a desired level of quality.

And now, with the growth of machine learning (ML), a new term called “MLOps” (short for “machine learning operations”) has been coined. And while you may frown at having to remember yet another “Ops” term, it’s important to understand the needs for a DevOps-like approach that addresses the unique issues, requirements, and challenges presented by ML.

Continuous Acronyms

Originating from the DevOps era, a number of methodologies are commonly employed including: Continuous Integration (CI), Continuous Delivery (CD), Continuous Deployment (also referred to as “CD”), and Continuous Testing (CT) that seek to formalize and automate their respective areas of focus.

MLOps embraces continuous integration and continuous delivery, but adds another important “continuous” methodology (and acronym): “continuous training” (also referred to as “CT”). Continuous training seeks to automatically and continuously retrain the model to support continuous delivery in an ML system. This involves the creation of a pipeline to automatically validate and train the data and model, and the development of a “trigger” that forces retraining.

The Machine Learning Workflow

To better understand the needs that these methodologies and MLOps address, let’s first consider a typical ML workflow:

An ML workflow begins with a “Data Management” phase, where training data is acquired, organized into a useful form, and may be labeled (e.g., for supervised learning) so that it can be classified by an ML system. Following this, a “Model Management” phase takes place, where an ML model is built, evaluated and tuned for behaviour and performance, and a version of the trained model is stored and made ready for deployment. Finally, “Serving Management” involves the deployment of that trained model into the real world, where it’s made available for inference, continually monitored, and its results are communicated to the business owner and back to the ML team. Note that “serving” can range from hosting a model in the cloud to embedding the model into a mobile device.

With MLOps, and more specifically continuous delivery and continuous training, numerous tool chains may be used to perform and even automate these processes, and are often referred to as “pipelines” when combined. For example, an organization might set up a pipeline to automatically train and deploy a model. And since models often suffer from “model decay”, additional automation can be configured to automatically trigger retraining and deployment of a new model, using updated data collected from the real world.

ML workflows are iterative, with the need to continually update and deploy new versions of the model. On top of this, the nature of ML workflows presents a number of unique requirements and attributes that necessitate formal approaches to successfully deploy enterprise-level ML projects. Some of these requirements and attributes include:

• The non-deterministic nature of ML which complicates testing as well as predictions as to how well a model will work, and continue to work, in the real world.
• The need to deploy a “trained” model, often in addition to code that runs inference on the model.
• New and different types of project collaborators that can span data scientists, ML engineers, ML Project Managers, ML researchers, developers, and even the business owner(s) of the system.
• The exploratory and experimental nature of model design, which is intrinsically iterative, and can evolve over time even after deployment.
• “Model decay” that occurs as the profile of real-world data changes, necessitating the need to retrain models or even redesign them.
• The need to explain how the ML model works.

MLOps Levels of Maturity

As with DevOps, the level of “maturity” of an organization’s MLOps can often be measured by the degree to which the organization’s processes and tools have been formalized and automated through methodologies like continuous integration, continuous delivery, and continuous training. In fact Google’s MLOps: Continuous delivery and automation pipelines in machine learning article formalizes MLOps into three distinct levels:

• MLOps level 1: generally describes organizations who are starting out with ML and do things manually. They may have little or no MLOps in place, meaning they may have little or no automation, and possibly limited collaboration between those involved with ML (e.g., data scientists and developers).
• MLOps level 2: organizations who have “more mature” MLOps. They may have developed automated pipelines for certain aspects of the ML workflow, such as a pipeline to automatically train and deploy a model.
• MLOps level 3: organizations who are fully engaged with MLOps. They have automated so much of the ML workflow, that data scientists can update some or all of the pipeline(s) with little to no developer intervention.

Of course MLOps is just as much about collaboration and formalizing processes as it is about automation. Today’s ML requires that teams be able to understand the inner workings of their models, why the models are producing certain results, and to have the ability to reproduce behaviours, while being able to easily iterate on new models with new data. In addition, cross-discipline collaboration requires tools, processes, and pipelines that are accessible and usable by all contributors in the ML workflow.

Where PerceptiLabs Fits In

PerceptiLabs visual modeling tool currently supports the “Architect and Build”, and “Evaluate & Tune” phases of “Model Management” in the workflow depicted above, offering a number of features and benefits that help organizations in their MLOps efforts.

The visual nature of PerceptiLabs provides a common environment and “language” around which multiple contributors, ranging from data scientists to ML engineers and project managers, can collaborate on models–a key foundation for both DevOps and MLOps. Data scientists can build and experiment with models with varying levels of help from developers (depending on their technical abilities), and view a rich set of statistics about the model’s performance. Developers can see exactly how the model has been designed, how each object is transforming the data, and view and edit the model’s underlying code. With this knowledge, developers can also develop and deploy any external code (e.g., JavaScript) that uses the model to perform inference, in lockstep, such that the model and external code can become formally packaged together.

Example of a simple neural network model in PerceptiLabs, and a popup to set parameters.

From this, all contributors can easily repeat the whole process of iterating on the model by adjusting parameters, modifying the model’s code, training the model, and viewing statistics about the model’s predictive performance.

These features are important for continuous training in MLOps, which requires that the model’s architecture and trained model be validated, in addition to running the “code-level” tests (e.g., unit tests, integration tests, etc.) traditionally performed in the DevOps era. Moreover, with continuous training in MLOps focused on automating the retraining of deployed models, PerceptiLabs’ inherent transparency helps stakeholders to quickly understand and compare the results of new training iterations.

PerceptiLabs’ drag-and-drop approach also enables continuous delivery by making it easier and hopefully quicker, to validate a model before deployment, or to develop semi-complete but functional experimental models for more rapid and continuous integration. And as new data becomes available, the process can be easily repeated, by feeding the new data into the model, and easily adjusting any parameters or code to accommodate changes in data schemas or business requirements.

Example showing some of the statistics presented in PerceptiLabs.

One key aspect to continuous delivery in MLOps is the notion that you’re developing more than just a single output (i.e., a software), but rather two outputs: a system for training and a system for inference. PerceptiLabs clearly contributes to the system for training, while trained models that have been packaged with external code, form the foundation for inference systems that will be deployed.

Currently, PerceptiLabs visual modeling tool outputs trained TensorFlow model files. In terms of continuous integration and delivery, a pipeline can be easily built to detect and pickup changes to these model files and the “source files” saved by PerceptiLabs as a new “version”, and automatically deploy them. Likewise, additional triggers can be built to automatically notify stakeholders when model decay is setting in. Perhaps most importantly though, is that both the visualization provided by PerceptiLabs, in conjunction with the packaged output, advances the explainability of an ML system, which is important should an audit be required.

So, just as traditional software development resulted in the evolution of “DevOps”, the unique aspects of ML are leading to the creation of “MLOps”. Despite its infancy, MLOps is gaining both recognition and popularity as ML grows, and ML workflows are sure to evolve as will PerceptiLabs’ solutions. And while we can’t eliminate the need to remember yet another “Ops” term, we are happy to say that PerceptiLabs is accelerating machine learning by helping teams architect, train, and generate trained models as part of their own “MLOps”.

For some practical tutorials on getting started with PerceptiLabs, be sure to check out the PerceptiLabs Video Tutorials.