MLOps

Overview

Our MLOps framework consists of three stages, each aimed at establishing a systematic machine learning operationalization process, validating model performance, and minimizing technical debt. By adopting complete DevOps practices at both a person and process level, we help organizations deliver and integrate ML models continuously in a production environment.

At Allerin, we believe that a disciplined approach to the model development cycle is key to promoting model integrity and robustness. That's why our MLOps services follow a variation of the cross-industry standard process for data mining (CRISP-DM) methodology, a simple and powerful development discipline that supports the creation of a baseline governance framework for technical and business dependencies.

Our MLOps services empower organizations to optimize their model deployment and maintenance. We offer comprehensive capabilities from feature curation, management, governance (via ModelOps), release activation & monitoring - leading in the way of minimizing operational failures as well as boosting transparency and sustainability for clients' machine learning models.

Our MLOps framework consists of three phases, the first of which is Model Development and Validation.

Model development and validation

At Allerin, we use a logical approach to feature stores to address the need for feature reusability, reproducibility, and reliability in ML portfolios. We store code used to create features for ML, version it, and accompany it with metadata. This approach enables us to create a repository of features from which users can pull training and testing datasets, accelerating ML model development. These feature stores can also orchestrate feature transformations and monitor data-serving models in production.

To ensure successful MLOps, we introduce validation checkpoints throughout the development and operationalization cycles of a project workflow. Collaboration among all team members and the ability to reproduce results with the model are key aspects of successful validation. Our model testing and validation step in the development cycle includes business and technical validation, ensuring that the model is developed based on specific desired business outcomes, operates within specified thresholds, and satisfies all interpretability requirements.

Model testing and validation are necessary steps before introducing the model to a preproduction environment. At Allerin, our MLOps framework offers several mechanisms and guardrails that need to be introduced based on the model type, model usage, domain, and industry-specific metrics. For instance, the testing of an NLP model would differ from a computer vision model, where an NLP model would be tested for minimum functionality, invariance, or directional expectation. In contrast, computer vision models are highly domain- and use-case-specific. NLP model testing is also very use-case-specific as the models and training and validation data vary based on the purpose of the model.

Model Operationalization Cycle Process

Allerin's Model Operationalization Cycle Process is designed to maximize the effectiveness of ML models in practical applications. It facilitates a seamless release and activation cycle, empowering businesses with streamlined data-driven decisions and automated tasks for optimal results. Through this process, companies gain invaluable insights from their deployed ML models - leading them on the path toward greater success.

After diligent development, the model is put to the test for verification before it can be released into held production. We must ensure that its version control and deployment are in alignment with business needs, followed by smooth integration of said model within our existing infrastructure - granting stakeholders and applications access when necessary.

The benefits of Allerin's Model Operationalization Cycle Process are significant, as it enables organizations to achieve their business objectives and improve their KPIs. It ensures that ML models are integrated seamlessly into the production environment, reducing the risk of errors and providing accurate insights. It also facilitates faster deployment, enabling businesses to react quickly to changes in the market and stay ahead of the competition.

Release Phase

The release phase is a critical sub-phase of the Model Operationalization Cycle Process that ensures the published model works effectively in real-world business conditions. Allerin's MLOps follows a comprehensive 7-step process during the release phase to guarantee model performance, minimize risks and enable successful deployment.

• Model Risk Review: The first step is to retest the machine learning model to ensure that it does not pose any business risks and has no open backdoors that can be exploited by attackers. A thorough security review of the model is necessary to ensure the safe operation of the systems. This includes built-in protection mechanisms against data poisoning, adversarial attacks, query attacks, and model manipulation attacks.
• Model Release: Once the model has passed the risk review, it is promoted to the release phase, where it is ready to be handed over to the operationalization team. At this stage, the model is labeled as a candidate model, which means that it has been development-vetted but is not yet fully production-ready.
• Endpoint Identification: This step involves validating the decision points where the machine learning model will deliver its insights. These endpoints could be within an existing application, a business process, an ensemble of models, or an input to another decision-making mechanism.
• Parameter Testing: Target business processes may be subject to technical constraints, which means that the velocity, shape, volume, and quality of the input data may not align with the data used to develop the model. This step tests the alignment of the target business processes and input data.
• Integration Testing: Assuming that the expected data matches the development assumptions, integration assumptions (such as REST APIs, microservices call, and code integration) also have to be tested to ensure proper performance.
• Instantiation Validation: As models in production are often part of model ensembles, even slight variations in those elemental models (such as connected models instantiated across multiple states or regions in the same country) can produce radically different results. This step ensures that the models are instantiated correctly in production.
• KPI Validation: Model performance should not only be measured against technical parameters (such as precision) but also against the agreed-on KPIs. Validate models for business drift, mathematical drift, and data drift to ensure that the model is still performing as expected and delivering value to the business. Business drift refers to deviations in business KPIs, mathematical drift refers to deviations in technical parameters, and data drift refers to shifts between the data used to build the model and the actual data in production.

Activation Phase

The Activation Phase is a critical step in the MLOps framework and involves operationalizing the ML model tested and validated in the previous phases. This phase comprises seven steps that must be followed to ensure the successful deployment of the ML model.

The first step is Management and governance. In this step, the ML model that is ready for activation is cataloged, documented, and versioned. This ensures that the model complies with the governance rules adopted by the data science team and the production committee, including the application product managers. The ML model is continuously integrated, deployed, tested, and verified while ensuring reproducibility, reusability, and reliability with the help of Allerin’s MLOps management and governance feature.

The next step is Model activation. The validated models are handed over to the production team as activated models. The models are fully documented, production-ready, and compliant with the governance rules.

Model deployment is the next step. Depending on the execution environment, such as on-premises, in the cloud, or both, measures need to be taken to ensure smooth processing of the transactions leveraging the model or the activated model's ensemble.Allerin’s MLOps framework supports multiple deployment strategies that can be undertaken depending on the ML model or application, but they’re broadly classified into following stages

1. Static Deployment:

• Basic deployment involves replacing the current ML model with the new model.
• Recreation involves scaling down the current version before scaling up the new version.
• Blue-green deployment has the current model live in the blue environment, while the green environment contains the new model for testing against live data. After testing, traffic is shifted to the green environment, and the blue is kept as a rollback or decommissioned.
• Canary deployment incrementally rolls out the new model until full deployment is achieved.
• A/B testing deploys different versions of the model with different features in production to compare performance and increase the conversion rate, and the best model is selected for deployment.
• Champion-challenger deployment has multiple models running in parallel against live data, competing for the best model performance, with the champion model always deployed.

2. Dynamic Deployment:

Multi-armed bandits is an advanced version of A/B testing that dynamically adjusts the traffic distribution based on model performance, allowing for faster deployment of the better-performing model.

Choosing the right deployment process can be tricky, but making a careful selection is essential to achieving business objectives. Evaluating both advantages and drawbacks will ensure that you find the option best suited for your needs.

Application integration is another crucial step in this phase. Insights are delivered through decision endpoints, which are mostly part of existing applications. Sometimes extra coding is required to embed the model or its input within an application, business process, or another set of insights to enhance the business outcome. This is where the model finally delivers its business value.

Production audit procedures are implemented in the next step. Model telemetry and various performance metrics such as accuracy, response time, input data variations, and infrastructure performance instrumentation are gathered from monitoring models in production. Various instruments such as A/B testing methods, multivariate testing, multi-armed bandits, and shadow models can be implemented to appreciate the performance of models.

Model behavior tracking is critical to track the model's performance in production. Alerts and monitoring methods are established to track performance thresholds and notification mechanisms to systematically flag any divergence or suspicious behavior.

Finally, KPI validation is extended from the release phase and fed by the two previous steps. It consistently measures the business contribution of the models or ensemble models in production to determine the business value that can be attributed to the model.