Sonamine has been deploying production grade machine learning for 15 years.  The projects have spanned the gamut from games to video streaming to national news publications. The platforms have also ranged from Console to PC to mobile.  We are proud to say that except for two customer who had layoffs and canceled projects, all our machine learning models made it into production, many of them for years.  

To Sonamine, productionizing a machine learning model does not end with generating inferences or scores in a scalable repeatable fashion.  Nor does it end with one single iteration of a model training run, or several for that matter.  A machine learning model is in production when it is demonstrably adding value in an existing business process.

From this vast experience, I have distilled 3 key lessons that we have learned.  My approach is to try to make sure these lessons are not overlapping.  In other words, each lesson can apply even when the other two do not.  

Here are the quick takeaways with greater detail in the subsequent paragraphs.

- Keep the focus on the business goal.
- Avoiding the fire-and-forget trap.
- Don't get stuck on statistical significance

Focus only on the business goal

This lesson is on making sure machine learning is not just a research project.

All machine learning models are tools and always incomplete.  So why use them?  The reason is that sometimes they can be useful to achieve some business goal.  This means that the machine learning output must be integrated into a business process seamlessly.  The theoretical exercise of building a model, while interesting in a research setting, is a waste of resources in a commercial setting.

However many teams focus solely on the data and machine learning aspects, while ignoring the business processes.  This is understandable, AI and data are exciting and new; business processes like running marketing campaigns can be quite mundane.  AI and data can be levers to opening the budget wallet.  While "improving business process" lacks that same allure.

The lesson here is to make sure that machine learning is used as a means to an end.  And the end goal is a better and more performant business process.  In practice this means we must ask the data scientists to learn and understand the business processes, not the other way around.  I have seen countless teams where the data team laments that the business team does not understand data, or can't write SQL.  If the data team does not try to learn and understand the business process, the project is almost always DOA.    

We keep the focus on the business goal, by highlighting the business process in every meeting.  Every data related meeting has a checkbox that asks "how is this relevant to the business process we are improving"?  If it is a content recommendation model, we ask how the front end is going to handle different the model output.  If it is a churn mitigation, we ask how the end user is going to be affected by the model output.  And so on. Every meeting.  If the data team cannot answer that, we end the meeting early.  

‍

Avoid the fire and forget trap

This lesson focuses on keeping the project going after the initial release.  

Many vendors and tools pitch their products as "self-learning".  You never have to check it once you have it running; that AI recommendation from the engagement tool, or the auto user segmentation is always accurate.  At least that is the claim.  It's a good sales pitch for naive customers, but most customers these days know that is too good to be true.  

The first lesson helps us avoid this trap if we are constantly keeping an eye on the key performance indicators.  Whether it's  watch depth of a recommended video, retention rate, spender conversion percentage or time spent read articles, reviewing these KPIs daily or weekly will keep the project on its toes.  Better yet, schedule a A/B test every month to make sure the model, the creative or any other part of the business process is as optimal as it could be.

Since the machine learning project is a multi-stakeholder project, one needs to avoid becoming stagnant or complacent in every step of the project.  On the data side of the project, keep checking for new user capabilities, data tracking and theoretical model performance metrics.  How frequently is the modeling training occurring? How are new data elements incorporated into the data pipeline? On the business process side, look for content staleness and increasing error rates.  Do new trends in marketing make our ads look dated?  

Stuck on Statistical Significance

This lesson is on not making bad business judgements based on limited statistics understanding.  

Many college students graduate with some notion of what statistical significance means. Every one "undestands" that to be real, findings must be significant at the 95% level.  Because this is repeated in the press and web, most machine learning projects require some KPIs to meet this threshold.  

There are at least two additional statistical concepts that are important to keep in mind.  One is the effect size.  Many experiments that involve statistics compare whether there are differences between two different groups.  The effect size is a way to compare how big the differences are.

Another is power.  Power is a way to tell whether the statistical test used will be capable of correctly telling you there is a difference between the groups.  The higher the power, the more likely the statistical test will be able to tease out a difference in the groups.

The judgement call on whether to continue the business value focused project usually comes to some combination of the three.  Focusing on only one can lead to some funny situations.   For example, say a machine learning enhanced business process has significance compared to a control, but the effect size is only 0.00001%, and the power is high.  It would be ludicrous to continue with the project since the effect size is so small.  Similarly it would be irresponsible to ignore the result if the effect size was 100% and the significance was "only" 80%.

‍

Remember to sign up for our newsletter.

Exploring how to use machine learning to solve your conversion, retention and engagement issues?  Contact us.

‍