Last week, I talked about Champion / Challenger and how this technique compares alternative strategies for a decision checkpoint. This week, I would like to talk about another usage for this technique. Using the same infrastructure, the same idea, you can test in Production your new and updated decision logic. This time it is not for evaluating its business performance. It is for testing it in Production, or what we call rolling out your deployment.
Rolling out a deployment… Why?
As part of the software development life cycle (SDLC), a critical aspect is testing. You do not want your software released out there without any quality assurance, of course. While many go over their test cases and more in the QA environment, it is frequent to roll out deployment in phases. When the software in question supports mission critical business operations, it is key to ensure it is performing as expected. There is only so much you can test in QA.
In the past, I have seen projects deployed to one state or geography in the first phase. If everything goes well, then the software gets deployed more widely. Some projects shadow the current execution with the new execution, just logging what the answer would have been, for an offline examination. I have also seen projects that were deployed to a segment of the incoming transactions, this time more randomly assigned.
Rolling Out a Deployment… How?
When looking at how to roll out deployments in more detail, you will inevitably see the parallel with Champion / Challenger. You have one champion: the existing service implementation. You have one challenger: the new and improved service implementation. Please do not feel confused by the term implementation. While there is software running, we are really talking about your old decision logic and your new decision logic. Business rules, maybe models, have changed, but the software implementation has not changed in our scenario. I am not talking about deploying actual software code.
At this point, you want to deploy the new business rules to only a fraction of your incoming business. Let’s say that you might want to start with 5% of your transactions. This is very similar to a Champion / Challenger setup with 95% of the transactions going to your champion, and 5% going to your challenger. You can monitor for days or weeks that the challenger segment behaves as expected. Then you can augment to 20% or 50%, and so on until you reach 100%.
Why would you use Champion / Challenger for that?
That’s a good question. Let me ask you this though: why not? Without being facetious, the alternative is to write the code infrastructure by hand. This means that you need to solve the issue of running two different releases of the same decision services concurrently. SMARTS, our decision management system allows you to do obviously — we had to solve that problem to enable Champion / Challenger. But other decision management systems might require that you clone your repository or something like.
Secondly, you need to hard-code the parameter of the Champion / Challenger experiment, aka the volume of transactions going to your current decision logic, and the volume of transactions going to your new decision logic. That part if not hard, but a change to this parameter implies a full software development life cycle. When you upgrade from 5% to 20%, you will need software engineers to make the change, and you will need QA to test, and you will need a formal production release. This can be heavy. A Champion / Challenger change only requires as much testing you would go through for a rule change. Wasn’t it the reason you decided to use business rules in the first place?
Finally, you will need to put in place a mechanism to keep track of the path you took, and whether it met your standards. Granted, you could simply check that transactions are processed without crashing the system… But I am sure you have higher QA standards. Champion / Challenger tracks out of the box which path was taken for each transaction.
Food for Thought
Rolling out your deployments is a step in the direction of more business transparency in the quality of your decisions. Once set up, it does not take much to start monitoring the business performance of your new decision logic. Why not start using Champion / Challenger?
Learn more about Decision Management and Sparkling Logic’s SMARTS™ Data-Powered Decision Manager
