What are Champion / Challenger Experiments in Decision Management?

on February 10, 2014
Champion-Challenger race

Have you ever heard of Champion/Challenger? If you are in the Financial Services industry, chances are you have and you might even have used this technique. If not, I wouldn’t be so sure you have.

What is Champion vs Challenger?

In a nutshell, in champion/challenger, you compare two or more strategies and promote the one that performs the best. The champion strategy is the current one while the challengers are the new strategies you would like to test.

With regulations or contracts, your business rules are pretty much set in stone and very little is left to interpretation. However, when you are making decisions like product recommendations or credit limit increases, there is no one right answer. In these cases, you have to deal with some degree of uncertainty. Should you target users based on age or type of smartphone? Should you set your threshold here or there?

In order to know which strategy performs the best, you have to actually test them in real life. Business rules make it easy to switch strategies. Even so, swapping them in production may introduce a level of risk that is not reasonable… It may also be tedious and biased to try one strategy, then another, then another sequentially… Not to mention the inconsistent behavior as seen from the outside…

Champion/Challenger addresses all of these issues by deploying all of them at once. You might wonder how this can be done since you usually have to pick ONE decision, not a bunch of them.

Champion Challenger is Like A/B Testing

You might be familiar with the concept of A/B testing in advertisement… The idea is to publish different ads to a market segment and measure which one performs the best. In order to protect the integrity of the experiment, you have to randomly select who gets A and who gets B, and potentially a few more alternatives. On websites, marketers try different headlines, styles, or content.

Champion/Challenger does the same. The champion is your Production strategy (aka decision logic, business rules). You can make it compete with one or more challengers (aka variants of the decision logic). You do have control as to how many transactions with go through one or another of these strategies within a segment, but you will let them be selected at random. As Champion/Challenger selects the strategy, it keeps track of the assignment for measuring its performance. You can track real-time how well each strategy is performing, and possibly shut down or tweak some experiments that are under-performing.

After minutes, days or weeks, you have gained enough evidence that one of the challengers is doing much better than the others, and promote it as the new champion. You may want to create new variants to compete against it once again.

What is a Champion Challenger Model?

In ModelOps or MLOps, it is common and necessary to replace over time. Strategies change and models will also naturally deteriorate. Rather than simply replacing the current model, Champion/Challenger enables you to test in production before your replace. Like with strategies, the current model is the champion model and any new models are the challenger models. New models can be a retrained version of the champion model or a completely new one. With a decision management platform like SMARTS™, not only can you run Champion/Challenger experiments on models, but you can also run them on any aspect of the decision logic (rules, scorecards, etc).

Flexibility in Experimental Design

The beauty of this mechanism is that you have full control over the percentages. If you are risk-adverse, you might want to run experiments on only 5% of your portfolio and exclude your VIP customers. If you are more aggressive, you might want to spread your eggs into multiple baskets equally and get faster results.

Knowing which strategy was applied is critical for consistent behavior over time. In some cases, regulations force you to keep track of the rules that were in force at the time of the decision so that you reuse the same business rules if you need to re-process the application. Even when regulation is not imposing it, you may want to run experiments across decision services through a customer session, or across sessions for a better user experience. Champion/Challenger provides the infrastructure to do just that.

Why can’t I just run a simulation like before?

Well, referring back to the illustration, the alternative is a coin toss when it comes to changing your strategy.

Complete coin toss? I sure hope not. I am sure that you do a fair amount of testing before publishing your business rules in Production. By running your historical transactions through several variants of your decision logic, you can get an idea of their business performance, assuming that past behavior is consistent with current behavior of course.

There are some aspects of your decision that you cannot anticipate in simulation though.

For example, you can check how many times you would have proposed offer A versus offer B, let’s say a free smartphone versus 12-month discounted service. But you cannot easily estimate how many customers would have accepted one offer or the other. The only way to know for sure is to actually offer the new ‘product’ to a some of your customers and see how they react.

How hard is it really?

Not as hard as you think… We believe that you should focus on what you want to experiment with, and not on what the plumbing looks like to assign the strategies and keep track of it. So the work is really just to define the variants and ‘sign them up’ for an experiment. The rest is magic!

See our most recent posts on Champion/Challenger experiments: “Champion/Challenger, It’s a Numbers Game” and “Another Usage for Champion/Challenger: Rolling Out Deployments“.

Learn more about Decision management and Sparkling Logic’s SMARTS™ Data-Powered Decision Manager

Search Posts by Category

ABOUT US

Sparkling Logic Inc. is a Silicon Valley-based company dedicated to helping organizations automate and optimize key decisions in daily business operations and customer interactions in a low-code, no-code environment. Our core product, SMARTS™ Data-Powered Decision Manager, is an all-in-one decision management platform designed for business analysts to quickly automate and continuously optimize complex operational decisions. Learn more by requesting a live demo or free trial today.