Integration with data is key to a successful decision application: Decision Management Systems (DMS) benefit from leveraging data to develop, test and optimize high value decisions.
This blog post focuses on the usage of data by the DMS for the development, testing and optimization of automated decisions.
Data is necessary at every stage of the life-cycle an automated decision. In particular,
- during development, you will incrementally build the business logic with simple test cases.
- while testing, you will use more elaborate test cases to fully check the decisions being made. If available, you will use existing production data to check their quality.
- in production, you will use data to make live decisions, and you will leverage the resulting decision data to improve the automated decisions in their next iterations.
Usual DMS implementations focus purely on the logic itself: flows, rules, etc. In some cases, they may allow you to start from a schema for the data.
However, we believe assessing the quality of the logic is as important, if not more, than implementing it according to the requirements. You can only assess the quality of a decision by measuring it. In other words, you’ll need to leverage the decision analytics capabilities of the DMS. And that, in turn, requires data on which to compute these analytics. The more important your decision, the more important that you measure its performance early.
Carole-Ann has written about this in the best practices series: Best Practices Series: Business Testing. The earlier you leverage data, and measure the quality of your decision, the better. If you can start with data, then the definition of the metrics you will use to assess the quality of the logic will put you in a much better position to implement decisions that won’t need to be corrected.
Starting with some sample data
Using sample data
You may start building a decision without any prior data and only referring to the schema for the data. But such an approach does not let you measure the quality of your decision –you can only be sure the decision is consistent with the schema, and consistent with the written requirements. However, that is not enough. The decision may be properly implemented from a technical perspective but totally fail from a business perspective. For example, the logic for a loan application may end up sending too many cases to manual review.
Furthermore, business analysts think in terms of logic on data, not schemas. Having data available to interact with as you develop the logic keeps you in your mindset, and does not force you to change to a programmer’s mindset.
For example, you will have some simple test cases which will help you in the implementation of your first few rules. If you already have lots of data, take a subset that would be relevant for the decision at hand. When test cases are small (in the tens of thousands of records at most), then having the DMS manage the data makes sense –in particular if that data is only used for decision development.
As the construction of the automated decision progresses, you will want to add more data for functional checks. You will then perhaps uncover new cases requiring more business logic. The DMS will ideally allow you to associate multiple such data sets to any decision.
Consequences for the DMS
To support this incremental build-up of the automated decision, the DMS will:
- provide support for managing data sets associated to decisions
- add new data and data sets on the fly
- support data formats commonly used in the enterprise (such as CSV, XML or JSON)
- provide decision analytics facilities to verify the quality of the decision using the data sets
One key challenge with using data is that data changes. We view that as an opportunity –in general, data changes because new data items are discovered and used to enrich it. Those new data items create further opportunities to implement better and richer decisions. It is thus very important that the DMS support easy updating of the data.
Moving forward with large data sets
Using large data sets in simulation and decision analytics
Once the implementation of the automated decision gets to a stable state, you will need to have access to more and more test cases. You will also want to determine whether these test cases are successfully executed.
When large data sets are available, the simulation capabilities of the DMS will verify the behavior of the automated decision, and give you relevant quality information through decision analytics. Usually, you will do this to determine how the decision behaves in general. Different variations of the same automated decision may also compete against one another to find out which variant behaves best. Leveraging these, you can ensure your decision improves in quality, or that your champion-challenger strategy, for example, is safe.
These data sets typically come from your data lakes, potentially fed from operational data. They may consist of large enriched historical data that you keep around for reporting, analysis, or machine learning purposes.
Consequences for the DMS
For such large data sets, you will want the data management to remain in the data lake, data mart or enterprise data environment. Duplicating data management for large data sets is expensive and potentially runs into security or compliance issues.
Thus, the DMS will ideally provide means to integrate with these without copying it or managing it within the DMS. Predefined connectors to databases, or data lakes can be useful for simpler access to existing stores. But a generic means to access data, by means of a web service, or using any standard data format through the FTP or http protocols, will guarantee access to anything.
Furthermore, the data sets can be very large. The DMS will ideally provide a simulation and decision analytics environment where the decisions and the metrics are computed without imposing boundaries on the size of the data set. For example, by providing a map-reduce and streaming simulation and decision analytics facility.
The DMS will:
- provide mechanisms to access enterprise data on-the-fly
- provide scalable simulation and decision analytics facilities that can cope with enterprise-size data sets
The enterprise will:
- manage large data sets relevant to the decision
- use the DMS facilities to leverage these large data sets to verify and enhance decisions
Improving the decision using large operational data sets
Using operational data
When the DMS executes an automated decision in a production system, you will want to collect interesting data points. You will then use them at a later time to determine why some results were obtained. You will also use them to improve these results by updating the decision.
Typically, your enterprise environment will include operational data stores, and data lakes where the data is merged for reporting, analysis and machine learning purposes. The more sophisticated enterprises will also include decision outcome databases and correlate business outcomes with decision outcomes in the data lakes.
Take the example of an application that offers promotions to an existing subscription customer. A good decision is such that its outcome is correlated to the fact that the customer:
- opened the offer
- accepted the offer
- is still a customer 6 months down the road
- has not had negative interactions with tech support in the 6 months following the offer
Using this data, you will be able to keep improving your decision and its value for the business. You can also use machine learning tools and extract knowledge from the accumulated data and incorporate it in your decision.
Consequences for the DMS
The DMS will:
- support storing decision outcomes
- provide mechanisms to access data lakes, operational data stores on the fly
- offer simulation and decision analytics facilities that scale
- ideally support model creation and/or model execution
The enterprise will:
- manage large data sets relevant to the decision
- use the DMS facilities to leverage these large data sets to verify and enhance decisions
This blog is part of the Technical Series, stay tuned for more!
In a later blog post, we’ll cover the various strategies to pass data to decisions at run time, including the retrieval of additional data while the decision executes.
In our last post, we looked at how predictive models are used in automated decisions. A key take away from that post is that a prediction is not a decision. Rather predictive models provide us with key insights based on historical data so we can make more informed decisions.
For example, a predictive model can identify customers that are likely to churn, transactions that are suspicious, and offerings and ads that are likely to have the most appeal. But, based on these predictions, we still need to decide the best response or course of action. A decision combines one or more predictions with business knowledge and expertise to define the appropriate actions.
From Predictions to Decisions
Determining how to take action based on predictions is not trivial . Most likely, there are multiple business options for the actions an organization could take based on a prediction. Consider for example charges that are identified as potentially fraudulent, a card issuer could report the case to the fraud team for further investigation, shut down the card to prohibit further charges, or text the cardholder to verify the charge.
Or in the case of customers who have been identified as having a high probability of switching to a competitor, a company may decide to contact them with a special incentive or renewal offer. But the company still needs to decide exactly how many customers will receive the offer and how much to offer. The company could target a flat percentage of customers, or could focus only on those with the highest projected CLTV (customer lifetime value). Targeting too many customers with too large off an offer might be too expensive to be worthwhile. The possible actions an organization can take based on predictions have different costs and benefits that need to be evaluated to determine the optimal decision. This is where decision simulation is applicable. Decision simulations help you identify the best decision strategy from amongst a set of alternatives.
Measure Your Decision Quality with KPIs and Metrics
The “best” decision strategy means the one that most closely meets your organization’s objectives. By defining KPIs and metrics that measure the quality of the decision in relation to these objectives, we have a basis to compare alternative decision approaches. Ideally these KPIs were identified early on, when you first decided to automate the decision.
Decision KPIs give us a clear understanding of how decision performance is related to business performance. They provide the basis for evaluating decision alternatives. To compare alternatives, you can run simulations using historical data. Using simulations you can compare one decision strategy to another, or you can compare how a given strategy performs on each of your customer segments as represented in your data.
Returning to the above customer churn example, we may decide we want to target customers who have an 80% or greater probability of churn based on our predictive model. One option would be to offer them a special 25% discount to attempt to re-engage and keep them as a customer. We can run a simulation against our historical data to learn how many customers fall into this bucket. From there, we can evaluate how much the discount offer would likely cost us. We can run multiple simulations using different thresholds, offers, and combinations until we find the best decision approach to deploy.
Decision Simulation Helps You Evaluate Alternative Decision Strategies
Decision simulations help us evaluate alternative decision strategies to narrow to the best approach. Modern decision management technologies, like SMARTS, make it easy to set up and run these simulations, even on very large data samples. Of course, the ultimate quality of the selected decision approach is related to its success once deployed- how many customers do we manage to retain and at what cost?
Once we deploy a decision we can monitor and track the KPIs but we have no way of knowing whether customers who did not accept our offer would have instead accepted a different offer. Or whether customers who did accept would also have accepted a 20% rather than a 25% discount. To answer these questions we need to use Champion / Challenger experiments. We’ll cover how Champion / Challenger works with decision management in an upcoming post.
Agility is a key focus and benefit in the discipline of decision management. Agility, in the decision management context, means being able to rapidly adjust and respond to business and market-driven changes. Decision management technologies allow you to separate the business logic from your systems and applications. Business analysts then manage and make changes to the business logic a separate environment. And they can deploy their changes with minimal IT involvement and without a full software development cycle. With decision management, changes can be implemented in a fraction of the time required to change traditional applications. This ability to address frequently changing and new requirements that impact key automated decisions makes your business more agile.
Being able to rapidly make and deploy changes is important. But how do you know what changes to make? Some changes, like those defined by regulations and contracts, are straightforward. If you implement the regulations or contract provisions accurately, the automated decision will produce the required results and therefore, make good decisions. However, many decisions don’t have such a direct and obvious solution.
When Agility Isn’t Enough
Frequently decisions depend on customer behavior, market dynamics, environmental influences or other external factors. As a result, these decisions involve some degree of uncertainty. For example, in a credit risk decision, you’re typically determining whether or not to approve a credit application and where to set the credit limit and interest rate. How do organizations determine the best decisions to help them gain customers while minimizing risk? The same applies to marketing decisions like making upsell and cross-sell offers. Which potential offer would the customer most likely accept?
Predictive Models Provide Data Insight
This is where predictive models help. Predictive models combine vast amounts of data and sophisticated analytic techniques to make predictions about the future. They help us reduce uncertainty and make better decisions. They do this by identifying patterns in historical data that lead to specific outcomes and detecting those same patterns in future transactions and customer interactions.
Predictive models guide many decisions that impact our daily lives. Your credit card issuer has likely contacted you on one or more occasions asking you to confirm recent transactions that were outside of your normal spending patterns. When you shop online, retailers suggest products you might want to purchase based on your past purchases or the items in your shopping cart. And you probably notice familiar ads displayed on websites you visit. These ads are directly related to sites you previously visited to encourage you to return and complete your purchase. All of these are based on predictive models that are used in the context of specific decisions.
How Predictive Models Are Built
Predictive modeling involves creating a model that mathematically represents the underlying associations between attributes in historical data. The attributes selected are those that influence results and can be used to create a prediction. For example, to predict the likelihood of a future sale, useful predictors might be the customer’s age, location, gender, and purchase history. Or to predict customer churn we might consider customer behavior data such as the number of complaints in the last 6 months, the number of support tickets over the last month, and the number of months the person has been a customer, as well as demographic data such as the customer’s age, location, and gender.
Assuming we have a sufficient amount of historical data available that includes the actual results (whether or not a customer actually purchased in the first example, or churned in the second) we can use this data to create a predictive model that maps the input data elements (predictors) to the output data element (target) to make a prediction about our future customers.
Typically data scientists build predictive models through an iterative process that involves:
- Collecting and preparing the data (and addressing data quality issues)
- Exploring and Analyzing the data to detect anomalies and outliers and identify meaningful trends and patterns
- Building the model using machine learning algorithms and statistical techniques like regression analysis
- Testing and validating the model to determine its accuracy
Once the model is built and validated it can be deployed and used in real-time to inform automated decisions.
Deploying Predictive Models in Automated Decisions
While predictive models can give us sound predictions and scores, we still need to decide how to act on them. Modern decision management platforms like SMARTS Decision Manager let you combine predictive models that inform your decisions with business rules that translate those decisions into concrete actions. SMARTS includes built-in predictive analytics capabilities and also lets you use models built using other analytics tools such as SAS, SPSS and R.
The use of predictive models is rapidly expanding and changing the way we do business. But it’s important to understand that predictions aren’t decisions! Real world business decisions often include more than one predictive model. For example, a fraud decision might include a predictive model that determines the likelihood that a transaction originated from an account that was taken over. It might also include a model that determines the likelihood that a transaction went into an account that was compromised. A loan origination decision will include credit scoring models and fraud scoring models. It may also include other models to predict the likelihood the customer will pay back early, or the likelihood they will purchase additional products and services (up-sell). Business rules are used to leverage the scores from these models in a decision that seeks to maximize return while minimizing risk.
In our next post, we’ll look at how modern decision management platforms, like SMARTS, help you evaluate alternative decision strategies. We’ll explore how you can use decision simulation to find the best course of action.
Decision Management has been a discipline for Business Analysts for decades now. Data Scientists have been historically avid users of Analytic Workbenches. The divide between these two crowds has been crossed by sending predictive model specifications across, from the latter group to the former. These specifications could be in the form of paper, stating the formula to implement, or in electronic format that could be seamlessly imported. This is why PMML (Predictive Model Markup Language) has proven to be a useful standard in our industry.
The fact is that the divide that was artificially created between these two groups is not as deep as we originally thought. There have been reasons to cross the divide, and both groups have seen significant benefits in doing so.
In this post, I will highlight a couple of use cases that illustrate my point.
A number of organizations have adopted the idea of making use of the Decision Management approach and technologies to problems such as risk, fraud, eligibility, maximizing and more. If you read this blog, you probably already know what Decision Management brings to the table.
Decision Management is all about automating repeatable decisions in a maintainable way so that they can be optimized in a continuous fashion.
Decision systems can use Business Rules Management Systems (BRMS), but they do not need to restrict themselves to just that: they can also be built on Predictive Analytics technology; or they can even consist of a combination of both. The increasing availability of data that can be used to test, optimize decisions, or extract insights from, makes it possible for decision-centric applications to combine expertise and data to levels not seen in previous generations of applications.
In this post, we’ll outline the evolution from pure Business Rules Systems to Prescriptive Analytics platforms for decision-centric applications.Read More »
In part 1, we saw that we could use knowledge, experience and intuition to build a model serving as a basis for making decisions. But when historical data is available, we can do more…
When large amounts of historical data are available (and the larger, the better), a predictive model can be built using predictive analytics: this basically uses statistics to comb through the data and find patterns. Such patterns can of course be found more easily when they occur frequently. It can be quite useful to make use of the results of BI (if available) to guide the predictive analytics algorithms so that they find the proper correlations.
When successful, the predictive model, used on new cases, will predict a given outcome –therefore based on past experience. Automation of the decision making, using the predictive model, can be performed by building business rules from that model.
And the resulting business rules can, as usual, be enriched using existing knowledge or future knowledge acquired over time (from human experience, or other predictive analytics “campaigns”).
When the results of predictive analytics are used in a number of simulation scenarios, we end up with a number of possible outcomes, a few of them possibly more optimal than others (and here we are talking business performance).
These simulation scenarios may be run continually, as new historical data becomes available, in order to constantly optimize the predictive models –and also so that they correspond to a reality that is more current.
The possibility of obtaining a number of possible decisions trying to maximize an expected outcome, all based on historical data (and possibly also on existing knowledge) leads to a real prescription: “something that is suggested as a way to do something or to make something happen” (Merriam-Webster dictionary).
Automatically providing advice on decisions to make to reach a given target is a very appealing and powerful idea: you don’t just rely on “gut feeling” or experience or past knowledge; you rely on all of these, simultaneously. And the suggestions evolve as time passes, allowing quick refocusing.
Making informed decisions
The ability of making decisions based on so many different aspects that evolve over time is already something we, humans, do at our own level (both consciously and unconsciously).
Scaling this up to tactical and strategic levels in the Enterprise requires the use of prescriptive analytics, backed by knowledge, experience, and big data. So that we can have some comfort that we made those decisions based on all that we had at our disposal.
Now, should I eat some Thai food for lunch, or some Japanese food?
We spend our lives, both personal and professional, making decisions, all day long; some without consequences, and some with long-lasting and even perhaps game-changing ones.
Should I eat some Thai food for lunch, or some Japanese food?
Do we make targeted offers to customers that have been with us for more than 2 years, or to those that have been with us for more than 5?
How do we reduce the time it takes us to fix defective devices?
Although sometimes not making a decision is worse than making the wrong one, we all strive to make the best decisions possible. And to make the best decisions, we rely on experience and whatever information is at hand. With experience in the subject matter, decisions can be made very quickly; when the matter is new or information is scarce, we usually require more time to evaluate a number of possibilities, to make a few computations, to balance the pros and cons.
All this is part of our daily lives. But when a large number of decisions need to be made in a short amount of time, or when the data available to us is limited, or on the other hand enormous, automation can come to the rescue. But how can we make informed decisions at a large scale?Read More »
As it is customary, let me share what I foresee as being big this new year… I would like to focus on just three points that are striking me as important in no particular order.
1. Predictive Analytics
Well, of course, we have been seeing that trend develop for a while. This is certainly not a surprising entry in this list.
The fact is that we see more and more projects combining predictive analytics and business rules. What is really interesting to me is the fact that more and more business analysts are getting trained to develop some of these predictive models.
Given the data scientist shortage, it make total sense. If you do not have a modeling team in-house or if it is swamped with high priority projects, you may as well look for other ways to leverage the available data to inform your decisions.
I am optimistic that we will see more business analysts add predictive analytics to their skill set.
2. Business Intelligence
Sticking with analytics at large, I see also a greater synergy between business intelligence and business rules. We have talked about ‘Operational BI’ for a while now, but there seems to be a lot of activity finally taking shape.
I believe that there will be more projects that actually combine both in 2014, allowing companies to act on the gained from monitoring historical trends.
3. Internet of things
When I was still in my early years, we dreamed of ‘intelligent’ equipment, cars and other things that would make our life easier. While embedding computers in all things around the house has been cost prohibitive for the mass market back then, the Cloud is now making it a reality.
The beauty of having ‘things’ that can communicate is that they are immediately candidate for ‘higher intelligence’. By hooking them up with a decision service on the cloud, we can seamlessly allow them to act more appropriately and subtly to signals they sense around around. They can better adapt since changing their behavior does not involve any hardware changes, or more generically any changes in-situ. The intelligence is located on the cloud, readily available for all connected things.
I am totally in awe with the progress we have made thus far, and the potential for a global ‘increase of intelligence’ of the things around us. The future is now!