Our Best Practices Series has focused, so far, on authoring and lifecycle management aspects of managing decisions. This post will start introducing what you should consider when promoting your decision applications to Production.
Make sure you always use release management for your decision
Carole-Ann has already covered why you should always package your decisions in releases when you have reached important milestones in the lifecycle of your decisions: see Best practices: Use Release Management. This is so important that I will repeat her key points here stressing its importance in the production phase.
You want to be 100% certain that you have in production is exactly what you tested, and that it will not change by side effect. This happens more frequently than you would think: a user may decide to test variations of the decision logic in what she or he thinks is a sandbox and that may in fact be the production environment.
You also want to have complete traceability, and at any point in time, total visibility on what the state of the decision logic was for any decision rendered you may need to review.
Everything they contributes to the decision logic should be part of the release: flows, rules, predictive and lookup models, etc. If your decision logic also includes assets the decision management system does not manage, you open the door to potential execution and traceability issues. We, of course, recommend managing your decision logic fully within the decision management system.
Only use Decision Management Systems that allow you to manage releases, and always deploy decisions that are part of a release.
Make sure the decision application fits your technical environments and requirements
Now that you have the decision you will use in production in the form of a release, you still have a number of considerations to take into account.
It must fit into the overall architecture
Typically, you will encounter one or more of the following situations
• The decision application is provided as a SaaS and invoked through REST or similar protocols (loose coupling)
• The environment is message or event driven (loose coupling)
• It relies mostly on micro-services, using an orchestration tool and a loose coupling invocation mechanism.
• It requires tight coupling between one (or more) application components at the programmatic API level
Your decision application will need to simply fit within these architectural choices with a very low architectural impact.
One additional thing to be careful about is that organizations and applications evolve. We’ve seen many customers deploy the same decision application in multiple such environments, typically interactive and batch. You need to be able to do multi-environment deployments a low cost.
It must account for availability and scalability requirements
In a loosely coupled environments, your decision application service or micro-service with need to cope with your high availability and scalability requirements. In general, this means configuring micro-services in such a way that:
• There is no single point of failure
○ replicate your repositories
○ have more than one instance available for invocation transparently
• Scaling up and down is easy
Ideally, the Decision Management System product you use has support for this directly out of the box.
It must account for security requirements
Your decision application may need to be protected. This includes
• protection against unwanted access of the decision application in production (MIM attacks, etc.)
• protection against unwanted access to the artifacts used by the decision application in production (typically repository access)
Make sure the decision applications are deployed the most appropriate way given the technical environment and the corresponding requirements. Ideally you have strong support from your Decision Management System for achieving this.
Leverage the invocation mechanisms that make sense for your use case
You will need to figure out how your code invokes the decision application once in production. Typically, you may invoke the decision application
• separately for each “transaction” (interactive)
• for a group of “transactions” (batch)
• for stream of “transactions” (streaming or batch)
Choosing the right invocation mechanism for your case can have a significant impact on the performance of your decision application.
Manage the update of your decision application in production according to the requirements of the business
One key value of Decision Management Systems is that with them business analysts can implement, test and optimize the decision logic directly.
Ideally, this expands into the deployment of decision updates to the production. As the business analysts have updated, tested and optimized the decision, they will frequently request that it be deployed “immediately”.
Traditional products require going through IT phases, code conversion, code generation and uploads. With them, you deal with delays and the potential for new problems. Modern systems such as SMARTS do provide support for this kind of deployment.
There are some key aspects to take into account when dealing with old and new versions of the decision logic:
• updating should be a one-click atomic operation, and a one-API call atomic operation
• updating should be safe (if the newer one fails to work satisfactorily, it should not enter production or should be easily rolled back)
• the system should allow you to run old and new versions of the decision concurrently
In all cases, this remains an area where you want to strike the right balance between the business requirements and the IT constraints.
For example, it is possible that all changes are batched in one deployment a day because they are coordinated with other IT-centric system changes.
Make sure that you can update the decisions in Production in the most diligent way to satisfy the business requirement.
Track the business performance of your decision in production
Once you have your process to put decisions in the form of releases in production following the guidelines above, you still need to monitor its business performance.
Products like SMARTS let you characterize, analyze and optimize the business performance of the decision before it is put in production. It will important that you continue with the same analysis once the decision is in production. Conditions may change. Your decisions, while effective when they were first deployed, may no longer be as effective after the changes. By tracking the business performances of the decisions in production you can identify this situation early, analyze the reasons and adjust the decision.
In a later installment on this series, we’ll tackle how to approach the issue of decision execution performance as opposed to decision business performance.
James Taylor, one of the leading experts on decision management, recently wrote an update on his blog featuring Sparkling Logic’s products. James is the CEO and a Principal Consultant of Decision Management Solutions. He is a well-known author and speaker on using decision modeling, business rules, and analytics to improve decision making to enable a more agile, analytic, and adaptive business.
In his recent blog post on Sparkling Logic, James does a great job of summarizing the key features of PENCIL Decision Modeler and the more recent features we’ve added to SMARTS Decision Manager since he last published an update on us. He highlights key features of PENCIL including the DMN decision diagram and glossary, and the ability to generate a project in SMARTS for execution, testing, simulation and deployment.
For SMARTS, he covers features we’ve added since his last update in 2013 including:
- Cascading or inherited decisions
- Native PMML support
- Lookup models
- Champion/Challenger testing
- Lifecycle management and task automation
Most of these are unique SMARTS features not found in other decision management products. **Spoiler Alert** He also mentions a feature in our upcoming release, Quebec, which supports graphical investigation of rules fired for specific transaction.
James’ “First Look” product updates are a great source to learn details about product offerings in the decision management space and we appreciate the recent update on our products. His blog and the Decision Management Solutions website are great resources to learn more about decision modeling and management.
We recently had the pleasure of meeting with Simon Halloway from Bloor to give him an update on our decision management and business rule products. Bloor is an independent analyst and research company based in the UK and Simon is their Practice Leader on Process Management and Sensory Devices.
Simon’s focus area includes the intelligent automation of processes using sensors, so he was particularly interested to learn about how some of our customers use SMARTS to analyze sensor data to drive automated decisions.
We were happy to see that he wrote a report following our meeting. In the report, called “Sparkling Logic brings SMARTS to Decisions”, Simon covers how PENCIL Decision Modeler and SMARTS Decision Manager work together. He explains that decision models created in PENCIL can be executed and tested with data in SMARTS. PENCIL let’s business analysts capture and document business rules and decisions using the Decision Model and Notation (DMN) standard and the decision model can be tested and validated in SMARTS.
The report also highlights some of the unique features in SMARTS:
He concludes, “Sparkling Logic’s SMARTS is definitely a solution in Bloor’s view that should be considered if an organization is looking at decision management automation across any sector, whilst still providing all the necessary support for business rules management”.
Thanks Simon, we couldn’t agree more! Get a copy of the report here.
In What is Lifecycle Management for a Business Rule, Carole-Ann touched upon the usefulness of rule versioning for traceability or backtracking. She also mentioned the power of a release, which basically represents the version of multiple rules (or any kind of item) at a given point in time, effectively giving the ability to travel back in time.
Such capabilities are essential during the implementation and deployment of decision logic; but they can also be extremely useful during decision modeling time, as we will see here.
The Decision Model and Notation (DMN) provides a number of ways to supply specific content to a model, i.e. some kind of information that is not directly related to the modeling or the decision implementation, but which can be relevant in your context nonetheless:
- All diagram elements (input data, decision, business knowledge model, knowledge source) can have a description
- Decisions have additional information such as a question that may characterize them and allowed answers, objectives, performance indicators, decision makers, decision owners, BPMN processes and BPMN tasks
- Knowledge sources also have additional information such as a location for the source of knowledge, and the type of that source of knowledge, as well as an owner
Pencil Decision Modeler adds more information to the mix, such as the volume of a decision (how frequently the decision is made), its frequency (how frequently it changes) and its latency (how much time is allowed to make and deploy changes). Finally, glossary categories and entries can also have a description.
While this is great, this may not be sufficient for your own needs: you may need more information to be provided either in the DMN diagram itself, or in decision logic.
As much as it is an ego boost to receive these acknowledgements, I would like to explain why we have been investing continuously on elicitation. After all, most of the research happened in the 1980’s and we are part of a very small group that has continued exploring new ways to simplify this task
As we hosted a recent webinar series on “Getting to the Best Decision”, focused on testing and improving business rules, several questions came up on KPIs — Key Performance Indicators. It is true that we tend to use many different terms for the same thing. You may have heard us or others speak of business objectives, KPIs, metrics, calculations, characteristics, features or variables. Are they synonyms?
I’d like to suggest ‘my’ definitions. Feel free to comment on whether you agree or not, and share the nuances your recommend.
Fraud has been on the rise lately with some recent high-profile cases like the Zappos leak a couple of weeks ago. The systems are unfortunately the target of fraudsters on all possible fronts:
- Origination or on-boarding: Can I trust this individual to do business with?
- Transactions or claims: Should I let it go through?
- Investigation: Is this transaction actually legitimate? Can I trust this individual?
- Management: How do I treat this flagged individual or transaction?
We often think of risk management as a Financial Services specialty but many if not all businesses can be the target of fraudsters. In my talk with eBay at BBC, Kenny and I discussed some specifics of Fraud Detection for a retail site. This is a significant problem they need to tackle very quickly, as you can imagine. Here are some numbers that are talk to the size of that problem:
- 2 rules deployments every week
- 20+ rules analysts around the globe depend on BRMS to innovate in fraud detection and risk management
- 110+ eBay user flows
- 300+ Rulesets
- 600 application servers running rules (in the slides), 1200 approved on the day of the talk!
- 1,000+ variables
- 15k+ rules
- 50M+ fired rules a day
- 140M+ rule sessions a day
Let me share of the key take-aways of the talk.
1. Fraudsters look for a good ROI
The same way that businesses consider the Return On Investment, fraudsters are on the look-out for the biggest bang for the buck. They continuously look for the weakness in the systems or procedures that can be exploited at large-scale. With that in mind, you could consider that the Fraud team’s job is not to make it impossible to abuse the system, but rather to make it *expensive*.
We have all received phishing emails, ranging from the African Dictator’s survivor to the Lottery Grand Prize. We know of credit card abuse, etc. Kenny shared some more unusual examples of fraud that eBay had to react to.
Account Take Over is a major issue. Originally fraudsters simply logged in to create new fraudulent listings. eBay started tracking the IP addresses in the account history and used it for comparison in case of new listings. Fraudsters eventually realized that they could instead revise the seller’s existing listings to the fraudulent ones. eBay introduced some delays in making the change visible to allow for verification. The fraudster found out that eBay, as a policy, did not delay those changes when made in the last 12 hours of the auction…
This feels very much like a chasing game. Kenny compares it to “catch the mouse”.
Here are some other “creative” moves from the fraudsters:
Fraudulent listings include contact information highlighted in the description to get the buyer to transact outside of eBay, by-passing the security measures of the commerce platform. eBay introduced a word search for email addresses at the time of posting. The fraudsters started posting their contact details as images!
A clever twist in the Fraud scheme caused an interesting puzzle for the Fraud Detection team. They realized that, after the fraudulent listings had been removed, they eventually reappeared despite the measures they took to block access… until they realized that, elsewhere in the account configuration, the fraudsters had made sure that non-sold items were automatically reposted. The automated rule repeated the fraud all by itself!
Fraudsters can get quite sophisticated. This “organized” crime organization moves fast and spreads everywhere through fraud rings and distribution channels.
2. The Intelligence to stop the fraudster
That is one fascinating aspect of the Fraud space: it is a moving target. You always need to solve new mysteries and devise plans to stop the fraud. If you love puzzles like I do, you cannot not be enticed by that challenge!
The rules analysts need to come up with rules that flag the fraudsters, all the fraudsters and only the fraudsters, as comprehensively as possible, as precisely as possible and as fast as possible. The metrics that are typically used to track the success of those business rules are the Hit Rate — when I flag a transaction, how likely is it that I catch an actually fraudulent transaction — and the Catch Rate — out of all the fraudulent transactions, how many do I catch.
Having clear objectives and ways to track them is a great start, but it does not solve the core issue of coming up with those business rules. The rules analysts have to rely both on their intuition, typically with the insight of the case workers, and lots of data insight of course. Analytics are critical tools in the Fraud Detection departments.
With this context in mind, the business case for Business Rules / Decision Management technology becomes obvious. The speed of change and the need to iterate to refine the fraud detection criteria are not at all compatible with traditional software development. If you played with the numbers that Kenny shared initially, you know that eBay makes about 20,000 changes per year. The only way to get this is done is by empowering those business analysts so that they can author the flagging rules on their own while the IT team focuses on improving the speed of data access and variable computation, which Kenny described in more details in his other talk.
In conclusion, the ROI for the companies that are fighting fraud is in getting the rules right and getting them fast.
Disclaimer: the examples of fraud I provided are not meant to encourage you to fraud… All of those schemes are now automatically flagged as fraudulent of course!