In this post, we explore decision testing and its crucial role in decision management. This continues our series on what’s new in SMARTS™ Washington.
What is Decision Testing?
Decision testing is the process of verifying the accuracy and performance of decision logic before deployment. In other words, testing enables you to check the following:
- Does the decision logic match what I intended the logic to be?
- Does my intended decision logic produce the outcomes that I expect?
Quality Assurance
Quality assurance (QA) is a systematic approach to ensuring that products (and services) meet pre-defined expectations (or quality standards). QA evolved from manufacturing to become essential in service and tech industries. In software development, QA testing ensures that the software meets requirements in terms of functionality, usability, and reliability before release. Typically this involves two types of testing:
- Regression testing ensures that recent code additions and changes do not adversely affect functionality. It checks both the code and the outputs of that code. Regression testing typically occurs frequently through out the development life cycle on a set of test cases where the expected outcomes are known in advance.
- Functional testing verifies each feature of the software functions as expected. It isn’t really concerned with verifying the individual lines of code but verifying the software as a whole. Functional testing usually occurs at key milestones in development such as completion of a feature and typically involves more test cases than regression testing to cover all known hypothetical situations.
QA in Decision Management
The same types of testing in software development can be applied to decision development. As you’re developing your decision logic, you should be conducting regression testing frequently to check for errors in your decision logic and decision strategy. You should also be running regressions tests frequently when you’re modifying existing decision logic.
In order to do so, you should have test cases readily available. Therefore, we always recommend that you start building your library of test cases before you start the decision logic development process. You should include common scenarios as well as edge cases. Once your decision logic has been developed, we recommend running large-scale simulations to test not only the overall “function” of your decision logic, but also your entire decisioning system.
Therefore, best practices for decision testing include the following:
- Build a test case library early
- Include diverse scenarios in your test cases
- Run simulations to test your entire decisioning system
Decision Testing in SMARTS™
Our SMARTS™ decision management platform was designed with decision testing in mind. SMARTS™ includes various tools to support both regression and functional testing. The SMARTS™ user interface (UI) enables users to develop decision logic in context of their data. Users can start a new decision project simply by uploading a file of test cases. The data model will either be extracted or inferred from the test cases and referenced in decision authoring. When not test cases exist, users can leverage AI Assistant, SMARTS™ generative AI tool, to iteratively create both the data model and test cases using natural language.
As decision logic is developed, users can quickly run regression tests on the test cases by clicking on the “Run” button in the UI. Users can review the decision flow path for each individual test case as well as aggregate results in the customizable Dashboard. By defining computations, business terms, and QA flags, users can easily measure decision quality and identify failed test cases. Further analysis can be achieved through SMARTS™ execution tracing and debugging tools. Once decision logic is complete, users can use the Simulations tool to run large-scale simulations on datasets of any size in various protocols and formats, synchronously or asynchronously. We’ve got clients that run millions of test cases!
Experimentation and Testing in Production
In addition to regression and functional testing, SMARTS™ also supports Champion/Challenger experiments. This enables users to test multiple decision strategies at once, such as using a Python model versus a scorecard. These experiments can be set up in both development and production. In cases where it’s not possible to fully measure decision quality in development, Champion/Challenger can be an effective way to mitigate risk in production. These experiments can also be deployed in-shadow or in the background. This allows organizations to continue to action off an existing strategy while collecting data on new strategies.
By incorporating testing into decision logic development, organizations can ensure compliance with regulatory requirements, verify decision strategy aligns with business objectives, and reduce the overall risk of costly decision errors. And through SMARTS™, organizations can perform testing easily, at high frequency, and with high volumes of test cases. See what testing could look like for your use case. Contact us today for a customized demo!