What is Data Integration?
Data integration is the process of combining data from multiple sources into a single, unified, and coherent view. The goal is to make data easier for both humans and machines to consume the data and act upon it. Often different sources will have different formats and inconsistencies will exist across the different sources. For example, one source may have dates stored in YYYY-MM-DD format while another may have dates stored as timestamps (YYYY-MM-DD HH:MI:SS). Or, the addresses in one source may not match the address in another source. This is why data cleansing and transformation is part of the data integration process. Here you would address typos, errors, conflicts, and any formatting issues. The final output should be the single source of truth for an organization, both in structure and content of the data. This integrated data is usually stored and managed in a database that can be queried by people and systems.
Data Integration in Decision Management
Decision Management Systems (DMS) like Sparkling Logic’s SMARTS™ enable organizations to efficiently and effectively manage decisions in automated processes and systems. Once data is integrated, Decision Management Systems (DMS) can benefit from leveraging this data to develop, test, and optimize high value decisions. Data can be used at every stage of the life-cycle an automated decision:
- During development, you can reference simple test cases to incrementally build your decision logic.
- In testing, you can use more elaborate test cases to validate your decision logic. Ideally, these test cases will come from existing production data.
- Through out production, you will ingest data to make live decisions and leverage the resulting outcome data to improve your decision logic.
While most DMS implementations focus on building the logic (ex. flows, rules), we at Sparkling Logic believe that assessing the quality of the logic is as important. Assessing requires analytics and analytics require data. This is why SMARTS™ comes with built-in decision analytics capabilities. Users can define Key Performance Indicators (KPIs) and QA flags and generate visual reports in our Dashboard. The more important your decision, the more you need to measure its performance early in the development. Carole-Ann has written about this in the best practices series: Best Practices Series: Business Testing.
Data for Development
Starting with Sample Data
Let’s say you’re building out decision logic for processing a loan application. While you don’t necessarily need data to build a decision, you at least need the data schema (or what we call form structure). The schema identifies all the relevant data fields and their corresponding formats in your integrated data. This ensures that your decision logic is referencing the appropriate pieces of data. For example, if birthdates are stored as dob, your decision logic should reference dob and not birthdate. If you use birthdate in your decision logic, it will fail because no data field of that name exists.
However, having the data schema only ensures that your decision logic is technically correct. This is why we recommend starting with sample data. The sample data will not only provide you the schema, but also content for which you can use to validate your decision logic. During development, you only need a small data sample, no more than 10,000. In most cases, 100 records is more than enough. Ideally you want a mix of records where you know the expected results ahead of time. In situations where no historical data exists such as you’re launching a new product, get test cases from your QA team and come up with some on your own.
SMARTS™ enables users to upload sample data into the UI which can then be referenced for building and testing rules. In the screenshot below, 5 records have been uploaded. One of those records is shown in the Document pane on the left. The Dashboard pane on the right shows that in the current state of the decision logic, 2 are approved and 3 are declined. Leveraging our RedPen™ point-and-click rules authoring tool, users can click on the appropriate fields in the sample record to write rules.
DMS Requirements to Leverage Data in Development
In order to fully leverage data in development, you should look for the following capabilities in a DMS:
- Enables management of data sets associated to decisions
- Enables ability to add new data and data sets on the fly
- Supports data formats commonly used in the enterprise (such as CSV, XML or JSON)
- Provide decision analytics facilities to verify the quality of the decision using the data sets
One key challenge with data is that data changes. Thus, the second requirement is crucial for your DMS. We see new data as an opportunity to further enhance your decisions and why me make it easy for users to add new data.
Data for Testing
Using Large Data Sets in Simulations
Once you’ve built out your decision logic, you will need to fully test it before you move to production. At this stage, you should work with large data sets that covers all the exceptions and unique cases. These data sets typically come from your data lakes, potentially fed from operational data. They may consist of large enriched historical data that you keep around for reporting, analysis, or machine learning (ML) purposes. Pro tip, since it may take time to generate all the test cases, start compiling them early in the project so that they’re available when you’re ready to test. Ideally, work with your QA team so that the same test cases that will eventually be used to test functionality can also be used to test validity of the logic.
With large data sets, duplication will be expensive and potentially cause security or compliance issues. Therefore, you will want the data management to remain in the data lake, data mart or enterprise data environment. Thus, the DMS will ideally provide a means to integrate with your data source without copying it or managing it within the DMS. Once integrated, you will most likely need to rely on simulation capabilities of the DMS. This paired with decision analytics, will enable you to get an aggregate view of how your decision logic performs. SMARTS™ includes not only simulation capabilities but also champion/challenger experiments which enable users to split the data set to run on different variations of the decision logic. For example, you may want to compare a new ML credit model against an existing credit scorecard. What’s great about these experiments is that they can run during production too, both in live and shadow environments.
DMS Requirements to Leverage Data in Testing
As previously mentioned, the DMS will need to integrate with your data source. Predefined connectors to databases or data lakes can be useful for simpler access to existing stores. However, as long as you have a generic means to access data, by means of a web service or using any standard data format through the FTP or http protocols, you should be able to access the data you need without issue. In addition, the simulation and decision analytics capabilities of the DMS must be able to handle large data sets. For example, SMARTS™ provides a map-reduce and streaming simulation and decision analytics facility to handle complex metrics computations at scale.
In summary, the DMS should provide the following:
- mechanism(s) to access enterprise data on-the-fly
- scalable simulation and decision analytics facilities that can cope with enterprise-size data sets
In addition, the enterprise will be responsible for the following:
- managing large data sets relevant to the decision
- using the DMS facilities to leverage these large data sets to verify and enhance decisions
Data for Production
Using Operational Data
Now that your loan evaluation decision logic is in production, you will want to collect interesting data points for further analysis. Typically, your enterprise environment will include operational data stores and data lakes where the data is merged for reporting, analysis and ML purposes. The more sophisticated enterprises will also include decision outcome databases and correlate business outcomes with decision outcomes in the data lakes. For example, let’s say you’re running an 80-20 champion/challenger with your existing credit scorecard and new ML credit model. After 6 months, you notice the following with applicants evaluated under the new model:
- Approvals are 15% higher
- Average loan sizes are about the same
- Defaults are 10% lower
Collectively, these metrics indicate that the new model is outperforming the old. You may decide to end the experiment and switch 100% to the new model. In addition to enabling access operational data stores, SMARTS™ provides real-time metrics to monitor performance in production. The same KPIs defined during development and testing can be used to measure live activity. In addition, users can create graphical reports to summarize results and create alert notifications when certain thresholds have been reached.
DMS Requirements to Leverage Data in Production
To leverage production data, the DMS should:
- support storing decision outcomes
- provide mechanisms to access data lakes and operational data stores on the fly
- offer simulation and decision analytics facilities that scale
- ideally support model creation and/or model execution
Meanwhile, the enterprise will:
- manage large data sets relevant to the decision
- use the DMS facilities to leverage these large data sets to verify and enhance decisions
This blog is part of the Technical Series, stay tuned for more! Future topics include the various strategies to pass data to decisions at run time, including the retrieval of additional data while the decision executes.
Learn more about Decision Management and Sparkling Logic’s SMARTS™ Data-Powered Decision Manager