SMARTS for data marketplaces

In my earlier blog post, I explained how decision management and business rules were suitable for micro-calculations, the type of computations that businesses often codify into large spreadsheets and use to score, rate, or price items. In this post, I explain how they are also suitable to simplify data integration, aggregation, and enrichment when building and running data marketplaces.

Data marketplaces

Data marketplaces are a shift from data warehouses where the goal is not only to store large volumes of data, but to make that data be consumed as a service without resorting to IT or prior knowledge of a query language. Data marketplaces are often organized into three layers:

• At the lowest layer, we find raw data stored in the form in which it was ingested from the data sources. Data sources can be global ERP and CRM systems, or even local MySQL databases and shared Excel spreadsheets.
• At the middle layer, we find integrated data from multiple sources that is reconciled to resolve disparities and inconsistencies found in the original data. Often, the source systems do not have the same format for dates, names, phone numbers, and addresses. Sometimes the same object can have different attributes in different data sources.
• At the topmost layer, we find aggregate data expressed in summarized forms, often to inform about groups rather than individuals. It is at this level that we also find data enriched by external data to make them directly consumable by the businesspeople.

To learn more about data marketplaces, I recommend the Eckerson Group white paper: The Rise of the Data Marketplace – Data as a Service by Dave Wells.

Easy to define, hard to construct

Defining a data marketplace as we have just done is simple, its construction is complicated for two reasons:

1) Data heterogeneity. It is not enough to bring together all the company’s data in a data marketplace for the data to be transformed into knowledge, forecasts, and decisions. Indeed, all data does not have the same age, the same structure, the same format, the same quantity, the same quality, and above all the same utility. If an attribute is important for a business line, it is not automatically important for another business line, yet within the same company.

Each business line has its vision of the product, of the customer, and of any entity managed by the various actors of the company. In the luxury sector for example, a dress, a bag, or a piece of jewelry, although it is a unique object, is seen through different attributes according to the databases where this same object is stored. Looking to exploit all the data available in a company to extract predictions and then decisions not only require integration but also transformation, unification, harmonization, and enrichment.

2) Data reorganization. A data marketplace would work better if data, information, and needs were always stable. But in a dynamic and rapidly changing business world, groups are reorganizing, and companies are acquired, merged or separated. For example, to simplify finance reporting, a country can change the region it was in a year ago, and it’s a safe bet that it will change yet again if a new boss is appointed, or a region is split or added. To be successful, data marketplaces must be implemented as change-tolerant projects.

These two reasons are representative of situations where decision management technologies are used: piecing together things that move independently to each other. Under the name of decision management, we group all the technologies that help organizations to automate all those simple but plentiful granular decisions and calculations that businesses often codify into decision tables, decision trees, or business rules.

Decisioning technologies to the rescue

Decisioning technologies can be used here. But one can use a database programming language or a general-purpose scripting language to automate the loading of data from the data sources into the raw data layer if the original format is kept. There is no real value using a business rules engine to do this straightforward job. The value of using a decisioning technology starts at the integration data layer, where attributes of objects are grouped together to form an updated version of an existing attribute or a new attribute.

Take the example of customer data from two different databases. Suppose that customers have their home address and business address in a first source database, and that they only have their home address in a second source database. Suppose also that the format of the addresses is not the same in the two databases. What should the integration data layer hold? An address? So, which one? And what format? Two addresses? So, what professional address to put for these customers who are only present in the second database with the home address? These questions can be easily answered through decision rules.

Now, suppose at the aggregate data layer, we want to add an average turnover with a client that buys from two business units. Here again, calculation rules can be easily used. One can use SMARTS’ look-model engine to automate such calculations.

SMARTS

SMARTS is our all-in-one low-code platform for data-driven decision-making. It unifies authoring, testing, deployment, and maintenance of micro-decisions and micro-calculations described in this article. SMARTS comes in the form of one product, four capabilities:

• A decision management platform that spans the entire life cycle of decisioning from modeling to deployment.
• A low-code no-code environment in which users express decisions and calculations through point-and-clicks and web forms.
• An AI & ModelOps environment that covers the full spectrum of ModelOps from importing existing models, to defining new ones, to executing learning tasks.
• A real-time decision analytics environment to manage the quality of decision and calculation performance.

SMARTS has been extensively used for decisions and calculations in finance, insurance, healthcare, retail, IoT, and utility sectors. To learn more about the product or our references, just contact us or request a free trial.

Wrap-up

• Data marketplaces promise to change the way data is consumed by businesses. Contrary to data warehouses, they are more complex to build and therefore deliver on their promises.
• Decision management technologies simplify data integration, aggregation, and enrichment when building and running data marketplaces. They make decisions and calculations explicit and therefore easy to change whenever situations change.
• SMARTS supplies multiple graphical representations* and engines to make such transformations, integrations, and enrichment easy to design, implement, test, deploy, and change according to situation changes.

* The following table, tree, and graph show three different representations of the same decision logic so that developers can use one that they are familiar with or that best fits the task at hand.

About

Sparkling Logic is a Silicon Valley company dedicated to helping businesses automate and improve the quality of their operational decisions with a powerful decision management platform, accessible to business analysts and ‘citizen developers.’ Sparkling Logic SMARTS customers include global leaders in financial services, insurance, healthcare, retail, utility, and IoT.

Sparkling Logic SMARTS^TM (SMARTS for short) is an all-in-one low-code platform for data-driven decision-making. It unifies authoring, testing, deployment, and maintenance of operational decisions. SMARTS combines the highly scalable Rete-NT inference engine, with predictive analytics and machine learning models, and low-code functionality to create intelligent decisioning systems.

Hassan Lâasri is a data strategy consultant, now leading marketing for Sparkling Logic. You can reach him at hlaasri@sparklinglogic.com.