Call to Action

Webinar: Take a tour of Sparkling Logic's SMARTS Decision Manager Register Now

Business Process Automation

SMARTS for data marketplaces


SMARTS for data marketplaces

In my earlier blog post, I explained how decision management and business rules were suitable for micro-calculations, the type of computations that businesses often codify into large spreadsheets and use to score, rate, or price items. In this post, I explain how they are also suitable to simplify data integration, aggregation, and enrichment when building and running data marketplaces.

Data marketplaces

Data marketplaces are a shift from data warehouses where the goal is not only to store large volumes of data, but to make that data be consumed as a service without resorting to IT or prior knowledge of a query language. Data marketplaces are often organized into three layers:

  • At the lowest layer, we find raw data stored in the form in which it was ingested from the data sources. Data sources can be global ERP and CRM systems, or even local MySQL databases and shared Excel spreadsheets.
  • At the middle layer, we find integrated data from multiple sources that is reconciled to resolve disparities and inconsistencies found in the original data. Often, the source systems do not have the same format for dates, names, phone numbers, and addresses. Sometimes the same object can have different attributes in different data sources.
  • At the topmost layer, we find aggregate data expressed in summarized forms, often to inform about groups rather than individuals. It is at this level that we also find data enriched by external data to make them directly consumable by the businesspeople.

To learn more about data marketplaces, I recommend the Eckerson Group white paper: The Rise of the Data Marketplace – Data as a Service by Dave Wells.

Easy to define, hard to construct

Defining a data marketplace as we have just done is simple, its construction is complicated for two reasons:

1) Data heterogeneity. It is not enough to bring together all the company’s data in a data marketplace for the data to be transformed into knowledge, forecasts, and decisions. Indeed, all data does not have the same age, the same structure, the same format, the same quantity, the same quality, and above all the same utility. If an attribute is important for a business line, it is not automatically important for another business line, yet within the same company.

Each business line has its vision of the product, of the customer, and of any entity managed by the various actors of the company. In the luxury sector for example, a dress, a bag, or a piece of jewelry, although it is a unique object, is seen through different attributes according to the databases where this same object is stored. Looking to exploit all the data available in a company to extract predictions and then decisions not only require integration but also transformation, unification, harmonization, and enrichment.

2) Data reorganization. A data marketplace would work better if data, information, and needs were always stable. But in a dynamic and rapidly changing business world, groups are reorganizing, and companies are acquired, merged or separated. For example, to simplify finance reporting, a country can change the region it was in a year ago, and it’s a safe bet that it will change yet again if a new boss is appointed, or a region is split or added. To be successful, data marketplaces must be implemented as change-tolerant projects.

These two reasons are representative of situations where decision management technologies are used: piecing together things that move independently to each other. Under the name of decision management, we group all the technologies that help organizations to automate all those simple but plentiful granular decisions and calculations that businesses often codify into decision tables, decision trees, or business rules.

Decisioning technologies to the rescue

Decisioning technologies can be used here. But one can use a database programming language or a general-purpose scripting language to automate the loading of data from the data sources into the raw data layer if the original format is kept. There is no real value using a business rules engine to do this straightforward job. The value of using a decisioning technology starts at the integration data layer, where attributes of objects are grouped together to form an updated version of an existing attribute or a new attribute.

Take the example of customer data from two different databases. Suppose that customers have their home address and business address in a first source database, and that they only have their home address in a second source database. Suppose also that the format of the addresses is not the same in the two databases. What should the integration data layer hold? An address? So, which one? And what format? Two addresses? So, what professional address to put for these customers who are only present in the second database with the home address? These questions can be easily answered through decision rules.

Now, suppose at the aggregate data layer, we want to add an average turnover with a client that buys from two business units. Here again, calculation rules can be easily used. One can use SMARTS’ look-model engine to automate such calculations.

SMARTS

SMARTS is our all-in-one low-code platform for data-driven decision-making. It unifies authoring, testing, deployment, and maintenance of micro-decisions and micro-calculations described in this article. SMARTS comes in the form of one product, four capabilities:

SMARTS has been extensively used for decisions and calculations in finance, insurance, healthcare, retail, IoT, and utility sectors. To learn more about the product or our references, just contact us or request a free trial.

Wrap-up

  • Data marketplaces promise to change the way data is consumed by businesses. Contrary to data warehouses, they are more complex to build and therefore deliver on their promises.
  • Decision management technologies simplify data integration, aggregation, and enrichment when building and running data marketplaces. They make decisions and calculations explicit and therefore easy to change whenever situations change.
  • SMARTS supplies multiple graphical representations* and engines to make such transformations, integrations, and enrichment easy to design, implement, test, deploy, and change according to situation changes.

* The following table, tree, and graph show three different representations of the same decision logic so that developers can use one that they are familiar with or that best fits the task at hand.

DecisionDecision treeDecision graph

About

Sparkling Logic is a Silicon Valley company dedicated to helping businesses automate and improve the quality of their operational decisions with a powerful decision management platform, accessible to business analysts and ‘citizen developers.’ Sparkling Logic SMARTS customers include global leaders in financial services, insurance, healthcare, retail, utility, and IoT.

Sparkling Logic SMARTSTM (SMARTS for short) is an all-in-one low-code platform for data-driven decision-making. It unifies authoring, testing, deployment, and maintenance of operational decisions. SMARTS combines the highly scalable Rete-NT inference engine, with predictive analytics and machine learning models, and low-code functionality to create intelligent decisioning systems.

Hassan Lâasri is a data strategy consultant, now leading marketing for Sparkling Logic. You can reach him at hlaasri@sparklinglogic.com.

Software industry trends behind the digital transformation revolution


This article presents the three software industry trends driving the digital transformation revolution: DevOps, low-code / no-code automation, vertical integration with digital decisioning.

Introduction

The pandemic changed tech priorities for many people both at work and home making a ‘hybrid’ work a top initiative. Where and how we do the work accelerated the need to improve customer digital experiences and efficiency across work, shopping, and everyday chores.

The data supports this new trend. The independent research firm Omdia compiled over 300 responses from executives at large companies indicated that working away from traditional offices will become the new norm. 58% percent of respondents said they will adopt a hybrid home/work. Even more interesting is that 68% of enterprises believe employee productivity has improved since the move to remote work.

Similarly, adoption of everyday on-line activities such as shopping, banking and entertainment further accelerated the pace of digital transformation. The need for improved applications increased the pressure on companies to relaunch efficient, friendly front-end customer apps with more intuitive UX. The back end now needs to support faster turnaround with the need to automate processes for the new on-line community of users demanding faster, cleaner, and more intelligent offerings.

To respond to this digital transformation, companies are rapidly adopting easy-to-use integrated enterprise software tools to optimize and accelerate development of these efficient digital products.

Several trends like DevOps, Low-code/automation and vertical integrations with integrated digital decisioning have emerged to help enterprises take the digital transformation journey faster and cheaper.

DevOps

DevOps is a software development concept bringing together historically disconnected functions in the lifecycle of the software development. Traditionally, business analysts would define the problem, developers would interpret the concept and build applications, and operations teams would test, report bugs and provide feedback. The disconnect between the functions, silo’d approach created inefficiencies, increased costs and slowed down application releases.

The emergence of integrated tools and processes which integrate this multiple aspect of software development and promote collaboration between these different functions supported growth of the DevOps industry.

In fact, the market data shows that these trends are supported by the investment community and exit activity. According to Venture Beat, in 2Q 2021, Venture funding for global DevOps startups reached $4 billion and the exit activity deal value was dominated by the IPOs of UiPath (robotic process automation) and Confluent, (data / application integration platform).

Low code / no code automation

Application development is also coming closer to non-developers with low/no-code approach and automation.

Software engineering, traditionally owned by IT and software engineers, has always been coveted by other, non-IT stakeholders in the enterprise. In 1991, Powerbuilder introduced a revolutionary concept of a development framework, aiming at democratizing development by allowing non-software professionals to get access to application development. Perhaps ahead of its time with clunky UX, WYSIWYG, Powerbuilder started the revolution of introducing emergence of ‘citizen-developers’, people who originally participated alongside IT in shaping the application and business models but could not code and create the applications themselves. It also introduced data integration with application logic and object-oriented concepts like inheritance and polymorphism and encapsulation, bringing software engineering to the masses.

Fast forward to 2020’s, virtually every enterprise tool platform and enterprise customer have adopted a low-code/no-code approach. The mission is the same as 30 years ago – to provide easy to use, graphical UI/UX, drag and drop concept to application development and allow business analysts, ‘citizen-developers’ and non-software engineers to create, test and even deploy enterprise applications.

Vertical integration with digital decisioning

The perennial challenge of allowing non-developers to create applications is the conundrum of how deep they can develop without coding and to what extent they can customize complex enterprise cloud applications without IT and coding.

To accelerate digital transformation, enterprise software vendors are emerging mostly from the workflow / BPA world, such as Pega and ServiceNow. They are applying a two prong approach – core tool collection and vertical integration. The workflow vendors have developed (or acquired) a collection of point tools in a core-component framework. Those components typically include AI/ML, reporting, workflow, RPA (Robotic Process Automation), case management, rules engine, decision management, knowledge bases, BPA (business process automation) and process orchestration. Those components typically feature common UI and work across a normalized data model and unified architecture.

But that is not enough. To satisfy modern rapid digital transformation needs, in case of fintech enterprise customers (i.e. banks, insurance companies and financial services) also now require pre-built workflow, data and application models. These vertical templates are higher level and more specific, providing out-of-the box, drag/drop solutions like credit card operations, loan management and payment operations. Using the low-code approach, a business analyst can graphically drag/drop pre-defined steps into a loan origination workflow with pre-defined commonly used tasks, created using best practices defined by the ‘centers of excellence’. Companies like UIPath have created a 3rd party marketplace for additional steps and templates created by analysts and consultants. (Those steps could be ‘get customer data’, ‘OCR input form’, ‘scrub customer data’, authorize user’, ‘assess risk profile’ etc.).

Beyond the top level tasks, the functionality ultimately becomes more complex and the sophisticated customer needs powerful decision capabilities to introduce their own business rules and implement proprietary features. The ‘secret-sauce’, which separtes most common steps from proprietary concepts distinguishes top corporations from the competition, requires more sophisticated digital decisioning tools. These digital decisioning tools enable non-developers to customize and manage decision logic, implement AI/ML features, run A/B testing and visualize performance results on training and production data in real time.

To satisfy most common customer base, digital workflow vendors typically provide rudimentary business rules integrated in their low-code platforms and further integrate them with the downstream workflow platforms and vertical ecosystem vendors (i.e. FiServ, Jack Henry, SAP, Salesforce and FIS in banking for example).

The most sophisticated and demanding customers, however, need a more sophisticated set of digital decisioning tools like standalone professional DM platforms. To simplify and visualize this complex decision management, a new generation of low-code digital decision management platforms like Sparkling Logic emerged. These platforms integrate historical business rules engine, data and AI, demystifying machine-learning and providing low-code approach to development and monitoring of application logic performance, continuously as the business logic and training data change and drift.

The pandemic, hybrid work and pervasiveness of the cloud computing have irreversibly changed the software application development. Enterprise customers are seeking and deploying better, faster, more integrated software tools. DevOps integration, low-code, vertical templates, integrated AI and digital decisioning are becoming a new normal while defining the next generation of applications, created not only by software engineers, but by mere mortals across the enterprise.

About

Davorin Kuchan is the CEO of Sparkling Logic, Inc, an AI-driven digital decision management enterprise tools platform. Major enterprise customers like Equifax, Centene, First American, Nike, SwissRE and Enova deploy and integrate Sparkling Logic SMARTS digital decision engine. Sparkling Logic, Inc is based in Sunnyvale, California. http://www.sparklinglogic.com


© 2022 SparklingLogic. All Rights Reserved.