What is a Data Marketplace?
A data marketplace is a “store” where individuals and organizations can exchange data. Data marketplaces offer a one-stop, self-service shopping experience. Data consumers can search for the data that they need and access it as a service (usually for a fee). Typically, data marketplaces act as an intermediary between data providers and businesses. However, many organizations are creating an enterprise data marketplace (EDM) to make their own data more accessible to their employees. EDM goes beyond traditional data warehouses where the focus is on data storage. In addition to data storage, EDM focuses on data retrieval. How can employees securely discover and access data without DBA/IT or being proficient in query language (ex. SQL)?
EDMs are often consists of three layers:
- Raw Data: The bottom layer contains the raw data from various data sources. Sources can be global such as ERP and CRM systems or local such as MySQL databases and shared Excel spreadsheets.
- Transformed Data: The middle layer contains data that has been transformed (cleaned and reformatted) so that it can be integrated. For example, one source may store the data and time as separate fields while another may store them together as datetime and yet another as timestamp. This middle layer will reformat the data so that they’re all consistent.
- Aggregated Data: The top layer aggregates data by summarizing the integrated data by attributes or groups defined by the business. For example, the count and sum of all purchases in the last year broken down by month and grouped by persona.
Enterprise Data Marketplace Challenges
While defining an enterprise data marketplace is straightforward, building one is complicated:
- Data is Messy. This is an ongoing problem which is not immediately solved by an EDM. Regardless if you ask a DBA to run a query or you pull a report from a user interface, if the data is bad, then the results are going to be bad.
- Beauty is in the Eye of the Beholder. Every business line and department has a different view of the company and therefore a different perspective on what data and information is important to them. Data integration is not just a reformatting exercise. Often it requires additional data enrichment. And defining the aggregations is a major project in itself. Is “month” calendar month or fiscal month? Does a “week” start on a Sunday or a Monday?
- Organizations are Dynamic. Businesses will eventually go through restructuring or reorganization even if they don’t undergo an acquisition or merger. Rules and regulations also change. Therefore, what data and information is considered important will also change.
Manage Complexity with Decision Management
While gathering requirements is no easy task, decision management platforms like SMARTS™ can help you transform those requirements into transparent business rules that are easy to manage. Decision management platforms enable businesses to organize their business rules in a manner that makes sense to those responsible for managing the logic rather than the machines that execute them. Is “mgail.com” a common error for “gmail.com”? Beside adding an email validator to your front end, you can write business rules to correct typos and other errors in the middle layer. Where the same attribute contains different information in different formats in different sources, you can write business rules to choose which source will be the ultimate source of truth and what format that data will be stored in the middle layer. In addition to if/then rules, you can write calculation rules to define your aggregations for the top layer.
The SMARTS™ Advantage
SMARTS™ not only provides a platform for authoring and managing business rules, but also enables users to do so in the context of their data and data analytics. Users can quickly run regression tests, simulations, and experiments to ensure the quality of their business rules before they deploy them. In addition, businesses can leverage SMARTS™ Orchestrations to automate data retrieval from the bottom layer, data transformations and storage in the middle layer, and data aggregations and storage in the top layer.
Learn how organizations leverage Sparkling Logic’s SMARTS™ Data-Powered Decision Manager for “micro-calculations” and rules-based personalization.