The Practical Guide
What is a Data Product?
Last updated on April 11, 2024
Table of Contents
A data product is a reusable data asset that bundles data together with everything needed to make it independently usable by authorized consumers.
06
The data product lifecycle
Data-driven enterprises have one thing in common: they build data products, as opposed to one-off data projects. Data products are reusable assets focused on business outcomes.
Every data product follows a lifecycle, similar to that of a software product, to iterate and assure that it delivers the desired business outcomes. It looks something like this:
Define
A data product is defined by its business objectives, governance constraints (security and privacy), and data asset inventories. Its design is a function of how the data is to be productized, for consumption via services.
Engineer
A data product is engineered by locating, accessing, and integrating the needed source data, and then processing it as required. Data services are created to provide consuming applications with access to the data, while data pipelines are engineered to deliver the data to authorized analytical data consumers. The data product is versioned and designed to comply with performance SLAs.
Test
Data products only add value once they’re run in production. But, before that can happen, they must be tested to ensure that the datasets they deliver perform as expected, and are fresh, cleansed, complete, compliant, and ready for high-scale consumption.
Deploy
The data product is deployed, monitored (for usage, performance, and reliability), maintained, and supported – to quickly address any issues that may arise.
Enter the data product manager
Similar to the software product manager, a data product manager is responsible for delivering business value and ROI from the data products – defining their goals and priorities (together with data engineers and consumers), and continuously working to ensure that the promised value is attained.
Why the cycle?
Data teams are constantly experimenting – implementing new services, deploying them, and monitoring the results. The quicker they go through the cycle, the quicker they learn, and the quicker they deliver incremental value to their customers.
07
Project vs product data mindset
Traditionally, most companies are project-driven when it comes to data.
For example, if a business domain requires a particular dataset to address a particular need, it typically raises a request with the central data engineering team. That request represents a project to identify, collect, prepare, and deliver the relevant dataset to the business domain. This same pattern is followed every time a new use case emerges, from any domain in the organization.
This “data as a project” approach has some major drawbacks, including slow time-to-delivery, lack of reuse, rigidity, and risk of delivering wrong, and/or incomplete data.
On the other hand, a product-driven approach keeps the entire enterprise’s data needs in mind.
Data products can be reused to support any number of use cases, serving any number of domains
.png?width=900&height=436&name=Untitled-1%20(1).png)
A project-driven approach to data drives greater complexity and minimal reuse,
compared to the simpler, more agile product-driven approach to data.
09
Getting started with data products
Deploy the right platform
K2view provides a Data Product Platform to engineer, test, deploy, and monitor data products, in serving a broad variety of workloads.
The platform’s Data Product Studio enables data teams to quickly define and maintain the metadata for data products, including the data schema, connectors, sync policies, data transformations, governance, and more.
Once deployed, a data product manages its dataset within its own hyper-performance Micro-Database™, to support enterprise scale, resilience, and agility.
A “Customer” data product collects data from all sources, prepares it,
and delivers it to authorized data consumers – end-to-end – in real time.
Appoint data product managers
Data product managers need a broad range of skills in the areas of data, analytics, enterprise applications, business analysis, and DataOps. Ultimately responsible for the entire data product lifecycle, they:
-
Develop data strategies, determine performance metrics, and promote data literacy across the company
-
Ensure business value from data, and maximize the Return On Data Investment
-
Close the gap between business and IT, by communicating the needs of data consumers across business domains and working with data engineers to improve data accessibility.
In the same way a software product manager defines user needs, prioritizes them, and works with R&D to assure delivery, data product managers collect the needs of data consumers, and collaborate with data engineers and data scientists to deliver on them.Data product managers are the ultimate definers of the data – and also the main champions of data products within the organization.
Go for flexibility
To maximize flexibility, enterprises should choose a platform that deploys on premises, in the cloud (iPaaS), or across hybrid environments – with support for all modern data architectures.
A data fabric architecture is a modular data management framework, which integrates with your existing data and analytics tools. It assumes that data products are defined by a central data and analytics organization, and adapt over time based on automated analysis of active metadata.
A data mesh architecture shifts data strategy to a federated data network. It gives business domains the autonomy and tools to create data products for their needs, and creates a common framework for building, and scaling, product-driven data solutions, in real time.
There are advantages and disadvantages to data mesh vs data fabric, but both architectures leverage data products as a fundamental construct.
