Metadata provides all the characteristics, properties, and context of data within a data fabric. Read on to learn how to optimize both with data products.
Table of Contents
What is Metadata?
What is Data Fabric?
Metadata in the Context of Data Fabric
What’s AI Got to Do with It?
What is Metadata?
Simply put, metadata is information about data. It provides the characteristics, properties, and context of data, and is frequently used to describe and document specific datasets.
The 3 most common types of metadata are listed below:
- Administrative metadata provides information about the management, maintenance, and use of data. It can include details about the ownership, rights, and permissions associated with the data, as well as information about its creation, modification, and preservation.
- Descriptive metadata provides basic information about a data element, such as its name, title, author, and subject matter. It is used to describe the content, context, and structure of the data.
- Structural metadata describes the organization and relationships of data elements within a dataset, including data types, formats, and structures.
Metadata can be stored and managed in many different ways, such as in databases, files, tables, or specialized metadata repositories. It’s often used in conjunction with a data catalog, data integration tools, data lakes, and data warehouses, to support the discovery, access, and use of data within an organization.
What is Data Fabric?
Data fabric is an architecture that enables the integration, management, and distribution of data from a wide variety of sources. It supports the flow of data between different systems and applications, and delivers a unified, 360-degree view of data for authorized data consumers and applications.
Data fabric architecture makes it easier for enterprises to access, analyze, and share data – and to enable agile development of applications. It includes data integration tools, and infrastructure components like data lakes, data warehouses, and cloud-based data processing and storage platforms.
Metadata in the Context of Data Fabric
In the context of data fabric, metadata supports:
- Data discovery, where metadata lets users and apps find and access the sought-after
- Data governance, where metadata is employed to store and enforce a company’s policies and rulesconcerning data management and usage.
- Data lineage, where metadata tracks data sources, transformations, and data pipeline
- Data quality, where metadata monitors and improves the accuracy, completeness, and compliance of the data– and recommend the use of data masking tools, when required.
Metadata plays a critical role in enabling a data fabric to persist, and provide access to, data – to support business applications and services.
What’s AI Got to Do with It?
Metadata drives data fabric, both passively and actively. According to Gartner, metadata is “data in context”, in terms of the what, when, where, who, and how aspects of data.
A metadata-driven data fabric listens, learns, and acts – on the metadata. It applies continuous analytics over metadata assets, and can reduce data management efforts, including design, deployment, and operations. It’s an amazing value proposition.
Gartner recommends that organizations leverage their metadata, by deploying active metadata, and applying AI, ML and knowledge graphs to optimize data management and access.
The problem is that organizations struggle to maintain an always up-to-date, rich, and well-managed metadata layer across all their data sources. This makes the deployment of active metadata with ML-driven data management recommendations a largely theoretical and unattainable goal.
But a data fabric, based on a data product approach, is more pragmatic and achievable.
