Page 160 - Building Big Data Applications

Chapter 9 Governance 159

The value of having metadata can be easily established in these situations by
measuring the cost impact with and without metadata:
Cost of commissioning new applications
Learning curve for new employees
Troubleshooting application problems
Creating new business intelligence and analytics applications
Data auditing
Compliance auditing
Traditionally in the world of data management, metadata has been often ignored and
implemented as a postimplementation process. When you start looking at big data, you
need to create a strong metadata library, as you will be having no idea about the content
of the data format that you need to process. Remember in the big data world, we ingest
and process data, then tag it, and post these steps consume it for processing.
Let us revisit the metadata subject area first and then understand how it integrates
into the world of big data.
Source of metadata include the following:

Metadata generated automatically for data and business rules
Metadata created by the designer of the systems
Metadata procured from third party sources
There are fundamentally five types of metadata that are useful for information
technology and data management across the enterprise from transaction processing to
analytical and reporting platforms, these include the following:
Technical metadatadconsists of metadata that is associated with data trans-
formation rules, data storage structures, semantic layers, and interface layers.
Metadata for data model and physical database includes length of a field, the
shape of a data structure, the name of a table, the physical characteristics of a
field, the number of bytes in a table, the indexes on a table, and DDL for the
table
Business processing metadata includes information about:
- The system of record for a specific piece of data,
- Transformations that were performed on what source data to produce data
in the data warehouse/data mart,
- Tables and columns used in the particular process in the data warehouse/
data mart and what do the transformations mean
Administrative metadata.
Business metadatadBusiness metadata refers to the data describing the content
available in the data warehouse/data mart and describes the following:
The structure of the data
The values that are stored within the attributes