Page 5 - Building Big Data Applications

Preface

In the world that we live in today it is very easy to manifest and analyze data at any given
instance. Space a very insightful analytics is worth every executive’s time to make decisions
that impact the organization today and tomorrow. Space this analytics is what we call Big
Data analytics since the year 2010, and our teams have been struggling to understand how to
integrate data with the right metadata and master data in order to produce a meaningful
platform that can be used to produce these insightful analytics.
Not only is the commercial space interested in this we also have scientific research and
engineering teams very much wanting to study the data and build applications on top off at.
The effort’s taken to produce Big Data applications have been sporadic when measured in
terms of success why is that a question that is being asked by folks across the industry. In my
experience of working in this specific space, what I have realized is that we are still working
with data which is lost in terms of volumes come on and it is produced very fast on demand
by any consumer leading to metadata integration issues. This metadata integration issue can
be handled if we make it an enterprise solution, and all renters in the space need not
necessarily worry about their integration with a Big Data platform. This integration is handled
through integration tools that have been built for data integration and transformation.
Another interesting perspective is that while the data is voluminous and it is produced very
fast it can be integrated and harvested as any enterprise data segment. We require the new
data architecture to be flexible, and scalable to accommodate new additions, updates, and
integrations in order to be successful in building a foundation platform. This data architec-
ture will differ from the third normal and star schema forms that we built the data warehouse
from. The new architecture will require more integration and just in time additions which are
more represented by NoSQL database architecture’s and how architectures do. How do we
get this go to success factor? And how do we make the enterprise realize that new approaches
are needed to ensure success and accomplishing the tipping point on a successful
implementation.
Our executives are always known for asking questions about the lineage of data and its
traceability. These questions today can be handled in the data architecture and engineering
provided we as an enterprise take a few minutes to step back and analyze why our past
journeys journeys were not successful enough, and how we can be impactful in the future
journey delivering the Big Data application. The hidden secret here is resting in the farm off
governance within the enterprise. Governance, it is not about measuring people it is about
ensuring that all processes have been followed and completed as requirements and that all
specifics are in place for delivering on demand lineage and traceability.
In writing this book there are specific points that have been discussed about the ar-
chitecture and governance required to ensure success in Big Data applications. The goal of
the book is to share the secrets that have been leveraged by different segments of people in
their big data application projects and the risks that they had to overcome to become
successful.
The chapters in the book present different types of scenarios that we all encounter, and
in this process the goals of reproducibility and repeatability for ensuring experimental

xiii