Big Data - The Data Deluge

In today’s world, almost every enterprise is seeing an explosion of data. They are getting huge amount of digital data generated daily. Almost every growing organization wantsto automate most of its business processes and is using IT to support every conceivable Business function. This is resulting into huge amount of data being generated in theform of transactions and interactions. Web has become an important interface for interactions with suppliers and customers generating the huge amount of data in the form ofemails etc. Besides this, there is a huge amount of data emitted automatically in the form of logs like network logs and web server logs.

Various Telecom Service Providers get huge amount of data in the form of conversations and Call Data Records. Various Social N/W Sites have started getting TBs of data everyday in the form of tweets, blogs, comments, photos and videos etc. Facebook generates 4TBs of compressed data every day. Web Companies like these get huge amount ofclick stream data generated daily as well. Hospitals have data about the patients, their diseases and the data generated by various medical devices as well. Sensors used invarious machines used for production keep generating so much of event data in seconds. Almost every sector like transport, finance is seeing a tsunami of Data.

Such huge amount of data needs to be stored for various reasons. Sometimes any compliance demands more historical data to be stored. Some times organizations want tostore, process and analyse this data for intelligent decision making to get the competitive advantage.For example analyzing CDR data can help a service provider know theirquality of service and then make the necessary improvements. A Credit Card company can analyze the customer transactions for fraud detection. Server logs can be analyzedfor fault detection. Web logs can help understand the user navigation patterns. Customer emails can help understand the customer behavior, interests and some time theproblems with the products as well.Now the important question that arises at this point of time is how do we store and process such huge amount of data most of which is Semi structured or Unstructured.