Big Data or Fast Data?

How big is Big Data? Really really big!

Why the recent hype over Big Data? What has brought us this change in data processing requirements? What about plain old operational and transactional data that was used to build the data warehouses of yesterday? What ever happened to good old ETL or extract, transform, and load. Is Big Data really that much different from a ye olde big data warehouse?

The short answer is no, and while big data does amount to a tremendous amount of raw bytes of data, perhaps a better fitting expression is fast and big data. This is because the rate of data collection has accelerated so much in the past few years that computer systems, databases, and analytics tools have struggled to keep up with the volumes of data accumulation. Just ask the NetApps and the EMCs of the world as they are selling more network based storage appliances then ever before.

Big Data describes the process of the collecting, storing, and analyzing of fragments of information that can be rapidly assembled to identify subtle macro trends or create actionable profiles that precisely target unique individuals. 

The size of the problem and the pending opportunity was characterized recently by IBM:

"Every day, we create 2.5 quintillion bytes of data - so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase-transaction records, and cell phone GPS signals, to name a few."

Big Data Leads to Big Market Opportunities

Include information about who individuals are, what they read, when they are in a particular place, where they shop, why they buy and how they feel about public policy and you have a wealth of valuable (Big) data. 

The commercial potential for Big Data has been one of the primary reasons Facebook achieved a record high valuation of $100 billion when they went public in their recent IPO.  The company is able to gather more personal information about people and their friends than any social network in operation. How they mine that personal data to drive more  profit is what analysts are trying to figure out.

For example, Facebook's future revenue most likely will not depend on selling banner advertising, whose pricing and efficacy will eventually decrease as the supply of page views expands exponentially. 

 Facebook will earn future revenue by innovating ways to commercialize the enormous amount of data it continues to accumulate.

It is difficult to predict what the winning Big Data app will be in this area. Facebook might not know but the $4 billion in cash collected at the IPO window equips the company to fund R&D for projects and business pilots for years to come. Eventually, Facebook, IBM or someone else will come up with the way to turn the growing nuggets of data into gold.  

As further evidence of the explosion of Big Data, the Pew Internet Project, in collaboration with Elon University’s Imagining the Internet Center, recently published a report on the future of big data, based on a “non-random online sample of 1,021 Internet experts and other Internet users.”

Fifty-three percent of the survey respondents agreed with the positive scenario presented to them: “Thanks to many changes, including the building of ‘the Internet of Things,’ human and machine analysis of large data sets will improve social, political, and economic intelligence by 2020. The rise of what is known as ‘Big Data’ will facilitate things like ‘nowcasting’ (real-time ‘forecasting’ of events); the development of ‘inferential software’ that assesses data patterns to project outcomes; and the creation of algorithms for advanced correlations that enable new understanding of the world.

Overall, the rise of Big Data is a huge positive for society in nearly all respects.” The report includes many specific quotes to support this view. For example, this one from Tiffany Shlain, director and producer of the film Connected and founder of The Webby Awards: “Big Data allows us to see patterns we have never seen before. This will clearly show us interdependence and connections that will lead to a new way of looking at everything. It will let us see the ‘real-time’ cause and effect of our actions. What we buy, eat, donate, and throw away will be visual in a real-time map to see the ripple effect of our actions. That could only lead to mores-conscious behavior.”

Not all survey respondents were so enthusiastic about big data. Thirty-nine percent agreed with the negative scenario presented to participants: “Thanks to many changes, including the building of ‘the Internet of Things,’ human and machine analysis of Big Data will cause more problems than it solves by 2020. The existence of huge data sets for analysis will engender false confidence in our predictive powers and will lead many to make significant and hurtful mistakes. 

Some believe analysis of Big Data will be misused by powerful people and institutions with selfish agendas who manipulate findings to make the case for what they want. And the advent of Big Data has a harmful impact because it serves the majority (at times inaccurately) while diminishing the minority and ignoring important outliers. Overall, the rise of Big Data is a big negative for society in nearly all respects.” Here’s a sample quote from Marcia Richards Suelzer, Wolters Kluwer, a global information services company:  “We can now make catastrophic miscalculations in nanoseconds and broadcast them universally. We have lost the balance inherent in 'lag time.'”


One thing is clear, Fast Big Data is here to stay. Companies are racing to drive profits from platforms that can accommodate the growing volumes of data. Whether the social outcomes from its use will be positive or negative only time will tell. 

Enjoyed the article?

Sign-up for our free newsletter to kick off your day with the latest technology insights, or share the article with your friends and contacts on Facebook, Twitter or Google+ using the icons below.

E-mail address


White Papers