The Journey To Big Data

The fastest growing new companies are building their business on data.  Most of them don't buy or sell data itself, but they gather massive amounts of data and use it to drive incredible growth rates in their "core business".  Let's look at three of them...

LinkedIn has transformed recruiting by by building a professional social network for employers and employees.  They use predictive recommendations to accelerate network building and try to match candidates with jobs.  LinkedIn recently completed a successful IPO with a valuation over $7B as of Dec. 7, 2011.

Facebook is transforming B2C and B2B marketing by gathering massive amounts of data about it's subscribers and sell advertisers information and opportunities to reach their members.  Facebook is rumored to be planning an IPO that could value the company at more than $100 Billion.

Zynga perfected A/B testing to grow their network and increase revenue per subscriber.  Zynga's apparent product is virtual games where they sell small objects for a few dollars each, but the real magic is their ability to grow the network and compare performance of products realtime based on tiny changes in the product appearance.

We are now starting to see enterprises in all sectors embrace this "big data" to transform their own business.  One retail bank embraced big data to increase revenues and reduce customer churn. This "old way" of recommending products to customer was based on the best guess of the teller, using little information about the bank's product margins or the specific customer.  

They first brought in product information so they could shift to "product-profit" based recommendations, picking the product with the best margin for the bank.  Then, they brought in customer profile information so they could start picking the most profitable product that the customer was actually likely to use.  Finally, they added realtime customer activity information to identify customers that are beginning to migrate away from the bank, so they could intercede and prevent customer churn.

Most companies I talk to would like to do the same thing.  So, why isn't everyone capitalizing on big data?  There are several challenges with the current model that require a new approach.  Technology, People & Process, and Applications.

Technology

Most traditional BI systems are based on scale-up physical architectures that are rigid and difficult to expand.  Traditional systems tend to also focus exclusively on structured data and it tends to be limited in size, so the repository has stale, summary data.

If your search engine ran like your data warehouse, it would look like this...  Call Google and tell them you want to search for "Graig Nettles".  A month later, they tell you the Nettles data is loaded.   You start your query.  You go to sleep, wake up, go for a jog, grab breakfast, shower, get dressed, and your query is half done.  Later that night, the query comes back and you want to drill in on the data for 1978.  Unfortunately, that isn't loaded, so you call Google again.

What you really want is a more iterative process that operates on all the data sources.  You want a Google-like experience from your analytics system.  You want to run hundreds of queries a day instead of one.  You want to realize that you can improve the results by tweaking your query terms.  You want to drill into the details of the data and ask another question.

Today's big data analytics systems have a fundamentally Google-like architecture to enable this kind of approach.  It includes all the data - structured and unstructured at Petabyte scale.  It can load data realtime and execute queries in seconds because the foundation is a scale-out virtual architecture providing massive scalability and elasticity.

Implementing this big data analytics platform is step 1 on the Journey to Big Data.  It's a technology focus and it needs to have cloud infrastructure and analytic engines for structured an unstructured data.  You will be able to analyze all the data an get faster answers.  That's Big Data Analytics.

EMC provides the Big Data Analytics platform in the Greenplum Database for structured data and Greenplum HD for unstructured data (Hadoop workloads).  Both are built on cloud architectures to provide the scalability needed for Big Data Analytics.

People And Process

Traditional analytics are based on an administrative bottleneck.  All requests for new reports or new data must pass through the gatekeeper.  Requests tend to take weeks or months to fulfill.  Finally, reports are run in isolation with little collaboration among analysts around what they found and what the could potentially find in only...

The collaboration model for big data should look more like Facebook - search for people you know, publish your latest activities and insights. 

Improving your collaboration is a start, but it's not enough.  You also need to build new skills.  The Data Scientist is a key role in leveraging the value of Big Data to transform your business.  To read more about Data Scientists and a recent EMC-sponsored Data Science survey, check out Chuck's Blog on the topic.

Changes to people and process are the pivotal component in phase 2 of the Big Data Journey.  Establish a data science practice and provide self-service and collaboration capabilities to all members of the data science team.  Phase 2 in the Journey moves organizations from Big Data Analytics to Agile Analytics.  

Today, EMC is launching a major step forward for Data Science teams - Greenplum Chorus.  Chorus is an Analytics Productivity Platform that brings Facebook-like collaboration to the world of Data Science.  Data Science team members can post updates, share their work, find and follow others or specific data sets and analysis.

Applications

As you exit phase 2, the right technology platform is in place and there is a Data Science team collaborating together and building new insights that can transform your strategy.  For these insights to have an impact, they must be pushed into every part of the business.  That's a departure from the traditional model.

The traditional model for predictive analytics started with the same stale, summary data that was in the warehouse.  The analytic insights were commonly deployed in isolated applications and not available to the majority of business users.  That means that most business decisions were made based on someone's "best guess".

One of the best places to see where this is done right is to look at Amazon.com.  Amazon.com has over 137,000,000 customers and 895,000,000 products, yet they make personalized recommendations to each and every one of their shoppers.  These recommendations are based on all the data and some pretty advanced predictive models.  The recommendations are also embedded within the storefront. Would you ever really visit another application at www.amazonrecommendations.com?

So the principles for scaling Big Data throughout an organization to truly transform your business is to embed the predictive analytics within the applications.  Predictions must also be data driven and real time.  If the system offers bad predictions or recommendations based on last week's activity, there won't be much improvement in business results.

Phase 3 in the Journey To Big Data is Big Data Enabled Applications.  In Phase 3, you transform from Agile Analytics to a Predictive Enterprise and monetize your data.  Phase 3 enables real time decisions by embedding predictive analytics into the applications that are used to run the business.

The Journey To Big Data Starts Now.  Today EMC is announcing the Greenplum Unified Analytics Platform - the essential Platform for the Journey To Big Data.  It includes the Greenplum Database, Greenplum HD and Greenplum Chorus.

Read more about Big Data at: http://www.emc.com/bigdata .