On October 16, 2014 I attended an afternoon of presentations and discussions on Big Data with a focus on information and insights for capital markets professionals. The event was a joint live broadcast of theCUBE and #BigDataNYC.
The meeting started with Wikibon Senior Analyst Jeff Kelly presenting compelling findings and insights on Big Data adoption in the enterprise and analysis of the companies best positioned to take advantage of the Big Data opportunity.Mr. Kelly’s presentation was followed by a panel discussion, facilitated by Wikibon CEO David Vellante. The panel discussion was focused on how Big Data executives are positioning their companies to compete and grow in this exploding market. Panelists include Wikibon's Jeff Kelly, Tresata CEO Abhi Mehta, Amy O'Connor, who ran Nokia's Big Data practice and former Cowen and Company Analyst Peter Goldmacher.
My goal in attending the presentation was to understand the Big Data trends and what credit unions should take away from Wikibon’s perspective on Big Data. Below are the 3 trends that credit union CEOs should be aware of:Visualization:
This is the software that allows a user to consume the data across a variety of platforms (e.g. -laptop, tablet, and phone). This space has traditionally been dominated by traditional business intelligence software companies like Cognos (now IBM), Business Objects (now SAP), Microsoft, and Oracle. The issue that has plagued these BI vendors has been the time and effort required to create reports for consumption by business users. Within the last 5 years, a new breed of self-service based reporting technology has emerged and is being led by vendors like Tableau and Qlik. This is great news for credit unions that often lack the resources to employ trained report writers.Storage Hardware:
The cost of processing power and storage goes up dramatically for companies that manage huge volumes data every day. Twitter, for example is estimated to generate about 12 terabytes of new data every day! The state-of-the-art blade server/SAN solution can get very expensive in these situations. This is where Hadoop comes in. Hadoop, developed by Apache Software, provides a means of distributing data across inexpensive (“commodity”) servers using a new file distribution technology combined with an advanced SQL query language called MapReduce. It is not surprising that Hadoop is used by Twitter to run real time analytics on the troves of structured and un-structured data that is generated every second.
Based on my experience with credit union data and analytics, I think that the need for Hadoop is quite a few years off for about 5% of credit unions and the rest may never need it in its current form. However, if one is interested in looking further into Hadoop, the current vendors are Cloudera, MAPR, and Horton Works.NoSQL:
According to Wikipedia, “A NoSQL database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. Motivations for this approach include simplicity of design, horizontal scaling and finer control over availability. The data structure (e.g. key-value, graph, or document) differs from the RDBMS, and therefore some operations are faster in NoSQL…Barriers to the greater adoption of NoSQL stores include the use of low-level query languages, the lack of standardized interfaces, and huge investments in existing SQL.”
The reality of credit union data today is that it is stored in relational databases that provide the structure necessary to process, retrieve, secure and store transactions. Given the regulatory and security demands placed on credit unions combined with the significant investment in relational database technology it will be a long time before NoSQL becomes a factor. Current NoSQL vendors are MarkLogic and MongoDB.
In evaluating the three Big Data trends above and their impact on credit unions, I came to the conclusion that it would be good for credit unions to keep an ear to the ground on Hadoop and NoSQL but I see no relevance in the next three to five years. Conversely, the great improvements that are being made by companies like Tableau and Qlik, in the area of self-service data visualization will be of great benefit to credit unions as they look for easier and less expensive ways to gain insights from the increasing mass of data they collect each day.
The conference ended with a great panel discussion on the topic of Big Data which I thought to be highly relevant to the credit union industry. Below is a summary of my 3 key takeaways:
- Data is the new infrastructure and what you get out of it is competitive advantage. Companies have already proven to us the high value of information, Google knows what you ask, Facebook knows who you like, Amazon knows what you buy, and Netflix knows what you watch.
- What data do you have or what data can you create to differentiate yourself? People who are thinking about how data can solve problems will be the winners.
- The Internet of things is in its infancy since 90% remains to be connected. We spend most of our time using the internet as a search tool. As more data is collected from the internet of “things” search will be replaced by suggestions based on the data that is collected from the ways that we interact with the world around us.
Big Data presents a great opportunity for credit unions to adapt to a changing marketplace but one must act to convert this opportunity into reality.