The Trellance Data Blog

Big Data/Analytics: The Curse of the Black Box

Posted by Peter Keers, PMP on Jan 29, 2015 1:11:00 PM


Credit unions seeking to improve their Big Data/Analytics capabilities face a classic choice: build (DIY) or buy?

The “buy” option is often perceived as the more expensive option. Many credit unions see prices of vendor solutions as budget busters.

The “build (DIY)” option appears much more attractive. Typically, a new position within the credit union is created to handle the new Big Data/Analytics initiative. Whether it is through a promotion or external hire, the cost seems much less and more controllable. Another perceived advantage is the effort will be tailored to the exact needs of the credit union. There will be no need to deal with a “one size fits all” vendor solution.

As these “build (DIY)” scenarios unfold, the best case situations result in improved reporting and analysis. Having an individual dedicated to building new reports and running ad hoc queries is a welcome improvement over the old way of doing things.

Yet, in many cases there is a dark side to this happy state of affairs. A “black box” is quietly forming behind the scenes.

Like many home grown solutions, there are strong temptations to skimp on documentation, employ unstable and informal processes, and rely on less than adequate tools (e.g. – MS Excel) to get the job done. The credit union’s Big Data/Analytics program is at risk because a lot of the important information about how it operates is dependent on one person. And with the current boom in Big Data and Analytics, that one person’s skills are in high demand in the labor market.

When that person leaves for another opportunity, the credit union’s promising Big Data/Analytics program is transformed into a nearly unusable black box.

Of course, the first response of the credit union is to fill the vacated position. The hiring process may be a shock because the cost of like talent is climbing quickly and finding a qualified person may take longer than expected.

Even if a new employee can be quickly brought on board, he or she is faced with making sense of the black box. The lack of documentation in particular can make for a very steep learning curve. In addition, there is the risk of the new hire transforming the old black box into his or her own black box.

In this situation, the “build (DIY)” choice doesn’t seem so cost efficient after all.

What advantages does that “expensive” vendor solution offer in contrast?

  • Vendor systems are standardized and well-documented.

  • Often the solution has been already successfully used in the credit union industry.

  • When a key employee leaves, the vendor can fill in temporarily and provide training for the new hire.

  • The software is supported by experts who have the knowledge and means to solve problems quickly.

  • A user community will typically form around the standardized solution where ideas and expertise can be shared.

Purchasing a vendor solution will not eliminate the Big Data/Analytics expert role at the credit union. That role is often crucial in improving the organization’s capabilities. Rather, such a solution will allow the initiative to grow in a much less risky fashion and prevent the formation of the dreaded black box.

New Call-to-action

Topics: Big Data, Data Integration, Analytic Data Model