Off the Charts
There is a fascinating collection of charts that have been shared across social media over the last year and half. Designed to look like subway maps, colorful lines snake around a page where metro stations represent the different skill sets a data scientist must master in order to be successful. It is, at the same time, an impressive and frightening amount of knowledge to command. And despite its expansiveness, the chart is not complete.
The chart leaves off intuition. In our rush to produce “skill ready” analysts and data scientists to fill the empty positions at companies, we’ve forgotten the art of research. It is analogous to trying to create more authors by just teaching grammar and spelling but not exploring the vital role of imagination and storytelling. Analytics and Data Science, in a world that now understands the value of all the information it stores, cannot simply revert to regurgitating information pulled from large databases. Without intuition to help drive the research and data discovery process, we provide a disservice to the decision makers we aim to help by not properly converting data and metrics into actionable intelligence or insight. Numbers can be illuminating but, provided in a vacuum with no context, are not useful and serve to undermine the premise that data can play an integral role in any decision. By passing data and metrics through an intuition filter, we increase the value and relevance of the information.
Intuition is that gut-feeling, that hunch, that subconscious reasoning that drives us to make connections that we don’t have evidence or analytic data to support. Intuition plays a critical role in many stages of data analysis and insight development. While it would appear that a numerically driven process and a subconscious process should be diametrically opposed, you will find that the most successful “quants” live and work in the space between intuition and data.
From the moment a project begins, at the definition and hypotheses formulation stage, intuition provides the over-arching direction
So, how do they not just coexist but support each other? To understand, we first must explore the different stages of business research.
From the moment a project begins, at the definition and hypotheses formulation stage, intuition provides the over- arching direction. Imagine, if among the terabytes of information to sift through, an analyst charged with finding interesting connections or answers to a problem, had no idea of where to start first. To reuse my earlier analogy, it’s the data science equivalent of a writer staring at a blank page without an inkling of what story to tell. Where to begin? Crunching through data at random hoping something useful might fall out would be a colossal waste of time and energy.
Intuition is what provides that initial thought on where to begin the deep data dive and, equally as important, where not to. In some cases that gut direction is as specific as “I’ve got a feeling something interesting is happening when these two events occur” and other times it’s vague, providing only a general guidance.
Early on in my current role as head of analytics at a financial services company, my team and I challenged key assertions in the narrative of our “typical” customer. The narrative was well accepted as company truth with many initiatives built around these premises. But something didn’t make sense. We couldn’t help but feel maybe we misunderstood our customer and that the poor results we were seeing in different campaigns were not failures in marketing or product design, but because the campaigns were designed around the wrong set of facts. It was listening to our intuition which guided us to begin a deep data dive in previously unexplored areas of our database. We discovered that our customer story was indeed off in several key places. Armed with this direction, we built a comprehensive and data-derived consumer portrait.
During the analysis stage, once all the information has been collected and studied, intuition provides an important check. A good analyst will always look at the results from a holistic perspective: does what I’ve found make sense? In other words, what does my intuition say about these results? One of the hardest challenges for anyone who spends time analyzing information is to doubt a conclusion drawn from data until it is proven beyond a reasonable doubt. Too often we believe what the numbers say without pausing to think if the narrative they are telling makes sense. By using intuition to check the meaning of what is derived from numerical calculations, we ensure the viability and usefulness of the information in a real world setting.
Around a year ago, my team and I inherited the responsibility for generating a suite of reports from another business unit. The reports seemed straightforward: join a few tables, pull some information and deliver the results to an operational unit. We realized something just didn’t “feel” right with the results; they didn’t match what we expected to see. We explored the issue and discovered a coding error returned an improperly derived value. Using our intuition as a filter to judge the integrity of the data helped us to isolate the information that needed further study.
For all the benefits that come with integrating intuitive thinking into data analytics, it isn’t done that often. It is a complicated skill to nurture and even a harder one to teach. When assembling a data science or analytics team, search for those people who can tap into their subconscious to guide their research and train them up with the balance of the necessary skills. As Hollywood learned long ago, it is easier to teach an actor a particular skill, like karate, than it is to take a karate expert and teach them to act. The same is true for analysts and data scientists.
Now, how do you find someone with that skill set? They are easy to spot once you know what to look for. Search for candidates who are analytically inclined but are blessed with an inexhaustible curiosity about how and why things work. In an interview, present real issues and problems and see if they challenge and test the underlying assumptions before even tackling the main question. These are the signs of someone relying upon their intuition to help them find a direction. Their backgrounds will be diverse—and not always from the traditional hard sciences or C.S. departments. This type of person will provide value to your organization by developing grounded and actionable insight as opposed to merely extracting numbers using impressive and complex techniques. Embrace their unusual ways of thinking about a problem, because in the end, they didn’t just make the stops on that subway map.