Data Scientists vs Tools

A recent FastCompany article pointed to a shortage of data scientists. It cites a McKinsey Global Institute report, which projects that by 2018, the US alone may face a 50-60% gap between supply and demand of deep analytical talent.

This shortage in statistical and analytical talent is part of the reason why we built Polychart. What can managers do when faced with such shortage? How can we empower those who do not have training in statistics and machine learning to analyze large amounts of data?

We believe that the answer lies in Advanced Data Visualization. Data visualization turns data -- something abstract and difficult to interpret -- into lines, points, and shapes -- objects that our visual system processes well. Patterns that are buried within millions of rows of data can surface easily with the right visual. This is because our brain is hardwired to detect certain visual patterns: can you tell whether class A or class B has a higher average, and whether the difference in mean is significant?

A 5.665396 A 5.300887 B 6.280863 B 6.283800 A 7.171414 B 6.417034 B 8.521771 A 4.706804 A 4.312119 A 4.354617 A 5.786552 A 5.438225 B 6.242482 B 6.265350 B 6.980730 A 3.767677 A 4.052122 A 6.778892 B 8.155159 B 6.252163 A 4.095930 A 6.145099 A 4.994415 A 7.626136 B 6.884315 B 5.589942 B 6.074620 B 5.449216 A 4.626119 A 5.545607 B 6.798265 B 8.806959 B 7.117702 A 4.973188 A 4.212082 A 4.825620 A 6.408064 B 6.982158 B 8.310192 B 8.389218 B 6.855587 B 7.015916 B 5.105175 A 3.963125 A 4.441499 A 7.290045 A 5.051415 B 7.489135 B 8.126187 B 7.077561 B 6.950186 B 8.080576 A 5.644856 A 5.116219 A 3.062682 A 3.980738 A 3.668895 B 7.537187 B 7.139165 B 6.891767

Well, how about now?

A statistician may run what is called a “t-test” to answer the above questions. The rest of us can still be fairly certain that the difference is pretty significant. This is what we call a “squinty eye test”: if you can see the difference in a chart when squinting your eyes, it’s probably significant.

So detecting patterns given the right visual is easy. The hard part, then, is finding the right visual. That’s where Polychart comes in.

A while ago Bret Victor, who designed user interfaces for Apple, gave a talk titled “Inventing on Principle”. One of the core principles behind all his demos is to have little resistance between having an idea and having that idea implemented. We think that this is an important issue in data analysis.

It’s one of the main reasons why we made it so easy to explore data iteratively in Polychart. All that’s necessary to iterate from one chart to the next is a simple drag-and-drop action. The new chart is created at the speed of thought.

Forbes recently noted the provocative title of a proposed SXSW panel: “The Data Scientist Will Be Replaced By Tools”. While we don’t think that tools can replace everything a data scientist does, we hope that new tools like Polychart will be able to help data scientists to do their jobs more effectively. Further, we hope that new tools will be accessible enough to those who do not have training in advanced statistics, and allow for greater democratization of data.