"Black swan" is a catch-all phrase for "outliers" or wildly unexpected events and processes: something such as the 2008 banking crisis, 9/11, the rise of Google et al.
There is an inherent danger that Big Data Analytics with algorithms galore, machine learning and predictive forecasting may fall foul of Black Swans. It may rely on the "narrative fallacy", the way past information is used to analyse the causes of events when so much history is actually "silent". It is the silence - the gap - the missing energy in the historical system, which produces the black swan.
The rise of the "citizen data scientist" could, I suggest, be the antidote to such dangers.
Could Operational and line of business workers (with their deep understanding of cause and effect on the front-line) be such practitioners? -
- Nurses and doctors on hospital wards
- Field-service engineers attending offshore turbine farms
- Quantity Surveyors on construction sites
- Loss-adjusters at accident sites
If BI and Analytics moves beyond beyond just presentation and visualisation to being intuitive self-service BI it can support a broad range of "citizen data scientists".
- Information Consumers
- Information Creators
- Analysts
Consumers
need standard KPI reporting but with the ability to personalise and re-prioritise to reflect daily requirements
Creators
need to test theories and ideas, quey data and author new dashboards and reports- and share them of course
Analysts
want to visually spot outliers and see if they are indeed random and unrepresentative or threatening black swans.
Look for a BI platform that offers this on a secure and scalable architecture e.g. Logi Info. Check out that is integrates closely with a Big Data Analytics platform like HPE's Vertica and IDOL.
That way you can apply an optimal mix of citizen and full-blown data scientists to gain real competitive advantage,
The shortage of data scientists and the resulting high salaries they command is giving rise to new self-service tools, automating all stages of data science so business analysts, marketing managers, IT staff and others could perform advanced analytics as part of their jobs. By 2017, Gartner says, the number of these citizen data scientists in small and large organizations will grow five times faster than the number of highly skilled data scientists. Forrester agrees that the “huge demand” for data scientists will not be met in the short term, “even as more degree programs launch globally.” And the demand for advanced data analysis will only increase in the coming years with the rise of the Internet of Things.
