There is a famous saying that there are “Lies, damn lies and statistics” and this can frustrate analysts who are already working hard to achieve validation and buy-in against more traditional challenges. Your ability to outright lie with statistics is actually limited when compared to other arenas, as you are bound to the unmovable facts of the raw data, but a clever and numerate individual can be creative and selective with what analysis is shown, and two different stories can emerge. To demonstrate I generated a random time-series for analysis and set about building two interpretations.
The random numbers alarmed the markets and public officials this quarter, when they were released this Friday. In Q2 2011, the growth rate had already decreased by 85%, and in Q3 the random numbers began a decline, losing 2.5% quarter-on-quarter.
The quarterly growth rate of the random numbers was down 3%, having decreased for a third straight quarter, spiking fears of a recession. If this trend continues, by mid-2012 the random numbers will have returned to their Q3 2004 value when they were first reported, erasing 7 years of progress.
After four-straight quarters of growth, the random numbers dipped back to their Q4 2010 level which was at the time the highest they had been in a year. Year-on-year growth was 3.5%.
The averaged random numbers over the past year are at their highest level since records began.
The random numbers have been through three cycles in the past 4 years, and some analysts are saying that growth should be expected to continue in the medium-term.
The actual time-series:
What gives statistics the ability to surpass damn lies is people’s tendency to believe them. As Operational Research/Business Analytics practitioners, we here at Figure it Out would advocate the application of analytics throughout business decision making in order to provide scientific evidence for making the best decision. A good analysis builds consensus and aligns an organisation for change, but what is to be done regarding the damn lies?
Obviously critical thinking by individuals is always important, but it is also important that you trust your analyst or your analytical supplier. You need to trust them to get the analysis right and not make mistakes, but you also need to trust them to be honest with you. A trustworthy analyst is not necessarily a stakeholder within your organisation. A trustworthy analyst is not necessarily an external body with commercial interests. On a case-by-case and person-by-person basis, you must evaluate their integrity and act accordingly.
A time-series offers many opportunities to be selective. By shrinking or growing the window of analysis, data points favourable or unfavourable to the aim can be included or excluded. The numbers themselves, their rate of change, their rate of change of rate of change, their proportional rate of change of rate of change, their absolute change, their annualised, de-seasonalised, year-on-year, quarter-on-quarter, moving average, etc. etc. statistics can all tell a different story. Statistically speaking (tongue-in-cheek), if you can generate enough different metrics, some will tell a compelling story that is contrary to the true trend.
A final note on being honest and being selective: A blog author can honestly claim to have generated and shared a random time-series, but that doesn’t mean they didn’t generate several and then select one that suited their purposes.