Transparency, big data and decision making

Anyone who knows me professionally will tell you that I'm a huge proponent of being transparent in communications.  I have firm expectations of transparency up, down and sideways through the org chart.  I tell my staff that I expect transparency in their communications with myself and their peers.  When asked by my management what I need to be successful, one of the top items I respond with is transparency from them with information.  I firmly believe that knowledge is power and that effective decisions can be made more accurately when people are better informed.

Transparency of information is a key element to data driven decision making.  Gut instincts will get some people a long ways however for the majority of people, the best decisions are made when people have as much information available to them.  Of course the rub is that you never have perfect information and you never can have complete information.  Or can you?

The world of big data is changing all of the paradigms around the decision making process.  When I started in graduate school working on a statistics degree nearly 20 years ago, the thought of having a complete data set for analysis was the ideal but not realistic.  We used the power of mathematics to develop confidence levels around decisions made based on sampling methods that reduced the error potential in reduced sample sizes.  A sample versus a census was the only feasible way to move from a theoretical exercise into an actionable decision.

The power of computation and the size of data warehouses today has become so enormous, the process of doing analysis and consequently making decisions around a census of the population is no longer part of theory.  It is practice.  This leads back to my fixation on transparency.  Just why is transparency so important?

As a business leader, I can either recognize that data is everywhere and virtually everyone has access to these data stores in one form or another.  If I'm not transparent, it is too easy for people I'm working with to find the information anyway.  Sometimes though when people discover the raw information on their own, they may not get complete data or they may resolve the data in ways that provide incorrect results.  The possibility of arriving at false conclusions is heightened.  When it comes to things that relate to the health of a business that can directly impact people's livelihoods, this can be a real problem.  People always tend to arrive at the worst possible conclusions and accept those as reality.  It is human nature to assume the sky is falling and left unchecked, these false decision outcomes can become a cancer within the business.

Treating information the same way that source code in the Open Source software world is the only way to mitigate these cancers.  Creating a culture of transparency in the workplace helps to alleviate these problems.  People don't always like to hear the honest truth and sometimes the whole story is not always welcome information.  I have found that the respect earned by consistently being transparent helps bridge over tough conversations and builds a level of trust that can be called upon when needed.  Not all FUD (Fear, Uncertainty and Doubt) can be removed by this effort but it certainly is minimized.

When people have been empowered with information through transparency, the probability of decisions made being successful increases dramatically.  Morale is also increased as people at all levels share in the ownership of the decision making process.