The Data & Analytics Evolution
In the Data & Analytics industry we’ve seen a lot of change over the years. We’ve moved on from Excel and Access databases to all the lakes, houses, and Fabrics. Our operating models have continually shifted – centralised within multiple domains, de-centralised hub & spoke, outsourced, insourced. New roles have now become commonplace – Data Scientist, Data Analyst, Data Translator, Data Engineer, Data Optimisation Specialist.
So. Many. Changes.
It’s hard to fault the technology investments that have delivered us unparalleled data capabilities – high speed data sourcing and cleansing through integration and data management, CDC processing, cloud-enabled complex data transformation at low cost, and seemingly unlimited automated report generation. And then there’s machine learning!
However, pick up any research firm report or listen to C-levels in most organisations and you’ll find that something isn’t quite connecting. (2, 3) We have all this innovation, advancement and dollars spent in the data space, yet there are few who feel their company is truly ‘data-driven’. HBR’s January 2023 article ‘Has progress on Data, Analytics and AI stalled in your company?’ even uses the term ‘back-sliding’ as a descriptor for the data journey some companies feel they are on. (1)
Spend Vs Value
The message is clear – Data & Analytics spend is increasing at a rate not matched by the increase to business value. Why?
My experiences with companies of all sizes, maturities, industries and data-spend has led me to identify a problem at the heart of this dilemma. Lack of engineering – environmental engineering, to be precise. Much like the environment of a prize-winning garden, organisations are complex ecosystems where success and growth depend on striking a delicate balance. This balance is best achieved through thoughtful planning and the sequencing of activities and inputs designed to target the key areas required for growth and long term sustainability.
A Tale of Growth
Let’s look at a young, innovative company experiencing rapid growth. Critical decisions were initially made on gut-feel by a few employees who were passionately invested in their vision.
Mandatory regulatory reporting was outsourced, and systems were not integrated. Little thought was given to data processes or data quality, as volumes were low enough that manual interventions could be applied to ensure the timeliness and accuracy of customer-facing communications. The company focused on delivering for their customers and were running so fast they were unable to optimise or strategize. Growth ensued and they became a multi-million dollar company, with scores of employees and thousands of customers. Those manual processes and data quality issues were now starting to cause serious issues. Manual data fix processes meant they also could no longer keep up with customer demand, regulatory reporting was too complex to outsource, and the reactiveness of their decision-making was causing significant issues in areas such as resourcing, financial forecasting and risk management. The garden was becoming unwieldy.
A project was established to improve visibility and control through reporting. A consultant was brought in to pull the data together from all the core systems and create the reports. However, unexpected problems with the data often caused the reports to fail, leaders disagreed on key metrics and their value in decision-making. Every leader demanded reporting to show the numbers they preferred – hundreds of aggregate counts and visualisations with little consistency or oversight into their accuracy or business value. Some leaders actively pushed against the reports and refused to use them – gut-feel (and 500 excel spreadsheets) have got us this far! – and that data doesn’t look right.
One year in, the project was far over budget, and failing in its goal of increasing the speed and accuracy of decision making. The reports were causing a significant amount of discourse amongst leadership, and the CEO and Board were frustrated by the inconsistency in numbers being presented to them. Focusing solely on the delivery of reports, this company had failed to make sure that the internal operating environment – including their leadership’s data literacy skills, governance frameworks and the company’s strategic decision-making requirements were engineered adequately. The reporting seeds that were scattered into the garden produced an overgrown environment that strangled their desire of helping the garden increase its sustainability and output.
The Path to Decision Intelligence
Much like growing vegetables, there is work that needs to be done before the seeds of reports can be sown. Like all organisational changes, it is best to start with people.
Culture, like good soil in the garden, is the key. Great soil retains moisture and nutrients, fosters deep root growth and biodynamic (complementary) organisms, and above all helps sustain a garden through turbulent environmental conditions above ground. Engaging leadership early to seek alignment and understanding of the upcoming changes is critical. This is best done by providing carefully curated data literacy training focusing on the usage of data for decision making.
- What are the right questions to ask, and how do we choose the right variables to measure?
- What are leading and lagging indicators, and how are they used?
- What is the optimal frequency of reporting to ensure it lines up with the corporate decision-making cycles and external factors?
- Why are data definitions and process modelling so important for decision making?
- What Board interaction is required regarding Risk Governance Frameworks?
From here we move on to Strategy. Trying to grow the garden you desire by throwing random seeds into the soil without a vision for what you want to grow and why, or even if those plants are suitable in your environment; is unlikely to lead to success. The same holds true for enterprise data and reporting projects.
- Which of these questions are best suited to help us achieve our organisational goals?
- How can we ensure the overall costs are within an acceptable level and delivering returns on our data investments?
- What changes do we need to make to our organisational structure to support the departments and individuals that need the data analytics output?
- What do we want our organisation and our decision making to look like in three years?
- Do we need to enable for mergers, acquisitions or new lines of business?
- Are we wanting to take advantage of innovations, like ChatGPT or perhaps start developing our own innovations?
Now that we have our soil prepared and we know what we should plant where and why, we can engineer the final two elements of environmental success in our garden – Technology and Governance. These are the tools and additives we need to have and the things we need to do to ensure our plants germinate and continue to grow. Much like fertiliser, and a watering and maintenance schedule.
We now have enough information to determine what technologies will best enable us now and into the next few years, and how best to integrate them. The information we may have could include:
- The datasets required and the frequency with which they need updating
- The exception triggers needed and who are they communicated to
- The level of democratisation that our tooling needs to support
- The security and risk governance mandates requiring compliance
- The permissions and role-based access needs of current and future resources
Strong consideration should be given to the enablement of a discovery, prototyping and deployment environment for the data scientists and senior analysts the business is looking to hire within the next 18 months. This is an important staff retention measure in its own right.
A Note on Data Governance
Before we start to build any reports or transform any data, we must also introduce our leaders and internal IT/Data staff to simplified data governance frameworks and processes. Assigning data steward and owner roles to the data in these early reports will ensure that decisions of data transformations, metrics, definitions and reporting requirements can be made early,reducing costly changes and rework down the track.
Introducing a ‘light-touch’ data governance framework also helps to increase the businesses’ confidence in the reporting being delivered and any future data initiatives. Ensuring that all data in the reports is clearly defined, modelled, tested and approved by the right resources, any required changes to data fields and processes are approved and implemented, will ensure new reports will maintain consistency over time. A simple process should be implemented to manage reporting changes and requests. Once the reporting floodgates are open, decisions on new builds can then be made strategically, and resources managed adequately.
Data steward/owners are now confident to act as internal advocates of the change, fully informed of the often significant work required to turn application and system data into meaningful business information. Light touch governance frameworks ensure they are not disheartened by cumbersome governance processes that can sometimes feel like unnecessary paperwork and roadblocks to delivery.
A Prize Winning Garden
Through using this repeatable process for environmental engineering, with a focus on well-sequenced activities across all four areas of Strategy, Culture, Governance and Technology, businesses are able to ensure continued growth and evolution of the decision intelligence capabilities. Your harmonious, beautiful and productive garden will lead to increased realised return on your data and analytics capabilities and improved business outcomes.
Register your interest for a no-obligation Decision Intelligence Discovery Session
Resources
1 Bean, R. (2023). Has Progress on Data, Analytics, and AI Stalled at Your Company? Harvard Business Review. Available at: https://hbr.org/2023/01/has-progress-on-data-analytics-and-ai-stalled-at-your-company
2 McKinsey & Company (2018). Five fifty: The data disconnect. McKinsey & Company. Available at: https://www.mckinsey.com/capabilities/quantumblack/our-insights/five-fifty-the-data-disconnect.
3 Belissent, J. (2018). Your data is worth nothing – unless you use it. Forrester. Available at: https://www.forrester.com/blogs/your-data-is-worth-nothing-unless-you-use-it/.
