In the late 1990’s, I started my data analytics career at a growing company that would become one of the leading vendors in the first generation of the BI (business intelligence) industry. The company’s founder had a remarkable ability to inspire us with grand visions of a future in which information would be available at our fingertips, enhancing the way we live and the way we work. Since this was before the arrival of Google or the iPhone, the notion of “data everywhere” was not as familiar as it is today.
One of his most memorable stories described a visit to the grocery store of the future. You’d walk into the store and pick up an item—like an egg—from the shelves and then all kinds of data would be presented to help you make an informed purchase. This “intelligent egg” could tell you about its nutritional value, the farm it came from, how the chickens are fed, reviews from other shoppers, and much more. It sounded like science fiction back then. Today, it’s as easy as scanning the item’s barcode using a phone camera.
The story of the intelligent egg comes to mind as I reflect upon the 25 years I’ve spent in the data space. I feel a shared sense of gratification when I see how far we’ve come in our journey to help people make better use of data to make decisions. From the early days of “mode 1” BI platforms, through the disruption of self-service data visualization, to the recent emergence of AI-powered tools, the data industry has made slow but steady progress towards its goal of democratizing data.
That being said, we, as an industry, still have a lot of work to do. Studies show that adoption of BI and analytics within organizations remains stagnant and only a quarter of the workforce actively uses analytics tools. I believe no organization should call itself “data driven” when 75% of its employees use their instincts rather than facts to make decisions. How can we fix this? In this article, I’d like to discuss the evolution of BI and share some ideas regarding where it’s headed.
A look back: The evolution of BI
Before we look at the road ahead, let’s review how we got here. The first era of BI, represented by platforms like BusinessObjects, Cognos, and MicroStrategy, was centered around IT (information technology) and data teams were responsible for every step in the data supply chain process, such as extraction, curation, organization, and publishing of insights in the form of reports and dashboards.
End users—people who make business decisions—relied entirely on IT to fulfill their analytics needs. Any new dataset, metric, or analysis had to go through the IT-controlled data supply chain before it reached business users. As you can imagine, this could be a lengthy process resulting in end-user frustration and, more importantly, hindering the organization’s ability to make timely business decisions.
A few years later, companies like Spotfire, Qlik, and Tableau fulfilled an unmet need of business teams that needed to move faster. These vendors ushered in the second era of BI, focusing on end users outside of IT—mainly data analysts—who wanted the ability to work with data on their own. They Introduced the self-service data visualization paradigm and completely disrupted the industry, displacing their predecessors (BusinessObjects, Cognos, and MicroStrategy) and establishing themselves as market leaders in BI.
All of a sudden, analysts who relied on Excel pivot tables or SQL scripts had access to technology that enabled them to combine disparate datasets, create data models, and build rich interactive visualizations and dashboards powered by those models. Self-service tools now empowered business teams to move fast and make data-driven decisions without depending on IT experts.
The third era: BI for the 75%
The first two eras of BI took us only so far. As mentioned earlier, analytics adoption remains stuck at a mere 25%. The third era, I believe, will be led by solutions specifically designed for the remaining 75%. Reaching that segment will require new ways of thinking about how people best consume information. Analytics providers must develop a deep understanding of the unique needs of this new audience, which differs from the needs of IT experts and data analysts that were served by the first two eras.
One key difference between the first two eras and the third one is how BI is viewed in relation to other business processes. In the first two eras, analytics was an entirely siloed process. Although it supported other functions like sales, marketing, or operations, it was nonetheless a separate discipline with specific tools, user audience, and skill requirements. In the first two eras, analytics proficiency was primarily the domain of IT experts and data analysts.
In the third era, however, BI is not a distinct process; instead, it’s another link in the business value chain, seamlessly integrated into processes and applications already in use. An end user should not require any specialized skills or have to go to a dedicated analytics tool to get insights. Instead, one should be able to get the insights they need in their everyday tools and workflows—without going to a separate analytics "destination".
Third-era analytics will become a natural part of the workflow for the millions of people that comprise the remaining 75% of users. The result will be a more inclusive model in which everyone, regardless of their role, background, or expertise, can benefit from data-driven insights.
Why now? An industry ripe for disruption
My career has given me a front row seat to the evolution of the BI market. I’ve seen dominant market leaders become followers (or disappear completely), outdone by emerging startups in the textbook definition of disruptive innovation. I’ve seen technology trends come and go, software delivery and consumption models transformed, and more recently, extraordinary leaps in artificial intelligence, alongside a global pandemic that accelerated years of technology innovation in a way that’s unlikely to have happened otherwise. I’ve never seen the kind of change we’re witnessing now – let’s take a look at the major shifts that are paving the way for the new era:
1. The arrival of "Gen D"
"Gen D", or generation data, comprises the millions of people who grew up surrounded by information at their fingertips. Immersed in data from the moment they first picked up a device, being data-driven feels as natural to them as receiving personalized recommendations on Netflix, making shopping decisions using reviews on Amazon, or setting fantasy football lineups each weekend. This new generation of the knowledge worker arrives with a natural inclination to make decisions with data.
2. The rise of AI
The tech space has a tendency to exaggerate things, so I won't blame you if you've dismissed some of the ChatGPT uproar as a lot of hype. But, having worked with experts in the field—brilliant minds who have studied and taught AI for decades—when I heard them describe LLMs (large language models) as "miraculous", I knew that this was more than just hype. I believe that we've reached a tipping point and that AI will fundamentally transform the way we live and the way we work. It's an incredibly exciting moment!
3. Emerging new vendors
I've found more than 50 startups that are recent entrants to the data space, with more than $700M raised between them. This figure doesn't even include all the startups I'm not aware of or new offerings from established vendors. The amount of investment pouring into this industry is astounding and encouraging. The next market leader is already out there, preparing to disrupt our industry once again.
4. The evolution of work
Despite "return to office" proclamations by tech CEOs, the debate about the pros and cons of working remotely is nowhere close to settled. Call me cynical, but I wonder how many of those RTO (return-to-office) mandates are largely motivated by empty real estate rather than concrete productivity metrics. There are numerous examples of successful companies that are fully distributed, along with studies that show an increase in productivity associated with remote work. It’s inevitable that such profound changes in where we work will significantly transform how we work.
What’s next?
The story of the intelligent egg illustrates a future in which data is all around us and becomes a natural part of our day-to-day existence. Today, we interact with data all the time without even thinking about it, so it’s not unreasonable to suggest that the future is already here. This is tremendous cause for excitement for anyone who believes that being better informed is fundamentally a good thing.
Whether or not this is true is a topic for another blog, but what’s undeniable is that we're entering the third era of analytics. Data consumption habits, workplace practices, and technology are all changing. Just like we saw a decade or so ago, the next industry leaders will be companies we probably haven't heard of yet. This is an industry that never ceases to reinvent itself and I'm thrilled to play a small part in its evolution.
.svg)
.svg)
.svg)
