In our work with B2B SaaS companies, we see this time and time again: companies that give the most attention to the extraction of value from raw data are in the best position to win in their respective markets.
The question becomes how to extract this value efficiently and effectively in the existing landscape that a company works within. As our analytics practice prepares for 2020 and beyond, we’re watching four key trends that are driving the business intelligence (BI) industry, and the companies they serve, forward.
The State of Business Intelligence in 2019
“While most organizations have already begun the work to extract value from their data, fewer than half of the leaders at the [North American Data Summit] had a true data strategy designed to deliver business results.”
— McKinsey
“There’s an unfortunate perception in the world of business analytics today. People widely believe that the solution to analytics is more and better tools. This is largely false.”
— Tristan Handy, Fishtown Analytics
Industry Trend One:
Improved Interactive Query Response Times for Analysis
The advent of cloud-based analytic databases such as Amazon Redshift, Snowflake and BigQuery fundamentally changed the analytics industry. These databases are classified as MPP—Massively Parallel Processing—meaning they can handle distributed operations across datasets, as opposed to Relational Database Management Systems (RDBMS), which are optimized for singular read/write operations.
This technology precipitated the rise of the modern data warehouse, allowing companies to store and process petabytes of information without having to worry about query response times for analysts. Prior to the advent of cloud MPP data warehouses, this technology was only available to enterprise companies with the budget and in-house expertise for on-premise hardware such as Vertica.
Although Amazon is the market leader in this space with Redshift, the space for cloud warehouses is now competitive, with BigQuery (Google), Azure DW (Microsoft), and Snowflake rapidly gaining ground (see recent benchmark reports). We expect the competition in this space to intensify in 2020, particularly with respect to query speeds and processing power.
What this means for BI leaders: For the platform vendors, selling a warehouse solution is another way to encourage vendor lock-in into their rest of the cloud offering. But today’s BI leaders can demand more flexibility. For example, you can run your micro-services architecture on AWS, while also leveraging the power of BigQuery as your BI engine—today, the multi-tenant, multi-cloud solution can also extend to a cloud warehouse solution.
This technology precipitated the rise of the modern data warehouse, allowing companies to store and process petabytes of information without having to worry about query response times for analysts. Prior to the advent of cloud MPP data warehouses, this technology was only available to enterprise companies with the budget and in-house expertise for on-premise hardware such as Vertica.
Although Amazon is the market leader in this space with Redshift, the space for cloud warehouses is now competitive, with BigQuery (Google), Azure DW (Microsoft), and Snowflake rapidly gaining ground (see recent benchmark reports). We expect the competition in this space to intensify in 2020, particularly with respect to query speeds and processing power.
What this means for BI leaders: For the platform vendors, selling a warehouse solution is another way to encourage vendor lock-in into their rest of the cloud offering. But today’s BI leaders can demand more flexibility. For example, you can run your micro-services architecture on AWS, while also leveraging the power of BigQuery as your BI engine—today, the multi-tenant, multi-cloud solution can also extend to a cloud warehouse solution.
Industry Trend Two:
Commoditization of ETL
In the past, running Extract-Transform-Load workflows was a complex, painful endeavor for IT Operations shops. Engineers were usually tasked with writing manual scripts to pull data from SaaS vendor apps (such as Salesforce or Google Analytics) into the data warehouse, and the maintenance and monitoring of these scripts was cumbersome and time-consuming.
The rise of ETL vendors such as Fivetran, Stitch, and Alooma (among others) has eased the burden of ETL on DevOps teams, and has allowed data engineering teams to focus on higher-value problems (such as transforming raw data collected in the data warehouse into insights).
What this means for BI leaders: ETL is a solved problem in 2019. So be sure you’re investing in the people, processes and tools that allow you to extract more and more value from your data and don’t ask your smart, talented developers to simply work on the plumbing.
The rise of ETL vendors such as Fivetran, Stitch, and Alooma (among others) has eased the burden of ETL on DevOps teams, and has allowed data engineering teams to focus on higher-value problems (such as transforming raw data collected in the data warehouse into insights).
What this means for BI leaders: ETL is a solved problem in 2019. So be sure you’re investing in the people, processes and tools that allow you to extract more and more value from your data and don’t ask your smart, talented developers to simply work on the plumbing.
Industry Trend Three:
The standardization of the modern data “stack”
Similar to how modern configuration management tools gave rise to the DevOps movement and the advent of the DevOps professional over the past 10 years, modern, adaptive tooling for data has resulted in the standardization of the data stack.
The modern data stack consists of:
Data collection infrastructure/Customer data platforms (Segment, Snowflake or Heap)
ETL tools (such as Stitch, Fivetran, Alooma)
MPP databases (such as Amazon Redshift or BigQuery)
Data modeling tooling and software (such as dbt, Talend or Dataform)
Business intelligence layer (query-building, visualization, and dashboarding, such as Looker, Mode, Periscope, or Tableau)
Integrations with other reporting tools (Google Sheets, Excel)
As a result of this standardization, there is a supply of data professionals who have successfully implemented modern data stacks in startups/enterprise companies. These professionals will migrate to newer, younger companies and look to disseminate these concepts as they grow in their careers.
What this means for BI leaders: There is no need to reinvent the data stack. Yes, it’s absolutely important to evaluate the vendor options and pick what’s right for you, but more than ever, BI leaders can focus on how their stack will help their team extract value, rather than if tech A will integrate with tech B.
The modern data stack consists of:
Data collection infrastructure/Customer data platforms (Segment, Snowflake or Heap)
ETL tools (such as Stitch, Fivetran, Alooma)
MPP databases (such as Amazon Redshift or BigQuery)
Data modeling tooling and software (such as dbt, Talend or Dataform)
Business intelligence layer (query-building, visualization, and dashboarding, such as Looker, Mode, Periscope, or Tableau)
Integrations with other reporting tools (Google Sheets, Excel)
As a result of this standardization, there is a supply of data professionals who have successfully implemented modern data stacks in startups/enterprise companies. These professionals will migrate to newer, younger companies and look to disseminate these concepts as they grow in their careers.
What this means for BI leaders: There is no need to reinvent the data stack. Yes, it’s absolutely important to evaluate the vendor options and pick what’s right for you, but more than ever, BI leaders can focus on how their stack will help their team extract value, rather than if tech A will integrate with tech B.
Industry Trend Four:
The Rise of the Analytics Engineer
Over the past five years, the data organizational structure has evolved to resemble something like this in most high-growth companies:
Data Engineers:
Professionals responsible for ETL workflows, data modeling, data warehouse, and build, maintenance, and performance tuning, etc
Data Analysts:
Professionals who use previously modeled data to surface insights to business stakeholders about past performance, and use basic statistical techniques to forecast or predict
Data Scientists:
Professionals adept at using raw data and modeled data in the data warehouse to develop machine learning models using advanced statistical techniques to predict and classify data
Director of Analytics/Data Science (sometimes, VP Data):
The leader of the data team, responsible for hiring and platform decisions, and working with key stakeholders at companies to determine analytics priorities and roadmapsHowever, a new class of data professionals, the Analytics Engineer, is beginning to surface at high-growth startups—a hybrid of the Data Engineer and Data Analyst, who leverage best practices from software engineering (version control, CI/CD, build pipelines) in their analytics workflow. This trend will permeate upwards through the mid-market to enterprise, and the analytic products that address the needs of the Analytics Engineer will eventually succeed.
What this means for BI leaders:
The Analytics Engineer’s role will ensure there are champions internally who recognize that data is a critical asset in an organization, and generating business insights from data is a core competency that can’t be outsourced.
Data Engineers:
Professionals responsible for ETL workflows, data modeling, data warehouse, and build, maintenance, and performance tuning, etc
Data Analysts:
Professionals who use previously modeled data to surface insights to business stakeholders about past performance, and use basic statistical techniques to forecast or predict
Data Scientists:
Professionals adept at using raw data and modeled data in the data warehouse to develop machine learning models using advanced statistical techniques to predict and classify data
Director of Analytics/Data Science (sometimes, VP Data):
The leader of the data team, responsible for hiring and platform decisions, and working with key stakeholders at companies to determine analytics priorities and roadmapsHowever, a new class of data professionals, the Analytics Engineer, is beginning to surface at high-growth startups—a hybrid of the Data Engineer and Data Analyst, who leverage best practices from software engineering (version control, CI/CD, build pipelines) in their analytics workflow. This trend will permeate upwards through the mid-market to enterprise, and the analytic products that address the needs of the Analytics Engineer will eventually succeed.
What this means for BI leaders:
The Analytics Engineer’s role will ensure there are champions internally who recognize that data is a critical asset in an organization, and generating business insights from data is a core competency that can’t be outsourced.
“More data is usually available. It takes time or money, but you can get more data. But you’re probably not using all the data you’ve already got. I’m guessing what you meant was, “I wish I had more certainty.” And that, unfortunately, isn’t available.”
— Seth Godin, I wish I had more data
Metrics That Matter
In September 2019, the Harvard Business Review wrote, “A company can easily lose sight of its strategy and instead focus strictly on the metrics that are meant to represent it.”
It’s an important perspective to remember, given the volume of data BI professionals can work with and analyze. As new tools emerge and the landscape continues to develop, we must remain committed to more than just clean, accurate data. BI professionals must provide executive teams with rapid actionable insights that help leaders make better decisions, faster. We must look beyond tools and examine what’s extracting the most value from data with the greatest efficiency.
BI experts can steer their companies and leadership through the complexities of technology, trends, and tools to ensure that data enhances strategy, rather than replaces it.
It’s an important perspective to remember, given the volume of data BI professionals can work with and analyze. As new tools emerge and the landscape continues to develop, we must remain committed to more than just clean, accurate data. BI professionals must provide executive teams with rapid actionable insights that help leaders make better decisions, faster. We must look beyond tools and examine what’s extracting the most value from data with the greatest efficiency.
BI experts can steer their companies and leadership through the complexities of technology, trends, and tools to ensure that data enhances strategy, rather than replaces it.
So what’s next?
Stay tuned for our 2020 predictions, where we’ll be digging into how BI leaders can win the escalating war for analytics talent, where to invest time and talent when it comes to your data, and what new questions executives will be asking of their analytics experts.
Contact our team to see how we can help you transform your data into valuable insights that work for marketing, and beyond.
