In the past year, growing organizations with up to 100 employees had the highest rate of business intelligence adoption.
Historically business intelligence has been a tactic exclusive to enterprise businesses that could afford data warehouses. Now, with easier BI system alternatives, it’s much more accessible. If you’re considering adding at BI system or changing from your current system, one of the first decisions you’re going to have to make is whether or not to invest in a data warehouse.
This post takes a fast, hard look at what it takes to get a data warehouse up and running, and then maintain it.
What a data warehouse does for your strategy
A data warehouse is used to store huge (HUGE) quantities of data. It’s most often used by enterprise businesses because, not only do they have the budget to maintain them, but they’re also generating data from dozens of reporting systems, in many different formats, over decades.
Before deciding how to tackle the process of implementing and maintaining a data warehouse, you need to understand where a data warehouse would fit into your strategy.
How the data warehouse fits into a BI workflow
At its most basic level, a data warehouse comes into play after the data has prepped, cleaned, and normalized in an ETL (Extract, Transform, Load) system. The data warehouse supplies data to your business intelligence system, which uses that data to perform queries, build metrics, and create visualizations.
BI workflow:
- Data sources and management systems send data to ETL.
- Data is cleaned, prepped, and normalized in ETL.
- Prepped data is sent to the data warehouse for storage.
- Business intelligence system pulls data from the data warehouse for queries and visualizations.
For a large enterprise (usually generating more than $50 million) a data warehouse is important to the business intelligence workflow. But for a smaller business that could easily use a database, or even their individual data sources to store their data, a data warehouse is unnecessary.
What it takes to set up and maintain a data warehouse
You have two choices for setting up your data warehouse:
- Install it on your own server. This can be easier for team members to access, but it’ll require a long installation period.
- Host it on a cloud server. There are data warehouse cloud services (such as Amazon Redshift and Panoply) that provide cloud support and data warehouse software.
Either way, it’s critical that you have an analyst experienced in data warehouse design leading the charge, and invest the needed time to gather all your data use-case requirements from each team in your organization.
It’s not unusual for the design and development phases of adopting a data warehouse to take upwards of 12 months or more.
The Hardware:
First, you’re going to need server space. If you’re self-hosting, you will need the server bandwidth to hold all your data and scale as more historical data is collected daily. You’ll also need physical space to store your hardware.
Alternatively, if you choose to host your data warehouse in a cloud service, storage costs will vary. You can read a more in-depth analysis of data warehouse storage costs here, but a conservative estimate of storage costs for a mid-sized data warehouse is $12,000 per year.
The People:
Servers don’t configure themselves, which brings us to the next part of setting up a data warehouse: the people
Your data warehouse won’t be self-sufficient. IT will need to configure and set up your data warehouse. Once it’s configured and ready to go, you will need data analysts to prepare data for the warehouse in your ETL system.
Is a data warehouse worth it?
A solution that requires a data warehouse will always cost more than one that doesn’t.
Just to build a data warehouse costs an average of $500,000 in startup costs. And once that’s set up, you still have to set up a team of dedicated analysts and IT experts to manage it perpetually.
Alternative BI solutions (called end-to-end solutions because they include the ETL and data visualization functions in one system) can provide the functionality you need without requiring a data warehouse. By connecting directly to your various systems (website, CRM, analytics) they pull data the source and put into your BI system for prepping, viewing, and analysis.
End-to-end BI systems are less expensive, require fewer people to manage, and can provide custom data insights.
Data warehouses are a huge commitment, and for some businesses, they’re absolutely necessary.
But most of the time, they’re not.
Read more about data warehouses in our blog or, if you have any questions about this blog post, feel free to ask us in the chatbox on the right!
