Excel sheets, a business intelligence tool and no data warehouse. When is a data warehouse typically introduced in an organization?
Terje Vatle, Chief Technology Officer at BI Builders, thinks that is done at a point when organizations need to professionalize and scale up the delivery of trusted, governed, well-documented, and quality assured data.
“The fact that most leading global organizations today employ some sort of data warehousing approach speaks to the considerable value contribution a data warehouse can have when done right.”, says Terje Vatle, Chief Technology Officer at BI Builders.
In this interview, he shared the importance of data strategy for business success. He also pointed out ways to overcome some of the challenges organizations face regarding data and analytics. Most importantly, the focus was given to the benefits of the data warehouse automation tools for the business needs and goals.
Hyperight: It is a pleasure to have BI Builders at this year’s Data Innovation Summit. Can you please tell us more about your organization and your professional background?
Terje Vatle: Back in 2011 BI Builders was founded when a major oil and gas development project was undertaken on the Norwegian continental shelf. The software Xpert BI was born out of the challenge of unifying the many information silos while working under massive time and cost constraints. The founders shared, and we still share, a passion to accelerate. The key idea was to provide a guided development experience when building a data platform, enforced by always updated metadata. Gartner calls it working on a higher abstraction level than typical point-and-click data integration tools.
In addition to making each developer much more productive the software was designed to manage frequent changes to the data platform. Standardization along best practices made the developed data platform more robust while minimizing operational risks. Soon, we discovered that several industries had similar challenges and today we provide significant business value across multiple industries including energy, banking, public sector, manufacturing, fisheries, and real estate.
I have always been fascinated with technology since my early days growing up in Norway. Studying computer science and working with data and analytics at a time when Google, Amazon and Facebook were founded, was truly inspirational. I soon realized it is not just about technology alone, but how it impacts our society and how it can improve our lives. And at the same time, we need to remain in charge to make sure technology is used for good.
As a person I love to meet new people, get new perspectives, and share ideas. It is always exciting to solve real world challenges using data and analytics. And most of all, innovate on how to solve them. I’m thrilled to work alongside great people at BI Builders that share the same passion for innovation and challenging the status quo.
Hyperight: How can traditional data warehouse methodologies and best practices come together to build a data solution that can support the business need for information?
Terje Vatle: Getting trusted, available and actionable information out of raw, siloed data is inherently a difficult task. The data warehouse methodologies and best practices are the collective experience from decades of data and analytics projects and the learnings from hundreds of successes and failures. Success lies in the combination of people, processes, organization, and technology. Thus, technology is merely an enabler. Still, technology should enforce these methodologies and best practices to avoid repeating any past mistakes.
With modern cloud data platforms and immature cloud data integration tools, it gets even more complex. For example, standardizing code, balancing price performance and achieving end-to-end data governance is extremely difficult. It is a strong call for an accelerator that can enforce such methodologies and best practices.
Hyperight: Organizations rely on data warehouse (DWH) and business intelligence (BI) tools for decision support, operations, and services. How can organizations better understand the role and value of a data warehouse?
Terje Vatle: The data warehouse is designed to hold a single version of the truth across the enterprise. It’s a technology independent storage architecture. As such, it provides a data foundation for BI-tools to create analyses and reporting to meet predefined business needs. It can help answer a wide range of well-known questions across different business domains.
Many organizations start out with excel sheets, a BI-tool and no data warehouse. A data warehouse is typically introduced at the point when there is a need to professionalize and scale up the delivery of trusted, governed, well-documented, and quality assured data.
To determine the value of a data warehouse in your organization, you should ask yourself: Where does the information you base your business decisions on come from and how trustworthy is it? What are the recurring efforts of making that information available? What is the value of making more accurate decisions before your competitors? What is the cost of bad data quality? The fact that most leading global organizations today employ some sort of data warehousing approach speaks to the considerable value contribution a data warehouse can have when done right.
Hyperight: Having a data strategy is an important step for business success. How can organizations know when the right time to have a data strategy is? What are your tips for organizations on how to collect and transform data aligned with their business goals?
Terje Vatle: A data strategy is the roadmap of all your data management priorities to reach a defined business goal. It covers people and organizations, process and governance, policy, and technology. Typically, as soon as the overall business strategy is defined, you may look at the current data management practice, your desired goals and define initiatives to bridge the gap.
When starting a data strategy, after assessing the as-is we always recommend setting a realistic ambition level. The gap will impact the budget required, expectations to be met, level of management buy-in, timeline needed, cultural change needed, which data you need to collect and transform, technology needed and much more. Together with our partners we help our customers through this process.
Hyperight: What are the key challenges in data and analytics today and what is BI Builders doing to mitigate it?
Terje Vatle: Before any actionable insights, AI or ML can be created, the organization needs quality assured data available. Data typically resides in silos and even with modern cloud data integration tools it takes too long, costs too much and is too risky. And when information is made available to users it is not always trusted and leveraged in decision making.
There is simply a lack of “how” to make the data available and manage the data lifecycle efficiently. It strongly limits organizations’ ability to optimize and innovate their businesses using data.
With Xpert BI, the methodology for working smarter while supporting data governance is ready from the start. Thus, developers can focus their efforts on solving business challenges rather than figuring out standards for coding, documentation, migration etc. It speeds up development and lets your project learn from past experiences. There is a great deal of flexibility, such as swapping key architectural components for better price performance without sacrificing robustness or having to rewrite code. Standardization and not having to invent your own automation framework massively reduces key personnel risks.
For users, there is a complete transparency of where data is coming from, how it is transformed and where it is used. We refer to it as end-to-end data lineage. Metadata is directly read out from SQL-code so the lineage is always updated and reflects the production environment. Exposed through a google-ish intranet search users can see the data, where it comes from, who is responsible, any business rules and any usage descriptions.
Last, but not least, a majority of a data platform TCO comes from managing changes. With the end-to-end data lineage, it’s easy to detect impacts and dependencies. Xpert BI guides developers and automates key steps of the change management process.
Hyperight: Many organizations want to automate more data delivery to reports and insights. Still, many have challenges not knowing how to get started. How is BI Builders helping organizations use data more extensively, and how do they get started with automating their data platform?
Terje Vatle: It is typically difficult to know where to start. That’s where methodology and best practices can help. Since that is already available in Xpert BI as part of a guided development experience, getting started is actually much easier and much safer. You don’t have to experiment, choose which standards to follow, enforce it among your developers or build your own accelerators. Instead, the methodology and best practices are ready, tested and documented – as they should be.
Starting off with an MVP you can later scale that up without throwing anything away. You may add more data sources, add more analytics deliveries, scale up or change the storage architecture and the way you process data without rewriting code. Future flexibility is one of the key reasons for choosing data automation such as Xpert BI.
Hyperight: Your company has developed Xpert BI to automate the design, implementation and maintenance of a data warehouse. In 2020, Xpert BI was recognized as a data warehouse automation tool by Gartner for its capabilities to significantly boost developer productivity. Can we know more about the benefits for organizations using this tool?
Terje Vatle: The recognition means a metadata-driven tool where developers are allowed to work on a higher abstraction level than with typical point-and-click data integration tools. Higher abstraction level means there are built in methodologies and best practices that guide the developer together with productivity accelerators and generation of code that is more standardized.
Productivity boost is one of several improvement aspects over data integration tools. Another benefit is agility. It means the ability to manage changes efficiently, such as changing business rules, adding new data sources, covering new analytics use cases, and even changing key architectural components without changing code.
Since a key reason for a data warehouse is reliable data delivery, data warehouse automation must also improve robustness. Data warehouse automation delivered as software that is developed for minimum downtime, is well tested, documented and used across several industries significantly reduces risk of operating the data warehouse.
Risk reduction is the final major benefit. Standardizing code means there are significantly less key personnel risks and it’s much easier to handover projects and code to other people.
For more on how this is different please check out our Guide to Data Warehouse Automation.
Hyperight: From your perspective, how do you see the progression of the data warehouse, its automation and BI? Any trends you see more of in the upcoming, let’s say, 12 months?
Terje Vatle: Cloud data warehouses are getting more complex with several different technologies being integral parts of the data lifecycle. Hybrid cloud and multi cloud is getting normal. Governance and cost control are getting more difficult to manage.
Organizations will more frequently ask themselves, where can I save money? How can I speed up data delivery? How can I trust the information and make the organization leverage the information for decision making?
We see a growing demand for ready data automation that can help manage the complexity, boost productivity, optimize for price performance and enable efficient data governance and easy to use documentation for non-tech people in the organization.
In summary, our data automation tool Xpert BI helps organizations meet these challenges and let them get more value out of their investments in cloud data platforms on Azure and AWS. Feel free to visit us at the booth D44 to discuss your challenges and if we can help your organization on the cloud data & analytics journey. We are also available at www.bi-builders.com.