Data virtualization as a solution — Part 1: the challenges of mixed IT landscapes

Every company eventually has to combine data from different systems — driven by separate tools across production, purchasing or HR, by subsidiaries and acquisitions, or simply by the need to bring in external data. This first part looks at the content and technological challenges that mixed IT landscapes create, and how data virtualization helps.

Every company has to face the challenge of combining data from different systems over the course of its life cycle.

Content challenges in combined reporting

Keeping data consistent is hard: customer, product and material number ranges differ across systems, exchange rates are missing, and dates and numeric formats diverge. Because data only becomes meaningful together with business logic, that logic must stay consistent too when it is used across the organization.

Technological challenges

Databases, warehouses and enterprise systems use proprietary technologies and interfaces that are difficult to combine, and even open APIs or REST endpoints usually can't be used by business users without programming. Network boundaries add another barrier when departments run their own infrastructure or data has to come from suppliers, customers and other third parties — so dynamic, automated integration into analyses is often simply not possible.

Conventional approaches and their cost

The usual answer is a data warehouse or data lake: data is copied from the sources into one or more new persistences through complex, multi-stage transformation and cleansing. These projects are time-consuming and expensive, later changes to the loading routines are avoided, and the business keeps changing anyway. When change requests are rejected, teams fall back on repetitive, error-prone manual workarounds and shadow systems that endanger the original IT investment.

Overcoming the challenges through data virtualization

Data virtualization stores only a virtual reference that points to the original data in the source system, instead of duplicating it. Because access is configured by the end user rather than programmed, the harmonization of number ranges, currencies and formats can be built by the people who need it. Data is always read live from the source — no load routines to monitor — and Single Sign-On reuses the source systems' authorization concepts, enabling secure access even across network boundaries.