November 25, 1998
Are multiple data warehouses too much of a good thing?
By W. H. Inmon
With the data warehouse proving to be one of the most important architectural phenomena to arrive on the information-processing scene in the 1990s, IT executives at many large corporations are wondering: If one data warehouse is good, are multiple data warehouses better? Or can you have too much of a good thing?
The answer is that certain types of data warehouses go well together and certain types don't. The system architect's job is to define clearly what is meant by "data warehouse" and formulate a sound architectural plan. Such a plan can spell the difference between symbiosis and disaster.
The growth of data warehousing has taken on tremendous momentum. The end user's hunger for information, the potential competitive advantages of information, and management's burning desire to do something constructive about legacy applications that have grown up and old in the last decade all propel the organization into the world of data warehousing.
An architectural sin?
In many corporations -- primarily large corporations -- a movement toward multiple data warehouses is afoot. A data warehouse administrator in these circumstances might well wonder whether it's an architectural sin to build more than one data warehouse -- whether the presence of multiple data warehouses in one corporation would be consistent with the notion of a sound Decision Support System (DSS) architecture.
It's all a matter of how you define your terms. The two most common types of data warehouses are the current-level detailed-data warehouse and the datamart. A current-level detailed-data warehouse is a storage facility in which very granular atomic data is gathered and integrated. It contains the foundation data that is common to many organizations in a corporation. Data warehouses of this type often store huge amounts of data -- as much as two years of detailed history.
|
To achieve a corporate lingua franca among the data warehouses, reference data must be standardized across a corporation's different implementations.
|
A datamart, on the other hand, is a customized subset of data peculiar to a department that collects data for its individual, parochial needs. It typically contains both summarized and customized data that reflect the individual tastes and needs of the sponsoring departments -- marketing, sales, finance, and so forth.
The two types of data warehouses have always traveled together in a symbiotic fashion. The current-level detailed-data warehouse makes an ideal foundation for the needs of the various departments. In a properly constructed architecture, the current-level detailed-data warehouse receives integrated and transformed data from the operational environment and passes the data to the many datamarts. In this sense, having multiple data warehouses in the same company is normal and natural.
But many companies are considering building multiple current-level detailed-databases. This is a very different issue.
For large multinational corporations, it makes sense to consider the possibility of multiple current-level detailed-data warehouses. In fact, multiple data warehouses of this type are probably inevitable if the business is wide and diverse. A company with loosely integrated units that engage in manufacturing, retail sales, and foreign sales, for instance, will need current-level detailed-data warehouses for each of those divisions.
If that's the case, there should be very little data overlap among these databases. Considerable overlap will make it difficult, if not impossible, to achieve a high degree of corporate integration. When two or more data warehouses contain more than a trivial amount of redundant data, integration is almost impossible to achieve because each data warehouse insists it has bedrock data, and is convinced that its own interpretation of that data is the one and only correct interpretation, regardless of what any other data warehouse may think. This fundamental conflict of integration and interpretation that occurs when there are massive amounts of redundant data among data warehouses greatly reduces the effectiveness of the data warehouse/DSS environment. Where there are multiple data warehouses, it's important that there be connectivity among them, achieved through the creation and support of foreign key relationships, whereby the key of one table appears as a reference in one or more other tables. Because the same key appears in multiple places, there is necessarily some redundancy of data.
Also, corporate reference data must be made consistent among the current-level detailed-data warehouses. Large companies almost always have many reference tables that encompass common data such as customer references, state and country references, and product references. In order to achieve a corporate lingua franca among the data warehouses, reference data must be standardized across a corporation's different implementations.
But other than connecting foreign key relationships and standardizing common reference data, there should be no overlap between the current-level detailed-data warehouses.
What about multiple platforms?
An important related issue is whether there should be a consistency of hardware platforms and database management systems among a corporation's data warehouses. Certainly, when it comes to datamarts and the supporting foundation of current-level detail, there will be different platforms and different DBMSs. Some technologies are optimal for managing the massive volumes of data found at the current level of detail, and other technologies are optimal for serving the highly flexible needs of the department. Because of the distinctive features of different platforms and DBMS technologies, there will be multiple platforms and multiple DBMSs spread among datamarts and the current-level detailed data.
But the larger issue is whether there should be multiple platforms and multiple database management systems spread among the different current-level detailed-databases. Obviously, corporations that standardize on a single set of technologies gain certain economies of scale and efficiencies of operation. Group purchase discounts, a minimal number of training classes, and a central focus for troubleshooting are some of the reasons why a single set of technologies for all current-level detailed data is appealing.
But the reality is that most corporations do not adopt a single set of platform and DBMS technologies for their multiple current-level detailed-data warehouses. The reasons probably have more to do with politics and budget than with the advantages of a single technology. In many cases, the different arms of a company are connected to the parent company at only the balance sheet. There is little or no business integration elsewhere. There are power centers throughout the corporation, and each power center makes its decisions independently of other units.
Datamarts alone
And then there's always the option of building just datamarts, with no underlying foundation of current-value data. While this approach has immediate appeal, its long-term effects are nothing short of disastrous. Datamarts should never be built directly on top of the legacy applications environment, for a number of good reasons:
There is massive redundancy of data from one datamart to another. Each datamart in such an architecture collects its own detailed data from the legacy application environment. Across the corporation as a whole, the same detailed data is collected over and over again.
There's no basis of integration. Each datamart makes its own individual interpretation of data, and there is no single foundation to turn to for reconciliation.
The interfaces between datamarts and legacy applications are many and complicated. The issue of interface management alone is enough to warrant building the datamarts from the current-level detail.
In short, some data warehouses work exceptionally well together,
while the combination of others may have disastrous consequences for your business.
How do you know what's right for you? You have to implement a sound architectural
plan that fits your company's business needs.
About the author:
Bill Inmon, the father of the data warehouse concept,
has written 32 books on data management, data warehouses, design review,
and management of data processing. He has published more than 250 articles in
many trade journals. In 1995, he founded Pine Cone Systems to create and market software for the management and control of the data warehouse, DSS environment, once the data warehouse has been built and populated.
| 
