The Challenge: A Legacy Load Process Holding the Data Warehouse Back
The client’s Data Warehouse ran on traditional on-premise hardware, with a load process that had become a structural bottleneck. Every table had to be loaded first into a separate staging database before being moved into the Data Warehouse itself — an extra hop that added time, complexity, and risk to every data integration project.
The real cost showed up whenever the business needed something new. Onboarding a single new table from a source system took between two and four days, with a developer fully occupied for the duration. In a telecom environment where source systems, campaigns, and integrations change constantly, that lag meant new reporting and analytics needs were permanently waiting in line behind manual ETL work.
The Discovery: An Embedded Partner for the Long Haul
IDS Consulting was brought in not for a single fix, but as an ongoing extension of the client’s Data Warehouse team. The mandate: modernize how data moved through the platform, and keep it evolving safely as the business — and the underlying infrastructure — kept changing around it.
The Solution: A Metadata-Driven, Cloud-Ready Data Platform
IDS redesigned the table-loading process from the ground up, replacing the old two-step, staging-database approach with a metadata-driven engine — and layered on the capabilities a modern telecom data platform needs.
- Metadata-driven table onboarding. New tables are now configured in a maximum of 20 minutes, with no second staging database required.
- Delta loading. The platform was upgraded to load only day-over-day differences instead of full data sets, cutting load times and infrastructure load.
- Streaming and API-based ingestion. IDS added the ability to ingest data directly from REST/web services and Kafka streams, alongside a dedicated engine that exposes Data Warehouse data back out to consumers through REST APIs.
- Automated data lifecycle management. Retention rules set at table and business-area level now automatically move data through its lifecycle — from database, to tape, to deletion — freeing up storage without manual intervention.
- Phased infrastructure evolution. IDS led the proof-of-concept and migration to Oracle Exadata, and later the full migration of the Data Warehouse to the cloud on Google BigQuery, while continuing to support day-to-day ETL and business-as-usual development throughout.
The Battle: Modernizing a Moving Target
None of this happened on a stable platform standing still. Every infrastructure change happening elsewhere in the company — new systems, upgrades, integrations — had to be supported by the Data Warehouse in parallel, to keep data flowing without gaps.
That constant motion forced some genuinely custom engineering. IDS built a bulletproof, custom rollback mechanism for Kafka queues to protect against message-processing failures. To stop critical systems from drifting out of sync with their source, the team implemented point-in-time table loading. And to keep storage costs under control for tables carrying large volumes of text, IDS built a custom compression system — avoiding a costly proprietary compression license that Oracle offered as the alternative.
The Results: A Faster, Leaner, Future-Proof Data Warehouse
- Table onboarding cut from days to minutes. What used to take two to four days, with a developer tied up throughout, now takes a maximum of 20 minutes — a reduction of over 99%.
- Daily processing finishing hours earlier. After the migration to Oracle Exadata, the daily batch — which used to complete as late as 11 PM — now finishes before 9 AM, giving the business same-day access to fresh data.
- Three generations of infrastructure, one continuous platform. The Data Warehouse evolved from on-premise Oracle, to Oracle Exadata, to Google BigQuery in the cloud — each transition led by IDS without interrupting the reporting and ETL processes the business depended on.
- Real-time and streaming-ready. Kafka and REST API ingestion opened the door to data sources the original platform was never built to handle.
- Storage and licensing costs kept in check. Automated data lifecycle management and a custom-built compression engine controlled storage growth without paying for proprietary add-ons.
- 8 years of continuous partnership. A lean team — never more than three IDS consultants, most of the time just two — sustained and modernized the platform across every architectural shift, from PL/SQL and Informatica to BigQuery and Airflow.