The Challenge: A Data Mart That Couldn’t Keep Up
The client — a banking group with both group-level and local operations — needed one thing above all: fast, centralized access to data for advanced analytics and reporting. What it had instead was data scattered across a Hadoop-based group Data Lake and a separate, Oracle-based data mart, with real integration problems between the two.
Reporting and advanced analytics for the Controlling function suffered as a result. Manual processes made reporting slow and error-prone, and performance issues on the Oracle side meant even routine analytical needs regularly went unmet.
The Discovery: Building on an Existing Partnership
IDS Consulting was already embedded in the client’s data environment, providing data integration and technical support on the group’s big data analytics platform. When the limitations of the Oracle-based data mart became a bottleneck for Controlling, extending that big data expertise into a dedicated solution was the natural next step — rather than continuing to patch the existing Oracle setup.
The Solution: A Dedicated Big Data Analytics Mart for Controlling
IDS designed and built a new analytical data mart purpose-built for Controlling, constructed entirely on big data technologies rather than the client’s Oracle stack. The new platform moved the mart off Oracle and onto Hadoop, consolidating group and local data into one centralized, scalable environment able to support both large-scale processing and advanced reporting.
The solution was built on:
- Processing & querying. PySpark, Python, SparkSQL, Hive, and Impala running on Hadoop.
- Operations & automation. Shell scripting for orchestration and pipeline management.
- Development & collaboration. JupyterHub for analysis, Git and Bitbucket for version control, and GitHub Copilot with multiple AI models to accelerate development.
The Battle: Stability Under Heavy Load
Migrating a Controlling-critical data mart from Oracle to Hadoop while keeping it running is one challenge; keeping it stable under continuous, intensive use is another. The biggest test for the IDS team was exactly that — ensuring the new platform held up under heavy day-to-day usage, and tracking down and resolving errors surfacing in the consumer layer, the point where Controlling and management actually draw their data.
No major external constraints slowed the project down; the challenge was squarely technical, resolved through the team’s ongoing debugging and stabilization work.
The Results: One Platform for Group and Local Analytics
- Consolidated, single-platform access. Group and local data now sit on one scalable big data platform, replacing the fragmented Oracle/Data Lake split.
- Materially faster production and reporting. Production runs that used to take hours on the Oracle-based mart now complete in a fraction of the time — an estimated 50–60% reduction in end-to-end processing time, with some reports that previously ran overnight now available same-day.
- Advanced analytics finally unblocked. Reporting and analytics needs that performance and integration issues had previously made impossible are now fully supported.
- Lean, embedded team. A 2-person IDS team designs, builds, and supports the platform.
- Built to keep evolving. The platform is currently migrating to Databricks (dbx), with continuous enhancements layered on top.