The Challenge: An On-Premises Data Mart Running Out of Room
The client’s analytical data mart ran on-premises on Hadoop — and it was showing its age in two specific ways.
First, resources were shared across every stakeholder using the platform. As production workloads and business demand kept growing, that shared pool became a bottleneck. On peak days, the system could seize up entirely, degrading service for production jobs and the consumer layer at the same time — with no way to isolate one from the other.
Second, the platform had no ACID transaction guarantees. Without that consistency layer, data reliability was left exposed, a growing risk for a bank where Controlling and management teams depend on trustworthy numbers.
The Discovery: A Technical Partner for the Cloud Transition
The client needed more than a lift-and-shift — it needed a partner who could translate an existing on-premises solution into a cloud-native one without losing what already worked. IDS Consulting was brought in to provide hands-on technical support and consultancy for the migration to Databricks.
The Solution: A Structured, Validated Migration to Databricks
IDS started by analyzing the client’s existing processes, framework, and current needs, to make sure the move to Databricks was coherent with how the business actually used the platform — not just a technical port.
From there, the team took on:
- Pipeline migration. Rebuilding and moving existing data pipelines onto Databricks, using PySpark and SparkSQL.
- Debugging and validation. Systematically checking that migrated processes ran correctly and that the resulting data matched expectations.
- Version-controlled delivery. Managing the migration through Git and Bitbucket, keeping the transition traceable and collaborative.
The Battle: Proving the Data Can Be Trusted
The single biggest challenge in the project has been data validation — confirming, step by step, that data migrated into the cloud is fully consistent with what the on-premises solution produced. For a banking client, that reconciliation isn’t optional; it’s the precondition for trusting the new platform at all. No other unexpected constraints have surfaced so far.
The Results: Built for Scale, Built for Consistency
The migration to Databricks directly targets the two structural problems of the on-premises setup:
- Elastic scalability and workload isolation. Databricks separates resources across workloads, removing the contention that used to let peak-day demand block both production and the consumer layer.
- ACID transactions. A capability the on-premises solution never had, now built into the platform — giving Controlling and management a consistency guarantee the old data mart couldn’t offer.
- A lean, focused delivery team. Two IDS consultants are driving the migration, supporting the client through analysis, pipeline rebuilding, and validation.
The project is currently in progress, with data validation as the active workstream. No unexpected obstacles have come up, and the engagement remains on track.
“IDS didn’t just move our code to a new platform — they took the time to understand the logic behind it first. That’s given Controlling and management real confidence in the numbers coming out of Databricks, even while the migration is still underway.”
— Head of Data & Controlling, Banking Client