ETL framework development and efficient cloud migration to GCP

Background

The client operates the world’s biggest advertising exchange, connecting hundreds of brands with global consumers. The client approached us because they are in need of improving profitability by reducing the operating costs involved in maintaining the existing on-premise infrastructure.

With certain complex interconnected systems, they faced challenges such as longer query processing time, scaling with various sources and ETL pipelines, inefficiency due to manual data analysis, etc.

Solution Implemented

We had several brainstorming sessions with the client to understand their business needs completely and created an ETL framework while facilitating existing databases and transferring data to a centralized repository warehouse from ad exchanges.
We built an Apache Spark-based ETL, hosted on the Google Cloud in order to set new pipelines quickly and manage feeds that are running in real-time.

Outcomes

2.5x
times improvement in data accuracy
$3.5
million Annual cost savings
10x
Faster data processing

The client was completely satisfied with the solution as it resulted in around $3.5 million annual cost savings and enhanced customer satisfaction with faster data processing. It also made sure that the system had minimal manual intervention and required low maintenance with better fault tolerance.