Background
Our customers are from logistic domain dealing with sensitive data covers customs and other information. This required higher level of security standards. On the other hand business demands on getting faster insights to data. Ability to scale quickly, horizontally as well as vertically also one of the main objective. Compliance required the customer to process and archive huge volume of data with secured access.
Solution Implemented
- Event Driven ETL framework for a Quick ETL using AWS Lambda, step functions, and AWS Glue pyspark. Reusable scripts or frameworks are built using a parameter-driven approach to accelerate development for known patterns.
- Extracted Data will be further transformed to apply business logic and process to build aggregated data.
- S3-based Data lake, Glue, and PySpark for the scalable and serverless solution.
- Fine-grained security (glue database level as well as for S3 folders) using AWS IAM.
- AWS KMS for Encryption (S3 and Glue).
- Qlik sense connects the AWS to provide near real-time Reports to the Collection Department.
- Sagemaker Auto Pilot for ML use cases.
Outcomes
A solid data platform is built with scalability, security and governance. Customers were able to scale with growing volume and extend the use cases to several in house data analytics requirements such as operational dashboards, compliance requirement, predictive models, etc.
Domain
Logistics
Methodology
Agile with sprint every 2 weeks
Duration
1 Year
Resources
10
Project Status
LIVE
AMC support
Technology Used
