Problem Statement:
The organization needed a robust and scalable data pipeline to handle large volumes of order data coming from live Amazon systems. Existing processes were not efficient for frequent data loads, leading to delays, inconsistencies, and limited visibility for Sales and Marketing reporting. The goal was to build an automated, high-performance pipeline ensuring accurate and timely data delivery.
Tools and Technologies:
Azure Databricks, PySpark, Azure Data Warehouse, SQL, Amazon Data Sources, Azure Data Engineering Services
Summary:
• Built end-to-end ETL pipelines using Azure Databricks (PySpark) for order data processing.
• Ingested high-volume transactional data from live Amazon systems on weekly and monthly schedules.
• Transformed and curated raw data into analytics-ready datasets for reporting use cases.
• Implemented data cleansing, validation, and quality checks to ensure accuracy and consistency.
• Optimized pipeline performance for large-scale data processing and efficient storage in Azure Data Warehouse.
• Supported Sales and Marketing teams with reliable datasets for business reporting and insights.
Architecture:
