Problem Statement:
The organization needed to migrate CRM data from Salesforce to Microsoft Dynamics 365 (Dataverse) across multiple production environments. The migration required complex data transformations, lookup and option-set mappings, BPF stage assignments, and Opportunity status handling. A reusable, configuration-driven framework was needed to automate the process, reduce manual effort, and ensure consistent, reliable migrations.
Tools and Technologies:
Azure Data Factory (ADF), Azure Data Lake Storage (ADLS) Gen2, Microsoft Dataverse / Dynamics 365, Salesforce, Azure Key Vault, SOQL, Dataverse Web API.
Summary:
- Designed and developed a config-driven Azure Data Factory (ADF) migration framework to extract, transform, and load Salesforce CRM data into Microsoft Dynamics 365 (Dataverse) across two production organizations.
- Built reusable Mapping Data Flows implementing field-level transformation logic, including direct mappings, value merging, lookups, and option-set translations, driven through configuration files instead of hard-coded logic.
- Developed automated pipelines for Business Process Flow (BPF) stage assignment and Opportunity Won/Lost closure using the Dataverse Web API.
- Implemented data validation, lookup resolution, and transformation logic to ensure accurate migration of complex Salesforce CRM data into Dataverse.
- Diagnosed and resolved migration failures involving schema mismatches, invalid status/state mappings, and lookup resolution issues using ADF skip-log analysis for efficient troubleshooting and rapid issue resolution.
- Used Azure Key Vault for secure management of credentials and connection details across the migration framework.
- Produced a comprehensive technical handover document covering solution architecture, configuration reference, operational procedures, and troubleshooting guidance to support project transition and ongoing maintenance.
Entity config:
Azure Data Lake:
Pipelines:
Master Pipeline:
Transformation Dataflow:
Key vault:
