Data Engineer

Key Responsibilities: Pipeline Design & Development Design and build robust, reusable, parameter-driven ingestion and transformation pipelines using Azure Data Factory, Synapse Pipelines, Data Bricks and/or Microsoft Fabric Data Factory. Implement medallion architecture (Bronze / Silver / Gold) on Azure Data Lake Storage Gen2 using Delta Lake, Parquet, and structured streaming patterns. Build performant ELT workflows that leverage pushdown to source systems (Synapse Dedicated SQL Pool, Azure SQL, Teradata) where appropriate. Develop and optimize PySpark notebooks and jobs on Azure Databricks or Synapse Spark. Data Modeling & Warehousin gDesign dimensional models (Kimball star/snowflake) and data vault patterns for analytics consumption .Implement Slowly Changing Dimensions (Type 1/2/3), Change Data Capture, and late-arriving data patterns .Tune distributed SQL workloads in Synapse Dedicated SQL Pool / Fabric Warehouse, including distribution keys, partitioning, and clustered column store indexes . Platform Engineering & DevO psImplement CI/CD for data pipelines using Azure DevOps (YAML pipelines, ARM/Bicep/Terraform) across Dev / SIT / UAT / Prod environment s.Instrument pipelines with robust logging, auditing, and monitoring using Azure Monitor, Log Analytics, and KQ L.Enterprise Data Warehouse (EDW) ETL/Informatica Develop erDefine and enforce coding standards, code review practices, branching strategies, and release managemen t. Migration & Modernizat ionLead or contribute to legacy-to-cloud migrations — e.g., Informatica PowerCenter to Azure Data Factory, on-premises Teradata / Oracle / SQL Server to Synapse or Fabr ic.Perform workload assessment, capacity planning, and cost modeling for target-state architectur es.production incident response for critical pipelin es. Required Qualificati ons:Deep hands-on expertise with Azure Data Factory: pipelines, datasets, linked services, triggers, parameterization, mapping data flows, and all three Integration Runtime types (Azure, Selfhosted, SS IS).Strong Experience in Data Bricks and PySp ark.Production experience with one or more of: Azure Synapse Analytics (Dedicated and Serverless SQL Pools, Spark Pools) OR Azure Databricks (Delta Lake, Unity Catalog) OR Microsoft Fa bric(Warehouse, Lakehouse, OneLa ke).Strong working knowledge of Azure Data Lake Storage Gen2 (hierarchical namespace, RBAC + ACLs, lifecycle management, securi ty).Experience with Azure Key Vault, Azure AD / Entra ID (including managed identities and service principals), and private networking (VNet integration, private endpoin ts).Monitoring and troubleshooting with Azure Monitor, Log Analytics, and KQL.Advanced SQL — window functions, CTEs, query optimization, execution plan analysis, performance tun ing.Strong Python for data engineering — pandas, PySpark, REST API integration, unit testing (pyte st).Proficient in T-SQL; familiarity with Spark SQL, KQL, PowerShell, and Bash shell script ing. Preferred Qualificat ions:5+ years of data warehouse development experi ence.5+ years of data modeling experience using ERWIN or similar t ools.2+ years of experience with Azure Data Factory and Snowf lake.Medicaid Domain Knowledge is a plus Show more Show less