data engineer

Job Description Principal Databricks Data Engineer Experience Required: 12?18 Years Key Requirements ? 12?18 years of overall Data Engineering experience ? 8+ years of experience with enterprise Data Warehouse and Data Lake platforms ? 5+ years of hands-on experience with Databricks and Apache Spark at scale ? Strong experience modernizing legacy Cloudera platforms including: ? CDH/CDP ? Hive ? HBase ? Impala ? Spark ? Modernize Cloudera platforms to Databricks Lakehouse architecture ? Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake ? Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines ? Ensure data reconciliation, audit integrity, and consistency during migration ? Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures ? Implement layered data architecture including: ? Raw / Landing Layer ? Curated / Conformed Layer ? Semantic / Consumption Layer ? Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures ? Strong experience with finance and risk data models including: ? General Ledger ? Sub-ledger ? Financial Hierarchies ? Credit Risk ? Liquidity Risk ? Market Risk ? Enable reporting capabilities including: ? Aggregation ? Drill-down ? Drill-back ? Build and manage semantic and consumption layers for BI, reporting, and analytics ? Define business metrics, dimensions, hierarchies, and KPIs ? Experience with: ? Databricks SQL ? Delta Tables ? dbt or similar frameworks ? Develop and optimize large-scale data pipelines using: ? PySpark ? Spark SQL ? Delta Lake ? Implement Medallion Architecture including: ? Bronze Layer ? Silver Layer ? Gold Layer ? Optimize workloads using: ? Z-ORDER ? OPTIMIZE ? Caching ? Cluster Configuration Tuning ? Implement: ? Data Governance ? Data Quality Frameworks ? Reconciliation Controls ? Exception Handling ? Establish data lineage and metadata management ? Ensure data security, access control, and compliance standards ? Experience with AWS or Azure cloud platforms ? Experience with CI/CD pipelines using: ? Git ? Terraform ? Jenkins ? Azure DevOps ? Familiarity with: ? Apache Airflow ? Databricks Workflows ? Experience with dbt is an advantage ? Act as a technical authority and lead enterprise architecture decisions ? Mentor senior engineers and establish engineering standards ? Collaborate with finance, risk, analytics, and governance stakeholders ? Translate complex data structures into business-ready insights Nice-to-Have Skills <d