data engineer

Big Data Engineer Mississauga, ON Hybrid(3 days office per week) *Fulltime * Job Description: JD: * Develop and maintain scalable data pipelines using PySpark and Spark SQL for processing large datasets efficiently. * Write clean, reusable, and optimized code in Python for data manipulation, analysis, and automation tasks. * Design and implement ETL workflows to extract, transform, and load data from various structured and unstructured sources. * Collaborate with data engineers, analysts, and stakeholders to understand data requirements and deliver solutions. * Optimize Spark jobs for performance tuning, resource utilization, and minimizing execution time. * Work with distributed computing frameworks to process and analyze big data in a cloud or on-premises environment. * Develop and maintain unit tests to ensure the accuracy and reliability of data pipelines and transformations. * Utilize Spark SQL for querying and managing large datasets stored in distributed systems like Hadoop or cloud storage. * Monitor and troubleshoot data pipeline issues, ensuring reliability and timely delivery of data. * Stay updated with the latest advancements in PySpark, Spark SQL, and big data technologies to improve existing systems. Pay: $100,000.00-$130,000.00 per year Benefits: * Flexible schedule Experience: * Big Data: 6 years (required) * SQL: 5 years (required) * PySpark: 1 year (required) Work Location: Hybrid remote in Mississauga, ON (L5B)