Apply for ETL Data Engineer - AWS & Spark
ETL Data Engineer - AWS & Spark
- categoryApplication Development and Maintenance-Digital Technology Transformation
- scheduleContract To Hire
- monetization_onNA
- location_onMelbourne, Sydney
Australia
Position |
|
Experience |
5-12 Yrs |
Skills
|
About the Job
- We are seeking highly skilled ETL Data Engineer with experience in ETL development and a solid understanding of the AWS ecosystem, particularly EMR, S3, Athena, and SQL. The ideal candidate will have expertise in Spark and Scala, ETL lifecycle management, and job scheduling using Autosys. Experience with framework builds and ETL job migration to AWS environments is highly desirable.
Key Roles & Responsibilities
- Execute ETL jobs and processes from Day 1 to ensure smooth data operations.
- Reconcile ETL jobs for a minimum of three runs, ensuring data accuracy and consistency.
- Triage, debug, and fix issues with ETL jobs while reconciling data discrepancies.
- Collaborate with multiple capability teams to debug and resolve job-related challenges effectively.
- Conduct knowledge transfer (KT) sessions to ensure offshore teams are equipped to manage and support ETL processes.
Experience & Qualifications
- 5-12 years of strong ETL development experience using Spark and Scala.
- Hands-on experience with AWS services, including EMR, S3, Athena, and SQL.
- Proficiency in Autosys for job scheduling.
- Familiarity with building reusable ETL frameworks.
- Prior experience in migrating ETL jobs to AWS environments.
- Comprehensive understanding of the ETL lifecycle, including running, debugging, testing, reconciling, and deploying jobs in an AWS environment.
- Strong problem-solving skills and ability to work effectively with cross-functional teams.
- Proactively run and execute jobs to ensure operational continuity.
- Perform reconciliation of jobs for at least three runs to validate data integrity.
- Debug, troubleshoot, and resolve issues related to ETL job failures.
- Engage and collaborate with multiple capability teams to address challenges and implement fixes.
- Provide thorough onsite knowledge transfer to offshore teams for smooth transition and support.
Additional Preferred Attributes
- Proactive and self-driven with a strong technical foundation in AWS and ETL processes.
- Excellent communication and collaboration skills to work with diverse teams.
- Proven ability to work under pressure and deliver high-quality solutions in a dynamic environment.