AWS Data Engineer

We are HCLTech, one of the fastest-growing large tech companies in the world and home to 219,000+ people across 54 countries, supercharging progress through industry-leading capabilities centered around Digital, Engineering and Cloud. The driving force behind that work, our people, are diverse, creative, and passionate, raising the bar for excellence on a regular basis. We, in turn, work hard to bring out the best in them as we strive to help them find their spark and become the best version of themselves that they can be.At HCLTech Australia, we value the unique perspective and contributions of all individual and we actively encourage applications from Aboriginal and Torres Strait Islander people to apply for this role. Are you ready to be an important part of this ever-transformational journey? About the roleWe are seeking AWS Data Engineer - Scale Migration Specialist to join our Data Team to drive the successful migration of on-premise Hadoop big data infrastructure to AWS, ensuring scalability, performance, and data integrity throughout the process. The AWS Data Engineer will be a key player in our Ninja Team, responsible for designing, developing, and implementing data processing and migration strategies. The role involves working with various big data technologies and AWS services to facilitate a seamless transition from on-premise Hadoop to a cloud-based environment. This includes data migration, optimization, and integration tasks to ensure robust and scalable data solutions. What you'll be doingDesign and implement data migration strategies from on-premise Hadoop to AWS.Develop and optimize Spark jobs for data processing on AWS EMR.Develop and optimize Spark jobs for data processing on AWS EMR.Manage and maintain S3 storage solutions for data lake architecture.Configure and manage AWS Glue for ETL processes and data cataloging.Ensure data quality and integrity throughout the migration process.Collaborate with stakeholders to understand and document data requirements.Perform data mapping and transformation tasks as part of the migration strategy.Implement and manage AWS Lake Formation for data governance and security.Troubleshoot and resolve data migration issues and performance bottlenecks.Monitor and optimize data workflows for performance and cost efficiency.Develop and maintain documentation for data processes and architecture.Coordinate with cross-functional teams to align migration activities with project goals.Provide technical guidance and support to junior team members.Conduct performance tuning and optimization of data processing tasks.Ensure compliance with data security and privacy policies.Design and implement automated data pipelines using AWS services.Perform regular data validation and verification checks.Collaborate with AWS support and engineering teams as needed.Stay up-to-date with the latest developments in big data and cloud technologies.Assist in training and knowledge transfer activities for team members. What we're looking forBachelor’s or Master’s degree in Engineering, Computer Science, or a related field.6-10 years of experience with the following technologies: Spark, Oracle 11g, Hive, HDFS, Hadoop, AWS services.4-6 years of experience specifically with data migration and big data technologies.Expertise in Apache Spark and AWS EMR.Strong experience with AWS services such as S3, Glue, and Lake Formation.Proficiency in HDFS, Hive, and data migration strategies.Solid understanding of data processing and optimization techniques.Experience with other AWS services such as Redshift and Snowflake.Familiarity with Hadoop ecosystem tools like Sqoop, Zookeeper, and Yarn.Knowledge of data governance and security best practices.AWS Certified Data Analytics – Specialty or AWS Certified Solutions Architect – Associate/Professional.Cloudera Certified Professional (CCP) in Data Engineering (preferred) Why Us We are one of the fastest-growing large tech companies in the world, with offices in 50+ countries across the globe and 219,000 employees.Our company is extremely diverse with 165 nationalities represented.We offer the opportunity to work with colleagues across the globe.We offer a virtual-first work environment, promoting a good work-life integration and real flexibility.We offer comprehensive benefits for all employees.We are a certified great place to work and a top employer in 17 countries, offering a positive work environment that values employee recognition and respect. Equality & Opportunity for All Representing 165 nationalities across the globe, we pride ourselves on being an equal opportunity employer, committed to providing equal employment opportunities to all applicants and employees regardless of race, religion, sex, colour, age, national origin, pregnancy, sexual orientation, physical disability or genetic information, military or veteran status, Aboriginal and Torres Strait Islander people or any other protected classification, in accordance with federal, state, and/or local law. Candidate Data Privacy Notice