Lead Data Engineer

1 day ago


Singapore EPAM Systems Full time

As a Lead Data Engineer, you will play a leading role in designing, building, and optimizing our data infrastructure, ensuring that it supports the advanced analytics need of the bank. You will oversee a team of data engineers, working closely with data analysts, DevOps team, infrastructure engineers, and other stakeholders to deliver high-quality data solution.

RESPONSIBILITIES
  • Lead the design, development and implementation of high-performance Spark Scala applications and data processing pipelines for large-scale data
  • Integrate Elasticsearch with Spark for efficient data indexing, querying and retrieval
  • Optimize Spark jobs for performance and scalability on OpenShift Container Platform (OCP)
  • Collaborate with cross-functional teams to translate data requirements into robust and scalable solutions
  • Implement and maintain CI/CD pipelines for data engineering workflows, ensuring platform stability and efficient resource utilization
  • Monitor and optimize data pipeline performance, troubleshoot issues and implement necessary enhancements
  • Establish and maintain data quality and integrity throughout the data processing lifecycle
  • Document data engineering processes, workflows and infrastructure configurations
  • Stay abreast of emerging technologies and best practices in data engineering and DevOps
  • Mentor and guide junior team members, fostering a culture of continuous learning and innovation
REQUIREMENTS
  • Bachelor's degree in Computer Science, Data Engineering, Information Technology or a related field
  • 10+ years of experience as a Data Engineer with expertise in Hadoop, Spark and related technologies (HDFS, Hive, Ranger)
  • Strong proficiency in Spark, Scala, and Python for data engineering tasks
  • Experience with containerization (Docker, Kubernetes) and orchestration platforms like OpenShift
  • Knowledge of DevOps practices, CI/CD pipelines and automation tools (Jenkins, Ansible, BitBucket)
  • Experience with monitoring and logging tools (Grafana, Prometheus, Splunk) is a plus
  • Strong problem-solving, troubleshooting, and communication skills
  • Ability to work effectively in a fast-paced, collaborative environment
  • Experience with cloud platforms (AWS, Azure, GCP) is a plus
WE OFFER
  • By choosing EPAM, you're getting a job at one of the most loved workplaces according to Newsweek 2021, 2022 & 2023
  • Employee ideas are the main driver of our business. We have a very supportive environment where your voice matters
  • You will be challenged while working side-by-side with the best talent globally. We work with top-notch technologies, constantly seeking new industry trends and best practices
  • We offer a transparent career path and an individual roadmap to engineer your future & accelerate your journey
  • At EPAM, you can find vast opportunities for self-development: online courses and libraries, mentoring programs, partial grants of certification, and experience exchange with colleagues around the world. You will learn, contribute, and grow with us


  • Singapore NTT Data Singapore Full time

    NTT Data Singapore is seeking a highly skilled Data Engineer to join our team. This role will be responsible for designing, developing, and maintaining large-scale data systems.Job Description:Key Responsibilities: Design, develop, and maintain large-scale data systems using Scala programming and functional programming principles. Collaborate with...


  • Singapore The Chemical Engineer Full time

    What’s the role?Your primary accountabilities as a Process Control Senior Engineer include: Improve robustness of Base Layer control performance, delivery of high uptime and low operator intervention rates. Conduct daily base layer performance monitoring, provide technical support and lead troubleshooting for control and safeguarding related issues....


  • Singapore NTT Data Singapore Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at NTT Data Singapore.This is an excellent opportunity for someone with experience in big data technologies to take their career to the next level and work on challenging projects.ResponsibilitiesDesign, develop, and deploy large-scale data processing systems using Hadoop ecosystem...


  • Singapore NTT DATA Full time

    Job OverviewThe Senior Principal Enterprise Architect at NTT DATA plays a pivotal role in making an impact with our company. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.This is...

  • Data Engineering Lead

    1 month ago


    Singapore Epam Full time

    We are looking for a highly skilled Data Engineering Lead to join our team at EPAM Singapore.About the RoleThis is an exciting opportunity to work on challenging projects and contribute to the growth of our data engineering capabilities. As a seasoned professional, you will lead the design, development, and maintenance of big data solutions using Cloudera...


  • Singapore NTT Data Singapore Full time

    NTT Data Singapore seeks a highly skilled Software Engineering Lead to spearhead the development of our enterprise-wide visualization platform. The successful candidate will have 3-7 years of experience in system development and be proficient in Python, SQL, and Oracle.Key Responsibilities:Design and implement a continuous integration and deployment (CI/CD)...


  • Singapore NTT DATA Full time

    About the RoleWe are seeking a skilled Senior Data Center Support Engineer to join our team at NTT DATA. As a key member of our technical support group, you will be responsible for providing high-level technical support and field engineering services to clients.Job ResponsibilitiesProvide second-line remote and onsite technical support to clients, resolving...


  • Singapore National University of Singapore Full time

    About NUS IT NUS Information Technology is the cornerstone to providing reliable, high-performance and secure IT solutions and effective IT governance for the campus. Here at NUS IT, we aim to transform NUS into a borderless computing community providing knowledge at its fingertips by enhancing the use of effective applications and services for teaching and...

  • Data Engineering Lead

    3 weeks ago


    Singapore SOURCEO PTE. LTD. Full time

    Job Description:Data EngineerWe are seeking a highly skilled Data Engineer to join our team at SOURCEO PTE. LTD.The ideal candidate will have 3-5 years of experience in data engineering or a related field, with a strong knowledge of data warehouse concepts, ETL processes, and data modeling techniques.A cloud-based data platform expert with experience in AWS...


  • Singapore Preferably Professional Engineer Full time

    Client: MNC, A1 Main Contractor. Mega Projects Staff Benefits: 17 days annual leave, Year-End Bonus + VB. Salary: Negotiable Based on Qualification, Experience & Capability Location: Central Job Requirements: Degree in Electrical or Mechanical Engineering, recognized by PE Board Singapore/ BCA / IES Min 10 year experience in Mechanical & Electrical...


  • Singapore OPUS IT SERVICES PTE LTD Full time

    About OPUS IT SERVICES PTE LTDWe are a leading provider of data engineering services, helping businesses unlock the full potential of their data. As a Data Engineering Team Lead, you will play a key role in driving our team's success and shaping the future of data-driven decision making.


  • Singapore Aboitiz Data Innovation Full time

    Aboitiz Data Innovation (ADI)is one of the leading up and coming start-ups in the field of Data Science and Artificial Intelligence. We believe data can drive change for a better world by advancing businesses across industries and communities. We are looking for 2 x Mid-Senior Data Scientists to develop and deploy multiple machine learning models leveraging...

  • Data Engineering Lead

    2 weeks ago


    Singapore SOURCEO PTE. LTD. Full time

    Job Title: Data Engineering Lead - Big Data ExpertAbout the Role:We are seeking an experienced Data Engineering Lead to join our team at SOURCEO PTE. LTD. as a Big Data Expert.Responsibilities:The successful candidate will be responsible for guiding and de-risking the delivery of data projects, promoting iterative and incremental value addition. They will...

  • Data Engineering Lead

    2 weeks ago


    Singapore MANPOWER STAFFING SERVICES (SINGAPORE) PTE LTD Full time

    About the RoleWe are seeking a highly skilled Data Engineer to lead our team in designing and developing innovative data solutions. The ideal candidate will have hands-on experience in Azure Synapse, Azure Data Factory, and PySpark.ResponsibilitiesDesign and implement data storage solutions using Azure Data Lake.Migrate on-prem SQL data warehouse to Azure...

  • Data Engineering Lead

    3 weeks ago


    Singapore UARROW PTE. LTD. Full time

    Job DescriptionWe are seeking an experienced Data Engineering Lead to join our team at UARROW PTE. LTD.About the RoleThis is a senior leadership position that requires strong technical expertise in Big Data technologies, including Hadoop, Spark, and PySpark. The successful candidate will lead the design, development, and maintenance of scalable data...


  • Singapore KEPPEL DATA CENTRES HOLDING PTE. LTD. Full time

    Job OverviewWe are seeking an experienced Data Centre Business Development Lead to join our team in Singapore. The successful candidate will drive data centre development projects in Singapore and overseas, contributing to the growth of our business.Estimated Salary: SGD 120,000 - 180,000 per annumThis is a challenging role that requires strong business...


  • Singapore United Overseas Bank Full time

    Join United Overseas Bank as a Data Engineering and Operations Lead to shape the future of data architecture and pipeline in support of AML uplift analytics needs. This role is part of our Compliance function, which plays a critical strategic partner and business enabler to the Board and senior management.As a Data Engineering and Operations Lead, you will...

  • Lead Data Engineer

    1 week ago


    Singapore D L RESOURCES PTE LTD Full time

    Roles & ResponsibilitiesKey Skills:Quantexa certification mandatoryJob ObjectivesWe seek individuals with highly developed conceptual, strategic, and analytical skills, capable of striking a balance between visionary thinking and practical solutions. The ability to comprehend, inspire, and mobilize others is crucial. A business-oriented mindset...

  • Lead Data Engineer

    3 months ago


    Singapore D L RESOURCES PTE LTD Full time

    Roles & ResponsibilitiesKey Skills:Quantexa certification mandatoryJob ObjectivesWe seek individuals with highly developed conceptual, strategic, and analytical skills, capable of striking a balance between visionary thinking and practical solutions. The ability to comprehend, inspire, and mobilize others is crucial. A business-oriented mindset...


  • Singapore ADECCO PERSONNEL PTE LTD Full time

    Job SummaryWe are seeking an experienced Cloud Data Engineering Lead to join our team at ADECCO PERSONNEL PTE LTD. The successful candidate will be responsible for designing and implementing data storage solutions, creating and managing data pipelines, and integrating data from various sources.About the RoleThis is a unique opportunity to leverage your...