Lead Data Engineer

3 months ago

Singapore D L RESOURCES PTE LTD Full time
Roles & Responsibilities

Key Skills:

Quantexa certification mandatory

Job Objectives

We seek individuals with highly developed conceptual, strategic, and analytical skills, capable of striking a balance between visionary thinking and practical solutions. The ability to comprehend, inspire, and mobilize others is crucial. A business-oriented mindset coupled with effective storytelling will drive your success. We are looking for self-starters ready to take on responsibilities with enthusiasm.

Key Responsibilities

As a Lead Data Engineer, you will play a leading role in designing, building, and optimizing our data infrastructure, ensuring that it supports the advanced analytics need of the bank. You will oversee a team of data engineers, working closely with data analysts, DevOps team, infrastructure engineers, and other stakeholders to deliver high-quality data solution. You will be working with Quantexa platform.



Your main responsibilities will include:

• Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data.

• Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data.

• Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch.

• Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions.

• Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques.

• Optimize data engineering workflows for containerized deployment and efficient resource utilization.

• Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability.

• Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements.

• Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure.

• Document data engineering processes, workflows, and infrastructure configurations for knowledge sharing and reference.

• Stay updated with emerging technologies, industry trends, and best practices in data engineering and DevOps.

• Provide technical leadership, mentorship, and guidance to junior team members to foster a culture of continuous learning and innovation to the continuous improvement of the analytics capabilities within the bank.




Key Requirements

• Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field.

• At least 10 years of experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments.

• Must be Quantexa certified data engineer / data architect and proficient with the tool.

• Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Ranger, etc)

• Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes.

• Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java.

• Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)

• Experience with Grafana, Prometheus, Splunk will be an added benefit

• Strong problem-solving and troubleshooting skills with a proactive approach to resolving technical challenges.

• Excellent collaboration and communication skills to work effectively with cross-functional teams.

• Ability to manage multiple priorities, meet deadlines, and deliver high-quality results in a fast-paced environment.

• Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services is a plus.


Tell employers what skills you have

Scalability
Scala
Azure
Pipelines
Architect
Hadoop
Automation Tools
Information Technology
Data Engineering
Logging
Python
Containerization
Visionary Thinking
Java
Orchestration

  • Data Engineer Specialist

    2 weeks ago

    Singapore NTT Data Singapore Full time

    NTT Data Singapore is seeking a highly skilled Data Engineer to join our team. This role will be responsible for designing, developing, and maintaining large-scale data systems.Job Description:Key Responsibilities: Design, develop, and maintain large-scale data systems using Scala programming and functional programming principles. Collaborate with...

  • The Chemical Engineer | Process Control Senior Technologist

    1 week ago

    Singapore The Chemical Engineer Full time

    What’s the role?Your primary accountabilities as a Process Control Senior Engineer include: Improve robustness of Base Layer control performance, delivery of high uptime and low operator intervention rates. Conduct daily base layer performance monitoring, provide technical support and lead troubleshooting for control and safeguarding related issues....

  • Data Engineering Expert for Big Data Technologies

    2 days ago

    Singapore NTT Data Singapore Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at NTT Data Singapore.This is an excellent opportunity for someone with experience in big data technologies to take their career to the next level and work on challenging projects.ResponsibilitiesDesign, develop, and deploy large-scale data processing systems using Hadoop ecosystem...

  • Strategic Enterprise Data Architect Lead

    1 month ago

    Singapore NTT DATA Full time

    Job OverviewThe Senior Principal Enterprise Architect at NTT DATA plays a pivotal role in making an impact with our company. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.This is...

  • Lead Data Engineer

    3 days ago

    Singapore EPAM Systems Full time

    As a Lead Data Engineer, you will play a leading role in designing, building, and optimizing our data infrastructure, ensuring that it supports the advanced analytics need of the bank. You will oversee a team of data engineers, working closely with data analysts, DevOps team, infrastructure engineers, and other stakeholders to deliver high-quality data...

  • Data Engineering Lead

    1 month ago

    Singapore Epam Full time

    We are looking for a highly skilled Data Engineering Lead to join our team at EPAM Singapore.About the RoleThis is an exciting opportunity to work on challenging projects and contribute to the growth of our data engineering capabilities. As a seasoned professional, you will lead the design, development, and maintenance of big data solutions using Cloudera...

  • Software Engineering Lead

    1 month ago

    Singapore NTT Data Singapore Full time

    NTT Data Singapore seeks a highly skilled Software Engineering Lead to spearhead the development of our enterprise-wide visualization platform. The successful candidate will have 3-7 years of experience in system development and be proficient in Python, SQL, and Oracle.Key Responsibilities:Design and implement a continuous integration and deployment (CI/CD)...

  • Senior Data Center Support Engineer

    1 month ago

    Singapore NTT DATA Full time

    About the RoleWe are seeking a skilled Senior Data Center Support Engineer to join our team at NTT DATA. As a key member of our technical support group, you will be responsible for providing high-level technical support and field engineering services to clients.Job ResponsibilitiesProvide second-line remote and onsite technical support to clients, resolving...

  • Data Engineering Lead

    1 week ago

    Singapore National University of Singapore Full time

    About NUS IT NUS Information Technology is the cornerstone to providing reliable, high-performance and secure IT solutions and effective IT governance for the campus. Here at NUS IT, we aim to transform NUS into a borderless computing community providing knowledge at its fingertips by enhancing the use of effective applications and services for teaching and...

  • Data Engineering Lead

    3 weeks ago

    Singapore SOURCEO PTE. LTD. Full time

    Job Description:Data EngineerWe are seeking a highly skilled Data Engineer to join our team at SOURCEO PTE. LTD.The ideal candidate will have 3-5 years of experience in data engineering or a related field, with a strong knowledge of data warehouse concepts, ETL processes, and data modeling techniques.A cloud-based data platform expert with experience in AWS...

  • Data Engineering Team Lead

    3 weeks ago

    Singapore OPUS IT SERVICES PTE LTD Full time

    About OPUS IT SERVICES PTE LTDWe are a leading provider of data engineering services, helping businesses unlock the full potential of their data. As a Data Engineering Team Lead, you will play a key role in driving our team's success and shaping the future of data-driven decision making.

  • M & e manager / a1 main con > perm

    2 months ago

    Singapore Preferably Professional Engineer Full time

    Client: MNC, A1 Main Contractor. Mega Projects Staff Benefits: 17 days annual leave, Year-End Bonus + VB. Salary: Negotiable Based on Qualification, Experience & Capability Location: Central Job Requirements: Degree in Electrical or Mechanical Engineering, recognized by PE Board Singapore/ BCA / IES Min 10 year experience in Mechanical & Electrical...

  • Aboitiz Data Innovation | Data Scientist

    6 days ago

    Singapore Aboitiz Data Innovation Full time

    Aboitiz Data Innovation (ADI)is one of the leading up and coming start-ups in the field of Data Science and Artificial Intelligence. We believe data can drive change for a better world by advancing businesses across industries and communities. We are looking for 2 x Mid-Senior Data Scientists to develop and deploy multiple machine learning models leveraging...

  • Data Engineering Lead

    3 weeks ago

    Singapore SOURCEO PTE. LTD. Full time

    Job Title: Data Engineering Lead - Big Data ExpertAbout the Role:We are seeking an experienced Data Engineering Lead to join our team at SOURCEO PTE. LTD. as a Big Data Expert.Responsibilities:The successful candidate will be responsible for guiding and de-risking the delivery of data projects, promoting iterative and incremental value addition. They will...

  • Data Engineering Lead

    3 weeks ago

    Singapore MANPOWER STAFFING SERVICES (SINGAPORE) PTE LTD Full time

    About the RoleWe are seeking a highly skilled Data Engineer to lead our team in designing and developing innovative data solutions. The ideal candidate will have hands-on experience in Azure Synapse, Azure Data Factory, and PySpark.ResponsibilitiesDesign and implement data storage solutions using Azure Data Lake.Migrate on-prem SQL data warehouse to Azure...

  • Data Engineering Lead

    3 weeks ago

    Singapore UARROW PTE. LTD. Full time

    Job DescriptionWe are seeking an experienced Data Engineering Lead to join our team at UARROW PTE. LTD.About the RoleThis is a senior leadership position that requires strong technical expertise in Big Data technologies, including Hadoop, Spark, and PySpark. The successful candidate will lead the design, development, and maintenance of scalable data...

  • Data Centre Business Development Lead

    3 weeks ago

    Singapore KEPPEL DATA CENTRES HOLDING PTE. LTD. Full time

    Job OverviewWe are seeking an experienced Data Centre Business Development Lead to join our team in Singapore. The successful candidate will drive data centre development projects in Singapore and overseas, contributing to the growth of our business.Estimated Salary: SGD 120,000 - 180,000 per annumThis is a challenging role that requires strong business...

  • Data Engineering and Operations Lead

    2 weeks ago

    Singapore United Overseas Bank Full time

    Join United Overseas Bank as a Data Engineering and Operations Lead to shape the future of data architecture and pipeline in support of AML uplift analytics needs. This role is part of our Compliance function, which plays a critical strategic partner and business enabler to the Board and senior management.As a Data Engineering and Operations Lead, you will...

  • Cloud Data Engineering Lead

    3 weeks ago

    Singapore ADECCO PERSONNEL PTE LTD Full time

    Job SummaryWe are seeking an experienced Cloud Data Engineering Lead to join our team at ADECCO PERSONNEL PTE LTD. The successful candidate will be responsible for designing and implementing data storage solutions, creating and managing data pipelines, and integrating data from various sources.About the RoleThis is a unique opportunity to leverage your...

  • Data Scientist and Engineering Lead

    3 weeks ago

    Singapore FPT ASIA PACIFIC PTE. LTD. Full time

    About FPT ASIA PACIFIC PTE. LTD.FPT Asia Pacific is a leading provider of IT services and solutions, with a strong presence in the region.Job SummaryWe are seeking an experienced Data Scientist and Engineering Lead to join our team in Singapore. The successful candidate will be responsible for designing, developing, and implementing data engineering...