Data Engineering Lead

6 days ago

Singapore Capgemini Full time $120,000 - $180,000 per year
Job Description

We are looking for an experienced Data Engineering Lead (Vendor) to support our platform modernization program. The resource will work with internal teams to migrate applications from Cloudera CDH to a Kubernetes-based global data platform and ensure timely delivery of high-quality data engineering solutions.

Key Responsibilities

  • Provide technical leadership for migration projects from Cloudera (Spark, Hive, Kafka, Control-M) to Kubernetes stack (Spark 3.5, DBT, Airflow, MinIO/S3, Kafka, Solace).
  • Lead a small team and internal engineers to deliver project deliverables.
  • Participate in design, architecture discussions, and migration planning with internal leads.
  • Build and review high-performance, production-ready pipelines.
  • Ensure adherence to standards, compliance, and governance requirements.
  • Provide status reporting, escalations, and delivery tracking to stakeholders.
  • Design and implement migration/acceleration framework to automate end to end migration.
  • Continuous enhancements to the frameworks to ensure the stability, scalability and support for diverse use cases and scenarios.
  • Work with various data applications to enable and support the migration process.
Required Skills
  • 9-12 years of experience in data engineering and big data ecosystems.
  • Strong hands-on expertise in Spark, Hive, Kafka, Solace.
  • Working experience with Kubernetes deployments and containerized data workloads.
  • Proficiency in Python, Scala, and/or Java.
  • Experience in orchestration tools (Airflow, Control-M) and SQL transformation frameworks (DBT preferred).
  • Familiarity with object stores (S3, MinIO).
  • Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).
  • Prior experience leading vendor or distributed teams for enterprise projects.

Engagement Expectations

  • Vendor resource will be expected to ramp up quickly and start contributing to active migration projects.
  • Must coordinate with internal leads, product owners, and platform teams to ensure project timelines are met.
  • Should be flexible to support multiple time zones (if required).

  • Data Engineer(Cloud Migration)

    2 weeks ago

    Singapore NTT DATA Full time

    Requirements / Qualifications - Bachelor's or master's degree in computer science, data engineering, or a related field. - Minimum 7 years of experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica IDMC. - Proficiency in programming languages such as Python, Java, or Scala for building data pipelines. - Evaluate...

  • Data Engineer

    2 weeks ago

    Singapore DOZER DATA PTE. LTD. Full time

    **About the job**: This position is needed to build a highly scalable, reliable, and efficient data platform that makes it easy for users to get deep insights from vast amounts of distributed data. This role will be responsible for driving key technical decisions to completion for the organization and also guiding other engineers. **Responsibilities**: In...

  • Azure Data engineer

    12 hours ago

    Singapore NTT Data Singapore Full time $60,000 - $180,000 per year

    Job Responsibilities:Data Analysis and Interpretation: Perform in-depth analysis of structured and unstructured data sets to identify trends, patterns, and valuable insights that drive business decisions and strategies. Data Visualization and Reporting: Develop intuitive and visually appealing dashboards, reports, and presentations to communicate complex...

  • Lead Data Engineer

    1 week ago

    Singapore VANGUARD SOFTWARE PTE. LTD. Full time

    **JOB SUMMARY** **JOB DUTIES** - Lead the design, development, and implementation of scalable and efficient data pipelines, databases, and data infrastructure. - Mentor, coach, and guide a team of data engineers, providing technical leadership and support. - Collaborate with data scientists, product managers, and business stakeholders to understand data...

  • DevSecOps Lead

    2 days ago

    Singapore NTT DATA Full time $120,000 - $200,000 per year

    Make an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.Your day at NTT DATAThe DevSecOps...

  • Lead Data Engineer

    5 days ago

    Singapore ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD. Full time

    **Your Role**: As a **Lead Data Engineer**, you will spearhead the development and optimization of our data infrastructure to enable advanced analytics capabilities. Collaborating with cross-functional teams, you will ensure seamless integration and delivery of robust, scalable, and efficient data solutions. In this leadership role, you will mentor and guide...

  • Data Engineering Lead

    3 days ago

    Singapore NUS Information Technology Full time

    **Posting Start Date**:12/12/2024 About NUS ITWe drive a culture that is forward-looking. With a strong passion for IT, our people are always striving to improve, push boundaries and innovate with a "can-do" attitude. We embrace collaboration, open communication and knowledge sharing. If you see yourself thriving in a dynamic environment and breaking new...

  • Lead Data Engineer

    5 days ago

    Singapore JERA Global Markets Pte. Ltd. Full time

    COMPANY DESCRIPTION **About the Company** JERA Global Markets (JERAGM) is a leading utility-backed seaborne energy trader specialising in LNG, Coal and Freight with a platform to trade across all key markets worldwide. JERAGM is a joint venture between majority shareholder JERA Inc. (JERA) and EDF Trading. JERAGM operates one of the largest seaborne...

  • Lead Data Engineer

    6 days ago

    Singapore Hays Full time

    **Your new company** Your new role is a leading technology business looking for a Lead Data Engineer to join one of their business units focusing on multiple data projects. **Your new role** You will: - Partner with business users to understand and translate them into technical requirements - Design and build resilient and efficient data pipelines for both...

  • Data Engineer Lead

    2 weeks ago

    Singapore Unison Consulting Pte Ltd Full time

    5 years of experience in data management, data engineering, or data visualization. Experience with Python, AWS, Glue, spark, redshift, Lambda, S3 and strong Sql (DWH). Excellent with AWS side of data engineering. Amazon S3 for storage, AWS Glue for ETL, Amazon Redshift for data warehousing, Amazon Kinesis for real-time data processing, and AWS Lambda for...