Lead Data Engineer

5 days ago

Singapore ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD. Full time

**Your Role**:
As a **Lead Data Engineer**, you will spearhead the development and optimization of our data infrastructure to enable advanced analytics capabilities. Collaborating with cross-functional teams, you will ensure seamless integration and delivery of robust, scalable, and efficient data solutions. In this leadership role, you will mentor and guide a team of engineers while contributing to the strategic direction of the organization’s data landscape.

**Key Responsibilities**:

- **Data Engineering & Development**:Design and implement large-scale data processing pipelines using **Spark Scala**, handling structured and unstructured data.
Integrate **Elasticsearch**with Spark for efficient indexing, querying, and data retrieval.
- **Containerization & Orchestration**:Deploy data solutions on **OpenShift Container Platform (OCP)**using containerization and orchestration techniques.
Optimize workflows for containerized environments and ensure efficient resource utilization.
- **DevOps Integration**:Collaborate with DevOps teams to streamline deployment using **CI/CD pipelines**and tools such as **Jenkins**, **Bitbucket**, and **Ansible**.
Implement robust monitoring and logging frameworks using tools like **Grafana**, **Prometheus**, and **Splunk**to maintain platform stability.
- **Collaboration & Stakeholder Management**:Work with data scientists, analysts, and infrastructure engineers to understand requirements and translate them into technical solutions.
Communicate technical designs and processes effectively to both technical and non-technical stakeholders.
- **Leadership & Mentorship**:Lead and mentor a team of data engineers, fostering a culture of continuous learning and innovation.
Advocate for best practices in data engineering, DevOps, and cloud technologies.
- **Documentation & Research**:Document workflows, pipelines, and configurations to ensure team knowledge sharing.
Stay abreast of emerging technologies and best practices in data engineering and cloud infrastructure.

**Requirements**:

- **Education**:Bachelor’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
- **Experience**:Minimum of 10 years in data engineering roles with expertise in **Hadoop**, **Spark**, and related tools (e.g., **HDFS**, **Hive**, **Ranger**).
Hands-on experience with **OpenShift Container Platform (OCP)**and container orchestration tools like **Kubernetes**.
Proven track record of building and scaling data infrastructure in large, complex environments.
- **Technical Skills**:Proficiency in programming languages such as **Scala**, **Python**, or **Java**.
Deep understanding of **DevOps**practices, including CI/CD pipelines and automation tools like **Docker**, **Jenkins**, and **Ansible**.
Familiarity with monitoring and logging tools such as **Grafana**, **Prometheus**, and **Splunk**.
Exposure to **cloud platforms**(AWS, Azure, GCP) and their data services is a plus.
- **Soft Skills**: Exceptional problem-solving abilities and a proactive approach to resolving technical challenges.
Strong communication and collaboration skills to work effectively across diverse teams.
Ability to balance multiple priorities and deliver high-quality results in a dynamic environment.

**Preferred Qualifications**:

- Experience in cloud-native data services (e.g., AWS Redshift, Azure Data Factory, GCP BigQuery).
- Knowledge of advanced analytics and machine learning pipelines.

  • Data Engineer(Cloud Migration)

    2 weeks ago

    Singapore NTT DATA Full time

    Requirements / Qualifications - Bachelor's or master's degree in computer science, data engineering, or a related field. - Minimum 7 years of experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica IDMC. - Proficiency in programming languages such as Python, Java, or Scala for building data pipelines. - Evaluate...

  • Data Engineer

    2 weeks ago

    Singapore DOZER DATA PTE. LTD. Full time

    **About the job**: This position is needed to build a highly scalable, reliable, and efficient data platform that makes it easy for users to get deep insights from vast amounts of distributed data. This role will be responsible for driving key technical decisions to completion for the organization and also guiding other engineers. **Responsibilities**: In...

  • Azure Data engineer

    19 hours ago

    Singapore NTT Data Singapore Full time $60,000 - $180,000 per year

    Job Responsibilities:Data Analysis and Interpretation: Perform in-depth analysis of structured and unstructured data sets to identify trends, patterns, and valuable insights that drive business decisions and strategies. Data Visualization and Reporting: Develop intuitive and visually appealing dashboards, reports, and presentations to communicate complex...

  • Lead Data Engineer

    1 week ago

    Singapore VANGUARD SOFTWARE PTE. LTD. Full time

    **JOB SUMMARY** **JOB DUTIES** - Lead the design, development, and implementation of scalable and efficient data pipelines, databases, and data infrastructure. - Mentor, coach, and guide a team of data engineers, providing technical leadership and support. - Collaborate with data scientists, product managers, and business stakeholders to understand data...

  • DevSecOps Lead

    3 days ago

    Singapore NTT DATA Full time $120,000 - $200,000 per year

    Make an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.Your day at NTT DATAThe DevSecOps...

  • Data Engineering Lead

    6 days ago

    Singapore Capgemini Full time $120,000 - $180,000 per year

    Job Description We are looking for an experienced Data Engineering Lead (Vendor) to support our platform modernization program. The resource will work with internal teams to migrate applications from Cloudera CDH to a Kubernetes-based global data platform and ensure timely delivery of high-quality data engineering solutions.Key ResponsibilitiesProvide...

  • Data Engineering Lead

    3 days ago

    Singapore NUS Information Technology Full time

    **Posting Start Date**:12/12/2024 About NUS ITWe drive a culture that is forward-looking. With a strong passion for IT, our people are always striving to improve, push boundaries and innovate with a "can-do" attitude. We embrace collaboration, open communication and knowledge sharing. If you see yourself thriving in a dynamic environment and breaking new...

  • Lead Data Engineer

    5 days ago

    Singapore JERA Global Markets Pte. Ltd. Full time

    COMPANY DESCRIPTION **About the Company** JERA Global Markets (JERAGM) is a leading utility-backed seaborne energy trader specialising in LNG, Coal and Freight with a platform to trade across all key markets worldwide. JERAGM is a joint venture between majority shareholder JERA Inc. (JERA) and EDF Trading. JERAGM operates one of the largest seaborne...

  • Lead Data Engineer

    7 days ago

    Singapore Hays Full time

    **Your new company** Your new role is a leading technology business looking for a Lead Data Engineer to join one of their business units focusing on multiple data projects. **Your new role** You will: - Partner with business users to understand and translate them into technical requirements - Design and build resilient and efficient data pipelines for both...

  • Data Engineer Lead

    2 weeks ago

    Singapore Unison Consulting Pte Ltd Full time

    5 years of experience in data management, data engineering, or data visualization. Experience with Python, AWS, Glue, spark, redshift, Lambda, S3 and strong Sql (DWH). Excellent with AWS side of data engineering. Amazon S3 for storage, AWS Glue for ETL, Amazon Redshift for data warehousing, Amazon Kinesis for real-time data processing, and AWS Lambda for...