Senior Data Engineer

4 days ago


Singapore OPENSOURCE PTE. LTD. Full time
Roles & Responsibilities

Job Title: Senior Data Engineer (5-10 Years of Experience)

Position Summary:

A company is seeking an experienced and skilled Senior Data Engineer with 5 to 10 years of hands-on experience in building and managing data pipelines and big data infrastructure. The ideal candidate will possess a strong technical background in data engineering tools, big data ecosystems, and programming languages. This role focuses on designing, implementing, and optimizing complex data workflows using technologies such as Apache Airflow, Apache NiFi, Kafka, PySpark, and Spark Scala. Candidates with a strong background in Python, Java, and shell scripting are highly preferred.

Responsibilities:
  • Design, build, and maintain scalable and reliable data pipelines to support real-time and batch processing requirements.
  • Manage, monitor, and optimize workflows in Apache Airflow to ensure data quality and integrity.
  • Develop and integrate data flows using Apache NiFi for seamless data ingestion and transformation.
  • Work extensively with Apache Kafka for data streaming and messaging across various data sources.
  • Implement data processing solutions using PySpark and Spark Scala to handle large-scale datasets and complex transformations.
  • Write efficient code in Python and Java to automate data workflows and support data engineering needs.
  • Utilize shell scripting for operational tasks and automation within the data environment.
  • Collaborate with cross-functional teams to understand data requirements and provide optimized data engineering solutions.
  • Ensure data security, compliance, and performance by following best practices in big data and distributed systems.
  • Continuously improve the performance, scalability, and reliability of data processing pipelines.
Required Skills and Experience:
  • Apache Airflow: Extensive experience in managing, scheduling, and monitoring data pipelines.
  • Apache NiFi: Strong experience in designing data flows for data ingestion and transformation.
  • Apache Kafka: In-depth knowledge of Kafka for real-time data streaming and messaging systems.
  • PySpark & Spark Scala: Proficiency in using PySpark and Spark Scala for large-scale data processing.
  • Programming Languages: Strong experience with Python and Java, with additional knowledge in shell scripting.
  • Big Data Knowledge: Familiarity with big data ecosystems and distributed data processing.
  • Ability to work independently and collaboratively in a fast-paced environment, handling complex data engineering challenges.
Educational Qualifications:
  • Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, or a related field.
Preferred Qualifications:
  • Experience in cloud environments (AWS, GCP, Azure) with big data components is a plus.
  • Experience with version control tools (e.g., Git) and CI/CD practices in data engineering.
  • Strong analytical, problem-solving, and communication skills.

Employment Type: Full-Time


Tell employers what skills you have

PySpark
Airflow
Scalability
Scala
Azure
Big Data
Pipelines
Scripting
Information Technology
Data Quality
Reliability
Data Engineering
Python
GCP
Java
Apache

  • Singapore The Chemical Engineer Full time

    About us At Exxon Mobil, our vision is to lead in energy innovations that advance modern living and a net-zero future. As one of the world’s largest publicly traded energy and chemical companies, we are powered by a unique and diverse workforce fueled by the pride in what we do and what we stand for. The success of our Upstream, Product Solutions and Low...


  • Singapore NTT Data Singapore Full time

    Requirements We are seeking a highly skilled Data Engineer to join our team at NTT Data Singapore. The ideal candidate will have a Bachelor's degree in Computer Science or a related field and at least 7 years of IT experience, with a minimum of 5 years in a similar role. Proficiency in distributed computing principles and strong experience with Hadoop,...


  • Singapore NTT Data Singapore Full time

    Job Requirements We are seeking a highly skilled Data Engineer to join our team at NTT Data Singapore. The ideal candidate will have a Bachelor's degree in Computer Science or a related field and at least 7 years of IT experience, with a minimum of 5 years in a similar role. Key Responsibilities • Proficient understanding of distributed computing...


  • Singapore KEPPEL DATA CENTRES HOLDING PTE. LTD. Full time

    About the RoleWe are seeking an exceptional Senior Manager, Technical Solutions to lead our Data Centre Engineering projects from inception to completion. This is a key position that requires strong technical expertise and leadership skills.Key Responsibilities:Develop and manage technical solutions for Data Centre Engineering projects, ensuring timely...


  • Singapore DC FRONTIERS PTE. LTD. Full time

    Roles & ResponsibilitiesHandshakes by DC Frontiers is an award-winning DataTech company that leverages data to empower safe, informed business decisions. We are located in 7 cities and are expanding our presence across the globe.We are hiring an experienced Senior Data Engineer to be based in our Singapore office. The Senior Data Engineer will be part of our...


  • Singapore SENIOR SPIRIT OF ROSELLE PARK Full time

    Responsibilities: Prepares key project documentation in executing the complete instrumentation engineering scope of work as per the project requirements in meeting customer’s expectations Prepares instrumentation system concepts, schedule of deliverables and plan activities in line with overall project schedule, delegate the tasks to instrumentation team...

  • Senior Data Engineer

    1 month ago


    Singapore TEKsystems (Allegis Group Singapore Pte Ltd) Full time

    Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at TEKsystems (Allegis Group Singapore Pte Ltd). As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining cloud-based data engineering solutions that meet the needs of our enterprise clients.Key Responsibilities:Develop and maintain...

  • Senior Data Engineer

    3 weeks ago


    Singapore RANDSTAD PTE. LIMITED Full time

    About the RoleThis Senior Data Engineer position offers an exciting opportunity to join a high-performing team within an established company. The ideal candidate will have a strong background in data engineering, with a proven track record in Azure Data Factory, Azure Synapse Analytics/Data Warehouse, and Python.Main ResponsibilitiesEstablish and manage data...


  • Singapore RAPSYS TECHNOLOGIES PTE. LTD. Full time

    Roles & Responsibilities2. Position Title: Senior Data EngineerExperience Required: 5-10 YearsPosition Summary:We are seeking an experienced Senior Data Engineer with 5 to 10 years of hands-on experience in building and managing data pipelines and big data infrastructure. The ideal candidate will have a strong technical background in data engineering tools,...


  • Singapore Careers@Gov Full time

    Senior Data Applications EngineerAs a Senior Data Applications Engineer at Careers@Gov, you will play a pivotal role in advancing technological innovation, particularly in managing and analysing data collected through distributed sensors and other channels.Leading industry and internal projects focused on innovative data application solutions, with a...

  • Senior Data Engineer

    2 months ago


    Singapore NEWPORT TANK CONTAINERS SINGAPORE PTE. LTD. Full time

    Roles & ResponsibilitiesOur CompanyGentco Logistics is a leading international liquid chemical logistics provider with three core business units: chemical tankers (5th largest fleet globally), chemical tank containers (2nd largest fleet globally), and terminal storage tanks. Our global headquarters are in Shanghai, China, supported by a regional headquarters...


  • Singapore Careers@Gov Full time

    Job Title: Senior Data Applications EngineerJob Summary:We are seeking a highly skilled Senior Data Applications Engineer to join our team at Careers@Gov. As a key member of our data engineering team, you will play a crucial role in advancing technological innovation, particularly in managing and analyzing data collected through distributed sensors and other...

  • Senior data engineer

    2 weeks ago


    Singapore Minden.ai Full time

    Who we are: minden.ai is a technology venture founded by Temasek in strategic partnership with DFI Retail Group and coalition partners Bread Talk Group, DBS Bank, PAssion Card, Mandai Wildlife Group, Singtel, Go Jek, Food Panda and Great Eastern. We are on a mission to redefine the engagement between brands and consumers in Southeast Asia. The way we work:...

  • Senior Data Engineer

    6 months ago


    Singapore MINDEN INTERNATIONAL PTE. LTD. Full time

    Roles & ResponsibilitiesWho we are:minden.ai is a technology venture founded by Temasek in strategic partnership with DFI Retail Group and coalition partners BreadTalk Group, DBS Bank, PAssion Card, Mandai Wildlife Group, Singtel, FoodPanda, GoJek and Great Eastern. We are on a mission to redefine the engagement between brands and consumers in Southeast...

  • Senior Data Engineer

    3 weeks ago


    Singapore ARROWPOINT INVESTMENT PARTNERS (SINGAPORE) PTE. LTD. Full time

    Roles & ResponsibilitiesAbout this roleWe are looking for a Senior Data Engineer who will be the first hire dedicated to data engineering within our multi-manager hedge fund. This is a unique opportunity for someone who thrives in a zero-to-one environment, where you’ll be responsible for building and scaling our data infrastructure from scratch. The ideal...


  • Singapore Careers@Gov Full time

    Job Title: Senior Data Applications EngineerJob Summary:We are seeking a highly skilled Senior Data Applications Engineer to join our team at Careers@Gov. As a key member of our organization, you will play a crucial role in advancing technological innovation, particularly in managing and analyzing data collected through distributed sensors and other...


  • Singapore NTT DATA Full time

    Job DescriptionMake an impact with NTT DATANavigate a company renowned for technical excellence and leading innovations, making a difference to our clients and society. Our workplace values diversity and inclusion – a place to grow, belong, and thrive.Your day at NTT DATAThe Senior Director of Design Management leads a team of Design Managers and oversees...


  • Singapore ABS CONNECT PTE. LTD. Full time

    About ABS Connect PTE. LTD.We are a cutting-edge technology firm seeking an experienced Senior Data Ingestion Engineer to join our dynamic team in Singapore. As a key member of our data platform, you will play a vital role in designing and implementing scalable data ingestion pipelines using Python.


  • Singapore Dell Full time

    Senior Analyst Data EngineeringIn the rapidly moving Artificial Intelligence era, few spaces are moving faster than the AI-enabled PC.  As a leading provider of world-class technology, this means bringing more intelligence into the PC ecosystem, enabling superior performance, enhanced productivity, and delightful experiences while maintaining privacy and...


  • Singapore Dell Full time

    Senior Analyst, Data Engineering (I6)In the rapidly moving Artificial Intelligence era, few spaces are moving faster than the AI-enabled PC.  As a leading provider of world-class technology, this means bringing more intelligence into the PC ecosystem, enabling superior performance, enhanced productivity, and delightful experiences while maintaining privacy...