Senior Data Engineer

2 weeks ago

Singapore OPENSOURCE PTE. LTD. Full time

**Job Title**:Senior Data Engineer (5-10 Years of Experience)

**Position Summary**:
**Responsibilities**:

- Design, build, and maintain scalable and reliable data pipelines to support real-time and batch processing requirements.
- Manage, monitor, and optimize workflows in **Apache Airflow **to ensure data quality and integrity.
- Develop and integrate data flows using **Apache NiFi **for seamless data ingestion and transformation.
- Work extensively with **Apache Kafka **for data streaming and messaging across various data sources.
- Implement data processing solutions using **PySpark **and **Spark Scala **to handle large-scale datasets and complex transformations.
- Write efficient code in **Python **and **Java **to automate data workflows and support data engineering needs.
- Utilize **shell scripting **for operational tasks and automation within the data environment.
- Collaborate with cross-functional teams to understand data requirements and provide optimized data engineering solutions.
- Ensure data security, compliance, and performance by following best practices in big data and distributed systems.
- Continuously improve the performance, scalability, and reliability of data processing pipelines.

**Required Skills and Experience**:

- **Apache Airflow**:Extensive experience in managing, scheduling, and monitoring data pipelines.
- **Apache NiFi**:Strong experience in designing data flows for data ingestion and transformation.
- **Apache Kafka**:In-depth knowledge of Kafka for real-time data streaming and messaging systems.
- **PySpark & Spark Scala**:Proficiency in using PySpark and Spark Scala for large-scale data processing.
- **Programming Languages**:Strong experience with Python and Java, with additional knowledge in shell scripting.
- **Big Data Knowledge**:Familiarity with big data ecosystems and distributed data processing.
- Ability to work independently and collaboratively in a fast-paced environment, handling complex data engineering challenges.

**Educational Qualifications**:

- Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, or a related field.

**Preferred Qualifications**:

- Experience in cloud environments (AWS, GCP, Azure) with big data components is a plus.
- Experience with version control tools (e.g., Git) and CI/CD practices in data engineering.
- Strong analytical, problem-solving, and communication skills.

**Employment Type**:Full-Time

  • Senior Data Scientist

    3 days ago

    Singapore NTT DATA Full time $120,000 - $180,000 per year

    Make an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.Your day at NTT DATAThe Senior Data...

  • Azure Data engineer

    1 week ago

    Singapore NTT Data Singapore Full time

    Job Responsibilities: Data Analysis and Interpretation: Perform in-depth analysis of structured and unstructured data sets to identify trends, patterns, and valuable insights that drive business decisions and strategies. Data Visualization and Reporting: Develop intuitive and visually appealing dashboards, reports, and presentations to communicate complex...

  • Azure Data engineer

    2 weeks ago

    Singapore NTT Data Singapore Full time $60,000 - $180,000 per year

    Job Responsibilities:Data Analysis and Interpretation: Perform in-depth analysis of structured and unstructured data sets to identify trends, patterns, and valuable insights that drive business decisions and strategies. Data Visualization and Reporting: Develop intuitive and visually appealing dashboards, reports, and presentations to communicate complex...

  • Senior Data Scientist

    6 days ago

    Singapore Data Analytics Full time

    Get to Know the Team Our Search and Personalisation Team focuses on advancing scalable machine learning and experimentation frameworks that power search relevance, ranking, and recommendations across Grab. By leveraging cutting-edge analytics and statistical methods, we continuously optimize how users connect with merchants, services, and content. Get to...

  • Senior Network engineer(Arista/Data Center)

    18 hours ago

    Singapore NTT Data Singapore Full time $120,000 - $200,000 per year

    The Implementation Engineer stands between the project manager, design engineer and operations. They support initiatives as they transition between design, implementation, and operations. They are active in all phases as an SME to support, solve program and facilitates decision making. The goal is to help the orderly and timely execution of projects and...

  • Data Center Project Engineer

    20 hours ago

    Singapore NTT DATA Full time $80,000 - $120,000 per year

    Make an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.Your day at NTT DATAThe Data Centre...

  • Senior Hyperscale Solutions

    1 week ago

    Singapore NTT DATA, Inc. Full time

    Senior Hyperscale Solutions & Sales Engineer, APAC Join to apply for the Senior Hyperscale Solutions & Sales Engineer, APAC role at NTT DATA, Inc. Overview The APAC Region Senior Hyperscale Solutions & Sales Engineer provides technical support to APAC Region Sales Engineering activities related to hyperscale clients and is responsible for developing and...

  • Senior Mechanical Engineer

    1 day ago

    Singapore Vantage Data Centers Full time

    About Vantage Data Centers Vantage is committed to being a workplace of inclusion, equity, respect and acceptance. We celebrate diversity and intentionally seek out opportunities to learn from one another’s experience. Vantage Data Centers powers, cools, protects and connects the technology of the world’s well-known hyperscalers, cloud providers and...

  • Data Centre Delivery Services Engineer

    19 hours ago

    Singapore NTT Data Singapore Full time $120,000 - $180,000 per year

    ResponsibilitiesRead, interpret, and validate data centre floor plans and technical drawingsDesign and plan DC hall layouts for newly allocated spaces, including rack positioning, structured cabling, power, and cooling designReview and assess vendor-submitted data centre plans, evaluate proposed technologies, and clarify design...

  • Data Engineer

    1 week ago

    Singapore NTT Data Singapore Full time

    **Competency Required**: - Possess a degree in Computer Science/Information Technology or related fields. - At least 3 years of experience in a role focusing on development and support of data ingestion pipelines. - **Experience with building on data platforms, e.g. Snowflake.**: - **Proficient in SQL and Python.**: - Experience in continuous integration...