Hadoop Data Engineer

18 hours ago

Singapore CONSULGURU PTE. LTD. Full time $120,000 - $180,000 per year

We are looking for an experienced and highly skilled Hadoop Data Engineer to join our dynamic team. The ideal candidate will have hands-on expertise in developing optimized data pipelines using Python, PySpark, Scala, Spark-SQL, Hive , and other big data technologies. You will be responsible for translating complex business and technical requirements into efficient data pipelines and ensuring high-quality code delivery through collaboration and code reviews.

Roles & Responsibilities:

Data Transformation & Pipeline Development:

  • Design and implement optimized data pipelines using PySpark, Python, Scala, and Spark-SQL .
  • Build complex data transformation logic and ensure data ingestion from source systems to Data Lakes (Hive, HBase, Parquet).
  • Produce unit tests for Spark transformations and helper methods.

Collaboration & Communication:

  • Work closely with Business Analysts to review test results and obtain sign-offs.
  • Prepare comprehensive design and operational documentation for future reference.

Code Quality & Review:

  • Conduct peer code reviews and act as a gatekeeper for quality checks.
  • Ensure quality and efficiency in the delivery of code through pair programming and collaboration.

Production Deployment:

  • Ensure smooth production deployments and perform post-deployment verification.

Technical Expertise:

  • Provide hands-on coding and support in a highly collaborative environment .
  • Contribute to development, automation, and continuous improvement practices.

System Knowledge:

  • Strong understanding of data structures, data manipulation, distributed processing , and application development .
  • Exposure to technologies like Kafka , Spark Streaming , and ML is a plus.

RDBMS & Database Management:

  • Hands-on experience with RDBMS technologies (MariaDB, SQL Server, MySQL, Oracle).
  • Knowledge of PLSQL and stored procedures is an added advantage.

Other Responsibilities:

  • Exposure to TWS jobs for scheduling.
  • Knowledge and experience in Hadoop tech stack , Cloudera Distribution , and CI/CD pipelines using Git, Jenkins .
  • Experience with Agile Methodologies and DevOps practices.

Technical Requirements:

  • Experience: 6-9.5 years of experience in Hadoop , Spark , PySpark , Scala , Hive , Spark-SQL , Python , Impala , CI/CD , and Git .
  • Strong understanding of Data Warehousing Methodology and Change Data Capture (CDC).
  • In-depth knowledge of Hadoop & Spark ecosystems with hands-on experience in PySpark and Hadoop technologies.
  • Proficiency in working with RDBMS such as MariaDB , SQL Server , MySQL , or Oracle .
  • Experience with stored procedures and TWS job scheduling .
  • Solid experience with Enterprise Data Architectures and Data Models .
  • Background in Core Banking or Finance domains is preferred; experience in AML (Anti-Money Laundering) domain is a plus.

Skills & Qualifications:

  • Strong hands-on coding skills in Python , PySpark , Scala , Spark-SQL .
  • Proficient in Hadoop ecosystem (Hive, HBase, etc.).
  • Knowledge of CI/CD , Agile , and DevOps methodologies.
  • Good understanding of data integration , data pipelines , and distributed data systems .
  • Experience with Oracle , PLSQL , and working with large-scale databases.
  • Strong analytical and problem-solving skills, with an ability to troubleshoot complex data issues.

  • Hadoop Operations Engineer

    1 day ago

    Singapore ENSIGN INFOSECURITY (CYBERSECURITY) PTE. LTD. Full time

    **Key Responsibilities** - As part of the security operation teams, operate, support and maintain a Hadoop Date Lake environment in a dedicated customer environment - Perform data engineering to extract, transform, load for data science analysis - Work with customer’s and internal data science team to design effective data models for optimal storage and...

  • DevOps Engineer

    6 days ago

    Singapore NTT SINGAPORE PTE. LTD. Full time

    DevOps Engineer (Big Data / Hadoop) – Contract Employment Type: Contract (12 months, renewable)Location: Central, Singapore Experience Required: 4–7 years Salary Range: Based on Experience Interested candidates are kindly requested to email their CV with their experience to Our client is a leading financial services and insurance group headquartered in...

  • Hadoop Engineer

    2 weeks ago

    Singapore ENSIGN INFOSECURITY (CYBERSECURITY) PTE. LTD. Full time

    **Job Description Summary The incumbent is responsible for delivering information security projects that are related to big data solutions, including the execution, maintenance, support and operation of all project activities. This competency tower’s cybersecurity includes NiFi, HDFS, Cloudera, Anaconda etc **Key Responsibilities** - Carry out...

  • Hadoop Developer

    7 days ago

    Singapore Power IT Services private limited Full time

    1. Enables different levels of Hadoop security at ecosystem level. 2. Perform POCs of new capabilities in Hadoop Platform. 3. Monitor Hadoop cluster connectivity and security. 4. Enhance data at rest and data at motion security. 5. Manage and review Hadoop log files and adhere to retention policy. 6. Perform backup and recovery tasks. 7. Implement and manage...

  • Hadoop Testing

    7 days ago

    Singapore SINGSOFT SOLUTIONS PTE. LIMITED Full time

    4 years exp **Roles & Responsibilities** - Build Hadoop/Big Data test methodologies high volume/frequency data testing, prepare Hadoop Test Plan, design Hadoop Test Cases, test Scenarios & test scripts - Prepare test data for test cases, execute Hadoop Test Cases. - Test workflow processes, detect bugs through the queries, finding bad data. - Automate...

  • Hadoop Security

    6 days ago

    Singapore ANTAS PTE. LTD. Full time $80,000 - $120,000 per year

    We are seeking a skilled Hadoop Security & Compliance Engineer to design and maintain secure, compliant, and high-performing enterprise data platforms. The role involves implementing and managing Hadoop ecosystem security , ensuring data governance , and supporting regulatory compliance across Cloudera CDP/CDH environments.Key Responsibilities:Configure and...

  • Hadoop Administrator

    7 days ago

    North-East Singapore Unison Consulting Pte Ltd Full time

    Plan, install, configure, and deploy Hadoop clusters based on business requirements and workload demands. - Monitor and manage the health of Hadoop clusters, addressing performance issues and ensuring optimal resource utilization. - Implement and maintain security protocols within the Hadoop ecosystem, including authentication, authorization, and...

  • Hadoop Administrator

    1 day ago

    Singapore ANTAS PTE. LTD. Full time

    Key Responsibilities: - Install, configure, and maintain Hadoop clusters (HDFS, YARN, Hive, Spark, Kafka, Zookeeper). - Perform capacity planning, performance tuning, upgrades, and patch management. - Manage user access, Kerberos authentication, Ranger/Sentry policies, and data security controls. - Monitor system health and troubleshoot performance or...

  • Database and Hadoop Admin

    7 days ago

    Central Singapore SKY & F PTE. LTD. Full time

    **Location** Singapore, Central Singapore **Job Type** Full Time **Salary** $4,000 - $7,500 Per Month **Date Posted** 4 hours ago Additional Details **Job ID** 133918 **Job Views** 1 **Job Description**: Roles & Responsibilities - Develop processes for optimizing database security. Manage database access. - Performance tuning of database...

  • Hadoop Administrator

    6 days ago

    Singapore ANTAS PTE. LTD. Full time $80,000 - $120,000 per year

    We are looking for an experienced Hadoop Administrator to manage, optimize, and support enterprise-level Big Data platforms running on Cloudera or Hortonworks distributions. The ideal candidate will ensure platform stability, performance, and security across complex data ecosystems.Key Responsibilities:Install, configure, and maintain Hadoop clusters (HDFS,...