Big Data Platform Engineer

7 days ago


Singapore HELIUS TECHNOLOGIES PTE. LTD. Full time
Roles & Responsibilities

Big Data Platform Engineer

  • 6 years of experience in building or designing large-scale, fault-tolerant, distributed systems, (for example: data lakes, delta lakes, data meshes, data lake houses, data platforms, data streaming solutions…)
  • In-depth knowledge and experience in one or more large scale distributed technologies including but not limited to: Hadoop ecosystem, Kafka, Kubernetes, Spark
  • Migration experience of storage technologies (e.g. HDFS to S3 Object Storage)
  • Integration of streaming and file based data ingestion /consumption (Kafka, Control M, AWA)
  • Experience in DevOps, data pipeline development, and automation using Jenkins and Octopus (optional: Ansible, Chef, XL Release, and XL Deploy)
  • Expert in Python and Java or another static language like Scala/R, Linux/Unix scripting, Jinja templates, puppet scripts, firewall config rules setup
  • VM setup and scaling (pods), K8S scaling, managing Docker with Harbor, pushing Images through CI/CD
  • Experience using data formats such as Apache Parquet, ORC or Avro Experience in machine learning algorithms is a plus.
  • Hands-on experience in integrating Data Science Workbench platforms (e.g. Dataiku)
  • Cloud migration experience
  • Experience of agile project management and methods (e.g., Scrum, SAFe)
  • Knowledge of financial sector and its products is beneficial

Responsibilities:

  • Operating Global Data Platform components (VM Servers, Kubernetes, Kafka) and applications (Apache stack, Collibra, Dataiku and similar).
  • Implement automation of infrastructure, security components, and Continuous Integration & Continuous Delivery for optimal execution of data pipelines (ELT/ETL).
  • Develop solutions to build resiliency in data pipelines with platform health checks, monitoring, and alerting mechanisms, quality, timeliness, recency, and accuracy of data delivery are improved
  • Apply DevSecOps & Agile approaches to deliver the holistic and integrated solution in iterative increments.
  • Liaison and collaborate with enterprise security, digital engineering, and cloud operations to gain consensus on architecture solution frameworks.
  • Review system issues, incidents, and alerts to identify root causes and continuously implement features to improve platform performance.
  • Be current on the latest industry developments and technology trends to effectively lead and design new features/capabilities.

Thanks, and Best Regards

Karanam Vijaya Kiran

(EA Registration no: R1443178)

Recruitment Manager

Hand Phone: +65 92333815

Helius Technologies Pte Ltd (EA Licence No: 11C3373)

Tell employers what skills you have

Puppet
Kubernetes
Big Data
DevOps
Pipelines
Cloud Migration
Hadoop
VM
Agile
Scripting
HDFS
Distributed Systems
Python
Continuous Integration
Docker
Data Science
Ansible
Java
S3
  • Big Data Engineer

    2 weeks ago


    Singapore INFINITE COMPUTER SOLUTIONS PTE LTD Full time

    Roles & ResponsibilitiesWe are seeking a highly skilled and experienced Big Data Engineer to join our team. The ideal candidate will have a minimum of 9 years of experience managing data engineering jobs in big data environment e.g., Cloudera Data Platform. The successful candidate will be responsible for designing, building, maintaining and supporting big...

  • Big Data Engineer

    2 weeks ago


    Singapore INFINITE COMPUTER SOLUTIONS PTE LTD Full time

    Roles & ResponsibilitiesWe are seeking a highly skilled and experienced Big Data Engineer to join our team. The ideal candidate will have a minimum of 9 years of experience managing data engineering jobs in big data environment e.g., Cloudera Data Platform. The successful candidate will be responsible for designing, building, maintaining and supporting big...

  • Big Data

    7 days ago


    Singapore EVO OUTSOURCING SOLUTIONS PTE. LTD. Full time

    Roles & ResponsibilitiesResponsibilities:Collaborate with cross-functional teams to understand business requirements and translate them into scalable and efficient big data platform solutions. Design and architect robust big data platforms, ensuring high performance, reliability, and scalability. Implement end-to-end data processing workflows, from data...

  • Big Data Engineer

    3 weeks ago


    Singapore ITCAN PTE. LIMITED Full time

    Roles & Responsibilities· Bachelor's or master's degree in computer science, Engineering, or a related field.· Proven experience as a Big Data Engineer or similar role.· Strong proficiency in Apache Spark and Scala programming.· Experience with big data technologies such as Hadoop, Hive, Kafka, etc.· Familiarity with data warehousing concepts and...

  • Big Data Engineer

    1 week ago


    Singapore ITCAN PTE. LIMITED Full time

    Roles & Responsibilities· Bachelor's or master's degree in computer science, Engineering, or a related field.· Proven experience as a Big Data Engineer or similar role.· Strong proficiency in Apache Spark and Scala programming.· Experience with big data technologies such as Hadoop, Hive, Kafka, etc.· Familiarity with data warehousing concepts and...

  • Big Data Engineer

    4 weeks ago


    Singapore ITCAN PTE. LIMITED Full time

    Roles & Responsibilities· Bachelor’s or master’s degree in computer science, Engineering, or a related field.· Proven experience as a Big Data Engineer or similar role.· Strong proficiency in Apache Spark and Scala programming.· Experience with big data technologies such as Hadoop, Hive, Kafka, etc.· Familiarity with data warehousing concepts and...

  • Big Data Engineer

    4 weeks ago


    Singapore ITCAN PTE. LIMITED Full time

    Roles & Responsibilities· Bachelor’s or master’s degree in computer science, Engineering, or a related field.· Proven experience as a Big Data Engineer or similar role.· Strong proficiency in Apache Spark and Scala programming.· Experience with big data technologies such as Hadoop, Hive, Kafka, etc.· Familiarity with data warehousing concepts and...

  • Big Data Engineer

    2 weeks ago


    Singapore DADACONSULTANTS PTE. LTD. Full time

    Roles & ResponsibilitiesMy ClientIt is a global cryptocurrency exchange dedicated to providing secure, efficient, and innovative digital asset trading services. The platform offers a wide range of services, including spot trading, derivatives trading, and wealth management. They are looking for experienced and motivated Mobile app developer to join the...

  • Big Data Engineer

    2 weeks ago


    Singapore DADACONSULTANTS PTE. LTD. Full time

    Roles & ResponsibilitiesMy ClientIt is a global cryptocurrency exchange dedicated to providing secure, efficient, and innovative digital asset trading services. The platform offers a wide range of services, including spot trading, derivatives trading, and wealth management. They are looking for experienced and motivated Mobile app developer to join the...

  • Big Data Engineer

    7 days ago


    Singapore DADACONSULTANTS PTE. LTD. Full time

    Roles & ResponsibilitiesKey Responsibilities: Ensure seamless deployment, configuration, and operation of applications on cloud servers and container platforms. Collaborate with development teams to design scalable and highly available system architectures. Contribute to the development and maintenance of internal platforms and tools. Perform daily...

  • Big Data Engineer

    4 weeks ago


    Singapore MICHAEL PAGE INTERNATIONAL PTE LTD Full time

    Roles & Responsibilities Organisation is a global leader in the technology and telecoms industry. Fast-paced environment, rapidly growing organisation.About Our ClientThe organisation is a global leader in the technology and telecoms industry. They are well-known for their innovative approach to tackling complex technical challenges and their commitment to...

  • Big Data Engineer

    2 weeks ago


    Singapore MICHAEL PAGE INTERNATIONAL PTE LTD Full time

    Roles & ResponsibilitiesOrganisation is a global leader in the technology and telecoms industry. Fast-paced environment, rapidly growing organisation.About Our ClientThe organisation is a global leader in the technology and telecoms industry. They are well-known for their innovative approach to tackling complex technical challenges and their commitment to...

  • Big Data Engineer

    2 weeks ago


    Singapore APAR TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesAre you passionate about big data and love working with cutting-edge technologies? Join our team as a Big Data Engineer and help us build scalable data solutions What You'll Do: Design, develop, and optimize big data pipelines Work with Apache Spark, Scala, Hadoop, Hive, Kafka, and more Manage and process large datasets efficiently...


  • Singapore PEOPLEBANK SINGAPORE PTE. LTD. Full time

    Roles & ResponsibilitiesData Platform Engineer12 months contract Work location: CentralThe OpportunityPerform operation, installation, and monitoring of big data platforms Ensure the reliability and normal operation of multiple core systems for big data and online computing, resolving issues within the agreed SLA Evaluate infrastructure requirements for...


  • Singapore PEOPLEBANK SINGAPORE PTE. LTD. Full time

    Roles & ResponsibilitiesData Platform Engineer12 months contract Work location: CentralThe OpportunityPerform operation, installation, and monitoring of big data platforms Ensure the reliability and normal operation of multiple core systems for big data and online computing, resolving issues within the agreed SLA Evaluate infrastructure requirements for...


  • Singapore MANPOWER STAFFING SERVICES (SINGAPORE) PTE LTD Full time

    Roles & ResponsibilitiesJob scopes· Design, build, deployment, of ETL/ELT data pipelines within big data ecosystems ( Impala, Hue, Spark, Talend and Cloudera)· Develop technical design, and ETL specification documents· Manage, monitor and fine tune Hadoop cluster architecture (HDFS) jobs for performance, security and resource management.· Create detailed...

  • Big Data Engineer

    2 weeks ago


    Singapore APAR TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesAre you passionate about big data and love working with cutting-edge technologies? Join our team as a Big Data Engineer and help us build scalable data solutions! What You’ll Do:✔ Design, develop, and optimize big data pipelines✔ Work with Apache Spark, Scala, Hadoop, Hive, Kafka, and more✔ Manage and process large datasets...

  • Big Data Engineer

    4 weeks ago


    Singapore MINDMAP TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesResponsibilities· Perform the role of a Data engineer on the Enterprise Data Architecture and Governance (EDAG) Programs, to ensure the business requirements and deliverables are aligned to business priorities· Design develop data pipelines on EDAG platform to support Wholesale bank data model.· Perform assessments and analysis on...

  • Big Data Engineer

    2 weeks ago


    Singapore MINDMAP TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesResponsibilities· Perform the role of a Data engineer on the Enterprise Data Architecture and Governance (EDAG) Programs, to ensure the business requirements and deliverables are aligned to business priorities· Design develop data pipelines on EDAG platform to support Wholesale bank data model.· Perform assessments and analysis on...

  • Big Data Engineer

    2 weeks ago


    Singapore BYTEDANCE PTE. LTD. Full time

    Roles & ResponsibilitiesAbout ByteDanceFounded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create...