Data Engineer(Spark,scala and Elastic Search)

6 days ago

Singapore Unison Consulting Pte Ltd Full time

**Responsibilities**:

- Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
- Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
- Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
- Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
- Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
- Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation

**Requirements**:

- Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field
- Proficiency in Scala programming language and familiarity with functional programming concepts
- In-depth understanding of Apache Spark architecture, RDDs, Data Frames, and Spark SQL
- Solid understanding of Elasticsearch data modeling, indexing strategies, and query optimization
- Experience with distributed computing, parallel processing, and working with large datasets
- Familiarity with big data technologies such as Hadoop, Hive, and HDFS
- Strong problem-solving and analytical skills with the ability to debug and resolve complex issues.
- Familiarity with version control systems (e.g., Git) and collaborative development workflows

  • Principal Solutions Architect, Public Sector

    2 weeks ago

    Singapore Elastic Full time

    Overview Senior Solutions Architect, Public Sector Join to apply for the Senior Solutions Architect, Public Sector role at Elastic . Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more...

  • Data Engineer

    7 days ago

    Singapore DYNAMIC HUMAN CAPITAL PTE. LTD. Full time

    **Job Descripion **Primary Skill** - Experience in scala / Java. - SPARK experience. - Experience in Kubernetes.( option) - at least a minimum of 3 years of experience. Staff Data Engineer to define, implement and run the Data platforms and analytics for an online banking project. **What will you be doing**: - Choosing the right technologies for our use...

  • Principal Solutions Architect

    6 days ago

    Singapore Elastic Full time

    Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays,...

  • SDR Manager, ASEAN

    4 hours ago

    Singapore Elastic Full time

    SDR Manager, ASEAN Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to...

  • Account Executive, Security Sales

    6 days ago

    Singapore Elastic Full time

    Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to accelerate the...

  • Principal Solutions Architect, Public Sector

    2 weeks ago

    Singapore Elastic Full time $120,000 - $200,000 per year

    Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to accelerate the...

  • Sdr Manager, Asean

    7 days ago

    Singapore Elastic Full time

    Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to accelerate the...

  • Senior Solutions Architect, Public Sector

    3 days ago

    Singapore Elastic Full time $120,000 - $200,000 per year

    Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to accelerate the...

  • Developer (Big Data-framework Engineer)

    6 days ago

    Singapore NodeFlair Full time

    **Job Summary**: **Salary** S$10,000 - S$13,000 / Monthly **Job Type** **Seniority** Senior **Years of Experience** At least 10 years **Tech Stacks** Experian Greenplum Struts Docker Jenkins Go Netezza Shell Script Spring Teradata Sqoop Oozie Elastic CI Kudu Cloudera Sas Flume HBase Shell OOP Control-M Apache SonarQube Logstash Groovy Presto Kibana Java...

  • Software Developer

    7 days ago

    Singapore QUESSCORP HOLDINGS PTE. LTD. Full time

    **Responsibilities**: - Responsible for Hadoop development and implementation for data ingestion on Hadoop technologies - Required to have sound knowledge and implementation experience to build Data warehouse / Data Mart using structured / unstructured data for both near real-time & batch processing - Hands on experience in complex code development and...