Data Engineer(spark,scala And Elastic Search)

1 month ago

Singapore search index Full time

We are seeking a talented and experienced Spark Scala Developer with expertise in Elasticsearch to join our team. As a Spark Scala Developer, you will play a crucial role in designing, developing, and optimizing big data solutions using Apache Spark, Scala, and Elasticsearch. You will collaborate with cross-functional teams to build scalable and efficient data processing pipelines and search applications. Knowledge and experience in the Compliance / AML domain will be a plus.

Responsibilities: Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation Requirements Bachelor\'s or Master\'s degree in Computer Science, Software Engineering, or a related field Min 5+ Years experience in developing Spark & Scala applications. Experience with Spark Streaming is a plus. Proficiency in Scala programming language and familiarity with functional programming concepts In-depth understanding of Apache Spark architecture, RDDs, Data Frames, and Spark SQL Experience integrating and working with Elasticsearch for data indexing and search applications. Solid understanding of Elasticsearch data modeling, indexing strategies, and query optimization Experience with distributed computing, parallel processing, and working with large datasets Familiarity with big data technologies such as Hadoop, Hive, and HDFS Proficient in performance tuning and optimization techniques for Spark applications and Elasticsearch queries Strong problem-solving and analytical skills with the ability to debug and resolve complex issues. Familiarity with version control systems (e.g., Git) and collaborative development workflows -

Additional Information

Career Level

Not Specified

Qualification

Not Specified

Job Specializations

,

Company Overview

Headquartered in Singapore, Unison Consulting was founded in September 2012 with the core motive to provide cutting-edge solutions and services in a variety of areas. At Unison Consulting, our mission is to empower businesses with our financial and insurance services.

Our consulting services help our clients implement risk management, risk analytics system, systems integration, business intelligence, information management, and custom development of IT solutions. We surpass all standards of quality for financial, insurance institutions and other verticals.

Our team views our clients as our partners and derives its success through the success of our clients. We are highly focused on exceeding our clients\' expectations and offering them the best possible solutions for their businesses.

Unison Consulting has a team of young, energetic, imaginative, and innovative individuals who never hold back from facing their challenges head-on. They push all boundaries to help our clients and exceed their expectations. We help our team to upscale their knowledge as required in the market, thereby to increase their personal and professional capabilities.

Additional Company Information

Registration No.

C

Industry

Computer / Information Technology (Software)

Be careful

This job ad has not been subjected to our hirer verification process. Proceed cautiously and do your own checks before providing any personal information. Search Index

  • Data Engineer

    3 weeks ago

    Singapore Tangspac APAC Full time

    Client Description:Job Description:Position: Data Engineer (Spark Scala Elastic)Employment Type: 1-year Agency Contract (Highly Renewable)Location: AlexandraJob ObjectivesWe are seeking a talented and experienced Spark Scala Developer with expertise in Elasticsearch to join our team. As a Spark Scala Developer, you will play a crucial role in designing,...

  • Senior Ecosystem Solution Architect

    2 weeks ago

    Singapore Elastic Full time

    Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays,...

  • Snr Manager, Solutions Architecture

    3 weeks ago

    Singapore Elastic Full time

    Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays,...

  • Customer Architect

    2 weeks ago

    Singapore Elastic Full time

    Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays,...

  • Rvp, Sales

    3 weeks ago

    Singapore Elastic Full time

    Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays,...

  • Partner Manager, Singapore

    3 weeks ago

    Singapore Elastic Full time

    Elastic is a free and open search company that powers enterprise search, observability, and security solutions built on one technology stack that can be deployed anywhere. From finding documents to monitoring infrastructure to hunting for threats, Elastic makes data usable in real-time and at scale. Thousands of organizations worldwide, including Barclays,...

  • Data Engineer

    3 weeks ago

    Singapore Krisvconsulting Services Pte Ltd Full time

    Key Responsibilities:Create Spark Scala/PySpark jobs for data transformation and aggregation.Produce unit tests for Spark transformations and helper methods.Use Spark and Spark-SQL to read parquet data and create tables in Hive using the Scala API.Work closely with Business Analysts to review test results and obtain sign-off.Prepare necessary design and...

  • Data Engineer

    3 weeks ago

    Singapore ELLIOTT MOSS CONSULTING PTE. LTD. Full time

    Roles & Responsibilities Create Spark Scala/Pyspark jobs for data transformation and aggregation. Produce unit tests for Spark transformations and helper methods. Use Spark and Spark-SQL to read the parquet data and create the tables in hive using the Scala API. Work closely with Business Analysts team to review the test results and obtain sign off Prepare...

  • Data Engineer

    3 weeks ago

    Singapore JONDAVIDSON PTE. LTD. Full time

    Job Title: Data Engineer (Python, Java or Scala)Welcome to our company! We are currently looking for a skilled Data Engineer to join our team. In this role, you will play a key part in designing, constructing, and managing data pipelines to support our data analytics and business intelligence projects. Collaboration with various teams is essential to create...

  • Data Engineer

    3 weeks ago

    Singapore Synpulse8 Full time

    Data EngineerAre you excited about transforming the way we bank by using the latest technology? We are tooAbout the job: Work with clients to solve business problems in fraud, compliance and financial crime and present project results Manage, transform, and cleanse high volume data Automate data processing to enable ongoing alerts on highrisk activity Work...

  • Data Security Analyst

    3 weeks ago

    Singapore UNISON CONSULTING PTE. LTD. Full time

    Build, deploy, and manage Data security products to secure and implement governance policies on the data assets in the organisation.Integrate security frameworks on modern high-performance analytics database and computation engines like Spark, Presto, Kafka, Elastic, and other technologies in our stackWorking with engineering teams to explore and create new...

  • Director - Development Manager (Big data)

    3 weeks ago

    Singapore TANGSPAC CONSULTING PTE LTD Full time

    Roles & ResponsibilitiesA well-established financial organization with a robust data team is in search of an Application Development Manager/Lead to take on the following key responsibilities:Overseeing an application development team consisting of over 20 membersSupervising the design of Big Data and enterprise BI applicationsEstablishing frameworks and top...

  • Elastic Engineer

    2 weeks ago

    Singapore Quess Corp Limited Full time

    Job Information:IndustryBanking***CitySingapore***State/ProvinceSingapore***CountrySingapore***Zip/Postal Code529542***- Upgrade a log analytics and infrastructure monitoring solution using the Elasticsearch. Deploy, configure, test, troubleshoot, maintain, and upgrade Elastic clusters, agents, nodes, and environments. Plan resources and continuously...

  • Senior Data Engineer

    2 weeks ago

    Singapore Claremont Consulting Full time

    Work type: Location:Singapore Discipline: Contact name: Posted date:25-Jul-2023Senior Data Engineer-Essential Skills:Strong experience in Data Engineering building strong, robust pipelines to ingest data from a variety of sources with a focus on real-time streaming Strong Spark / Spark Streaming experience Experience in Java, Kafka and Streaming...

  • Data Engineer

    3 weeks ago

    Singapore NodeFlair Full time

    Job Summary:SalaryS$5,000 - S$10,000 / MonthlyJob TypeSeniorityMidYears of ExperienceAt least 3 yearsTech StacksAWS Qliksense Analytics Glue RedShift Java Hive Spark SQL Scala Hadoop PythonWe believe in positively impacting the lives of people around us and the environment we live in through the use of technology. Hence, we are committed to providing a...

  • Senior Engineer, Data Engineer

    3 weeks ago

    Singapore NodeFlair Full time

    Job Summary:SalaryS$6,000 - S$10,000 / MonthlyJob TypeSenioritySeniorYears of ExperienceAt least 2 yearsTech StacksDruid Java Hive Spark Linux kafka Flink Scala Hadoop PythonResponsible for the collection, analysis and mining of massive data in the game business and building an intelligent game system. Collaborate with the business team for game data...

  • Senior Data Platform Engineer

    3 weeks ago

    Singapore NodeFlair Full time

    Job Summary:SalaryS$8,33- S$16,666 / Monthly ESTJob TypePermanentSenioritySeniorYears of ExperienceAt least 5 yearsTech StacksSnowflake Spring BigQuery Java Spark kafka Flink Springboot Kotlin SQL Scala Hadoop Python In this data engineering role, we are seeking engineers with a background in data engineering to join us in various aspects of data...

  • Data Engineer Intern

    3 weeks ago

    Singapore Shopee Full time

    DepartmentEngineering and Technology- LevelInternship- LocationSingaporeThe Engineering and Technology team is at the core of the Shopee platform development. The team is made up of a group of passionate engineers from all over the world, striving to build the best systems with the most suitable technologies. Our engineers do not merely solve problems at...

  • IT Engineer

    2 weeks ago

    Singapore CUTECH QUALITY SOLUTIONS PTE. LTD. Full time

    Roles & ResponsibilitiesResponsibilities:Lead big Data Design and Development Activities Design, Development and Maintenance of Big Data MIS Application while working in an Agile team. Work with system analysts/business analysts to understand business/functional requirements and implement those in Java Spark/Scala Spark Identify and implement non-functional...

  • Backend Engineer, Technology Platform

    3 weeks ago

    Singapore NodeFlair Full time

    Job Summary:SalaryS$8,20- S$15,042 / Monthly ESTJob TypePermanentSeniorityMidYears of ExperienceAt least 3 yearsTech StacksElastic ELK Logstash RabbitMQ Kibana Java NoSQL kafka Kotlin Scala Cassandra Elasticsearch Redis Python In this fastchanging digital world that is increasingly becoming borderless, our mission is to empower businesses of all sizes to...