Hiring for Big data Consultant

2 weeks ago


Singapore PATH INFOTECH PTE. LTD. Full time

We are hiring a Data Engineer at our company



- Write Spark code to Extract, Transform and Load data from different sources (such as CSV, JSON files, databases, and streaming like Kafka) into Hive tables in various layers

- Create complex queries using Joins, Sub Queries, Temporary Views, CTE, Map, Group Map in Hive

- Handle necessary aggregations, rank data using window functions, eliminate duplicates from source data with spark for banking reports

- Perform Join operations in spark for merging results from multiple source files to construct a data warehouse

- Implement data pivoting, unpivoting, exception handling, and logging

- Validate data quality using Python and Spark

- Extract parameters and variables into spark code from configuration files specific to the environment

- Enhance performance in hive through partitioning and bucketing

- Set up run parameters for spark job like memory settings and cluster resources availability

- Data Migration and reconciliation among different banking core systems

- Develop a solid framework to facilitate product enhancements in each development sprint

- Optimize performance and monitor job efficiency promptly


Skills:


  • JSON
  • MSSQL
  • Oracle 11g
  • Postgres
  • Database Administration
  • ETL
  • Logging
  • Data Migration
  • Performance Tuning
  • MSSQL Server
  • Data Visualization


  • Singapore PATH INFOTECH PTE. LTD. Full time

    Roles & Responsibilities· Write Spark code to Extract, Transform and Load data from heterogenous source (like CSV, JSON files, RDBMS and streaming viz Kafka) into Hive tables in different layers· Writing complex queries using Joins, Sub Queries, Temporary Views and CTE, Map, Group Map in Hive· Perform necessary aggregations, ranking data using window...


  • Singapore PATH INFOTECH PTE. LTD. Full time

    Write Spark code to Extract, Transform and Load data from heterogenous source (like CSV, JSON files, RDBMS and streaming viz Kafka) into Hive tables in different layers Writing complex queries using Joins, Sub Queries, Temporary Views and CTE, Map, Group Map in Hive Perform necessary aggregations, ranking data using window functions, removing duplicates from...

  • Big Data Engineer

    2 weeks ago


    Singapore HELIUS TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesPosition Type: Contract deputed to ClientRequired skillsMinimum 4 years of experience in Data Engineering Experience in Cloudera Data Platform Experience in computing, distributed, big data systems: Hadoop, HDFS, Nginx, Kubernetes, Docker, OpenStack, Spark, Flink Understands Linux operating system, storage, network IO and other...


  • Singapore Databricks Full time

    As a Resident Solutions Architect (RSA) in our Professional Services team ,you will work with clients on short to medium term customer engagements on their big data challenges using the Databricks platform. You will provide data engineering, data science, and cloud technology projects which require integrating with client systems, training, and other...

  • Big Data Developer

    2 weeks ago


    Singapore Pan Asia Group Resources Full time

    JD: Minimum 5 years of total IT experience; at least 4 years of experience in development with PySpark & Python; Good knowledge and experience on Big Data platform, Big Data Eco system tools, SPARK, Python, Etc. Experience with Banking domain is preferable. Strong understanding of SDLC processes including all the project documentations. Ability to work under...

  • Big Data Developer

    1 month ago


    Singapore Robert Walters Full time

    A global financial services company is seeking a Big Data Developer (12-month renewable contract). About the Big Data Developer (12-month renewable contract) Role: In this job, you will build data capabilities for multiple financial systems.Key Responsibilities: Study, evaluate and provide feedback on functional specifications. Provide detailed estimation...

  • Big Data Developer

    4 weeks ago


    Singapore Robert Walters Full time

    A global financial services company is seeking a Big Data Developer (12-month renewable contract). About the Big Data Developer (12-month renewable contract) Role: In this job, you will build data capabilities for multiple financial systems.Key Responsibilities: Study, evaluate and provide feedback on functional specifications. Provide detailed estimation...

  • Big Data Developer

    2 weeks ago


    Singapore Robert Walters Full time

    A global financial services company is seeking a Big Data Developer (12-month renewable contract). About the Big Data Developer (12-month renewable contract) Role: In this job, you will build data capabilities for multiple financial systems.Key Responsibilities: Study, evaluate and provide feedback on functional specifications. Provide detailed estimation...


  • Singapore PATH INFOTECH PTE. LTD. Full time

    About the Job:This job posting is for a Senior Solution Architect position where you will be responsible for migrating the current MOH analytics platform to a GCC (government commercial cloud) platform.Roles & Responsibilities:Analyze factors related to business problems and provide solutions on the current Hadoop and future AWS cloud platform.Understand...


  • Singapore Grab Full time

    Job Description:Life at Grab:At Grab, every Grabber is guided by The Grab Way, which spells out our mission, how we believe we can achieve it, and our operating principles - the 4Hs: Heart, Hunger, Honour and Humility. These principles guide and help us make decisions as we work to create economic empowerment for the people of Southeast Asia.Get to know our...


  • Singapore Quantexa Full time

    Big Data Graduate ProgrammeFounded in 2016 with only a handful of individuals, Quantexa was built with a purpose that through a greater understanding of context, better decisions can be made. 7 years, 15 locations and 700+ employees later we still believe that today. We connect the dots within our Customers data using dynamic entity resolution and advanced...

  • Senior Data Engineer

    4 weeks ago


    Singapore TANGSPAC CONSULTING PTE LTD Full time

    Roles & ResponsibilitiesWorking in a large Big data and datawarehouse team of a bank, you will be responsible for carrying out the following Understand functional requirements translate user specs to FSD Carrying out solution design Carrying out data analysis and mapping carrying out development of DW and Big data solutionsRequirements MIn 7 years of...

  • Big Data Engineer

    2 weeks ago


    Singapore D L RESOURCES PTE LTD Full time

    BIG DATA ENGINEER / DATA SCIENTIST, FCC ANALYTICSJob Description:Big Data EngineeringCreate, build and maintain the data infrastructure required for Extraction, Transformation and Loading of data from a wide variety of sources.Ensure high-quality data is delivered to support Compliance / FCC model development.Support the implementation of Data Analytics /...

  • Big Data Sre

    2 weeks ago


    Singapore COGNOSPHERE PTE. LTD. Full time

    Programming & Technology|Singapore|Full-timeJob descriptionWho are we?At HoYoverse, we are committed to creating immersive virtual world experiences for players around the world.In addition to game products such as Genshin Impact, Honkai Impact 3rd, Tears of Themis, and Honkai: Star Rail, HoYoverse also launched the dynamic desktop software N0va Desktop, the...

  • Big Data Engineer

    2 weeks ago


    Singapore NodeFlair Full time

    Job Summary:SalaryS$6,500 - S$13,000 / MonthlyJob TypeSeniorityJuniorYears of ExperienceAt least 1 yearTech StacksETL ClickHouse Presto Java Hive Spark Flink SQL Scala Hadoop Python**ResponsibilitiesAbout TikTokTikTok is the leading destination for short-form mobile video. Our mission is to inspire creativity and bring joy. TikTok has global offices...

  • Big Data Developer

    2 weeks ago


    Singapore Cognizant Full time

    Responsibility: Develop and maintain data pipelines implementing ETL processes in Bigdata environment Take responsibility for Hadoop development and implementation Experience in Python, Spark, PySpark, and Hive Understanding of data warehousing and data modeling techniques Knowledge on Data lake architectureRequirements: Bachelor's degree in Computer Science...


  • Singapore NodeFlair Full time

    Job Summary:SalaryS$6,500 - S$13,000 / MonthlyJob TypeSeniorityJuniorYears of ExperienceAt least 1 yearTech StacksETL ClickHouse Metastore Analytics R Flume Presto Hive Spark kafka Flink SQLResponsibilitiesTikTok will be prioritizing applicants who have a current right to work in Singapore, and do not require TikTok's sponsorship of a visa.TikTok is the...


  • Singapore TANGSPAC CONSULTING PTE LTD Full time

    Roles & ResponsibilitiesA large financial institution with large data team is looking for Application Development Manager / Lead with the following responsibilities : Managing an applicaiton development team of around 20+ members Managing the solutioning of Big Data and enterprise BI applications Building frameworks, best practices for the engineering...


  • Singapore TANGSPAC CONSULTING PTE LTD Full time

    Roles & ResponsibilitiesA large financial institution with large data team is looking for Application Development Manager / Lead with the following responsibilities : Managing an applicaiton development team of around 20+ members Managing the solutioning of Big Data and enterprise BI applications Building frameworks, best practices for the engineering...


  • Singapore TANGSPAC CONSULTING PTE LTD Full time

    Roles & ResponsibilitiesA well-established financial organization with a robust data team is in search of an Application Development Manager/Lead to take on the following key responsibilities:Overseeing an application development team consisting of over 20 membersSupervising the design of Big Data and enterprise BI applicationsEstablishing frameworks and top...