Data Engineer

3 weeks ago

Singapore ANTAS PTE. LTD. Full time
Roles & Responsibilities

POSITION OVERVIEW : Software Development Analyst

POSITION GENERAL DUTIES AND TASKS :

Job Description: We are seeking a skilled Data Engineer to do the migration of our existing data warehouse and data model from MariaDB to the Hadoop Big Data Cloudera on-premise platform. The ideal candidate must be proficient in SQL, Hive SQL, Spark, and data modeling. Additionally, they should possess a strong understanding of the production deployment process, including design, development, testing, UAT, and production deployment. Experience of scheduling jobs using Autosys is essential for this role.

Key Responsibilities

• Migrate existing data warehouse and data model from MariaDB to Hadoop Big Data Cloudera on-premises platform.
•Develop and optimize SQL, Hive SQL, and Spark scripts to ensure efficient data processing.
•Design and implement data models to support business requirements and optimize performance.
•Collaborate with cross-functional teams to understand data requirements and ensure data integrity throughout the migration process.

•Develop and execute test plans to validate data accuracy and system performance.

•Coordinate with stakeholders (internal) to plan and execute production deployments.

•Schedule and monitor jobs using Autosys to ensure timely execution and minimize downtime.

•Provide technical expertise and guidance to team members throughout the migration project.

•Document processes, procedures, technical specifications, and best practices to facilitate knowledge sharing and ensure scalability.

•Create and maintain unit test case documents to ensure code quality and reliability.

Qualifications

• Bachelor's (or Higher) degree in computer science, Engineering, or a related field.
•Proven experience of 5+ years in data engineering and migration projects for big data (Hortonworks / Cloudera)
•Strong hands-on experience on Cloudera and related ecosystem components.
•Strong experience in implementing ETL (Extract, Transform, Load) processes.

•Highly proficient in SQL, Hive SQL, Spark, and data modeling.

•Strong understanding of the production deployment process.

•Experience with scheduling jobs using Autosys or similar.

•Experience with version control systems such as Bitbucket, GIT etc.

•Ability to troubleshoot and resolve data related issues efficiently.

•Good communication and interpersonal skills

•Experience with Dataiku is a plus, but not mandatory.

Tell employers what skills you have

cloudera
Version Control
UAT
Scalability
Big Data
Bitbucket
Data Modeling
Pipelines
Hadoop
ETL
MariaDB
Reliability
Data Engineering
Spark
SQL
Data Warehouse Architecture
Autosys
Teradata Data Warehouse
Software Development
Business Requirements

  • Azure Data engineer

    2 weeks ago

    Singapore NTT Data Singapore Full time $120,000 - $180,000 per year

    Job Responsibilities:Data Analysis and Interpretation: Perform in-depth analysis of structured and unstructured data sets to identify trends, patterns, and valuable insights that drive business decisions and strategies. Data Visualization and Reporting: Develop intuitive and visually appealing dashboards, reports, and presentations to communicate complex...

  • Azure Data engineer

    1 week ago

    Singapore NTT Data Singapore Full time

    Job Responsibilities: Data Analysis and Interpretation: Perform in-depth analysis of structured and unstructured data sets to identify trends, patterns, and valuable insights that drive business decisions and strategies. Data Visualization and Reporting: Develop intuitive and visually appealing dashboards, reports, and presentations to communicate complex...

  • Senior Data Scientist

    2 weeks ago

    Singapore Data Analytics Full time

    Get to Know the Team Our Search and Personalisation Team focuses on advancing scalable machine learning and experimentation frameworks that power search relevance, ranking, and recommendations across Grab. By leveraging cutting-edge analytics and statistical methods, we continuously optimize how users connect with merchants, services, and content. Get to...

  • Lead Data Scientist

    2 weeks ago

    Singapore Data Analytics Full time

    Get to Know the Team At Grab, the Market Insights team drives impactful data-driven decisions by uncovering insights about the region's users, merchants, and driver partners. By analysing vast datasets across transactional and behavioural activities, we bridge the gap between raw data science capabilities and actionable business strategies. Our work is...

  • Lead Data Scientist

    3 days ago

    Singapore Data Analytics Full time $180,000 - $250,000 per year

    Get to Know the TeamAt Grab, the Market Insights team drives impactful data-driven decisions by uncovering insights about the region's users, merchants, and driver partners. By analysing vast datasets across transactional and behavioural activities, we bridge the gap between raw data science capabilities and actionable business strategies. Our work is...

  • Production Support Engineer with Appdynamics,Splunk,Data Dog

    4 weeks ago

    Singapore NTT DATA North America Full time

    Production Support Engineer with Appdynamics,Splunk,Data Dog You will be responsible for supporting production systems with a 16*5 support model, including being on call during weekends. Your main focus will be on utilizing Observability tools such as App Dynamics, Data Dog, and Splunk for triaging issues, monitoring, and creating dashboards.It is essential...

  • Ms Engineer

    16 hours ago

    Singapore NTT DATA Full time

    **Make an impact with NTT DATA** Join a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion - it’s a place where you can grow, belong and thrive. **Your day at NTT DATA** The...

  • Senior Engineer/Engineer, Facility Management

    3 weeks ago

    Singapore Keppel Data Centres Full time

    Senior Engineer/Engineer, Facility Management Join to apply for the Senior Engineer/Engineer, Facility Management role at Keppel Data Centres Senior Engineer/Engineer, Facility Management Join to apply for the Senior Engineer/Engineer, Facility Management role at Keppel Data Centres Maintain a positive and professional working relationship with internal...

  • Full Stack Engineer

    2 weeks ago

    Central Singapore Singularity Data Full time

    **One-line description** Design and implement RisingWave Cloud. **What will you will be working on after joining our company?** - Implement cross-platform web-based features, user interfaces, and services for enterprise-grade cloud platforms; **What are the essential skills?** - BS/MS in Computer Science or a related technical field preferred; - Strong...

  • Customer Success Services Engineer

    3 days ago

    Singapore Data Direct Networks Full time

    Overview: This is an incredible opportunity to be part of a company that has been at the forefront of AI and high performance data storage innovation for over two decades. DDN Storage is a global market leader renowned for powering many of the world's most demanding AI data centers, in industries ranging from life sciences and healthcare, to financial...