Data Engineer(IDMC, Databricks)

2 weeks ago

Singapore ANTAS PTE. LTD. Full time
Roles & Responsibilities

* Position: Data Engineer(IDMC, Databricks)

Experience: +6 Years

Skills: AWS, IDMC, Databricks and Tableau

Roles And Responsibilities:

* Design and architect data storage solutions, including databases, data lakes, and warehouses, using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks' Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging.

* Create, manage, and optimize data pipelines for ingesting, processing, and transforming data using AWS services like AWS Glue, AWS Data Pipeline, and AWS Lambda, Databricks for advanced data processing, and Informatica IDMC for data integration and quality.

* Integrate data from various sources, both internal and external, into AWS and Databricks environments, ensuring data consistency and quality, while leveraging Informatica IDMC for data integration, transformation, and governance.

* Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and enrich data, making it suitable for analytical purposes using Databricks' Spark capabilities and Informatica IDMC for data transformation and quality.

* Monitor and optimize data processing and query performance in both AWS and Databricks environments, making necessary adjustments to meet performance and scalability requirements. Utilize Informatica IDMC for optimizing data workflows.

* Implement security best practices and data encryption methods to protect sensitive data in both AWS and Databricks, while ensuring compliance with data privacy regulations. Employ Informatica IDMC for data governance and compliance.

* Implement automation for routine tasks, such as data ingestion, transformation, and monitoring, using AWS services like AWS Step Functions, AWS Lambda, Databricks Jobs, and Informatica IDMC for workflow automation.

* Maintain clear and comprehensive documentation of data infrastructure, pipelines, and configurations in both AWS and Databricks environments, with metadata management facilitated by Informatica IDMC.

* Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and deliver appropriate solutions across AWS, Databricks, and Informatica IDMC.

* Identify and resolve data-related issues and provide support to ensure data availability and integrity in both AWS, Databricks, and Informatica IDMC environments.
• Optimize AWS, Databricks, and Informatica resource usage to control costs while meeting performance and scalability requirements.

* Stay up-to-date with AWS, Databricks, Informatica IDMC services, and data engineering best practices to recommend and implement new technologies and techniques.

Requirements / Qualifications:

* Bachelor's or master's degree in computer science, data engineering, or a related field.

* Minimum 5 years of experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica IDMC.

* Proficiency in programming languages such as Python, Java, or Scala for building data pipelines.

* Evaluate potential technical solutions and make recommendations to resolve data issues especially on performance assessment for complex data transformations and long running data processes.
• Strong knowledge of SQL and NoSQL databases.

* Familiarity with data modeling and schema design.

* AWS certifications (e.g., AWS Certified Data Analytics - Specialty, AWS Certified Data Analytics - Specialty), Databricks certifications, and Informatica certifications are a plus.

Tell employers what skills you have

Tableau
RDS
Scala
Amazon RDS
Data Modeling
Amazon Redshift
Informatica
Data Transformation
Data Integration
Databricks
Data Governance
Data Engineering
Metadata
Cataloging
Encryption
AWS Lambda
S3
Amazon Dynamodb

  • Data Engineer

    5 days ago

    Singapore EXASOFT PTE. LTD. Full time

    Roles & Responsibilities* Primary skills/competency: Informatica IDMC/IICS, Databricks - Data Engineering, AWS Cloud* Secondary skills/competency: Python – PysparkResponsibilities:· Manage the end-to-end migration process from informatica PowerCenter (CDI PC) to Informatica IDMC, ensuring minimal disruption to business operations.· Hands on experience in...

  • Data Engineer

    2 weeks ago

    Singapore ANTAS PTE. LTD. Full time

    Roles & ResponsibilitiesPOSITION OVERVIEW : Industry Consulting ConsultantPOSITION GENERAL DUTIES AND TASKS :Roles And Responsibilities:• Design and architect data storage solutions, including databases, data lakes, and warehouses, using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks' Delta Lake....

  • Data Engineer(Cloud Migration)

    1 week ago

    Singapore NTT DATA Full time

    Requirements / Qualifications - Bachelor's or master's degree in computer science, data engineering, or a related field. - Minimum 7 years of experience in data engineering, with expertise in AWS services, Databricks, and/or Informatica IDMC. - Proficiency in programming languages such as Python, Java, or Scala for building data pipelines. - Evaluate...

  • Cloud Native Engineer

    3 days ago

    Singapore ZENITH INFOTECH (S) PTE LTD. Full time

    Zenith Infotech (S), a tech recruitment firm with 27 years of presence in Singapore, is lookingo ut for **Clould Native Engineer, 5 years experience with proficiency in Databricks, Python and AWS. **Job Scope** - Lead end to end data migration project from on-premises environments to Databricks with mínimal downtime. - Work with architects and lead solution...

  • Data Engineer

    5 hours ago

    Singapore Synapxe Full time

    Position Overview Do you love Healthtech and Singapore? Synapxe is a multiple award-winning Healthcare IT Leader that digitises, connects, and analyses Singapore's health ecosystem. Its ultimate aim is to improve the Singapore population's health and health administrations by integrating intelligent, highly resilient, and cost-effective technologies with...

  • Cloud Data Engineer

    18 hours ago

    Singapore Unison Consulting Pte Ltd Full time

    Design and architect data storage solutions, including databases, data lakes, and warehouses, using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks' Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging. - Create, manage, and optimize data pipelines for ingesting,...

  • Cloud Data Engineer

    3 days ago

    Singapore Unison Consulting Pte Ltd Full time

    Design and architect data storage solutions, including databases, data lakes, and warehouses, using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks' Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging. Create, manage, and optimize data pipelines for ingesting, processing,...

  • Cloud Data Engineer

    4 days ago

    Singapore Unison Consulting Pte Ltd Full time $90,000 - $120,000 per year

    Design and architect data storage solutions, including databases, data lakes, and warehouses, using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks' Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging. Create, manage, and optimize data pipelines for ingesting, processing,...

  • Advanced Data Infrastructure Specialist

    4 days ago

    Singapore beBeeDataEngineering Full time $180,000 - $220,000

    Key Data Engineering Role Job Overview: A skilled Data Engineer is required to build and maintain a robust data infrastructure that leverages analytics platforms. This role involves designing, architecting, and integrating data storage solutions using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB. The ideal candidate will...

  • Data Pipeline Specialist

    4 days ago

    Singapore beBeeData Full time $150,000 - $180,000

    Key to our organization's success is having skilled professionals like you on board. As a Data Engineer, you will play a crucial role in shaping the data landscape.Data Engineers design, build, and maintain the complex systems that handle massive amounts of data. This involves developing ETL processes to cleanse, transform, and enrich data using Databricks'...