Data Engineering Expert

2 weeks ago

Singapore 3 CUBED BUSINESS CONSULTING PTE. LTD. Full time
Roles & Responsibilities

1. Data Engineering & Platform Knowledge (Must)


• Strong understanding of Hadoop ecosystem: HDFS, Hive, Impala, Oozie, Sqoop, Spark (on YARN).


• Experience in data migration strategies (lift & shift, incremental, re-engineering pipelines).


• Knowledge of Databricks architecture (Workspaces, Unity Catalog, Clusters, Delta Lake, Workflows).

2. Testing & Validation (Preferred)


• Data reconciliation (source vs. target).


• Performance benchmarking.


• Automated test frameworks for ETL pipelines.

3. Databricks-Specific Expertise (Preferred)


• Delta Lake: ACID transactions, time travel, schema evolution, Z-ordering.


• Unity Catalog: Catalog/schema/table design, access control, lineage, tags.


• Workflows/Jobs: Orchestration, job clusters vs. all-purpose clusters.


• SQL Endpoints / Databricks SQL: Designing downstream consumption models.


• Performance Tuning: Partitioning, caching, adaptive query execution (AQE), photon runtime.

4. Migration & Data Movement (Preferred)


• Data migration from HDFS/Cloudera to cloud storage (ADLS/S3/GCS).


• Incremental ingestion techniques (Change Data Capture, Delta ingestion frameworks).


• Mapping Hive Metastore to Unity Catalog (metastore migration).


• Refactoring HiveQL/Impala SQL to Databricks SQL (syntax differences).

5. Security & Governance (Nice to have)


• Mapping Cloudera Ranger/SSO policies → Unity Catalog RBAC.


• Azure AD / AWS IAM integration with Databricks.


• Data encryption, masking, anonymization strategies.


• Service Principal setup & governance.

6. DevOps & Automation (Nice to have)


• Infrastructure as Code (Terraform for Databricks, Cloud storage, Networking).


• CI/CD for Databricks (GitHub Actions, Azure DevOps, Databricks Asset Bundles).


• Cluster policies & job automation.


• Monitoring & logging (Databricks system tables, cloud-native monitoring).

7. Cloud & Infra Skills (Nice to have)


• Strong knowledge of the target cloud (AWS/Azure/GCP):

o Storage (S3/ADLS/GCS).

o Networking (VNETs, Private Links, Security Groups).

o IAM & Key Management. 9. Soft Skills


• Ability to work with business stakeholders for data domain remapping.


• Strong documentation and governance mindset.


• Cross-team collaboration (infra, security, data, business).

Tell employers what skills you have

Azure
Pipelines
Key Management
Hadoop
Reengineering
ETL
Data Integration
Databricks
Data Governance
Data Domain
Data Engineering
SQL
Data Migration
Data Architecture
Performance Tuning
Data Science
Encryption
Orchestration
Data Analytics
Performance Benchmarking

  • Data Engineer

    2 weeks ago

    Singapore NTT Data Singapore Full time

    **The primary responsibilities**: - Lead architecture design, develop, document, and implement end-to-end data pipelines and data-driven solutions. - Define roadmap to transform data architecture focusing on scalability, performance, and stability for the entire data lifecycle; - Build data flow for data acquisition, aggregation, and modelling, using both...

  • Data Engineering Expert

    2 weeks ago

    Singapore beBeeDataEngineering Full time $80,000 - $150,000

    Job Title: Data Engineering ExpertAs a seasoned data engineer, you will be responsible for designing and implementing efficient data processing pipelines using Apache Spark and Hive. Your expertise in SQL query optimization and data quality assurance will ensure seamless data flows across systems.Key Responsibilities:Design and optimize complex data...

  • Data Engineering Expert

    2 weeks ago

    Singapore beBeeDataEngineering Full time $90,000 - $120,000

    Job Title:Data Engineering Expert Job Description:We are seeking a highly skilled Data Engineer to join our team.The ideal candidate will have experience with designing, developing and deploying data pipelines and architectures.They should be proficient in programming languages such as Python, SQL and Hadoop.A strong understanding of big data technologies...

  • Data Engineering Expert

    2 weeks ago

    Singapore beBeeEngineering Full time $80,000 - $120,000

    Job Opportunity:We are seeking a skilled Data Engineering Expert to join our team.About the Role:Craft, test and validate data pipelines for seamless deployments.Verify data integrity and completeness post-deployments.Troubleshoot pipeline issues efficiently.Implement fixes and document root causes of problems encountered.Enhance pipeline performance and...

  • Expert Cloud Data Engineer

    2 weeks ago

    Singapore beBeeData Remote Work Freelance Full time $160,000 - $200,000

    Cloud Data Engineering ExpertWe are seeking an expert in cloud data engineering to design, develop, and maintain large-scale data processing systems. This role requires a strong understanding of cloud-based data services, including AWS and Databricks.The ideal candidate will have experience with data pipelines, data warehousing, and ETL processes, as well as...

  • Expert Cloud Data Engineer

    2 weeks ago

    Singapore beBeeData Full time

    Cloud Data Engineering Expert We are seeking an expert in cloud data engineering to design, develop, and maintain large-scale data processing systems. This role requires a strong understanding of cloud-based data services, including AWS and Databricks. The ideal candidate will have experience with data pipelines, data warehousing, and ETL processes, as...

  • Expert Cloud Data Engineer

    1 week ago

    Singapore beBeeData Full time

    Cloud Data Engineering Expert We are seeking an expert in cloud data engineering to design, develop, and maintain large-scale data processing systems. This role requires a strong understanding of cloud-based data services, including AWS and Databricks. The ideal candidate will have experience with data pipelines, data warehousing, and ETL processes, as...

  • Azure Data engineer

    3 days ago

    Singapore NTT Data Singapore Full time

    Job Responsibilities: Data Analysis and Interpretation: Perform in-depth analysis of structured and unstructured data sets to identify trends, patterns, and valuable insights that drive business decisions and strategies. Data Visualization and Reporting: Develop intuitive and visually appealing dashboards, reports, and presentations to communicate complex...

  • Azure Data engineer

    1 week ago

    Singapore NTT Data Singapore Full time $120,000 - $180,000 per year

    Job Responsibilities:Data Analysis and Interpretation: Perform in-depth analysis of structured and unstructured data sets to identify trends, patterns, and valuable insights that drive business decisions and strategies. Data Visualization and Reporting: Develop intuitive and visually appealing dashboards, reports, and presentations to communicate complex...

  • Senior Associate Infrastructure Engineering and Operations Engineer

    1 week ago

    Singapore NTT DATA, Inc. Full time

    Overview Senior Associate Infrastructure Engineering and Operations Engineer at NTT DATA, Inc. Make an impact with NTT DATA. Join a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and...