Senior Data Engineer

We are looking for an experienced Senior Data Engineer to design, build, and operate scalable data pipelines and reusable data products using AWS and Databricks. The successful candidate will provide technical leadership across data ingestion, transformation, modelling, data quality, orchestration, monitoring, and performance optimisation.

Key Responsibilities
• Design and build reliable batch and streaming data pipelines using AWS and Databricks.
• Develop Bronze, Silver, and Gold data layers using Delta Lake and lakehouse architecture.
• Integrate data from databases, files, APIs, and event streams.
• Implement incremental loading, change data capture, backfills, replay, and recovery processes.
• Establish automated data quality checks, schema validation, reconciliation, metadata, and lineage.
• Build and manage workflows using Databricks and relevant AWS services.
• Implement automated testing, CI/CD, and infrastructure as code.
• Monitor pipeline performance, reliability, data freshness, and operational issues.
• Optimise Spark and SQL workloads for performance, scalability, and cost efficiency.
• Apply appropriate cloud security, access control, encryption, and audit logging practices.
• Review technical designs and code, establish engineering standards, and mentor junior engineers.
• Work closely with data owners, analysts, data scientists, architects, and technology teams.
• Maintain technical documentation, data models, mappings, and operational procedures. Requirements
• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems, or a related discipline.
• Around 6–10 years of relevant data or software engineering experience.
• At least 2 years of experience working with cloud-based data platforms.
• Strong hands-on experience with Python, PySpark, and SQL.
• Production-level experience with Databricks, Apache Spark, and Delta Lake.
• Hands-on AWS experience with services such as S3, Glue, Lake Formation, IAM, Lambda, Step Functions, CloudWatch, Redshift, MSK, or Kinesis.
• Experience designing lakehouse or data-lake architectures and reusable data products.
• Experience with workflow orchestration, Git, CI/CD, automated testing, and infrastructure as code such as Terraform.
• Knowledge of data quality, metadata management, lineage, monitoring, and pipeline observability.
• Good understanding of cloud security, access controls, encryption, secrets management, and audit logging.
• Strong communication, analytical, troubleshooting, and stakeholder-management skills.
• Experience providing technical guidance, conducting code reviews, or mentoring engineers. Good to Have
• Experience with Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing, or Databricks Asset Bundles.
• Exposure to Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis, or AWS DMS.
• Experience with data products, data contracts, semantic layers, master data, or data-mesh models.
• Exposure to feature engineering, ML datasets, feature stores, or MLOps pipelines.
• Relevant certifications such as:
• Databricks Certified Data Engineer Professional
• AWS Certified Data Engineer – Associate
• AWS Certified Solutions Architect – Associate or Professional 📩 Interested candidates may apply or send their updated CV via LinkedIn. #Hiring #SeniorDataEngineer #DataEngineering #Databricks #AWS #PySpark #Python #SQL #DeltaLake #SingaporeJobs #TechJobs