Senior Data Engineer

**Senior Data Engineer**:
**Date**:26 Nov 2024

**Location**: Singapore, Singapore

**Company**:Singtel Group

**Be a Part of Something BIG**

We are seeking a Senior Data Engineer who will be responsible to design and deliver data engineering products, solutions, and run operations. Ensure the efficient and sustainable operation of Singtel Unified Data Platform and Event Streaming Platform, and to build and maintain large-scale, highly available, high-performance distributed systems based on system availability and performance. You will be part of Group IT - Data & Platform Management (DPM) team. In this position you will work with business, IT, and data professionals to drive Singtel data and insights transformation for our business and customers.
- Develop new data solutions and accelerators that help to deploy our data platform and
- engineering services at scale
- Manage the full life cycle from requirement gathering and analysis, design of the architecture and deployment
- Design and build scalable and reliable data pipelines for both batch and real-time streaming data
- Drive optimization, testing and tooling to improve data quality and high availability
- Maintenance and troubleshooting of data pipelines and services as per SLA
- Review and approve solution design of data pipelines

**Make An Impact By**
- Design, develop and automate large scale, high-performance distributed data processing systems (batch and/or real-time streaming)
- Practice high quality data engineering/software engineering towards building data platform infrastructure and data pipelines at scale to deliver Big Data Analytics and Data-Science initiatives.
- Lead data engineering projects End data pipelines delivery with reliable, efficient, testable, & maintainable artifacts, involves ingest & process data from a large number and variety of data sources.
- Drive Cloud data engineering practices and Cloud Lake-house re-platform drive to build & scale Modern Data Platform & Infrastructure.
- Design data models for optimal storages across data layers, workload and presentation retrieval to meet critical business requirements and platform operational efficiency.
- Develop data pipelines for both real-time stream processing and batch processing on the Data platform (Cloud and On-Prem) that meet functional / non-functional business requirements
- Build/Optimize and Contribute to shared Data Engineering Frameworks/Tools, Data Products & standards to improve the productivity and quality of output for Data Engineers across the data platform group.
- Design and build scalable Data APIs to host Operational data and Unified Data Platform assert in Data-Mesh / Data Fabric Architecture.
- Partnered with Business domain experts, data scientists, and solution designers to identify relevant data-assets, domain data model and data solutions. Also, communicate with product development engineers to coordinate backlog feature development of data pipelines patterns.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing data products for greater scalability
- Deliver high level & detailed design to ensure that the solution delivers to the business needs and align to the data architecture principles and technology stacks. Also patriciate in peer design Review and approval process.
- Support and contribute to Date engineering product and data pipeline documentations, development guidelines & standards for data-pipeline, data model and layer design.
- Drive and delivery industry standard Devops (CI/CD ) best practices, automate development and release management.
- Drive Modern Data Platform operations using Data Ops, ensure data quality, monitoring the data system. Also support Data science MLOps platform.

**Skills for Success**:

- Minimum of 8 years of experience in Data Engineering, Unified Data Platform (Datarbicks) Infrastructure, Data
- Warehousing, Data Analytics tools or related, in design and developing of end to end scalable data pipelines and data products.
- Experience designing, building and operating robust distributed data systems, and deploying high performance with reliable system with monitoring and logging practices
- Experience Build and manage the data products and pipelines using some of the most scalable and resilient open source big data technologies - Spark, Delta -Lake, Kafka, Flink, Airflow, Pestro and related distributed data processing.
- Experience building scalable and reliable Data pipelines/ ETL for large number and variety of data sources ingestion, processing and data integrations
- Build and deploy performant modern data engineering & automation frameworks using programming languages such as Scala/Python and automate the big data workflows such as ingestion, aggregation, ETL processing etc.
- Proficiency programming languages like Scala ,Python, Java, Go, Rust or scripting languages like Bash.
- Experience in handling large Unified Data P