Data Engineer

COMPANY DESCRIPTION

NE Digital is the digital, data and technology organization that serve as a center of excellence to drive digital transformation for our group of NTUC Social Enterprises to meet the critical social needs of Singapore's community. Delivering innovative products and solutions, we empower our people to lead a better and meaningful life through digital services in the area of daily essentials, health and community care, childcare and education as well as financial services.

**The Team**

We believe that diversity is key to driving an innovative, cohesive, productive and fun workplace Hence, at NE Digital our people join us from all around the world. Be sure to be soaked in an environment with different ethnic groups driving innovation and injecting some creative juice as one

Contributing to a social purpose through technology, our team of passionate and dedicated folks are spread into different social enterprises such as NTUC Fairprice Group, NTUC First Campus, NTUC Health and among others

**Creating technologies that impacts**

DESIGNATION : Data Engineer

RESPONSIBILITIES

This role is part of the newly minted Central Data Platform & Infrastructure team, within the office of Chief Data Officer of NTUC Group. This team will be responsible for building and maintaining mission-critical data engineering platforms and associated infrastructure of the whole NTUC Group.

Other core responsibilities will involve, but not limited to:

- Designing Central Data Architecture
- Upholding data engineering standards for the entire group.
- Working closely with various Data Engineering teams across social enterprises
- Maintain cognizance of Data Technologies and Processes used in the organization.
- Data Security and Privacy

**Responsibilities**

The Data Engineer will be responsible for the design, develop, and maintain:

- Best Practices for Data Architecture and Design Patterns for Data Engineering Use-Cases.
- Real-time data feeds from database sources like MySQL, Oracle, and MS SQLServer using the “Change Data Capture Engine” aka “CDC”.
- Sub-Second Real-Time Data Pipelines using Google Dataflow (Apache Beam), Kinesis, Lambda & Google PubSub.
- Batch Pipeline Orchestration using on Google Cloud Composer (Apache Airflow) and Jenkins.
- Auto Scalable platform using Kubernetes on GKE and EKS.
- Org Wide Master Data Management, Data Catalog Engine, and Data Quality Engine.
- Code Repo Pipeline to automate Continuous Integration and Continuous Deployment (CI / CD).
- Structure Tables with Partitioning and Clustering to increase Cost & Performance Benefits.
- Guide Data Analysts and Data Scientists to write efficient queries and workloads.
- Data Sharing with On-Demand Encryption/Decryption which can operate at Scale.
- Running Containerized ETL workflow at scale.

QUALIFICATIONS
- Have strong fundamentals in Computer Science concepts like Cloud Computing Architecture, Distributed Computing, High-Velocity Data Processing, Lambda Architecture, etc
- Strong Data modeling and managing Distributed Computing Platforms for Data Processing.
- Advance knowledge of SQL and writing resource-efficient queries.
- Have at least 2+ years of professional programming experience in Python.
- Have at least 2+ years of experience in running a data processing pipelines on either of these: Google BigQuery, Redshift, Hadoop, Presto, Spark, or KSQL.
- Have at least 2+ years of experience in writing Sub-Second Real-Time pipelines using Google DataFlow(Apache Beam), PubSub, Kinesis Stream, Lamda, etc...
- Have a good understanding of how Kubernetes clusters work and scale on-demand.
- Have adequate experience using Containers for Data Engineering workload.
- Implemented manual or automated tools for Data Quality, Catalog, and Lineage.
- Uphold the sense of Frugality across Data Engineering teams.
- Have Good Interpersonal and Presentation Skills to explain and promote Best Practices across the organization with both technical as well as non-technical stakeholders.