Data Engineer

The Data Engineer supports the design, implementation and maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information in a scalable, repeatable and secure manner. Key responsibilities are:

- Define optimal solutions for data collection, processing and warehousing
- Design, code and test data systems and work on implementing those into the internal infrastructure or cloud platform to support software developers and data scientists on data activities such as data analytics, ML and AI
- Focus on collecting, parsing, managing, analyzing and visualizing large sets of data to get information ready for insight discovery
- Such other duties and responsibilities as may be reasonably requested by your immediate supervisor / the head of department from time to time

Key tasks include:

- Identify business needs
- Identify suitable data structures based on business needs to ensure availability and accessibility of data
- Determine technical system requirements based on data needs
- Keep abreast of latest technologies and products in data stores and data processing software, and technologies
- Build and maintain data pipeline
- Assist in building scalable data pipelines to extract, transform, load and integrate data
- Develop codes and scripts to process structured and unstructured data in real-time from a variety of data sources
- Test data pipelines for scalability and reliability to process high data volume, variety and velocity
- Consolidate and create data storage solutions for storage and retrieval of information
- Develop prototypes and proof-of-concepts for data solutions
- Monitor data system performance
- Support the handling and logging of errors
- Develop backup data archiving systems to ensure system continuity
- Implement and monitor data security and privacy measures on existing data solutions
- Optimize solution performance
- Assist in the integration of data systems with existing infrastructure
- Develop tools to improve data flows between internal and/or external systems and the data warehouse
- Automate the data collection and analysis processes, data releasing and reporting tools
- Test data system configurations to increase efficiency

JOB REQUIREMENTS
- Bachelor’s Degree in computer science, Statistics, Informatics, Information Systems, Engineering, or relevant field with at least 5 years working experience in Data Engineering area
- Completed Data Engineering training with certification will be advantageous
- Familiar with the popular cloud platforms such Azure, AWS, Google, etc.
- Passionate about numbers and works with large data sets
- Advanced working SQL knowledge and experience working with relational databases
- Strong analytical skills related to working with unstructured datasets
- Experience with (2 or more) programming and/or scripting languages: Python, Java, Javascript, C++, etc
- Experience building and optimizing data pipelines, architectures and data sets
- Competent in performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
- Experience in building processes supporting data transformation, data structures, metadata, dependency and workload management.
- Experience in manipulating, processing and extracting value from large-disconnected datasets.
- Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores
- Experience supporting and working with cross-functional teams in a dynamic environment
- Experience with data pipeline and workflow management tools such as Google Dataflow, AWS Data Pipeline, Azure Data Pipeline, etc
- Experience with stream-processing systems/services such as Kafka, etc. will have added advantage
- Domain knowledge on plant operation and asset maintenance will also be an added advantage

BUSINESS UNIT

Keppel Infrastructure