Lead Big Data Engineer

**Job Summary**:
**Salary**
S$8,000 - S$11,000 / Monthly

**Job Type**

**Seniority**

Lead

**Years of Experience**
At least 8 years

**Tech Stacks**
ETL MariaDB Docker Data Extraction play Shell Script Analytics Sqoop Dataiku Cloudera HDFS Shell Java Hive Airflow Spark Bitbucket SQL Scala Hadoop Python

**Job Purpose**

At Great Eastern, we are building a data platform to realize the full value of our data. We seek an operational personnel to join our Group Data Division.

The primary purpose of this role encompass designing, building, and maintaining the systems and processes required to manage and analyses large volumes of data in an efficient, reliable, and secure manner. This involves working with other teams such as data scientists, data engineers, and data analysts to ensure that data is processed, transformed, and delivered in a timely manner to meet the needs of the business.

This is a hands-on role which requires technical expertise to deliver data and analytical services including data ingestion, transformation, storage, and reporting, and strives to continuously develop new and improved data engineering capabilities including data platform monitoring and operations, administration of the analytics platforms including Big Data Platform, Data Science platform and industry leading business intelligence and visualization products and leveraging a variety of DataOps, data warehousing and system administration methodologies and disciplines

**The Job**
- Drive optimization, testing and tooling to improve data quality and review solution design for data pipelines
- Design, create and maintain optimal data pipeline solutions to meet business, technical, user requirements.
- Ensure that proposed solutions are aligned and conformed to the big data architecture guidelines and roadmap.
- Support the data landscape and development best practices.
- Participate in proofs-of-concepts on platform innovation, and effectively transition and scale those concepts into production at scale.
- Contribute to platform enhancements, capacity management, performance monitoring, troubleshooting and resolution of technical issues.
- Identify, design, and implement process improvements including automating manual processes and optimizing data delivery.
- Technical ownership for data and data pipelines to ensure compliance with data standards, architectural standards, and achievement of business requirements.
- Participate in the management and automation of data pipeline, deployment, testing, release, security and monitoring processes.
- Participate in implementations, upgrades, updates, patches, enhancements, and customizations of the analytics systems in the data ecosystem.
- Assess and make optimal use of Big Data platforms especially Cloudera Data Platform CDP, AI/ML Platform Dataiku, MariaDB and Tableau.
- Improve the performance, security and organization of data assets.
- Identify and recommend hardware, software, and service upgrades where needed.
- Monitor and improve the performance, efficiency, effectiveness, and orchestration of our analytics systems across the data ecosystem.
- Provision access to analytics systems and data assets according to applicable policies and procedures.
- Participate in the creation of standards, procedures, design patterns and technical frameworks.
- Takes accountability in considering business and regulatory compliance risks and takes appropriate steps to mitigate the risks.
- Maintains awareness of industry trends on regulatory compliance, emerging threats and technologies in order to understand the risk and better safeguard the company.
- Highlights any potential concerns /risks and proactively shares best risk management practices.

**Our Requirements**

Bachelor’s degree in IT, Computer Science, Software Engineering, Business Analytics or equivalent with 8+ years of experience in distributed system such as Big Data Hadoop.
- **Proficiency in Hadoop**: Experience in Cloudera Data Platform CDP (Spark, HDFS, Hive, Sqoop, Airflow, Docker)
- **Managing data infrastructure**: Experience in monitoring of data platform infrastructure and incident handling.
- **Knowledge of SQL**: Experience in MariaDB or any other relational database management system.
- **Experience with ETL**: Experience in designing and implementing ETL processes, which includes data extraction, transformation, and loading from various sources into a centralized data warehouse.
- **Data warehousing**: Good understanding of data warehousing concepts, including data architecture, dimensional modelling, and ETL processes.
- Analytical skills: They should be able to analyse complex data sets, identify patterns, and develop insights to support business decisions.
- **Experience with DevOps**: Experience in DevOps tools and environment (BitBucket, Jira, Bomboo)
- Experience in **Agile Methodology**
- Strong in User Requirements Gathering, Maintenance and Support
- Ability to analyze and understand complex problems
- **Co