Senior Data/ml Engineer

Job Posted
- Empty
- Location
Singapore
-Full time
-Remote
-Non-Remote

**Who Are We Looking For**:
This role requires someone familiar with the **dynamic nature of a startup**, capable of rapidly designing and implementing scalable solutions. You'll work closely with research teams to optimize performance and ensure seamless integration of systems, handling data from **financial institutions, government agencies, consumer brands, and internet companies**.

**Key Responsibilities**:
**Strong understanding of ML concepts and algorithms**:
Practical experience working with models in production settings in AI / data science teams to transform AI / data science code into scalable, production-ready systems.

**Data Ingestion & Integration**:
Ingest data from **enterprise relational databases** such as **Oracle**, **SQL Server**, **PostgreSQL**, and **MySQL**, as well as **enterprise SQL-based data warehouses** like **Snowflake**, **BigQuery**, **Redshift**, **Azure Synapse**, and **Teradata** for large-scale analytics.

**Data Validation & Quality Assurance**:
Ensure ingested data conforms to predefined **schemas**, checking data types, missing values, and field constraints.

Implement **data quality checks** for nulls, outliers, and duplicates to ensure data reliability.

**Data Transformation & Processing**:
**Design scalable data pipelines** for **batch processing**, deciding between **distributed computing** tools like **Spark**, **Dask**, or **Ray** when handling extremely large datasets across multiple nodes, and **single-node tools** like **Polars** and **DuckDB** for more lightweight, efficient operations. The choice will depend on the size of the data, system resources, and performance requirements.

**Leverage Polars** for high-speed, in-memory data manipulation when working with large datasets that can be processed efficiently in-memory on a single node.

**Utilize DuckDB** for on-disk query execution, offering SQL-like operations with mínimal overhead, suitable for environments that need a balance between memory use and query performance.

Seamlessly transform **Pandas-based research code** into **production-ready pipelines**, ensuring efficient memory usage and fast data access without adding unnecessary complexity.

**Data Storage & Retrieval**:
Work with internal data representations such as **Parquet**, **Arrow**, and **CSV** to support the needs of our generative models, choosing the appropriate format based on **data processing and performance needs**.

**Distributed Systems & Scalability**:
Ensure that the system can **scale efficiently from a single node to multiple nodes**, providing **graceful scaling** for users with varying compute capacities.

Optimize **SQL-based queries** for performance and scalability in **enterprise SQL environments**, ensuring efficient querying across large datasets.

**GPU Acceleration & Parallel Processing**:
Utilize **GPU acceleration** and **parallel processing** to improve performance in large-scale model training and data processing.

**Data Lineage & Metadata Management** (Reduced Emphasis)**:
Implement **basic data lineage** for auditability, ensuring traceability in data transformations when required.

Manage metadata as needed to document pipelines and workflows.

**Error Handling, Recovery, & Performance Monitoring**:
Design robust **error handling** mechanisms, with **automatic retries** and **data recovery** in case of pipeline failures.

Track performance metrics such as **data throughput**, **latency**, and **processing times** to ensure efficient pipeline operations at scale.

**Documentation & Reporting**:
Create clear **documentation** of data pipelines, workflows, and system architectures to enable smooth handovers and collaboration across teams.

**Essential Skills and Qualifications**:
**High Priority**:
Hands-on experience **scaling data pipelines** and **machine learning systems** to handle **hundreds of millions to billions of rows** in enterprise environments.

4+ years of experience in building scalable data solutions with **Python** and distinct libraries such as:
**Data Science Libraries**:Pandas**, **NumPy**, **Scikit-learn**.

**Scaling Libraries**:Polars** for in-memory processing and **DuckDB** for efficient on-disk queries.

Ability to **choose the right framework** (e.g., **Dask**, **Ray**, **Polars**, **DuckDB**) depending on the workload and environment, with a focus on balancing simplicity and scalability.

Experience in **data validation** and ensuring data quality with tools like **Pandera** or **Pydantic**.

Proficiency in building **ETL/ELT pipelines** and managing data across **relational databases**, **data warehouses**, and **cloud storage**.

Strong knowledge of **GPU parallelization** for deep learning models using **PyTorch**.

**Good to Have**:
Experience with logging and monitoring in production environments.

Understanding of **data lineage** and **metadata management** systems to support data transparency.

Famil