Data Infrastructure Architect

Job Title: Data Infrastructure Architect

We are seeking an experienced Data Infrastructure Architect to build and maintain scalable data infrastructure, converting research into production-ready solutions for synthetic tabular data generation.

You will architect and operate large-scale data curation, scraping, and cleaning pipelines to deliver massive datasets for pretraining and finetuning large language models on tabular and unstructured domains.

This is an individual contributor role suited for someone who thrives in a fast-paced environment. The ideal candidate has experience scaling data and machine learning systems to handle billions of records and can build complex data pipelines for enterprise applications.

You'll work closely with software, machine learning, and applied research teams to optimize performance and ensure seamless integration of systems.

1. Data Infrastructure Development:

• Design and build data ingestion pipelines from enterprise relational databases (e.g. Oracle, SQL Server, PostgreSQL, MySQL) and files (e.g. Parquet, CSV) for large-scale synthetic data pipelines.
• Architect and maintain data warehouses and data lakes (e.g. Delta Lake) optimized for synthetic data training and generation workflows.
• Seamlessly transform Pandas-based research code into production-ready pipelines.
• Build automated data quality monitoring and validation systems to ensure data integrity throughout the pipeline lifecycle.
• Implement comprehensive data lineage tracking and audit capabilities for regulatory compliance and privacy validation.
• Design robust error handling mechanisms, with automatic retries and data recovery in case of pipeline failures.
• Track performance metrics such as data throughput, latency, and processing times to ensure efficient pipeline operations at scale.
• Implement monitoring and alerting (e.g. Prometheus, Grafana) for pipeline health, throughput, and data quality metrics.

• Design and build distributed web scraping clusters to extract data from millions of pages.
• Build LLM-aided data filtering systems combining automated model scoring to evaluate and prioritize high-quality content.

• Fair understanding of machine learning concepts, training workflows, and algorithms, with familiarity in tools like PyTorch and Hugging Face.

• Bachelor's degree in Computer Science, Software Engineering, Data Engineering, or related field with strong foundation in distributed systems and data processing.
• Expert proficiency at scaling data pipelines and machine learning systems to handle billions of rows in enterprise environments.
• 3+ years of experience in building scalable data solutions with Python and distinct libraries such as Pandas, NumPy, Scikit-learn, PyTorch, Spark, Dask, Airflow, Dagster, etc.
• Expertise in automated data quality frameworks including rule-based and AI-based automation for format validation, anomaly detection, statistical validation.
• Proficiency in building ETL/ELT pipelines and managing data across relational databases, data lakes, and cloud storage.
• Experience in building data monitoring and alerting systems.
• Hands-on experience with web scraping tools (Scrapy, Selenium, Puppeteer).
• Experience building ML data pipelines and supporting infrastructure for training and deploying machine learning models at scale.

• Experience with data governance frameworks and compliance requirements (GDPR, CCPA, PDPA) in data processing systems.
• Experience with containerization and orchestration using Docker, Kubernetes, and cloud-native deployment strategies.

This is a unique opportunity for someone looking to actively build and scale systems in a fast-moving start-up. If you've successfully scaled machine learning and data systems to billions of rows and thrive in a dynamic environment, this role is for you.

• Flexible time-off arrangements.
• Flexible work arrangements - work from office at One North or WFH on some days.