Pyspark Developer
2 weeks ago
Job Posting: Spark Developer (Data Modernization Project)About the Role: We are seeking multiple skilled and motivated Spark Developers to join a dynamic data engineering team. You will be a key contributor to a strategic, large-scale data modernization initiative for a leading global financial institution. This project involves the refactoring, upgrading, and deployment of a significant portfolio of PySpark scripts to modernize a critical data platform. This is a fantastic opportunity to work on a high-impact project, enhance your skills with the latest Spark technologies, and gain invaluable experience in the financial services domain. Key Responsibilities: Refactor and upgrade legacy PySpark scripts to be modular, reusable, and compliant with Spark 3.3+ and Python 3.10+. Optimize Spark jobs for high performance using techniques like broadcast joins, effective partitioning, and predicate pushdown. Replace deprecated APIs (e.g., RDDs, legacy UDFs) with optimized DataFrame and Pandas UDF implementations. Implement robust code with structured logging, comprehensive error handling, and alerting mechanisms. Ensure data quality and integrity through schema enforcement, consistent data typing, and correct SCD (Slowly Changing Dimensions) logic. Collaborate within an Agile team, participating in code reviews, sprint planning, and daily stand-ups. Support the integration of code into CI/CD pipelines and contribute to automated testing frameworks. Qualifications and Experience: Education: Bachelor's or Master's degree in Software Engineering, IT, Computer Science, or a related field. Experience: 3 to 5 years of hands‐on experience in PySpark development. Mandatory Technical Skills: PySpark Development: 3-5 years of proven experience in refactoring and developing efficient PySpark scripts using DataFrame APIs. Spark Optimization: 2-3 years of practical experience in performance tuning (e.g., broadcast joins, partitioning strategies, predicate pushdown). PySpark Migration: Hands‐on experience with PySpark migration or modernization projects. Banking & Financial Data Models: Understanding of financial data concepts, including SCD logic, surrogate keys, and schema evolution. Good‐to‐Have Skills: Testing Frameworks (e.g., Pytest, Great Expectations). Data Governance & Compliance (e.g., PII/PHI handling, data lineage). Operational Readiness (e.g., backfill support, idempotent writes). #J-18808-Ljbffr
-
AWS Pyspark Developer
1 week ago
Singapore Trades Workforce Solutions Full timeAWS Pyspark Developer What's on Offer: Industry: Consulting Location: Singapore 12 months contract role (with the possibility of extension)Competitive Compensation Job Description: Design, develop, and maintain data pipelines and ETL processes using PySpark on distributed systems. Implement scalable solutions on AWS cloud services (e.g., S3, EMR, Lambda)....
-
Data Engineer
2 weeks ago
Singapore APAR TECHNOLOGIES PTE. LTD. Full timeJob Description We are looking for a skilled Data Engineer with strong hands‐on experience in PySpark, Python, Big Data , and performance tuning. The role involves building and optimizing scalable data pipelines, collaborating with business users and technical stakeholders, and ensuring high‐quality data delivery in a distributed environment....
-
AWS Pyspark Developer
4 days ago
Singapore ENFACTUM PTE. LTD. Full timeKey Responsibilities Design, develop, and maintain data pipelines and ETL processes using PySpark on distributed systems. Implement scalable solutions on AWS cloud services (e.g., S3, EMR, Lambda). Optimize data workflows for performance and reliability. Collaborate with data engineers, analysts, and business stakeholders to deliver high-quality solutions....
-
Data Engineer
2 days ago
Singapore NodeFlair Full time**Job Summary**: **Salary** S$7,500 - S$11,500 / Monthly **Job Type** **Seniority** Senior **Years of Experience** At least 6 years **Tech Stacks** ETL AWS Analytics pySpark Apache Azure Spark NoSQL SQL Python **Position Overview**: As a PySpark Data Engineer, you will be responsible for designing, implementing, and maintaining data processing...
-
AWS PySpark Developer — Cloud Data Pipelines
1 week ago
Singapore Trades Workforce Solutions Full timeA consulting firm is seeking an AWS Pyspark Developer located in Singapore. In this 12-month contract role, you will design, develop, and maintain data pipelines using PySpark and AWS services such as S3, EMR, and Lambda. Ideal candidates should have strong programming skills in Python, expertise in distributed data processing with PySpark, and proficiency...
-
Lead Financial Reporting Developer
1 week ago
Singapore IDC TECHNOLOGIES (SINGAPORE) PTE. LTD. Full timeA financial technology firm in Singapore is seeking an experienced IT specialist to develop and support reporting for financial systems. The ideal candidate will have strong skills in report development, proficiency in PySpark and SQL, and hands-on experience with tools like SSRS and Power BI. A Bachelor's degree in Computer Science or related field is...
-
Data Engineer(Python, Pyspark, Spark
2 weeks ago
Singapore Unison Consulting Pte Ltd Full time**Key Responsibilities**: - Design, build, and maintain efficient, scalable, and reliable data pipelines using Python, PySpark, Spark, and SQL. - Collaborate with data scientists, analysts, and other stakeholders to understand data needs and deliver appropriate data solutions. - Develop and optimize ETL (Extract, Transform, Load) processes to ensure data...
-
Azure Data Engineer: ETL/PySpark Pipelines
2 weeks ago
Singapore CLOUD KINETICS CONSULTING PTE. LTD. Full timeA leading data solutions firm located in Singapore is looking for a Data Engineer to develop and maintain ETL/ELT pipelines for processing both structured and unstructured data. The ideal candidate will have a minimum of 1-2 years of data engineering experience with Azure-native implementations and demonstrated skills in building pipelines using ADF,...
-
Data Engineer
2 weeks ago
Singapore APAR TECHNOLOGIES PTE. LTD. Full timeA data solutions company in Singapore is seeking a skilled Data Engineer with strong hands-on experience in PySpark, Python, and Big Data. The role involves developing and optimizing ETL/ELT pipelines, ensuring data workflow performance, and collaborating with cross-functional teams. Ideal candidates will have over 4 years of experience in Data Engineering...
-
12200273 - Data Engineer (Python/pyspark)
2 weeks ago
Singapore Capgemini Full time**Responsibilities**: - Develop enterprise grade Data Products using Spark and other Big Data frameworks - Benchmark existing solutions, identify gaps, and propose solutions to eliminate them - Be able to work within a team of data engineers, data scientists, Devops engineers, business analysts, project managers around the proposed solution and contribute...