Pyspark Developer

2 weeks ago


Singapore FLARE CONSULTING PTE. LTD. Full time

Job Posting: Spark Developer (Data Modernization Project)About the Role: We are seeking multiple skilled and motivated Spark Developers to join a dynamic data engineering team. You will be a key contributor to a strategic, large-scale data modernization initiative for a leading global financial institution. This project involves the refactoring, upgrading, and deployment of a significant portfolio of PySpark scripts to modernize a critical data platform. This is a fantastic opportunity to work on a high-impact project, enhance your skills with the latest Spark technologies, and gain invaluable experience in the financial services domain. Key Responsibilities: Refactor and upgrade legacy PySpark scripts to be modular, reusable, and compliant with Spark 3.3+ and Python 3.10+. Optimize Spark jobs for high performance using techniques like broadcast joins, effective partitioning, and predicate pushdown. Replace deprecated APIs (e.g., RDDs, legacy UDFs) with optimized DataFrame and Pandas UDF implementations. Implement robust code with structured logging, comprehensive error handling, and alerting mechanisms. Ensure data quality and integrity through schema enforcement, consistent data typing, and correct SCD (Slowly Changing Dimensions) logic. Collaborate within an Agile team, participating in code reviews, sprint planning, and daily stand-ups. Support the integration of code into CI/CD pipelines and contribute to automated testing frameworks. Qualifications and Experience: Education: Bachelor's or Master's degree in Software Engineering, IT, Computer Science, or a related field. Experience: 3 to 5 years of hands‐on experience in PySpark development. Mandatory Technical Skills: PySpark Development: 3-5 years of proven experience in refactoring and developing efficient PySpark scripts using DataFrame APIs. Spark Optimization: 2-3 years of practical experience in performance tuning (e.g., broadcast joins, partitioning strategies, predicate pushdown). PySpark Migration: Hands‐on experience with PySpark migration or modernization projects. Banking & Financial Data Models: Understanding of financial data concepts, including SCD logic, surrogate keys, and schema evolution. Good‐to‐Have Skills: Testing Frameworks (e.g., Pytest, Great Expectations). Data Governance & Compliance (e.g., PII/PHI handling, data lineage). Operational Readiness (e.g., backfill support, idempotent writes). #J-18808-Ljbffr



  • Singapore Trades Workforce Solutions Full time

    AWS Pyspark Developer What's on Offer: Industry: Consulting Location: Singapore 12 months contract role (with the possibility of extension)Competitive Compensation Job Description: Design, develop, and maintain data pipelines and ETL processes using PySpark on distributed systems. Implement scalable solutions on AWS cloud services (e.g., S3, EMR, Lambda)....

  • Data Engineer

    2 weeks ago


    Singapore APAR TECHNOLOGIES PTE. LTD. Full time

    Job Description We are looking for a skilled Data Engineer with strong hands‐on experience in PySpark, Python, Big Data , and performance tuning. The role involves building and optimizing scalable data pipelines, collaborating with business users and technical stakeholders, and ensuring high‐quality data delivery in a distributed environment....


  • Singapore ENFACTUM PTE. LTD. Full time

    Key Responsibilities Design, develop, and maintain data pipelines and ETL processes using PySpark on distributed systems. Implement scalable solutions on AWS cloud services (e.g., S3, EMR, Lambda). Optimize data workflows for performance and reliability. Collaborate with data engineers, analysts, and business stakeholders to deliver high-quality solutions....

  • Data Engineer

    2 days ago


    Singapore NodeFlair Full time

    **Job Summary**: **Salary** S$7,500 - S$11,500 / Monthly **Job Type** **Seniority** Senior **Years of Experience** At least 6 years **Tech Stacks** ETL AWS Analytics pySpark Apache Azure Spark NoSQL SQL Python **Position Overview**: As a PySpark Data Engineer, you will be responsible for designing, implementing, and maintaining data processing...


  • Singapore Trades Workforce Solutions Full time

    A consulting firm is seeking an AWS Pyspark Developer located in Singapore. In this 12-month contract role, you will design, develop, and maintain data pipelines using PySpark and AWS services such as S3, EMR, and Lambda. Ideal candidates should have strong programming skills in Python, expertise in distributed data processing with PySpark, and proficiency...


  • Singapore IDC TECHNOLOGIES (SINGAPORE) PTE. LTD. Full time

    A financial technology firm in Singapore is seeking an experienced IT specialist to develop and support reporting for financial systems. The ideal candidate will have strong skills in report development, proficiency in PySpark and SQL, and hands-on experience with tools like SSRS and Power BI. A Bachelor's degree in Computer Science or related field is...


  • Singapore Unison Consulting Pte Ltd Full time

    **Key Responsibilities**: - Design, build, and maintain efficient, scalable, and reliable data pipelines using Python, PySpark, Spark, and SQL. - Collaborate with data scientists, analysts, and other stakeholders to understand data needs and deliver appropriate data solutions. - Develop and optimize ETL (Extract, Transform, Load) processes to ensure data...


  • Singapore CLOUD KINETICS CONSULTING PTE. LTD. Full time

    A leading data solutions firm located in Singapore is looking for a Data Engineer to develop and maintain ETL/ELT pipelines for processing both structured and unstructured data. The ideal candidate will have a minimum of 1-2 years of data engineering experience with Azure-native implementations and demonstrated skills in building pipelines using ADF,...

  • Data Engineer

    2 weeks ago


    Singapore APAR TECHNOLOGIES PTE. LTD. Full time

    A data solutions company in Singapore is seeking a skilled Data Engineer with strong hands-on experience in PySpark, Python, and Big Data. The role involves developing and optimizing ETL/ELT pipelines, ensuring data workflow performance, and collaborating with cross-functional teams. Ideal candidates will have over 4 years of experience in Data Engineering...


  • Singapore Capgemini Full time

    **Responsibilities**: - Develop enterprise grade Data Products using Spark and other Big Data frameworks - Benchmark existing solutions, identify gaps, and propose solutions to eliminate them - Be able to work within a team of data engineers, data scientists, Devops engineers, business analysts, project managers around the proposed solution and contribute...