Pyspark Developer

Job Posting: Spark Developer (Data Modernization Project) About the Role: We are seeking multiple skilled and motivated Spark Developers to join a dynamic data engineering team. You will be a key contributor to a strategic, large-scale data modernization initiative for a leading global financial institution. This project involves the refactoring, upgrading,...

**Roles and Responsibilities: - **Develop Spark Scala and PySpark jobs for data transformation and aggregation.** - **Create unit tests for Spark transformations and helper methods.** - **Utilize Spark and Spark SQL to read parquet data and create tables in Hive using the Scala API.** - **Collaborate closely with the Business Analysts team to review test...

About Sembcorp Sembcorp is a leading energy and urban solutions provider headquartered in Singapore. Led by its purpose to drive energy transition, Sembcorp delivers sustainable energy solutions and urban developments by leveraging its sector expertise and global track record. Join us in shaping a sustainable energy future Drive Asia's energy transition with...

About Sembcorp Sembcorp is a leading energy and urban solutions provider headquartered in Singapore. Led by its purpose to drive energy transition, Sembcorp delivers sustainable energy solutions and urban developments by leveraging its sector expertise and global track record. Play a role in Powering Asia's Energy Transition Drive Asia's energy transition...

A dynamic technology consultancy in Singapore is seeking a Data Engineer (Azure) with expertise in Azure Data engineering, Pyspark, Python, and Power BI. The ideal candidate will design and develop ETL pipelines, dashboards, and manage data integration processes. Candidates should have a Bachelor's Degree in Computer Science or Engineering and 3-5 years of...

**Responsibilities**: - Develop enterprise grade Data Products using Spark and other Big Data frameworks - Benchmark existing solutions, identify gaps, and propose solutions to eliminate them - Be able to work within a team of data engineers, data scientists, Devops engineers, business analysts, project managers around the proposed solution and contribute...

**Responsibilities**: - Architect and implement robust ETL/ELT pipelines to ingest and transform large volumes of data from heterogeneous systems (e.g., Mainframe, Oracle, SQL Server, DB2, Teradata) into distributed data platforms such as Hadoop and cloud-based data lakes. - Utilize tools such as Apache Spark, PySpark, and Hive for batch and stream data...

JD: - Minimum 5 years of total IT experience; at least 4 years of experience in development with PySpark & Python; - Good knowledge and experience on Big Data platform, Big Data Eco system tools, SPARK, Python, Etc. - Experience with Banking domain is preferable. - Strong understanding of SDLC processes including all the project documentations. - Ability to...

Good understanding of concurrent software systems and building them in a way that is scalable, maintainable, and robust - Deep understanding of the concepts in Hive, HDFS, yarn, Spark, Spark sql, Scala and Pyspark - HDFS file formats and their use cases (eg Parquet, ORC, Sequence etc) - Good knowledge in data warehousing system - Experienced in any scripting...

**Responsibilities**: 4+ years of work experience with ETL, Data Modelling, and Data Architecture Proficient in ETL optimization, designing, coding, and tuning big data processes using Pyspark Extensive experience in building data platforms on AWS using core AWS services Step function, EMR, Lambda, Glue and Athena, Redshift, Postgres, RDS etc and...