Senior AWS Data Engineer

Overview
Join to apply for the
Senior AWS Data Engineer
role at
PALO IT .
PALO IT is a global technology consultancy that crafts tech as a force for good. We design, develop and scale digital and sustainable products and services to unlock value across the triple bottom line: people, planet, profit. We are World Economic Forum New Champion and a B Corp-certified company.
We are small enough to care locally, big enough to deliver globally (5 continents, 18 offices, +650 experts from +50 nationalities)
We are robust and resilient (100% independent and 0 debt)
We are entrepreneurs and passionate experts: We invest in what we believe genuinely and work as a collective intelligence
We are positive, courageous, caring, doers and committed to excellence
Your Role
The Senior Data Engineer will work on acquiring, storing, governing and processing huge sets of structured and unstructured data. You will bring a vision of the big data solutions landscape for choosing optimal architecture components and a set of data engineering skills for implementing enterprise data foundations such as data lakes. You will collaborate with other PALO IT experts in Data Intelligence, Research, UX Design, Digital Technology and Agile teams to enable intelligence in the digital mesh of our clients and deliver big impact.
Responsibilities
The project is executing a data pipeline, handling data download, decryption, quality checking, cleaning, processing, and prediction, including comprehensive error handling, error reporting, and self-recovery mechanisms. The pipeline is currently deployed on an on-premises server; the project team will monitor operations and performance and align on enhancements using industry-standard tools.
Experience with migrating data pipelines to AWS Cloud using AWS managed components
Proficient in Python, SQL, and Shell scripts (Linux)
Proficient in Airflow or similar tooling to create pipelines
Proficient in AWS services such as S3, Glue, EMR, Lambda, Step Functions, Fargate, and Batch
Familiar with tools like AWS Batch, Athena, Parquet formats; NoSQL knowledge and tools like AWS OpenSearch
Who You Are / Qualifications
Bachelor’s, Master’s, or Ph.D. in IT, Information Management, Computer Science with at least 7 years of experience
Knowledge of big data technologies and distributed storage/computing concepts
Experience with big data frameworks (e.g., Hadoop, Spark) and distributions (Cloudera, Hortonworks, MapR)
Experience with batch ETL jobs to ingest and process data from multiple sources
Experience with NoSQL databases (e.g., Cassandra, MongoDB, Neo4J, ElasticSearch)
Experience with querying tools (e.g., Hive, Spark SQL, Impala)
Experience with Power BI
Experience or willingness to explore real-time streaming using Kafka, AWS Kinesis, Flume, and/or Spark Streaming
Interest in DevOps and DataOps principles (Infrastructure as Code, automating data pipelines)
Understanding of Data Science concepts and methodologies
Passionate about technology with a drive for continuous learning
We will be prioritizing applicants who have current right to work in Singapore and do not require visa sponsorship.
More About Palo IT
We’re eager to adapt to change, learn from our experiences and move to meet our planet’s urgent needs. Our action areas include: climate net-zero, target revenue from impact projects, workforce training on impact, achieving B Corp certification, and measuring employee happiness.
What We Offer
Stimulating working environments
Unique career path
International mobility
Internal R&D projects
Knowledge sharing
Personalized training
For more on our team culture and benefits, check out our careers page.
PALO IT Singapore is an equal opportunity employer. Employment decisions are based on merit, qualifications and abilities. Palo IT SG does not discriminate in employment opportunities or practices on the basis of race, colour, religion, sex, sexuality, national origin, age, disability, marital status or any other characteristics protected by law.
Your personal data will be processed for recruitment purposes, including interviews, tests, evaluation and related activities. Please consult our Privacy Notice to learn how we collect, use and