Big Data Platform Architect

2 days ago

Singapur, Singapore U3 Full time
Job Title: Big Data Platform Engineer/Architect

Job Summary:

The Big Data Platform Engineer/Architect will be responsible for designing and implementing a scalable and secure big data platform on hybrid cloud environments. This includes defining architecture and strategy for batch workloads, enabling multitenancy and capacity planning at enterprise scale, component stress and load testing, deployment automation, monitoring, and governance.

Key Responsibilities:

  • Design and develop batch data processing on on-prem or public cloud
  • Develop and maintain an advanced platform to manage Kubernetes clusters lifecycle for batch processing
  • Understand the on-prem VM-based big data ecosystem, designing and developing migration plans to on-prem cloud-native deployment
  • Determine security and segregation of lab and factory/BAU workloads
  • Develop tools for automated build, test, deployment, and management of the platform
  • Determine approach for multitenancy and capacity planning at enterprise scale
  • Improve continuous integration and delivery systems
  • Monitor system events to ensure health, maximum system availability, and service quality
  • Maintain documentation regarding configurations, operations, and troubleshooting procedures
  • Participate in the definition of standards, guidelines, and best practices
  • Assist in evaluating new requirements, technical design, and standards
  • Participate in R&D efforts, bringing new ideas and new technologies

Qualifications:

  • University degree or equivalent experience in computer information systems, computer science, mathematics, or a related field
  • Relevant experience: architect with 8+ years of experience in big data, hands-on with Spark, Hive, and Presto; developer with 2+ years of experience in Kubernetes/OpenShift

Skills:

  • Good understanding and in-depth knowledge of distributed systems
  • Professional experience building enterprise big data applications using Spark, Hadoop, Hive, etc.
  • Hands-on knowledge of containerization and deploying open-source products on K8s/OpenShift
  • Maintaining and building in-house-based open-source platforms like Hadoop, Spark, Hive, Presto
  • Skills in developing techniques and methodologies to resolve unprecedented problems or situations
  • Ability to communicate effectively with all levels of management
  • Ability to communicate with remote teams
  • Ability to make complex information accessible
  • Must be a self-starter that requires only limited supervision/guidance
  • Candidate must present with honesty, integrity, and maintains confidentiality at all times
  • Be a team player
  • Have good interpersonal communication skills
  • Ability to work effectively in a fast-paced changing environment
  • Ability to multi-task and prioritize
  • Thorough experience in defining and applying design standards, patterns, quality matrix depending on the solutions
  • Deep knowledge of OOPs, data structure, and algorithm
  • Ability and willingness to learn technologies at pace

Knowledge:

  • Expertise with Java or any coding languages
  • Linux scripting (Bash, Python, etc.)
  • Expertise with source control management (Git, Bitbucket, etc.)
  • Expertise with CI/CD (Jenkins/ArgoCD)
  • Expertise with configuration management (Ansible, Terraform, etc.)
  • Expertise with monitoring (Prometheus, etc.)
  • In-depth knowledge with Linux environments (RHEL)
  • Knowledge on Protegrity
  • Knowledge of cloud infrastructures (AWS EKS, Google GKS, Azure AKS, etc.)
  • Knowledge of web servers (Nginx, Apache, etc.)
  • Knowledge in RESTful API design and implementation
  • Knowledge in the development of high-performance and fault-tolerant systems
  • Knowledge in distributed systems programming
  • Experience with Java or Go programming language an asset
  • Experience with Docker an asset
  • Experience running Kubernetes in production an asset
  • Experience in large-scale production deployments an asset

  • Data Engineer

    2 weeks ago

    Singapur, Singapore NTT Data Singapore Full time

    Job SummaryWe are seeking a highly skilled and experienced Big Data Engineer to lead our data engineering efforts. The ideal candidate must have a minimum of 4 years of experience managing data engineering jobs in big data environments, such as Cloudera Data Platform. Overall experience can be 4 to 10 years. Consultant/Senior Consultant preferably.Key...

  • Data Engineer with Big Data Expertise

    2 days ago

    Singapur, Singapore NTT Data Singapore Full time

    Job SummaryWe are seeking a highly skilled and experienced Big Data Engineer to lead our data engineering team. The ideal candidate must have a minimum of 4 years of experience managing data engineering jobs in big data environments, such as Cloudera Data Platform. Overall experience can be 4 to 10 years. Consultant/Senior Consultant preferably. The...

  • Big Data Global Manager

    4 months ago

    Singapur, Singapore TikTok Full time

    About the team This role is a member of our Data Platform team focusing on Data products and solutions to enterprise services, leveraging TikTok's cutting-edge data infrastructure and data application. You will be helping enterprise customers to build their next generation data warehouse and data analytics, enabling them to do things that were not possible...

  • Data Architect

    15 hours ago

    Singapur, Singapore Databricks Full time

    Job Title: Solutions ArchitectAs a Pre-sales Solutions Architect (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a technical advisor internally to the sales team, and work with the product team as an advocate of your customers in the field. You will...

  • Senior Data Architect Consultant

    3 weeks ago

    Singapur, Singapore ITCAN Full time

    Job DescriptionAbout the RoleWe are seeking a highly skilled Senior Data Architect Consultant to join our team at ITCAN. As a key member of our data analytics team, you will be responsible for designing and implementing large-scale enterprise Big Data solutions in client data centers and cloud environments.Key ResponsibilitiesTranslate business requirements...

  • Senior Data Architect

    2 weeks ago

    Singapur, Singapore ITCAN Full time

    Job Title: Senior Network ConsultantsAbout the Role:We are seeking a highly skilled Senior Network Consultant to join our team at ITCAN. As a key member of our team, you will be responsible for translating business requirements into technical solutions, leveraging your strong business acumen to drive innovation and growth.Key Responsibilities:Lead...

  • Data Architect

    3 weeks ago

    Singapur, Singapore Databricks Full time

    About the Role: We are seeking a highly skilled Data Architect to join our team at Databricks. As a Data Architect, you will play a key role in guiding the technical evaluation phase in a hands-on environment throughout the sales process. You will be a technical advisor internally to the sales team, and work with the product team as an advocate of your...

  • Data Architect

    2 weeks ago

    Singapur, Singapore Infosys Singapore & Australia Full time

    Senior Data ArchitectWe are seeking a highly skilled and experienced Senior Data Architect to lead our data engineering efforts. The ideal candidate will have a strong background in designing and implementing high-performance, scalable, and secure data architectures.Key Responsibilities:Design and implement data pipelines and workflows using ELT principles...

  • Senior Principal Enterprise Architect

    3 weeks ago

    Singapur, Singapore NTT DATA Full time

    About the RoleWe are seeking a highly skilled Senior Principal Enterprise Architect to join our team at NTT DATA. As a key member of our organization, you will play a critical role in driving business growth and innovation through the development and implementation of strategic enterprise architecture solutions.Key ResponsibilitiesSupport the client manager...

  • Big Data Engineer

    5 days ago

    Singapur, Singapore Virtusa Full time

    Big Data EngineerVirtusa is seeking a highly skilled Big Data Engineer to join our team. As a Big Data Engineer, you will be responsible for designing, developing, and implementing large-scale data processing systems using big data technologies such as Hadoop, Spark, and NoSQL databases.Key Responsibilities:Design and develop data pipelines using ETL tools...

  • Data Architect

    2 weeks ago

    Singapur, Singapore Databricks Full time

    About the RoleWe are seeking a highly skilled Data Architect to join our team at Databricks. As a Data Architect, you will play a key role in guiding the technical evaluation phase in a hands-on environment throughout the sales process.Key ResponsibilitiesLead technical evaluations and provide expert advice to sales teams on data engineering and cloud...

  • Data Architect

    4 months ago

    Singapur, Singapore Infosys Singapore & Australia Full time

    We are looking for a highly experienced and skilled Data Architect to join our team. The ideal candidate will have 12-15 years of experience in architecting solutions of data engineering, focusing on ELT and PySpark/Hadoop workloads. In addition to strong solution and delivery skills, the ideal candidate will also have a view on business growth and managing...

  • Big Data Architect

    3 weeks ago

    Singapur, Singapore TikTok Full time

    **About the Team** Our **Recommendation Architecture Team** is responsible for building and optimizing the architecture for recommendation systems to provide the most stable and best experience for our TikTok users. We cover almost all short-text recommendation scenarios in TikTok, such as search suggestions, the video-related search bar, and comment...

  • Data Architect

    2 days ago

    Singapur, Singapore Databricks Full time

    Job Title: Solutions ArchitectJob Summary:We are seeking a highly skilled Solutions Architect to join our team at Databricks. As a Solutions Architect, you will play a key role in guiding the technical evaluation phase in a hands-on environment throughout the sales process. You will be a technical advisor internally to the sales team, and work with the...

  • Senior Enterprise Architect

    2 weeks ago

    Singapur, Singapore NTT DATA Full time

    About the RoleWe are seeking a highly skilled Senior Principal Enterprise Architect to join our team at NTT DATA. As a key member of our organization, you will play a pivotal role in shaping the future of our clients' businesses through innovative technology solutions.Key ResponsibilitiesSupport client managers in account planning, opportunity seeding,...

  • Big Data Engineer

    4 months ago

    Singapur, Singapore Virtusa Full time

    Big Data Engineer - 123101 Description Responsibilities Good understanding and completion of projects using waterfall/Agile methodology. • Good understanding of analytics and data warehouse implementations. • Ability to troubleshoot complex issues ranging from system resource to application stack traces. • Strong SQL, data modelling and data...

  • Big Data Developer

    1 week ago

    Singapur, Singapore Helius Full time

    Job Title: Big Data DeveloperHelius is seeking a highly skilled Big Data Developer to join our team. As a key member of our data integration team, you will be responsible for designing, developing, and deploying production-grade Talend Big Data jobs that meet our organization's architecture standards.Key Responsibilities:Design and develop Talend Big Data...

  • Big Data Specialist

    3 weeks ago

    Singapur, Singapore Epam Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at EPAM. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining data ingestion and processing jobs in big data environments.Key ResponsibilitiesAnalyze and document business data requirements to ensure seamless data access to...

  • Big Data Architect/Technical Lead

    3 weeks ago

    Singapur, Singapore ITCAN Full time

    Job DescriptionAbout the RoleWe are seeking a highly skilled and experienced IT professional to join our team at ITCAN as a Big Data Architect/Technical Lead. This is a key role that will be responsible for designing, implementing, and managing our cloud infrastructure and data center operations.Key ResponsibilitiesDesign and implement cloud infrastructure...

  • Data Engineer

    2 weeks ago

    Singapur, Singapore DBS Bank Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at DBS Bank. As a Data Engineer, you will play a key role in designing, building, and maintaining our Big Data platform.Key ResponsibilitiesDesign and implement scalable data pipelines using Big Data tools and technologiesCollaborate with cross-functional teams to define and deliver...