Software Engineer

2 days ago


Singapore HyperGAI Full time

We are seeking a highly motivated and experienced WebCrawler Engineer to develop exciting Generative AI (GenAI) products. Candidates should have at least 3 to 5 years web crawler experience, and experience in multimodal (text, image, video) data is preferred.As a WebCrawler engineer, you will work with other teams to ensure the smooth progress of product iterations, continuously improve the user experience of the product through in-depth understanding of the business and products, and use technology to drive business growth. Your expertise will play a key role to build our products and achieve our vision.
Responsibilities
Designing and Implementing Web Crawlers: Develop scalable and efficient web crawling systems to gather data from various online sources. This involves understanding the structure of different websites, implementing crawling algorithms, handling dynamic content, and ensuring compliance with legal and ethical guidelines.
Data Extraction and Parsing: Write scripts or develop algorithms to extract relevant information from web pages. This may involve parsing HTML/XML documents, using regular expressions, or employing advanced parsing techniques such as natural language processing (NLP) to extract structured data from unstructured sources.
Data Quality Assurance: Implement mechanisms to ensure the quality and reliability of crawled data. This includes error handling, data validation, deduplication, and dealing with inconsistencies or missing data.
Scalable Data Storage and Management: Design and develop backend systems to store, organize, and manage large volumes of crawled data efficiently. This may involve selecting appropriate databases (e.g., relational databases, NoSQL databases), optimizing database schemas, and implementing data caching and indexing strategies for faster retrieval.
Performance Optimization: Optimize the performance of web crawling and data management systems to handle large-scale data processing efficiently. This includes optimizing algorithms, minimizing resource usage, and parallelizing data processing tasks.
Monitoring and Maintenance: Implement monitoring tools and logging mechanisms to track the health and performance of web crawling and data management systems. Proactively identify and resolve issues such as crawling failures, performance bottlenecks, or data inconsistencies.
Security and Compliance: Ensure that the web crawling and data management systems adhere to security best practices and regulatory requirements. Implement mechanisms to protect against security threats such as XSS (Cross-Site Scripting) attacks, CSRF (Cross-Site Request Forgery) attacks, and data breaches.
Qualifications
Possess a Bachelor's degree or above in Computer Science, Software Engineering, Technical, Science, E-Commerce, Information Technology, Mathematics or software related majors.
More than 3 years of web crawler experience, experience in large-scale web crawling and multimodal (text, image, video) data is preferred.
Proficient in one or more programming languages such as Golang/Python/PHP/Java, with strong architectural capabilities and good coding standards.
Familiar with common databases, such as MySQL, Redis, HBase, etc.
Experience in web crawling libraries like Scrapy, Beautiful Soup, Selenium, Apache Nutch.
Prior experience in Search engine companies is a plus.
#J-18808-Ljbffr


  • SOFTWARE ENGINEER

    2 days ago


    Singapore CLOUDPILOTS Software & Consulting GmbH Full time

    SOFTWARE ENGINEER CLOUD CLOUDPILOTS | Location Berlin, Cologne or Home Office As a Software Engineer* Cloud, you act as a technical contact person and analyze existing systems, define migration strategies, and take over the setup and customization of the cloud environment for our customers. You help to master technical and organizational challenges when...


  • Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesJOB DESCRIPTION· The incumbent will be responsible for developing and installing robust software solutions.· Key responsibilities include designing, implementing, and delivering high-quality software projects using JAVA, Spring Boot, SQL/Oracle, J2EE, Junit, Mockito, JQuery, Javascript, and other JAVA technologies.· The...


  • Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesJOB DESCRIPTION· The incumbent will be responsible for developing and installing robust software solutions.· Key responsibilities include designing, implementing, and delivering high-quality software projects using JAVA, Spring Boot, SQL/Oracle, J2EE, Junit, Mockito, JQuery, Javascript, and other JAVA technologies.· The...

  • Software Engineer

    2 days ago


    Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesOur company is seeking a game-changer Software Engineer who can conceptualize and execute complex problem statements in less than half the time that most people think is possible.We are looking for highly passionate individuals who love their workand have been champions at what they do.The Incumbent will:• Developing and installing...

  • Software Engineer

    14 hours ago


    Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full time

    Roles & ResponsibilitiesOur company is seeking a game-changer Software Engineer who can conceptualize and execute complex problem statements in less than half the time that most people think is possible.We are looking for highly passionate individuals who love their workand have been champions at what they do.The Incumbent will:• Developing and installing...

  • Software Engineer

    2 days ago


    Singapore THE BOSTON SOFTWARE SOLUTIONS INTERNATIONAL PTE LTD Full time

    Roles & ResponsibilitiesWe are looking for a candidate with the following qualifications and experience:• Experience: At least 8 years of SAP MM and EWM implementation experience • Technical Expertise: Proficiency in EWM scenarios and barcode integration.• Certifications: SAP certification is preferred.• Broad Knowledge: Hands-on experience with data...

  • Software Engineer

    15 hours ago


    Singapore THE BOSTON SOFTWARE SOLUTIONS INTERNATIONAL PTE LTD Full time

    Roles & ResponsibilitiesWe are looking for a candidate with the following qualifications and experience:• Experience: At least 8 years of SAP MM and EWM implementation experience • Technical Expertise: Proficiency in EWM scenarios and barcode integration.• Certifications: SAP certification is preferred.• Broad Knowledge: Hands-on experience with data...


  • Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full time

    JOB DESCRIPTION The incumbent will be responsible for developing and installing robust software solutions. Key responsibilities include designing, implementing, and delivering high-quality software projects using JAVA, Spring Boot, SQL/Oracle, J2EE, Junit, Mockito, JQuery, Javascript, and other JAVA technologies. The individual should possess knowledge of...

  • Software Engineer

    2 days ago


    Singapore Sioux High Tech Software Full time

    Software Engineer (Machine Control – Python)Hybrid Singapore, Central Singapore, Singapore About Sioux Sioux is a strategic high-tech solutions provider that develops, innovates, and assembles complex high-tech systems. We have a team of over 1,200 dedicated engineers who either support our clients or serve as their Research and Development department....


  • Singapore TECHVIFY Software Full time

    Responsibilities Work directly with customers to design and execute automated test plans and test cases to ensure the quality of our software products. Perform automation testing, including functional, regression, integration, and system testing. Collaborate with product managers, developers, and other stakeholders to understand requirements and identify...


  • Singapore Sioux High Tech Software Full time

    About Sioux Sioux is a strategic high-tech solutions provider that develops, innovates and assembles complex high-tech systems. We have a team of over 1,200 dedicated engineers who either support our clients or serve as their Research and Development department. Our strong productivity allows us to speed up product development, giving our clients a...


  • Singapore Preferably Professional Engineer (Electrical); LEW(Grade 9) or Professional Engineer (Mechanical) Full time

    Client: MNC, A1 Main Contractor. Mega Projects Staff Benefits: 17 days annual leave, Year-End Bonus + VB. Salary: Negotiable Based on Qualification, Experience & Capability Location: Central Job Requirements: Degree in Electrical or Mechanical Engineering, recognized by PE Board Singapore/ BCA / IES Min 10 year experience in Mechanical & Electrical services...


  • Singapore Success Software Full time

    Primary Role of Positions Software Engineers work in project groups of the Delivery organization and are responsible for the implementation of software applications Job Duties and Responsibilities Participate in design/code reviews. Coding, implementing, maintaining, and troubleshooting software Aid in creating prototype/test applications. Working with...


  • Singapore Veeam Software Full time

    Veeam, the #1 global market leader in data protection and ransomware recovery, is on a mission to empower every organization to not just bounce back from a data outage or loss but bounce forward. With Veeam, organizations achieve radical resilience through data security, data recovery, and data freedom for their hybrid cloud. The Veeam Data Platform...


  • Singapore Sioux High Tech Software Ltd. Full time

    ABOUT SIOUX Sioux is a strategic high-tech solutions provider that develops, innovates and assembles complex high-tech systems We have a team of over 1,200 dedicated engineers who either support our clients or serve as their Research and Development department. Our strong productivity allows us to speed up product development, giving our clients a...


  • Singapore Sioux High Tech Software Ltd. Full time

    ABOUT SIOUX Sioux is a strategic high-tech solutions provider that develops, innovates, and assembles complex high-tech systems. We have a team of over 1,200 dedicated engineers who either support our clients or serve as their Research and Development department. Our strong productivity allows us to speed up product development, giving our clients a...

  • Software Engineer

    5 days ago


    Singapore Simulation Software & Technology (S2T) Pte Ltd Full time

    S2T is an innovative Tech company that develops cutting-edge cyber-intelligence solutions for Government and Enterprise Customers. We're looking for a talented and passionate Javascript Software Engineer (Full Stack)who is looking for their next challenge to help us bring our vision to life!Responsibilities: Design, develop, and maintain scalable web...


  • Singapore VANGUARD SOFTWARE PTE. LTD. Full time

    Roles & ResponsibilitiesABOUT THE TEAM:QA team is responsible for the quality and efficient delivery of software solutions to our clients, with a wide range of testing scope including functional test through API testing, static analysis, end-to-end test, as well as non-functional testing scope including performance, security, scalability, and...


  • Singapore VANGUARD SOFTWARE PTE. LTD. Full time

    Roles & ResponsibilitiesABOUT THE TEAM:QA team is responsible for the quality and efficient delivery of software solutions to our clients, with a wide range of testing scope including functional test through API testing, static analysis, end-to-end test, as well as non-functional testing scope including performance, security, scalability, and...

  • Project Manager

    2 days ago


    Singapore VANGUARD SOFTWARE PTE. LTD. Full time

    Roles & ResponsibilitiesJOB SUMMARYWe are seeking experienced Project Managers to take ownership of our suite of in-house products, elevate, rejuvenate and relaunch them as our next generation integrated best-of-suite solutions for our clients. As a Project Manager, you will have the opportunity to implement the strategic vision for the product and lead a...