Software Engineer
2 days ago
We are seeking a highly motivated and experienced WebCrawler Engineer to develop exciting Generative AI (GenAI) products. Candidates should have at least 3 to 5 years web crawler experience, and experience in multimodal (text, image, video) data is preferred.As a WebCrawler engineer, you will work with other teams to ensure the smooth progress of product iterations, continuously improve the user experience of the product through in-depth understanding of the business and products, and use technology to drive business growth. Your expertise will play a key role to build our products and achieve our vision.
Responsibilities
Designing and Implementing Web Crawlers: Develop scalable and efficient web crawling systems to gather data from various online sources. This involves understanding the structure of different websites, implementing crawling algorithms, handling dynamic content, and ensuring compliance with legal and ethical guidelines.
Data Extraction and Parsing: Write scripts or develop algorithms to extract relevant information from web pages. This may involve parsing HTML/XML documents, using regular expressions, or employing advanced parsing techniques such as natural language processing (NLP) to extract structured data from unstructured sources.
Data Quality Assurance: Implement mechanisms to ensure the quality and reliability of crawled data. This includes error handling, data validation, deduplication, and dealing with inconsistencies or missing data.
Scalable Data Storage and Management: Design and develop backend systems to store, organize, and manage large volumes of crawled data efficiently. This may involve selecting appropriate databases (e.g., relational databases, NoSQL databases), optimizing database schemas, and implementing data caching and indexing strategies for faster retrieval.
Performance Optimization: Optimize the performance of web crawling and data management systems to handle large-scale data processing efficiently. This includes optimizing algorithms, minimizing resource usage, and parallelizing data processing tasks.
Monitoring and Maintenance: Implement monitoring tools and logging mechanisms to track the health and performance of web crawling and data management systems. Proactively identify and resolve issues such as crawling failures, performance bottlenecks, or data inconsistencies.
Security and Compliance: Ensure that the web crawling and data management systems adhere to security best practices and regulatory requirements. Implement mechanisms to protect against security threats such as XSS (Cross-Site Scripting) attacks, CSRF (Cross-Site Request Forgery) attacks, and data breaches.
Qualifications
Possess a Bachelor's degree or above in Computer Science, Software Engineering, Technical, Science, E-Commerce, Information Technology, Mathematics or software related majors.
More than 3 years of web crawler experience, experience in large-scale web crawling and multimodal (text, image, video) data is preferred.
Proficient in one or more programming languages such as Golang/Python/PHP/Java, with strong architectural capabilities and good coding standards.
Familiar with common databases, such as MySQL, Redis, HBase, etc.
Experience in web crawling libraries like Scrapy, Beautiful Soup, Selenium, Apache Nutch.
Prior experience in Search engine companies is a plus.
#J-18808-Ljbffr
-
SOFTWARE ENGINEER
2 days ago
Singapore CLOUDPILOTS Software & Consulting GmbH Full timeSOFTWARE ENGINEER CLOUD CLOUDPILOTS | Location Berlin, Cologne or Home Office As a Software Engineer* Cloud, you act as a technical contact person and analyze existing systems, define migration strategies, and take over the setup and customization of the cloud environment for our customers. You help to master technical and organizational challenges when...
-
Senior Software Engineer
2 days ago
Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full timeRoles & ResponsibilitiesJOB DESCRIPTION· The incumbent will be responsible for developing and installing robust software solutions.· Key responsibilities include designing, implementing, and delivering high-quality software projects using JAVA, Spring Boot, SQL/Oracle, J2EE, Junit, Mockito, JQuery, Javascript, and other JAVA technologies.· The...
-
Senior Software Engineer
15 hours ago
Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full timeRoles & ResponsibilitiesJOB DESCRIPTION· The incumbent will be responsible for developing and installing robust software solutions.· Key responsibilities include designing, implementing, and delivering high-quality software projects using JAVA, Spring Boot, SQL/Oracle, J2EE, Junit, Mockito, JQuery, Javascript, and other JAVA technologies.· The...
-
Software Engineer
2 days ago
Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full timeRoles & ResponsibilitiesOur company is seeking a game-changer Software Engineer who can conceptualize and execute complex problem statements in less than half the time that most people think is possible.We are looking for highly passionate individuals who love their workand have been champions at what they do.The Incumbent will:• Developing and installing...
-
Software Engineer
14 hours ago
Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full timeRoles & ResponsibilitiesOur company is seeking a game-changer Software Engineer who can conceptualize and execute complex problem statements in less than half the time that most people think is possible.We are looking for highly passionate individuals who love their workand have been champions at what they do.The Incumbent will:• Developing and installing...
-
Software Engineer
2 days ago
Singapore THE BOSTON SOFTWARE SOLUTIONS INTERNATIONAL PTE LTD Full timeRoles & ResponsibilitiesWe are looking for a candidate with the following qualifications and experience:• Experience: At least 8 years of SAP MM and EWM implementation experience • Technical Expertise: Proficiency in EWM scenarios and barcode integration.• Certifications: SAP certification is preferred.• Broad Knowledge: Hands-on experience with data...
-
Software Engineer
15 hours ago
Singapore THE BOSTON SOFTWARE SOLUTIONS INTERNATIONAL PTE LTD Full timeRoles & ResponsibilitiesWe are looking for a candidate with the following qualifications and experience:• Experience: At least 8 years of SAP MM and EWM implementation experience • Technical Expertise: Proficiency in EWM scenarios and barcode integration.• Certifications: SAP certification is preferred.• Broad Knowledge: Hands-on experience with data...
-
Singapore NEWGEN SOFTWARE TECHNOLOGIES PTE. LTD. Full timeJOB DESCRIPTION The incumbent will be responsible for developing and installing robust software solutions. Key responsibilities include designing, implementing, and delivering high-quality software projects using JAVA, Spring Boot, SQL/Oracle, J2EE, Junit, Mockito, JQuery, Javascript, and other JAVA technologies. The individual should possess knowledge of...
-
Software Engineer
2 days ago
Singapore Sioux High Tech Software Full timeSoftware Engineer (Machine Control – Python)Hybrid Singapore, Central Singapore, Singapore About Sioux Sioux is a strategic high-tech solutions provider that develops, innovates, and assembles complex high-tech systems. We have a team of over 1,200 dedicated engineers who either support our clients or serve as their Research and Development department....
-
Singapore TECHVIFY Software Full timeResponsibilities Work directly with customers to design and execute automated test plans and test cases to ensure the quality of our software products. Perform automation testing, including functional, regression, integration, and system testing. Collaborate with product managers, developers, and other stakeholders to understand requirements and identify...
-
Sioux High Tech Software | Software Engineer
2 days ago
Singapore Sioux High Tech Software Full timeAbout Sioux Sioux is a strategic high-tech solutions provider that develops, innovates and assembles complex high-tech systems. We have a team of over 1,200 dedicated engineers who either support our clients or serve as their Research and Development department. Our strong productivity allows us to speed up product development, giving our clients a...
-
Preferably Professional Engineer
2 days ago
Singapore Preferably Professional Engineer (Electrical); LEW(Grade 9) or Professional Engineer (Mechanical) Full timeClient: MNC, A1 Main Contractor. Mega Projects Staff Benefits: 17 days annual leave, Year-End Bonus + VB. Salary: Negotiable Based on Qualification, Experience & Capability Location: Central Job Requirements: Degree in Electrical or Mechanical Engineering, recognized by PE Board Singapore/ BCA / IES Min 10 year experience in Mechanical & Electrical services...
-
Success Software | SENIOR FRONTEND DEVELOPER
2 days ago
Singapore Success Software Full timePrimary Role of Positions Software Engineers work in project groups of the Delivery organization and are responsible for the implementation of software applications Job Duties and Responsibilities Participate in design/code reviews. Coding, implementing, maintaining, and troubleshooting software Aid in creating prototype/test applications. Working with...
-
Veeam Software | Senior Territory Manager
2 days ago
Singapore Veeam Software Full timeVeeam, the #1 global market leader in data protection and ransomware recovery, is on a mission to empower every organization to not just bounce back from a data outage or loss but bounce forward. With Veeam, organizations achieve radical resilience through data security, data recovery, and data freedom for their hybrid cloud. The Veeam Data Platform...
-
Singapore Sioux High Tech Software Ltd. Full timeABOUT SIOUX Sioux is a strategic high-tech solutions provider that develops, innovates and assembles complex high-tech systems We have a team of over 1,200 dedicated engineers who either support our clients or serve as their Research and Development department. Our strong productivity allows us to speed up product development, giving our clients a...
-
Singapore Sioux High Tech Software Ltd. Full timeABOUT SIOUX Sioux is a strategic high-tech solutions provider that develops, innovates, and assembles complex high-tech systems. We have a team of over 1,200 dedicated engineers who either support our clients or serve as their Research and Development department. Our strong productivity allows us to speed up product development, giving our clients a...
-
Software Engineer
5 days ago
Singapore Simulation Software & Technology (S2T) Pte Ltd Full timeS2T is an innovative Tech company that develops cutting-edge cyber-intelligence solutions for Government and Enterprise Customers. We're looking for a talented and passionate Javascript Software Engineer (Full Stack)who is looking for their next challenge to help us bring our vision to life!Responsibilities: Design, develop, and maintain scalable web...
-
IT Senior Software Tester
2 days ago
Singapore VANGUARD SOFTWARE PTE. LTD. Full timeRoles & ResponsibilitiesABOUT THE TEAM:QA team is responsible for the quality and efficient delivery of software solutions to our clients, with a wide range of testing scope including functional test through API testing, static analysis, end-to-end test, as well as non-functional testing scope including performance, security, scalability, and...
-
IT Senior Software Tester
16 hours ago
Singapore VANGUARD SOFTWARE PTE. LTD. Full timeRoles & ResponsibilitiesABOUT THE TEAM:QA team is responsible for the quality and efficient delivery of software solutions to our clients, with a wide range of testing scope including functional test through API testing, static analysis, end-to-end test, as well as non-functional testing scope including performance, security, scalability, and...
-
Project Manager
2 days ago
Singapore VANGUARD SOFTWARE PTE. LTD. Full timeRoles & ResponsibilitiesJOB SUMMARYWe are seeking experienced Project Managers to take ownership of our suite of in-house products, elevate, rejuvenate and relaunch them as our next generation integrated best-of-suite solutions for our clients. As a Project Manager, you will have the opportunity to implement the strategic vision for the product and lead a...