Web Scraping and Senior Data Acquisition Engineer
2 weeks ago
Responsibilities
- Work closely with cofounding team to define priorities and develop information sourcing roadmaps
- Lead the effort to design and implement the architecture of a largescale crawling system (100+ crawlers)
- Design, implement, and maintain various components of data acquisition infrastructure (building new crawlers, maintaining existing crawlers, data cleaners & loaders)
- Build pragmatic, scalable, and statistically rigorous solutions to largescale web and data infrastructure problems by leveraging or developing statistical and machine learning methodologies
- Effectively advocate technical solutions to research, engineering teams and business audiences
Requirements:
- Bachelors degree in quantitative field (e.g. Computer Science, Engineering, Mathematics, Statistics, Operations Research or other related field)
- 3+ years of experience with Python for data wrangling and cleaning
- Expertise in running, monitoring and maintaining all aspects of a scraping pipeline end to end (building and maintaining 100+ spiders, avoiding bot prevention techniques, data cleaning and pipelining); familiarity with scraping libraries and monitoring tools highly recommended (BeautifulSoup, Xpaths, Selenium, Puppeteer, Splash)
- Experience in extracting data from multiple disparate sources including HTML, XML, REST, GraphQL, PDF, and spreadsheets
- Experience in using techniques to protect web scrapers against bot detection, site ban, IP leak, browser crash, CAPTCHA and proxy failure
- OOP, SQL and Django ORM basics
What's next?
Don Chan
- Senior Consultant_
EA Personnel Number:
R1763146
EA License Number: 20C0292
Salary:
$8, $16,550.00 per month
Schedule:
- Monday to Friday
Work Location:
One location
-
Data Engineer
2 weeks ago
Singapore SIMULATION SOFTWARE & TECHNOLOGY (S2T) PTE. LTD. Full timeThe Role: Develop and maintain efficient scripts to scrape data from social media platforms. Clean and organize data sets for analysis. Work with the data science team to implement data models and algorithms. Continuously monitor the performance of the scraping process and make improvements as necessary. Reverse engineer undocumented APIs and navigate...
-
Data Engineer
2 weeks ago
Singapore Oxford Knight Full timeOur client is a leading quant and systematic hedge fund, leveraging their deep knowledge of trading, technology and operations to deliver high-quality, uncorrelated returns. Seeking a Data Engineer to join their Singapore team who has experience collaborating with global colleagues to develop industrial-strength software components.As part of the Data...
-
Clojure Developer
1 month ago
Singapore 068914 Digital Commerce Intelligence Pte Ltd Full timeAt dci.ai we are building Southeast Asia\xe2\x80\x99s most advanced Digital Commerce Intelligence platform that enables Brands & Retailers to better understand and effectively drive their eCommerce performance. What we do: We are applying advanced Data Engineering & Science techniques to capture digital data points from the web, social and mobile world and...
-
Data Engineer
2 weeks ago
Singapore Combined Insurance Full timeResponsibilities: Responsible for using Data wrangling techniques to convert one 'raw' form into another including data visualization, data aggregation, training a statistical model etc Work with various relational and nonrelational data sources with the target being Azurebased SQL Data Warehouse & Cosmos DB repositories Clean, unify and organize messy and...
-
Senior Data Engineer
2 weeks ago
Singapore REGASK PTE. LTD. Full time**OUR COMPANYRegASK is an international, diverse and rewarding place to work. If you are a builder, highly autonomous in your working style, and eager to problem solve, we would love to hear from you.**THE POSITION_Title: _Senior Data EngineerRegASK is a digital platform, using artificial intelligence for augmenting regulatory research. Our technology...
-
Web Engineer
1 month ago
Singapore TENTEN PARTNERS Full time\xe2\x80\x8bWe\'re working with a globally renowned blue-chip insurance firm who have recently signed off over S$72million to revamp and modernize their entire technology infrastructure in Singapore.They are looking for a passionate web and mobile developer who is comfortable with Flutter to join their Product Front End Engineering division. You will help in...
-
Junior Data Analyst
2 weeks ago
Singapore CREDIT AGRICOLE CORPORATE AND INVESTMENT BANK Full timeRoles & ResponsibilitiesPosition AvailableJoin our team as a Junior Data Analyst in the Data & Analytics team at the Digital Excellence Center (DEC).This role offers an exciting opportunity for a skilled team player to contribute to a technology team responsible for a range of solutions across different business sectors of the bank.Main...
-
Traffic & Acquisition Intern
2 weeks ago
Singapore Try and Review Full timeTry & Review is looking for a Traffic & Acquisition Intern to support its performance marketing and CRM strategy.Key Responsibilities In the context of our ongoing product test campaigns, collaborate with our Campaign Managers to schedule, execute and optimize acquisition campaigns to guarantee the achievement of our clients' KPIs Build and execute reporting...
-
Clojure Developer
1 month ago
Singapore Digital Commerce Intelligence Pte Ltd Full timeAt we are building Southeast Asia\xe2\x80\x99s most advanced Digital Commerce Intelligence platform that enables Brands & Retailers to better understand and effectively drive their eCommerce performance. What we do: We are applying advanced Data Engineering & Science techniques to capture digital data points from the web, social and mobile world and use...
-
Associate Data Analyst
2 weeks ago
Singapore Singtel Full timeAssociate Data Analyst - #catalystWSP:Date:06-Feb-2023Location: Singapore, SingaporeCompany:Singtel GroupResponsibilities:As a Data Analyst, you will play a crucial role in extracting valuable insights from large sets of data and driving business decisions. You will be responsible for retrieving data from various sources such as MSSQL and Oracle databases,...
-
Senior Web Developer
1 month ago
Singapore A PLUS MANPOWER SERVICES PTE. LTD. Full timeAnnual Remuneration Package: SGD 60K-70K Location: City Hall Working Hours: Monday-Friday, 9:00am to 6:00pmJob Highlights: Flexi work arrangement, Flexi health benefits, Annual company trip, other staff benefitss Collaborate closely with UX/UI designer to create highly interactive and user-friendly web applications Prepare test plans, and conduct SIT and UAT...
-
Senior Web Developer
1 month ago
Singapore A PLUS MANPOWER SERVICES PTE. LTD. Full timeAnnual Remuneration Package: SGD 60K-70K Location: City Hall Working Hours: Monday-Friday, 9:00am to 6:00pmJob Highlights: Flexi work arrangement, Flexi health benefits, Annual company trip, other staff benefitss Collaborate closely with UX/UI designer to create highly interactive and user-friendly web applications Prepare test plans, and conduct SIT...
-
Senior Data Engineer
2 weeks ago
Singapore NodeFlair Full timeJob Summary:SalaryS$7,000 - S$10,000 / MonthlyJob TypeSenioritySeniorYears of ExperienceAt least 7 yearsTech StacksHANA ETL Snowflake AWS Data Extraction Analytics Teradata CI R RedShift Java Scala PythonThe Data and AI teams at TE are part of the TE Information Solutions (TEIS) Organization and Corporate Technology and are responsible for driving organic...
-
Senior Talent Acquisition Specialist
2 weeks ago
Singapore Al-Futtaim Full timeWe are seeking a dynamic and strategic Senior Talent Acquisition Specialist to join our Retail team. As a Senior Talent Acquisition Specialist, you will be responsible for driving the end-to-end recruitment process, developing talent acquisition strategies, and partnering with hiring managers to fulfill staffing needs. You will play a crucial role in...
-
Data Engineer
2 weeks ago
Singapore NodeFlair Full timeJob Summary:SalaryS$4,000 - S$7,000 / MonthlyJob TypeSeniorityMidYears of ExperienceAt least 5 yearsTech StacksETL Analytics Hive Impala Spark kafka SQL Hadoop PythonRequirement 5+ years of relevant experience in data engineering/analytics space Expertise in SQL and data analysis and strong handson expertise with at least one programming language: Python....
-
Data Engineer
2 weeks ago
Singapore DOZER DATA PTE. LTD. Full timeAbout the job:This position is needed to build a highly scalable, reliable, and efficient data platform that makes it easy for users to get deep insights from vast amounts of distributed data.This role will be responsible for driving key technical decisions to completion for the organization and also guiding other engineers.Responsibilities:In this role,...
-
Talent Acquisition Specialist
1 month ago
Singapore Best Careers and Opportunities Full timeJOB LOCATION: SINGAPOREJOB TITLE: TALENT ACQUISITION SPECIALISTJob Description- The Talent Acquisition Specialist will support one of our European Banking client in Singapore. In this role, the successful incumbent will ensure that we are delivering the highest levels of customer satisfaction, and it will involve liaising with client on a day-to-day basis as...
-
Talent Acquisition Specialist
1 month ago
Singapore Best Careers and Opportunities Full timeJOB LOCATION: SINGAPORE JOB TITLE: TALENT ACQUISITION SPECIALIST Job Description - The Talent Acquisition Specialist will support one of our European Banking client in Singapore. In this role, the successful incumbent will ensure that we are delivering the highest levels of customer satisfaction, and it will involve liaising with client on a day-to-day...
-
Senior Data Engineer
1 month ago
Singapore Xylem Full timeKey ResponsibilitiesWork closely with Product Owners, Architects and other team members in Scrum team to achieve common goalsCollaborate with Engineers and Scientist on innovative analytical featuresDesign & implement a modern ELT and custom data integrationsAnalyse existing data sets and future business needs to design and implement data management...
-
Web Marketer
2 weeks ago
Singapore Info-communications Media Development Authority Full timeWhat the role isWhat you will be working on Search Engine Optimisation (SEO)Manage all SEO activities (i.e., on-page, off-page and technical SEO implementation) to improve and maintain favourable SERP rankings and drive organic traffic to the websiteEnsure website content adheres to best SEO practices and maintenance of meta data hygiene (i.e., meta data...