Web Scraping and Senior Data Acquisition Engineer

2 weeks ago


Singapore Adstify Search Pte Ltd Full time
Our client is an AI-powered research platform. They organize unstructured information in crypto, making it accessible to investors and researchers

Responsibilities

  • Work closely with cofounding team to define priorities and develop information sourcing roadmaps
  • Lead the effort to design and implement the architecture of a largescale crawling system (100+ crawlers)
  • Design, implement, and maintain various components of data acquisition infrastructure (building new crawlers, maintaining existing crawlers, data cleaners & loaders)
  • Build pragmatic, scalable, and statistically rigorous solutions to largescale web and data infrastructure problems by leveraging or developing statistical and machine learning methodologies
  • Effectively advocate technical solutions to research, engineering teams and business audiences

Requirements:

  • Bachelors degree in quantitative field (e.g. Computer Science, Engineering, Mathematics, Statistics, Operations Research or other related field)
  • 3+ years of experience with Python for data wrangling and cleaning
  • Expertise in running, monitoring and maintaining all aspects of a scraping pipeline end to end (building and maintaining 100+ spiders, avoiding bot prevention techniques, data cleaning and pipelining); familiarity with scraping libraries and monitoring tools highly recommended (BeautifulSoup, Xpaths, Selenium, Puppeteer, Splash)
  • Experience in extracting data from multiple disparate sources including HTML, XML, REST, GraphQL, PDF, and spreadsheets
  • Experience in using techniques to protect web scrapers against bot detection, site ban, IP leak, browser crash, CAPTCHA and proxy failure
  • OOP, SQL and Django ORM basics

What's next?
Don Chan

  • Senior Consultant_

EA Personnel Number:
R1763146

EA License Number: 20C0292

Salary:
$8, $16,550.00 per month

Schedule:

  • Monday to Friday

Work Location:
One location
  • Data Engineer

    2 weeks ago


    Singapore SIMULATION SOFTWARE & TECHNOLOGY (S2T) PTE. LTD. Full time

    The Role: Develop and maintain efficient scripts to scrape data from social media platforms. Clean and organize data sets for analysis. Work with the data science team to implement data models and algorithms. Continuously monitor the performance of the scraping process and make improvements as necessary. Reverse engineer undocumented APIs and navigate...

  • Data Engineer

    2 weeks ago


    Singapore Oxford Knight Full time

    Our client is a leading quant and systematic hedge fund, leveraging their deep knowledge of trading, technology and operations to deliver high-quality, uncorrelated returns. Seeking a Data Engineer to join their Singapore team who has experience collaborating with global colleagues to develop industrial-strength software components.As part of the Data...

  • Clojure Developer

    1 month ago


    Singapore 068914 Digital Commerce Intelligence Pte Ltd Full time

    At dci.ai we are building Southeast Asia\xe2\x80\x99s most advanced Digital Commerce Intelligence platform that enables Brands & Retailers to better understand and effectively drive their eCommerce performance. What we do: We are applying advanced Data Engineering & Science techniques to capture digital data points from the web, social and mobile world and...

  • Data Engineer

    2 weeks ago


    Singapore Combined Insurance Full time

    Responsibilities: Responsible for using Data wrangling techniques to convert one 'raw' form into another including data visualization, data aggregation, training a statistical model etc Work with various relational and nonrelational data sources with the target being Azurebased SQL Data Warehouse & Cosmos DB repositories Clean, unify and organize messy and...

  • Senior Data Engineer

    2 weeks ago


    Singapore REGASK PTE. LTD. Full time

    **OUR COMPANYRegASK is an international, diverse and rewarding place to work. If you are a builder, highly autonomous in your working style, and eager to problem solve, we would love to hear from you.**THE POSITION_Title: _Senior Data EngineerRegASK is a digital platform, using artificial intelligence for augmenting regulatory research. Our technology...

  • Web Engineer

    1 month ago


    Singapore TENTEN PARTNERS Full time

    \xe2\x80\x8bWe\'re working with a globally renowned blue-chip insurance firm who have recently signed off over S$72million to revamp and modernize their entire technology infrastructure in Singapore.They are looking for a passionate web and mobile developer who is comfortable with Flutter to join their Product Front End Engineering division. You will help in...

  • Junior Data Analyst

    2 weeks ago


    Singapore CREDIT AGRICOLE CORPORATE AND INVESTMENT BANK Full time

    Roles & ResponsibilitiesPosition AvailableJoin our team as a Junior Data Analyst in the Data & Analytics team at the Digital Excellence Center (DEC).This role offers an exciting opportunity for a skilled team player to contribute to a technology team responsible for a range of solutions across different business sectors of the bank.Main...


  • Singapore Try and Review Full time

    Try & Review is looking for a Traffic & Acquisition Intern to support its performance marketing and CRM strategy.Key Responsibilities In the context of our ongoing product test campaigns, collaborate with our Campaign Managers to schedule, execute and optimize acquisition campaigns to guarantee the achievement of our clients' KPIs Build and execute reporting...

  • Clojure Developer

    1 month ago


    Singapore Digital Commerce Intelligence Pte Ltd Full time

    At we are building Southeast Asia\xe2\x80\x99s most advanced Digital Commerce Intelligence platform that enables Brands & Retailers to better understand and effectively drive their eCommerce performance. What we do: We are applying advanced Data Engineering & Science techniques to capture digital data points from the web, social and mobile world and use...


  • Singapore Singtel Full time

    Associate Data Analyst - #catalystWSP:Date:06-Feb-2023Location: Singapore, SingaporeCompany:Singtel GroupResponsibilities:As a Data Analyst, you will play a crucial role in extracting valuable insights from large sets of data and driving business decisions. You will be responsible for retrieving data from various sources such as MSSQL and Oracle databases,...

  • Senior Web Developer

    1 month ago


    Singapore A PLUS MANPOWER SERVICES PTE. LTD. Full time

    Annual Remuneration Package: SGD 60K-70K Location: City Hall Working Hours: Monday-Friday, 9:00am to 6:00pmJob Highlights: Flexi work arrangement, Flexi health benefits, Annual company trip, other staff benefitss Collaborate closely with UX/UI designer to create highly interactive and user-friendly web applications Prepare test plans, and conduct SIT and UAT...

  • Senior Web Developer

    1 month ago


    Singapore A PLUS MANPOWER SERVICES PTE. LTD. Full time

    Annual Remuneration Package: SGD 60K-70K Location: City Hall Working Hours: Monday-Friday, 9:00am to 6:00pmJob Highlights: Flexi work arrangement, Flexi health benefits, Annual company trip, other staff benefitss Collaborate closely with UX/UI designer to create highly interactive and user-friendly web applications Prepare test plans, and conduct SIT...

  • Senior Data Engineer

    2 weeks ago


    Singapore NodeFlair Full time

    Job Summary:SalaryS$7,000 - S$10,000 / MonthlyJob TypeSenioritySeniorYears of ExperienceAt least 7 yearsTech StacksHANA ETL Snowflake AWS Data Extraction Analytics Teradata CI R RedShift Java Scala PythonThe Data and AI teams at TE are part of the TE Information Solutions (TEIS) Organization and Corporate Technology and are responsible for driving organic...


  • Singapore Al-Futtaim Full time

    We are seeking a dynamic and strategic Senior Talent Acquisition Specialist to join our Retail team. As a Senior Talent Acquisition Specialist, you will be responsible for driving the end-to-end recruitment process, developing talent acquisition strategies, and partnering with hiring managers to fulfill staffing needs. You will play a crucial role in...

  • Data Engineer

    2 weeks ago


    Singapore NodeFlair Full time

    Job Summary:SalaryS$4,000 - S$7,000 / MonthlyJob TypeSeniorityMidYears of ExperienceAt least 5 yearsTech StacksETL Analytics Hive Impala Spark kafka SQL Hadoop PythonRequirement 5+ years of relevant experience in data engineering/analytics space Expertise in SQL and data analysis and strong handson expertise with at least one programming language: Python....

  • Data Engineer

    2 weeks ago


    Singapore DOZER DATA PTE. LTD. Full time

    About the job:This position is needed to build a highly scalable, reliable, and efficient data platform that makes it easy for users to get deep insights from vast amounts of distributed data.This role will be responsible for driving key technical decisions to completion for the organization and also guiding other engineers.Responsibilities:In this role,...


  • Singapore Best Careers and Opportunities Full time

    JOB LOCATION: SINGAPOREJOB TITLE: TALENT ACQUISITION SPECIALISTJob Description- The Talent Acquisition Specialist will support one of our European Banking client in Singapore. In this role, the successful incumbent will ensure that we are delivering the highest levels of customer satisfaction, and it will involve liaising with client on a day-to-day basis as...


  • Singapore Best Careers and Opportunities Full time

    JOB LOCATION: SINGAPORE JOB TITLE: TALENT ACQUISITION SPECIALIST Job Description - The Talent Acquisition Specialist will support one of our European Banking client in Singapore. In this role, the successful incumbent will ensure that we are delivering the highest levels of customer satisfaction, and it will involve liaising with client on a day-to-day...

  • Senior Data Engineer

    1 month ago


    Singapore Xylem Full time

    Key ResponsibilitiesWork closely with Product Owners, Architects and other team members in Scrum team to achieve common goalsCollaborate with Engineers and Scientist on innovative analytical featuresDesign & implement a modern ELT and custom data integrationsAnalyse existing data sets and future business needs to design and implement data management...

  • Web Marketer

    2 weeks ago


    Singapore Info-communications Media Development Authority Full time

    What the role isWhat you will be working on Search Engine Optimisation (SEO)Manage all SEO activities (i.e., on-page, off-page and technical SEO implementation) to improve and maintain favourable SERP rankings and drive organic traffic to the websiteEnsure website content adheres to best SEO practices and maintenance of meta data hygiene (i.e., meta data...