Web Scraping and Senior Data Acquisition Engineer

1 week ago


Singapore Adstify Search Pte Ltd Full time

Our client is an AI-powered research platform. They organize unstructured information in crypto, making it accessible to investors and researchers

**Responsibilities**
- Work closely with co-founding team to define priorities and develop information sourcing roadmaps
- Lead the effort to design and implement the architecture of a large-scale crawling system (100+ crawlers)
- Design, implement, and maintain various components of data acquisition infrastructure (building new crawlers, maintaining existing crawlers, data cleaners & loaders)
- Build pragmatic, scalable, and statistically rigorous solutions to large-scale web and data infrastructure problems by leveraging or developing statistical and machine learning methodologies
- Effectively advocate technical solutions to research, engineering teams and business audiences

**Requirements**:

- Bachelors degree in quantitative field (e.g. Computer Science, Engineering, Mathematics, Statistics, Operations Research or other related field)
- 3+ years of experience with Python for data wrangling and cleaning
- Expertise in running, monitoring and maintaining all aspects of a scraping pipeline end to end (building and maintaining 100+ spiders, avoiding bot prevention techniques, data cleaning and pipelining); familiarity with scraping libraries and monitoring tools highly recommended (BeautifulSoup, Xpaths, Selenium, Puppeteer, Splash)
- Experience in extracting data from multiple disparate sources including HTML, XML, REST, GraphQL, PDF, and spreadsheets
- Experience in using techniques to protect web scrapers against bot detection, site ban, IP leak, browser crash, CAPTCHA and proxy failure
- OOP, SQL and Django ORM basics

**What's next?**

Don Chan
- Senior Consultant_
EA Personnel Number: R1763146
EA License Number: 20C0292

**Salary**: $8,166.00 - $16,550.00 per month

Schedule:

- Monday to Friday

Work Location: One location


  • Web Scrape Engineer

    2 weeks ago


    Singapore Veeva Systems Full time

    Veeva Systems is a mission-driven organization and pioneer in industry cloud, helping life sciences companies bring therapies to patients faster. As one of the fastest-growing SaaS companies in history, we surpassed $2B in revenue in our last fiscal year with extensive growth potential ahead. At the heart of Veeva are our values: Do the Right Thing,...

  • Data Engineer

    5 days ago


    Singapore SIMULATION SOFTWARE & TECHNOLOGY (S2T) PTE. LTD. Full time

    **The Role**: - Develop and maintain efficient scripts to scrape data from social media platforms. - Clean and organize data sets for analysis. - Work with the data science team to implement data models and algorithms. - Continuously monitor the performance of the scraping process and make improvements as necessary. - Reverse engineer undocumented APIs and...

  • Data Engineer

    2 weeks ago


    Singapore Canalasset Full time

    Responsibilities Web Scraping and Data Extraction : Design and implement robust web scraping solutions to collect structured and unstructured data from websites, APIs, and other online sources. Database Management : Design and maintain databases to efficiently store and manage large volumes of scraped data. Implement data storage strategies, indexing, and...

  • Data Engineer

    2 weeks ago


    Singapore PSA Singapore Full time

    We are seeking a skilled Data Engineer to join the Insights, Digitalization & Analytics department. The ideal candidate will design, develop, and maintain scalable data solutions to drive analytics, machine learning, and business insights. Responsibilities include building data pipelines, APIs, and dashboards, deploying Machine Learning projects, and...

  • Data Engineer

    2 weeks ago


    Singapore Cognizant Full time

    Responsibility - Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization. - Perform data extraction, cleaning, transformation, and flow. Web scraping may be also a part of the work scope in data extraction. - Design, build, launch and maintain efficient and reliable large-scale...

  • Data Engineer

    1 week ago


    Singapore PSA Corporation Full time

    We are seeking a skilled Data Engineer to join the Insights, Digitalization & Analytics department. The ideal candidate will design, develop, and maintain scalable data solutions to drive analytics, machine learning, and business insights. Responsibilities include building data pipelines, APIs, and dashboards, deploying Machine Learning projects, and...

  • Data Engineer

    2 weeks ago


    Singapore CDG ZIG PTE. LTD. Full time

    The Engineer, Data should have a proven track record of delivering data pipeline solutions and architecture. He/She should also understand business requirement and able to build reliable data infrastructure using big data technologies. Ideally, you are someone who enjoys optimizing data pipeline, automating and building from scratch. Job Responsibilities :...

  • Data Engineer

    1 day ago


    Singapore Oxford Knight Full time

    Global Investment Manager Singapore New Listing Our client is a leading quant and systematic hedge fund, leveraging their deep knowledge of trading, technology and operations to deliver high-quality, uncorrelated returns. Seeking a Data Engineer to join their Singapore team who has experience collaborating with global colleagues to develop...

  • Senior Data Engineer

    2 weeks ago


    Singapore NITYO 3P SOLUTIONS PTE. LTD. Full time

    Roles & Responsibilities Job Description & Requirements Developing and supporting of data pipelines to process data from various sources Experience in BAU tasks and working closely with data platform users Development of dashboards for multiple stakeholders Designing and developing various ETL/ELT data pipelines, automation processes for data collection...

  • Data Engineer

    6 days ago


    Singapore CDG ZIG PTE. LTD. Full time

    Roles & Responsibilities The Engineer, Data should have a proven track record of delivering data pipeline solutions and architecture. He/She should also understand business requirement and able to build reliable data infrastructure using big data technologies. Ideally, you are someone who enjoys optimizing data pipeline, automating and building from scratch....