Web Scraping and Senior Data Acquisition Engineer
1 week ago
Our client is an AI-powered research platform. They organize unstructured information in crypto, making it accessible to investors and researchers
**Responsibilities**
- Work closely with co-founding team to define priorities and develop information sourcing roadmaps
- Lead the effort to design and implement the architecture of a large-scale crawling system (100+ crawlers)
- Design, implement, and maintain various components of data acquisition infrastructure (building new crawlers, maintaining existing crawlers, data cleaners & loaders)
- Build pragmatic, scalable, and statistically rigorous solutions to large-scale web and data infrastructure problems by leveraging or developing statistical and machine learning methodologies
- Effectively advocate technical solutions to research, engineering teams and business audiences
**Requirements**:
- Bachelors degree in quantitative field (e.g. Computer Science, Engineering, Mathematics, Statistics, Operations Research or other related field)
- 3+ years of experience with Python for data wrangling and cleaning
- Expertise in running, monitoring and maintaining all aspects of a scraping pipeline end to end (building and maintaining 100+ spiders, avoiding bot prevention techniques, data cleaning and pipelining); familiarity with scraping libraries and monitoring tools highly recommended (BeautifulSoup, Xpaths, Selenium, Puppeteer, Splash)
- Experience in extracting data from multiple disparate sources including HTML, XML, REST, GraphQL, PDF, and spreadsheets
- Experience in using techniques to protect web scrapers against bot detection, site ban, IP leak, browser crash, CAPTCHA and proxy failure
- OOP, SQL and Django ORM basics
**What's next?**
Don Chan
- Senior Consultant_
EA Personnel Number: R1763146
EA License Number: 20C0292
**Salary**: $8,166.00 - $16,550.00 per month
Schedule:
- Monday to Friday
Work Location: One location
-
Web Scrape Engineer
2 weeks ago
Singapore Veeva Systems Full timeVeeva Systems is a mission-driven organization and pioneer in industry cloud, helping life sciences companies bring therapies to patients faster. As one of the fastest-growing SaaS companies in history, we surpassed $2B in revenue in our last fiscal year with extensive growth potential ahead. At the heart of Veeva are our values: Do the Right Thing,...
-
Data Engineer
5 days ago
Singapore SIMULATION SOFTWARE & TECHNOLOGY (S2T) PTE. LTD. Full time**The Role**: - Develop and maintain efficient scripts to scrape data from social media platforms. - Clean and organize data sets for analysis. - Work with the data science team to implement data models and algorithms. - Continuously monitor the performance of the scraping process and make improvements as necessary. - Reverse engineer undocumented APIs and...
-
Data Engineer
2 weeks ago
Singapore Canalasset Full timeResponsibilities Web Scraping and Data Extraction : Design and implement robust web scraping solutions to collect structured and unstructured data from websites, APIs, and other online sources. Database Management : Design and maintain databases to efficiently store and manage large volumes of scraped data. Implement data storage strategies, indexing, and...
-
Data Engineer
2 weeks ago
Singapore PSA Singapore Full timeWe are seeking a skilled Data Engineer to join the Insights, Digitalization & Analytics department. The ideal candidate will design, develop, and maintain scalable data solutions to drive analytics, machine learning, and business insights. Responsibilities include building data pipelines, APIs, and dashboards, deploying Machine Learning projects, and...
-
Data Engineer
2 weeks ago
Singapore Cognizant Full timeResponsibility - Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization. - Perform data extraction, cleaning, transformation, and flow. Web scraping may be also a part of the work scope in data extraction. - Design, build, launch and maintain efficient and reliable large-scale...
-
Data Engineer
1 week ago
Singapore PSA Corporation Full timeWe are seeking a skilled Data Engineer to join the Insights, Digitalization & Analytics department. The ideal candidate will design, develop, and maintain scalable data solutions to drive analytics, machine learning, and business insights. Responsibilities include building data pipelines, APIs, and dashboards, deploying Machine Learning projects, and...
-
Data Engineer
2 weeks ago
Singapore CDG ZIG PTE. LTD. Full timeThe Engineer, Data should have a proven track record of delivering data pipeline solutions and architecture. He/She should also understand business requirement and able to build reliable data infrastructure using big data technologies. Ideally, you are someone who enjoys optimizing data pipeline, automating and building from scratch. Job Responsibilities :...
-
Data Engineer
1 day ago
Singapore Oxford Knight Full timeGlobal Investment Manager Singapore New Listing Our client is a leading quant and systematic hedge fund, leveraging their deep knowledge of trading, technology and operations to deliver high-quality, uncorrelated returns. Seeking a Data Engineer to join their Singapore team who has experience collaborating with global colleagues to develop...
-
Senior Data Engineer
2 weeks ago
Singapore NITYO 3P SOLUTIONS PTE. LTD. Full timeRoles & Responsibilities Job Description & Requirements Developing and supporting of data pipelines to process data from various sources Experience in BAU tasks and working closely with data platform users Development of dashboards for multiple stakeholders Designing and developing various ETL/ELT data pipelines, automation processes for data collection...
-
Data Engineer
6 days ago
Singapore CDG ZIG PTE. LTD. Full timeRoles & Responsibilities The Engineer, Data should have a proven track record of delivering data pipeline solutions and architecture. He/She should also understand business requirement and able to build reliable data infrastructure using big data technologies. Ideally, you are someone who enjoys optimizing data pipeline, automating and building from scratch....