DataMiner Scraper is a data extraction tool that lets you scrape any HTML web page. You can extract tables and lists from any page and upload them to Google Sheets or Microsoft Excel. With Scraper you can export web pages into XLS, CSV, XLSX or TSV files (.xls.csv.xlsx.tsv) You can use Dataminer Scraper for FREE in our starter subscription plan. Mkdir /headless-web-scraping. Cd /headless-web-scraping. Env/bin/activate # activate the environment which populates the shell's PATH. So now we can launch a browser, open a page (a tab in chrome) and navigate to a website and wait for javascript to finish loading/executing then close the browser with the above code. Web scraper chrome extension is one of the most powerful tools for extracting web data. Using the extension, you can devise a plan or sitemap regarding how a particular web site of your choice should be navigated. Web scraper chrome extension will, then, follow the navigation design accordingly and scrape the data. Feb 01, 2021 Our web scraping API handles thousands of requests per second without ever being blocked. If you don’t want to lose too much time setting everything up, make sure to try ScrapingBee. The first 1k API calls are on us:). We recently published a guide about the best web scraping tools on the market, don't hesitate to take a look!

1. Open Web Scraper Chrome Web
2. Web Scraper Download
3. Google Web Scraper
4. Open Web Scraper Chrome Browser
5. Google Chrome Web Scraper
6. Web Scraper For Chrome

Web scraping is the best method to gather data from websites. Scraping tools such as Web Scraper help users to scrape websites easily. In this post we will show you how to scrape data using the Web Scraper Chrome Extension.

Prerequisites

Open Web Scraper Chrome Web

• Google Chrome Browser – You will need to download the Chrome browser. The extension requires Chrome 49+.
• Web Scraper Chrome Extension – The Web Scraper extension can be downloaded from the Chrome Web Store. After downloading the extension you will see a spider icon in your browser toolbar.

Creating a Sitemap

After downloading the Web Scraper Chrome extension you’ll find it in developer tools and see a new toolbar added with the name ‘Web Scraper’. Activate the tab and click on ‘Create new sitemap‘, and then ‘Create sitemap‘. Sitemap is the Web Scraper extension name for a scraper. It is a sequence of rules for how to extract data by proceeding from one extraction to the next. We will set the start page as the cellphone category from Amazon.com and click ‘Create Sitemap’. The GIF illustrates how to create a sitemap:

Navigating from root to category pages

Right now, we have the Web Scraper tool open at the _root with an empty list of child selectors

Click ‘Add new selector’. We will add the selector that takes us from the main page to each category page. Let’s give it the id category, with its type as link. We want to fetch multiple links from the root, so we will check the Multiple box below. The ‘Select button’ gives us a tool for visually selecting elements on the page to construct a CSS selector. ‘Element Preview’ highlights the elements on the page and ‘Data Preview’ pops up a sample of the data that would be extracted by the specified selector.

Click select on one of the category links and a specific CSS selector will be filled on the left of the selection tool. Click one of the other (unselected) links and the CSS selector should be adjusted to include it. Keep clicking on the remaining links until all of them are selected. The GIF below shows the whole process on how to add a selector to a sitemap:

A selector graph consists of a collection of selectors – the content to extract, elements within the page and a link to follow and continue the scraping. Each selector has a root (parent selector) defining the context in which the selector is to be applied. This is the visual representation of the final scraper (selector graph) for our Amazon Cellphone Scraper:

Here the root represents the starting URL, the main page for Amazon Cellphone. From there the scraper gets a link to each category page and for each category, it extracts a set of product elements. Each product element, extracts a single name, a single review, a single rating, and a single price. Since there are multiple pages we need the next element of the scraper to go into every page available.

Running the scraper

Click Sitemap to get a drop-down menu and click Scrape as shown below

The scrape pane gives us some options about how slowly Web Scraper should perform its scraping to avoid overloading the web server with requests and to give the web browser time to load pages. We are fine with the defaults, so click ‘Start scraping’. A window will pop up, where the scraper is doing its browsing. After scraping the data you can download it by clicking the option ‘Export data as CSV’ or save it to a database.

Download the Data

To download the scraped data as a CSV file that you can open in Microsoft Excel or Google Sheets, go to the Sitemap drop down > Export as CSV > Download Now.

We can help with your data or automation needs

Turn the Internet into meaningful, structured and usable data

Learn how to scrape webpages using Puppeteer and Serverless Functions built with OpenFaaS.

Introduction to web testing and scraping

In this post I’ll introduce you Puppeteer and show you how to use it to automate and scrape websites using OpenFaaS functions.

There’s two main reasons you may want to automate a web browser:

• to run compliance and end-to-end tests against your application
• to gather information from a webpage which doesn’t have an API available

When testing an application, there are numerous options and these fall into two categories: rendered webpages, running with JavaScript and a real browser, and then text-based tests which can only parse static HTML. As you may imagine, loading a full web-browser in memory is a heavy-weight task. In a previous position I worked heavily with Selenium, which has language bindings for C#, Java, Python, Ruby and other languages. Whilst our team tried to implement most of our tests in the unit-testing layer, there were instances where automated web tests added value, and mean that the QA team could be involved in the development cycle by writing User Acceptance Tests (UATs) before the developers had started coding.

Selenium is still popular in the industry, and it inspired the W3C Working Draft of a Webdriver API that browsers can implement to make testing easier.

The other use-case is not to test websites, but to extract information from them when an API is not available, or does not have the endpoints required. In some instances, you see a mixture of both usecases, for instance - a company may file tax documents through a web-page using automated web-browsers, when that particular jurisdiction doesn’t provide an API.

Kicking the tires with AWS Lambda

I learned more recently of a friend who offers a search for Trademarks through his SaaS product, and for that purpose he chose a more modern alternative to Selenium called Puppeteer. In fact if you search StackOverflow or Google for “scraping and Lambda” you will likely see “Puppeteer” mentioned along with “headless-chrome.” I was curious to try out Puppeteer with AWS Lambda, and the path was less than ideal, with friction at almost every step of the way.

• The popular aws-chrome-lambda npm module is over 40MB in size because it ships a static binary binary, meaning it can’t be uploaded as a regular Lambda zip file, or as a Lambda layer
• The zip file needs to be uploaded through a separate AWS S3 bucket in the same region as the function
• The layer can then be referenced from your function.
• Local testing is very difficult, and there are many StackOverflow issues about getting the right combination of npm modules

I am sure that this can be done, and is being run at scale. It could be quite compelling for small businesses if they don’t spend too much time fighting the above, and can stay within the free-tier.

Getting the title of a simple webpage - 15.5s

That said, OpenFaaS can run anywhere, even on a 5-10 USD VPS and because OpenFaaS uses containers, it got me thinking.

Is there another way?

So I wanted to see if the experience would be any better with OpenFaaS. So I set out to see if I could get Puppeteer working with OpenFaaS, and this isn’t the first time I’ve been there. It’s something that I’ve come back to from time to time. Today, things seem even easier with a pre-compiled headless Chrome browser being available from buildkite.com.

Typical tasks involve logging into a portal and taking screenshots. Anecdotally, when I ran a simple test to navigate to a blog and take a screenshot, this took 15.5s in AWS Lambda, but only 1.6s running locally within OpenFaaS on my laptop. I was also able to build and test the function locally, the same way as in the cloud.

Walkthrough

We’ll now walk through the steps to set up a function with Node.js and Puppeteer, so that you can adapt an example and try out your existing tests that you may have running on AWS Lambda.

OpenFaaS features for web-scraping

What are the features we can leverage from OpenFaaS?

• Extend the function’s timeout to whatever we want
• Run the invocation asynchronously, and in parallel
• Get a HTTP callback with the result when done, such as a screenshot or test result in JSON
• Limit concurrency with max_inflight environment variable in our stack.yml file to prevent overloading the container
• Trigger the invocations from cron, or events like Kafka and NATS
• Get rate, error and duration (RED) metrics from Prometheus, and view them in Grafana

OpenFaaS deployment options

We have made OpenFaaS as easy as possible to deploy on a single VM or on a Kubernetes cluster.

• Deploy to a single VM if you are new to containers and just want to kick the tires whilst keeping costs low. This is also ideal if you only have a few functions, or are worried about needing to learn Kubernetes.

  See also: Bring a lightweight Serverless experience to DigitalOcean with Terraform and faasd

• This is the standard option we recommend for production usage. Through the use of containers and Kubernetes, OpenFaaS can be deployed and run at scale on any cloud.

  Many cloud providers have their own managed Kubernetes services which means it’s trivial to get a working cluster. You just click a button and deploy OpenFaaS, then you can start deploying functions. The DigitalOcean and Linode Kubernetes services are particularly economic.

Deploy Kubernetes and OpenFaaS on your computer

In this post we’ll be running Kubernetes on your laptop, meaning that you don’t have to spend any money on public cloud to start trying things out. The tutorial should take you less than 15-30 minutes to try.

For the impatient, our arkade tool can get you up and running in less than 5 minutes. You’ll just need to have Docker installed on your computer.

The arkade info openfaas command will print out everything you need to log in and get a connection to your OpenFaaS gateway UI.

Create a function with the puppeteer-node12 template

Let’s get the title of a webpage passed in via a JSON HTTP body, then return the result as JSON.

Now edit ./scrape-title/handler.js

Deploy and test the scrape-title function

Deploy the scrape-title function to OpenFaaS.

You can run faas-cli describe FUNCTION to get a synchronous or asynchronous URL for use with curl along with whether the function is ready for invocations. The faas-cli can also be used to invoke functions and we’ll do that below.

Try invoking the function synchronously:

Running with time curl was 10 times faster than my test with AWS Lambda with 256MB RAM allocated.

Alternatively run async:

Run async, post the response to another service like requestbin or another function:

Example of a result posted back to RequestBin

Each invocation has a unique X-Call-Id header, which can be used for tracing and connecting requests to asynchronous responses.

Take a screenshot and return it as a PNG file

One of the limitations of AWS Lambda is that it can only return a JSON response, whilst there may be good reasons for this approach, OpenFaaS allows a binary input and response for functions.

Let’s try taking a screenshot of the page, and capturing it to a file.

Web Scraper Download

Edit ./screenshot-page/handler.js

Now deploy the function as before:

Invoke the function, and capture the response to a file:

Now open screenshot.png and check the result.

Produce homepage banners and social sharing images

You can also produce homepage banners and social sharing images by rendering HTML locally, and then saving a screenshot.

Unlike a SaaS service, you’ll have no month fees to pay, and get unlimited use, you can also customise the code and trigger it however you like.

The execution time is very quick at under 0.5s per image and could be made faster by preloading the Chromium browser and re-using it. if you cache the images to /tmp/ or save them to a CDN, you’ll have single-digit latency.

Edit ./banner-gen/handler.js

Deploy the function:

Example usage:

Note that the inputs are URLEncoded for the querystring. You can also use the event.body if you wish to access the function programmatically, instead of from a browser.

This is an example image generated for my GitHub Sponsors page which uses a different HTML template, that’s loaded from disk.

HTML: sponsor-cta.html

Deploy a Grafana dashboard

We can observe the RED metrics from our functions using the built-in Prometheus UI, or we can deploy Grafana and access the OpenFaaS dashboard.

Access the UI at http://127.0.0.1:3000 and login with admin/admin.

See also: OpenFaaS Metrics

Hardening

If you’d like to limit how many browsers can open at once, you can set max_inflight within the function’s deployment file:

A separate queue can also be configured in OpenFaaS for web-scraping with a set level of parallelism that you prefer.

See also: Async docs

You can also set a hard limit on memory if you wish:

See also: memory limits

Long timeouts

Whilst a timeout value is required, this number can be as large as you like.

See also: Featured Tutorial: Expanded timeouts in OpenFaaS

Getting triggered

If you want to trigger the function periodically, for instance to generate a weekly or daily report, then you can use a cron syntax.

Users of NATS or Kafka can also trigger functions directly from events.

See also: OpenFaaS triggers

Google Web Scraper

Wrapping up

You now have the tools you need to deploy automated tests and web-scraping code using Puppeteer. Since OpenFaaS can leverage Kubernetes, you can use auto-scaling pools of nodes and much longer timeouts than are typically available with cloud-based functions products. OpenFaaS plays well with others such as NATS which powers asynchronous invocations, Prometheus to collect metrics, and Grafana to observe throughput and duration and share the status of the system with others in the team.

Open Web Scraper Chrome Browser

The pre-compiled versions of Chrome included with docker-puppeteer and aws-chrome-lambda will not run on a Raspberry Pi or ARM64 machine, however there is a possibility that they can be rebuilt. For speedy web-scraping from a Raspberry Pi or ARM64 server, you could consider other options such as scrapy.

Google Chrome Web Scraper

Ultimately, I am going to be biased here, but I found the experience of getting Puppeteer to work with OpenFaaS much simpler than with AWS Lambda, and think you should give it a shot.

Web Scraper For Chrome

Find out more: