Scrapling

What happened

The plain fetching is excellent. Twenty books in 0.4 seconds, thirty quotes across three pages in under a second. On the JavaScript-only page, a normal HTTP request saw 0 quotes; the built-in browser saw all 10 in 3 seconds.

The headline feature is “adaptive” scraping: save what an element looks like, and when the site is redesigned, find it again without new selectors. We built two versions of the same shop page. Version 2 renamed every class and changed the structure. With the old selectors, a normal scraper finds 0 of 15 fields on version 2.

Scrapling found 10 of 15:

• Product names: 5 of 5.
• Prices: 5 of 5, even though “$89.00” became “USD 89.00”.
• Ratings: 0 of 5. Four times it said it found nothing above its 40% similarity threshold, which is fine. Once it returned a wrong value without any warning: another product’s price instead of the rating.

That last one is the reason for “use with care”. A scraper that says “not found” is easy to handle. A scraper that quietly returns the wrong number ends up in your spreadsheet.

Two things to know before you install it

• Adaptive data location. By default it saves its adaptive data next to the installed package. As a normal (non-root) user that folder isn’t writable and you get a database error on the first request. Point it somewhere writable with storage_args={"storage_file": "..."}.
• Version 0.5 renamed things. Older tutorials use DynamicFetcher; in 0.5.x the browser fetcher is StealthyFetcher.

Who should use it

Use it for fetching and parsing, including JavaScript pages. Use adaptive tracking as a helper, not as truth: check the values it relocates before you trust them. We did not test the bot-protection bypass, and we won’t on sites we don’t own.