Get it to work with browserless

[Boss-man]

Has anyone got Hoarder to work with browserless.io ?

I run browserless.io in docker on IP:4000, and Hoarder in LXc on Proxmox.
I edited the.env file to point browser_websockert_url to my browserless instance with ws:\IP:40

In the log on browserless I see Hoarder connect and it starts op a Chrome session, but on Hoarder the logs repeatedly says :

[Crawler] The puppeteer browser got disconnected. Will attempt to launch it again.

What do I do wrong ?

[MohamedBassem]

Has anyone got Hoarder to work with browserless.io ?

Almost everyone using unraid are using the browserless container.

What do I do wrong ?

Are you just annoyed from the log line, or is crawling actually broken?

[Boss-man]

Are you just annoyed from the log line, or is crawling actually broken?

Annoyed that it constantly tries to spin up a chrome session as it gets disconnected or not getting a response back.

I see in browser.io logs, that it actually do some crawling, but Hoarder never gets satisfied, and just retries many times.
I do have my own puppeter script using the same browser.io instance and that works.

I probably just need to get the right config set up. But unsure what to change.

[MohamedBassem]

Can you show me the logs?

[Boss-man]

This is from Hoarder

Jan 12 19:54:55 hoarder pnpm[295]: 2025-01-12T18:54:55.403Z info: [Crawler][581] Will crawl "https://travel.rakuten.com/contents/usa/en-us/guide/things-to-do-shinjuku/" for link with id "tgynf8x1kbxo1lr2tgvsiq5b"
Jan 12 19:54:55 hoarder pnpm[295]: 2025-01-12T18:54:55.403Z info: [Crawler][581] Attempting to determine the content-type for the url https://travel.rakuten.com/contents/usa/en-us/guide/things-to-do-shinjuku/
Jan 12 19:55:00 hoarder pnpm[295]: 2025-01-12T18:55:00.405Z error: [Crawler][581] Failed to determine the content-type for the url https://travel.rakuten.com/contents/usa/en-us/guide/things-to-do-shinjuku/: AbortError: The operation was aborted.
Jan 12 19:55:00 hoarder pnpm[295]: 2025-01-12T18:55:00.405Z info: [Crawler] Connecting to existing browser websocket address: ws://192.168.1.97:3000
Jan 12 19:55:01 hoarder pnpm[295]: 2025-01-12T18:55:01.699Z info: [Crawler][581] Successfully navigated to "https://travel.rakuten.com/contents/usa/en-us/guide/things-to-do-shinjuku/". Waiting for the page to load ...
Jan 12 19:55:06 hoarder pnpm[295]: 2025-01-12T18:55:06.387Z info: [Crawler][581] Finished waiting for the page to load.
Jan 12 19:55:06 hoarder pnpm[295]: 2025-01-12T18:55:06.404Z info: [Crawler][581] Successfully fetched the page content.
Jan 12 19:55:06 hoarder pnpm[295]: 2025-01-12T18:55:06.571Z info: [Crawler][581] Finished capturing page content and a screenshot. FullPageScreenshot: false
Jan 12 19:55:06 hoarder pnpm[295]: 2025-01-12T18:55:06.580Z info: [Crawler][581] Will attempt to extract metadata from page ...
Jan 12 19:55:06 hoarder pnpm[295]: 2025-01-12T18:55:06.679Z info: [Crawler][581] Will attempt to extract readable content ...
Jan 12 19:55:06 hoarder pnpm[295]: 2025-01-12T18:55:06.939Z info: [Crawler][581] Done extracting readable content.
Jan 12 19:55:06 hoarder pnpm[295]: 2025-01-12T18:55:06.958Z info: [Crawler][581] Stored the screenshot as assetId: 4621f6fb-f845-49ee-a6fc-b02269328909
Jan 12 19:56:25 hoarder pnpm[295]: 2025-01-12T18:56:25.405Z error: [Crawler][581] Crawling job failed: Error: Timed-out after 90 secs
Jan 12 19:56:25 hoarder pnpm[295]: Error: Timed-out after 90 secs
Jan 12 19:56:25 hoarder pnpm[295]:     at Timeout._onTimeout (/opt/hoarder/apps/workers/utils.ts:2:1025)
Jan 12 19:56:25 hoarder pnpm[295]:     at listOnTimeout (node:internal/timers:594:17)
Jan 12 19:56:25 hoarder pnpm[295]:     at process.processTimers (node:internal/timers:529:7)

I tried to setup the timeout, but still time out.

And this is from browserless after is actually did scrape the url

browserless.io:limiter:warn  Job has hit timeout after 30,013ms of activity. +2m
browserless.io:limiter:info  Calling timeout handler +30s
browserless.io:router:error  Websocket job has timedout, sending 429 response +2m
browserless.io:limiter:info  (Running: 0, Pending: 0) All jobs complete.  +1ms
browserless.io:router:trace  WebSocket Request handler has finished. +30s
browserless.io:browser-manager:info  0 Client(s) are currently connected, Keep-until: 0, force: false +2m
browserless.io:browser-manager:info  Closing browser session +0ms

Tried looking at the crawler code and seems it times out getting metadata. But not sure.
Also did set BROWSER_CONNECT_ONDEMAND=true, that helped on trying to create a new session all the time.

[MohamedBassem]

Ah yes, checked the unraid template and they default to BROWSER_CONNECT_ONDEMAND=true. I think I created it for them, so that's the right fix.

As for the timeout extracting metadata, does this happen in all URLs? I've seen this happen in some URLs because of metascraper (e.g. #806) but it should be quite rare.

[Boss-man]

Most url's do work, but browserless is still reporting a timeout.
I have 4-5 urls that dont work.

Will try to look more into the problem tomorrow.

[Boss-man]

So I had a look at it, and it does work :)
However the timeout in Browserless is because the browser is not closed or closed too late.
The context is closed and then browserless just times out and closes the browser. Maybe that's just okay.

I also got my urls that didn't work to work, by commenting out metascraperLogo. Not sure what I miss out on, but the hoards seems to look fine.