adapter system
scraping
Web-scraping adapter — fetch a page under the shared SSRF guard, size cap and rate limiter, then extract text, tables, contacts, JSON-LD, links, feeds, headers, sitemaps and robots. Native fetch +...
scraping (adapter system)
Web-scraping core. Ported from leumas-middleware/lib/a.scraping, keeping the source's SSRF guard (private-IP / localhost block), protocol allow-list, and 5 MB size cap. Reimplemented on native fetch + regex to drop the heavy deps: puppeteer (headless browser), cheerio (link parse), xml2js (sitemap parse), and robots-parser. The express server, Vite frontend, and docs were stripped. Runs anywhere with zero native installs.
Tools (adapters)
| tool | args | result |
|---|---|---|
fetchHtml | { url, timeoutMs?, headers? } | { html, status, finalUrl } |
extractLinks | { url } or { url, html } | { links: [...] } (absolute, deduped) |
pageMetadata | { url } or { html } | { title, description, canonical, ogTitle, ogImage } |
sitemap | { url } (sitemap.xml url) | { urls: [...] } |
robots | { url } (any page on the site) | { found, agents, sitemaps } |