# scraping (adapter system)

Web-scraping adapter — fetch a page under the shared SSRF guard, size cap and rate limiter, then extract text, tables, contacts, JSON-LD, links, feeds, headers, sitemaps and robots. Native fetch +...


Web-scraping core. Ported from `leumas-middleware/lib/a.scraping`, keeping the source's **SSRF guard**
(private-IP / localhost block), protocol allow-list, and 5 MB size cap. Reimplemented on native
`fetch` + regex to drop the heavy deps: `puppeteer` (headless browser), `cheerio` (link parse),
`xml2js` (sitemap parse), and `robots-parser`. The express server, Vite frontend, and docs were
stripped. Runs anywhere with zero native installs.

## Tools (`adapters`)

| tool | args | result |
|------|------|--------|
| `fetchHtml` | `{ url, timeoutMs?, headers? }` | `{ html, status, finalUrl }` |
| `extractLinks` | `{ url }` or `{ url, html }` | `{ links: [...] }` (absolute, deduped) |
| `pageMetadata` | `{ url }` or `{ html }` | `{ title, description, canonical, ogTitle, ogImage }` |
| `sitemap` | `{ url }` (sitemap.xml url) | `{ urls: [...] }` |
| `robots` | `{ url }` (any page on the site) | `{ found, agents, sitemaps }` |


---
Source: shared/engines/adapters/domain/scraping/README.md
Canonical: https://docs.leumas.tech/p/adapters/adapter-scraping