{
  "schema": "leumas.docs.page/1",
  "id": "pkg:@leumas/adapter-scraping",
  "slug": "adapters/adapter-scraping",
  "kind": "capabilities",
  "bucket": "package",
  "title": "scraping (adapter system)",
  "name": "scraping",
  "eyebrow": "adapter system",
  "chip": null,
  "summary": "Web-scraping adapter — fetch a page under the shared SSRF guard, size cap and rate limiter, then extract text, tables, contacts, JSON-LD, links, feeds, headers, sitemaps and robots. Native fetch +...",
  "keywords": [
    "adapter-scraping",
    "web-scraping",
    "sitemaps",
    "robots",
    "ssrf",
    "adapter scraping api",
    "leumas adapter scraping",
    "headers"
  ],
  "audience": "both",
  "funnel": {
    "product": null,
    "cta": null
  },
  "body": "# scraping (adapter system)\n\nWeb-scraping core. Ported from `leumas-middleware/lib/a.scraping`, keeping the source's **SSRF guard**\n(private-IP / localhost block), protocol allow-list, and 5 MB size cap. Reimplemented on native\n`fetch` + regex to drop the heavy deps: `puppeteer` (headless browser), `cheerio` (link parse),\n`xml2js` (sitemap parse), and `robots-parser`. The express server, Vite frontend, and docs were\nstripped. Runs anywhere with zero native installs.\n\n## Tools (`adapters`)\n\n| tool | args | result |\n|------|------|--------|\n| `fetchHtml` | `{ url, timeoutMs?, headers? }` | `{ html, status, finalUrl }` |\n| `extractLinks` | `{ url }` or `{ url, html }` | `{ links: [...] }` (absolute, deduped) |\n| `pageMetadata` | `{ url }` or `{ html }` | `{ title, description, canonical, ogTitle, ogImage }` |\n| `sitemap` | `{ url }` (sitemap.xml url) | `{ urls: [...] }` |\n| `robots` | `{ url }` (any page on the site) | `{ found, agents, sitemaps }` |\n",
  "source": {
    "path": "shared/engines/adapters/domain/scraping/README.md",
    "blobSha": "",
    "commit": "",
    "committedAt": "",
    "provenance": "no-git",
    "bytes": 1151,
    "hash": "eac310ae1e857aa485c9d1f5c26a5aa895388a5e"
  },
  "urls": {
    "html": "/p/adapters/adapter-scraping",
    "json": "/docs/adapters/adapter-scraping.json",
    "md": "/docs/adapters/adapter-scraping.md"
  },
  "links": {
    "composes": [],
    "usedBy": [],
    "product": [],
    "howTo": [],
    "skills": []
  },
  "exports": null
}