{
  "schema": "leumas.docs.page/1",
  "id": "pkg:@leumas/osint-sdk",
  "slug": "engines/osint-sdk",
  "kind": "capabilities",
  "bucket": "package",
  "title": "@leumas/osint-sdk",
  "name": "@leumas/osint-sdk",
  "eyebrow": null,
  "chip": null,
  "summary": "Does a public profile exist for this handle? The OSINT probe engine behind Leumas - a source-neutral rule evaluator, ONE shared per-host limiter, a concurrent multi-handle job registry and an SSE...",
  "keywords": [
    "osint-sdk",
    "concurrent",
    "osint",
    "limiter",
    "handle",
    "osint sdk api",
    "leumas osint sdk",
    "how to use osint sdk"
  ],
  "audience": "both",
  "funnel": {
    "product": null,
    "cta": null
  },
  "body": "# @leumas/osint-sdk\n\nDoes a public profile exist for this handle? Asked 481 times, politely.\n\n## What it is\n\nThe probe engine behind Leumas OSINT. The **site list** comes from\n[sherlock-project/sherlock](https://github.com/sherlock-project/sherlock) (MIT) and lives in\n`data/sites.json`, regenerated by `tools/sync-sites.mjs` and pinned to an upstream commit in\n`data/provenance.json`. That list is the part with real accumulated value — hundreds of people\nestablishing which URL pattern and which failure signal each site actually uses.\n\n**No upstream code is vendored.** The engine is original, because Sherlock is Python and this has to\nrun in the API process, in a packaged EXE, with no interpreter and no pip. There is nothing to\ninstall, nothing to detect at runtime, and no API key.\n\n## What it absorbs\n\nBefore this package, the probe lived inside the `osint` adapter and the Information controller had a\n**second, serial loop** over it, one site at a time. Two implementations of \"run a sweep\" that could\ndrift on the next manifest change. Now there is one engine and three thin consumers:\n\n| consumer | what it adds |\n|---|---|\n| `shared/engines/adapters/domain/osint` | HTTP, MCP and AI-functioncall surfaces, for free |\n| `./router`, mounted at `/api/information/sherlock` | SSE — start a job, watch it, cancel it |\n| `products/leumas-api` `controllers/information.js` | audit rows and dossier findings |\n\n## Public API\n\n```js\nimport { sweepUsername, probeOne, planSites, emailProfile, normalize, status } from '@leumas/osint-sdk';\nimport { createOsintJobs } from '@leumas/osint-sdk/jobs';\nimport { createOsintRouter } from '@leumas/osint-sdk/router';\nimport { splitHandles, STATUS_TEXT, progressOf } from '@leumas/osint-sdk/client';  // browser-safe\n```\n\n- `planSites({ categories, q, includeNsfw })` — what **would** be checked. No network. Call this\n  before paying for a sweep.\n- `sweepUsername({ username, categories, concurrency })` — the one-shot fan-out. Returns once, at the\n  end. For an agent or an MCP client.\n- `createOsintJobs()` — the streaming registry. One job = **N handles × M sites**, interleaved by\n  handle so every handle progresses at once. For a human watching a screen.\n- `probeOne({ username, site })` — one site. The unit a job emits and a retry re-runs, so a job and a\n  single check can never disagree about what a hit means.\n\n## The three rules that are the whole algorithm\n\nEach site declares how it signals absence:\n\n| `errorType` | absence looks like |\n|---|---|\n| `status_code` | the declared `errorCode` (usually 404); anything else means the page is there |\n| `message` | the site answers 200 for everything, so absence is a **string in the body** |\n| `response_url` | a missing profile redirects to a known URL, usually the homepage |\n\nPlus `regexCheck`, which runs **before any request**: a site whose charset cannot hold the handle is\nreported `invalid`, not `absent`, and costs nothing.\n\n## Import the right half\n\n`./client` is the **browser-safe** half. React imports that, never the barrel — the barrel reaches\n`node:crypto`, `node:dns` and the SSRF guard, and `./router` reaches express. `./client` also does\nnot import `./sites.js`, because shipping a 123 KB manifest to a browser to render 23 category chips\nis not a trade worth making.\n\n## What a match means\n\nA hit means **a page exists at that URL**. It does not mean the person you are researching owns it.\nHandles collide constantly, and popular handles collide on purpose. Every result carries the URL it\ncame from so a human can look; nothing here should be promoted to a fact about a person without one\ndoing so.\n\nThis package reads only what is already public. It never logs in, never submits a form, never follows\na paywall and never touches a private endpoint — every probe is the same unauthenticated request a\nbrowser makes by visiting the profile URL. If you are profiling an identifiable person, establishing\na lawful basis for it and honouring erasure requests is the operator's responsibility.\n\n## Politeness\n\nOne module-scoped limiter, shared by every verb and **every concurrent job in the process**: 2\nrequests/sec per host, 20 in flight overall. `MAX_CONCURRENCY` is 32, and it is a politeness number\nrather than a performance one — past the limiter, raising it only lengthens the queue. A site that\nrate-limits or blocks returns `{ status: 'error' }` **inside the results**; it never throws, because\na caller who learns to retry the whole sweep on any failure is what gets an IP banned.\n\n## Refreshing the site list\n\n```sh\nnode shared/engines/osint/tools/sync-sites.mjs --dry   # print the diff\nnode shared/engines/osint/tools/sync-sites.mjs         # write it\n```\n\nCommit `data/sites.json` and `data/provenance.json` together. Never hand-edit either.\n\n## Tests\n\n```sh\nnode --test shared/engines/osint/test/*.test.js\n```\n\nOffline — `fetch` is stubbed throughout. The drain budget in the job tests is seconds rather than\nmilliseconds on purpose: the limiter paces even a one-site job, and a loop tuned to how fast the code\n*could* go is a flaky test.\n",
  "source": {
    "path": "shared/engines/osint/README.md",
    "blobSha": "",
    "commit": "",
    "committedAt": "",
    "provenance": "no-git",
    "bytes": 5262,
    "hash": "04d6f2ff22b769019801987f21a9b90f552409e4"
  },
  "urls": {
    "html": "/p/engines/osint-sdk",
    "json": "/docs/engines/osint-sdk.json",
    "md": "/docs/engines/osint-sdk.md"
  },
  "links": {
    "composes": [],
    "usedBy": [
      "pkg:@leumas/studio"
    ],
    "product": [
      "pkg:@leumas/studio"
    ],
    "howTo": [],
    "skills": []
  },
  "exports": null
}