# @leumas/osint-sdk

Does a public profile exist for this handle? The OSINT probe engine behind Leumas - a source-neutral rule evaluator, ONE shared per-host limiter, a concurrent multi-handle job registry and an SSE...


Does a public profile exist for this handle? Asked 481 times, politely.

## What it is

The probe engine behind Leumas OSINT. The **site list** comes from
[sherlock-project/sherlock](https://github.com/sherlock-project/sherlock) (MIT) and lives in
`data/sites.json`, regenerated by `tools/sync-sites.mjs` and pinned to an upstream commit in
`data/provenance.json`. That list is the part with real accumulated value — hundreds of people
establishing which URL pattern and which failure signal each site actually uses.

**No upstream code is vendored.** The engine is original, because Sherlock is Python and this has to
run in the API process, in a packaged EXE, with no interpreter and no pip. There is nothing to
install, nothing to detect at runtime, and no API key.

## What it absorbs

Before this package, the probe lived inside the `osint` adapter and the Information controller had a
**second, serial loop** over it, one site at a time. Two implementations of "run a sweep" that could
drift on the next manifest change. Now there is one engine and three thin consumers:

| consumer | what it adds |
|---|---|
| `shared/engines/adapters/domain/osint` | HTTP, MCP and AI-functioncall surfaces, for free |
| `./router`, mounted at `/api/information/sherlock` | SSE — start a job, watch it, cancel it |
| `products/leumas-api` `controllers/information.js` | audit rows and dossier findings |

## Public API

```js
import { sweepUsername, probeOne, planSites, emailProfile, normalize, status } from '@leumas/osint-sdk';
import { createOsintJobs } from '@leumas/osint-sdk/jobs';
import { createOsintRouter } from '@leumas/osint-sdk/router';
import { splitHandles, STATUS_TEXT, progressOf } from '@leumas/osint-sdk/client';  // browser-safe
```

- `planSites({ categories, q, includeNsfw })` — what **would** be checked. No network. Call this
  before paying for a sweep.
- `sweepUsername({ username, categories, concurrency })` — the one-shot fan-out. Returns once, at the
  end. For an agent or an MCP client.
- `createOsintJobs()` — the streaming registry. One job = **N handles × M sites**, interleaved by
  handle so every handle progresses at once. For a human watching a screen.
- `probeOne({ username, site })` — one site. The unit a job emits and a retry re-runs, so a job and a
  single check can never disagree about what a hit means.

## The three rules that are the whole algorithm

Each site declares how it signals absence:

| `errorType` | absence looks like |
|---|---|
| `status_code` | the declared `errorCode` (usually 404); anything else means the page is there |
| `message` | the site answers 200 for everything, so absence is a **string in the body** |
| `response_url` | a missing profile redirects to a known URL, usually the homepage |

Plus `regexCheck`, which runs **before any request**: a site whose charset cannot hold the handle is
reported `invalid`, not `absent`, and costs nothing.

## Import the right half

`./client` is the **browser-safe** half. React imports that, never the barrel — the barrel reaches
`node:crypto`, `node:dns` and the SSRF guard, and `./router` reaches express. `./client` also does
not import `./sites.js`, because shipping a 123 KB manifest to a browser to render 23 category chips
is not a trade worth making.

## What a match means

A hit means **a page exists at that URL**. It does not mean the person you are researching owns it.
Handles collide constantly, and popular handles collide on purpose. Every result carries the URL it
came from so a human can look; nothing here should be promoted to a fact about a person without one
doing so.

This package reads only what is already public. It never logs in, never submits a form, never follows
a paywall and never touches a private endpoint — every probe is the same unauthenticated request a
browser makes by visiting the profile URL. If you are profiling an identifiable person, establishing
a lawful basis for it and honouring erasure requests is the operator's responsibility.

## Politeness

One module-scoped limiter, shared by every verb and **every concurrent job in the process**: 2
requests/sec per host, 20 in flight overall. `MAX_CONCURRENCY` is 32, and it is a politeness number
rather than a performance one — past the limiter, raising it only lengthens the queue. A site that
rate-limits or blocks returns `{ status: 'error' }` **inside the results**; it never throws, because
a caller who learns to retry the whole sweep on any failure is what gets an IP banned.

## Refreshing the site list

```sh
node shared/engines/osint/tools/sync-sites.mjs --dry   # print the diff
node shared/engines/osint/tools/sync-sites.mjs         # write it
```

Commit `data/sites.json` and `data/provenance.json` together. Never hand-edit either.

## Tests

```sh
node --test shared/engines/osint/test/*.test.js
```

Offline — `fetch` is stubbed throughout. The drain budget in the job tests is seconds rather than
milliseconds on purpose: the limiter paces even a one-site job, and a loop tuned to how fast the code
*could* go is a flaky test.


---
Source: shared/engines/osint/README.md
Canonical: https://docs.leumas.tech/p/engines/osint-sdk