@leumas/osint-sdk

Does a public profile exist for this handle? Asked 481 times, politely.

What it is

The probe engine behind Leumas OSINT. The site list comes from sherlock-project/sherlock (MIT) and lives in data/sites.json, regenerated by tools/sync-sites.mjs and pinned to an upstream commit in data/provenance.json. That list is the part with real accumulated value — hundreds of people establishing which URL pattern and which failure signal each site actually uses.

No upstream code is vendored. The engine is original, because Sherlock is Python and this has to run in the API process, in a packaged EXE, with no interpreter and no pip. There is nothing to install, nothing to detect at runtime, and no API key.

What it absorbs

Before this package, the probe lived inside the osint adapter and the Information controller had a second, serial loop over it, one site at a time. Two implementations of "run a sweep" that could drift on the next manifest change. Now there is one engine and three thin consumers:

Public API

import { sweepUsername, probeOne, planSites, emailProfile, normalize, status } from '@leumas/osint-sdk';
import { createOsintJobs } from '@leumas/osint-sdk/jobs';
import { createOsintRouter } from '@leumas/osint-sdk/router';
import { splitHandles, STATUS_TEXT, progressOf } from '@leumas/osint-sdk/client';  // browser-safe

• planSites({ categories, q, includeNsfw }) — what would be checked. No network. Call this

before paying for a sweep.

• sweepUsername({ username, categories, concurrency }) — the one-shot fan-out. Returns once, at the

end. For an agent or an MCP client.

• createOsintJobs() — the streaming registry. One job = N handles × M sites, interleaved by

handle so every handle progresses at once. For a human watching a screen.

• probeOne({ username, site }) — one site. The unit a job emits and a retry re-runs, so a job and a

single check can never disagree about what a hit means.

The three rules that are the whole algorithm

Each site declares how it signals absence:

Plus regexCheck, which runs before any request: a site whose charset cannot hold the handle is reported invalid, not absent, and costs nothing.

Import the right half

./client is the browser-safe half. React imports that, never the barrel — the barrel reaches node:crypto, node:dns and the SSRF guard, and ./router reaches express. ./client also does not import ./sites.js, because shipping a 123 KB manifest to a browser to render 23 category chips is not a trade worth making.

What a match means

A hit means a page exists at that URL. It does not mean the person you are researching owns it. Handles collide constantly, and popular handles collide on purpose. Every result carries the URL it came from so a human can look; nothing here should be promoted to a fact about a person without one doing so.

This package reads only what is already public. It never logs in, never submits a form, never follows a paywall and never touches a private endpoint — every probe is the same unauthenticated request a browser makes by visiting the profile URL. If you are profiling an identifiable person, establishing a lawful basis for it and honouring erasure requests is the operator's responsibility.

Politeness

One module-scoped limiter, shared by every verb and every concurrent job in the process: 2 requests/sec per host, 20 in flight overall. MAX_CONCURRENCY is 32, and it is a politeness number rather than a performance one — past the limiter, raising it only lengthens the queue. A site that rate-limits or blocks returns { status: 'error' } inside the results; it never throws, because a caller who learns to retry the whole sweep on any failure is what gets an IP banned.

Refreshing the site list

node shared/engines/osint/tools/sync-sites.mjs --dry   # print the diff
node shared/engines/osint/tools/sync-sites.mjs         # write it

Commit data/sites.json and data/provenance.json together. Never hand-edit either.

Tests

node --test shared/engines/osint/test/*.test.js

Offline — fetch is stubbed throughout. The drain budget in the job tests is seconds rather than milliseconds on purpose: the limiter paces even a one-site job, and a loop tuned to how fast the code could go is a flaky test.