# nlp — natural-language analysis pack

Natural-language analysis pack: statistical corpus analysis over text with zero ML dependencies. Lexicon-based sentiment (AFINN-style), RAKE/TF keyword extraction, extractive summarization, n-gram...


Statistical / lexicon-based corpus analysis over text. **Pure JS, zero npm deps, no ML models** —
everything is deterministic (frequency counts, an AFINN-style lexicon, stopword profiles, Flesch
formulas).

## Tools

| Tool | What it does |
|---|---|
| `sentiment({ text })` | Lexicon-based (AFINN-style, `data/afinn.json`) with negation flipping → `{ score, comparative, label:'positive'\|'negative'\|'neutral', positive:[], negative:[] }`. |
| `keywords({ text, top=10 })` | RAKE-style keyword/keyphrase extraction (candidate phrases scored by word degree/frequency, stopwords excluded) → ranked terms with scores. |
| `summarize({ text, sentences=3 })` | Extractive summary: sentences scored by content-word frequency, top-N returned **in original order**. |
| `ngrams({ text, n=2, top? })` | n-gram frequency table. |
| `languageDetect({ text })` | Stopword-profile heuristic across en/es/fr/de/it/pt/nl (+ `unknown`) → `{ language, confidence, scores }`. |
| `readability({ text })` | Full **Flesch Reading Ease** + **Flesch-Kincaid grade**, syllable estimate, avg sentence length. |
| `tokenize({ text, mode='word'\|'sentence' })` | Word or sentence tokenization. |
| `wordFrequency({ text, stopwords=true, top=20 })` | Word-frequency table (optionally excluding stopwords). |
| `entities({ text })` | Heuristic entities: capitalized sequences → proper nouns, plus regex for emails / urls / dates / money / numbers, grouped. |
| `stats({ text })` | chars / words / sentences / paragraphs / avgWordLength / uniqueRatio. |

## Usage

```js
import nlp from './index.js';

nlp.adapters.sentiment({ text: 'I love this, it is great and wonderful' });
// → { score: 10, comparative: 1.25, label: 'positive', positive: ['love','great','wonderful'], negative: [], tokens: 8 }

nlp.adapters.languageDetect({ text: 'el gato está en la casa y la comida es muy buena' });
// → { language: 'es', confidence: ..., scores: { en, es, fr, de, it, pt, nl } }

nlp.adapters.readability({ text: 'The cat sat on the mat. It was a sunny day.' });
// → { fleschReadingEase: <number>, fleschKincaidGrade: <number>, ... }
```

Every tool takes one args object (`text` required) and returns a plain JSON-serializable object.

## DRY boundary vs `domain/a-text`

- **`a-text`** = per-string **transforms**: case conversion, base64, ciphers, digests, extraction,
  a *basic* `readabilityScore`, and stemming on a single word/string.
- **`nlp`** (this pack) = statistical **corpus analysis**: sentiment, keyword/summary extraction,
  language detection, the **full** readability metrics (Flesch Reading Ease + Flesch-Kincaid grade,
  where a-text only offers a basic readability score), n-grams, entity/stat roll-ups over a whole
  document.

Rule of thumb: need a **number about the text** → `nlp`; need to **reshape a string** → `a-text`.

## Data

- `data/afinn.json` — ~230-word AFINN-style sentiment lexicon (scores -5..+5).
- `data/stopwords.json` — English stopword list (shared by keywords / summarize / wordFrequency / entities).
- `data/lang-profiles.json` — top-function-word profiles for the 7 detected languages.


---
Source: shared/engines/adapters/domain/nlp/README.md
Canonical: https://docs.leumas.tech/p/adapters/domain/nlp