Bulk Content Analysis: 86 Languages in a Pipeline

A catalogue of forty thousand pages cannot be read by an editor, and a sample of forty tells you what the sample says. Bulk content analysis scores the whole export overnight, sorts it by grade, and hands an editor a list instead of a feeling.

Code: ReadSightPy, MIT licensed.

readsight for Python 3.10 measures text in 86 languages with 17 formulas, and it is built for the job where the text arrives from a database rather than from a text box.

What a pipeline asks of a scorer

Three things, in this order: the same number twice, no network, and fast enough to finish before the morning.

Determinism comes from the data: language files and TeX hyphenation patterns are fixed, compiled patterns are cached on disk, and nothing is fetched. Speed is in the same league as the surrounding work: about a millisecond for a 45-word text, ten milliseconds to build an engine from cache, sixty for a German engine on first load.

Against a whole export, the reading time of the database dominates. That is the right shape for a pipeline component.

The report that comes out

A table with one row per document: language, formula set, grade, and the threshold for that audience. Spanish gets eleven formulas, German eight, Thai five, and the table keeps them apart instead of averaging them into one meaningless figure.

Sort by grade within a language and the worst hundred pages are on top. That list is the deliverable: an editor can work it down, and the next run shows whether the work moved the number.

The gate that earns its keep

The most valuable use is a filter in a translation pipeline. Compare the target text with its source and refuse anything more than two grades harder. A translation that is technically correct and unreadable passes every other check in the chain.

Two grades is a starting threshold, not a law. Pick it from your own corpus, then keep it fixed so the number means the same thing next quarter.

Two dependencies, both named

The package needs regex for Unicode letter classes and platformdirs to place the pattern cache in the right directory per operating system. Nothing else at runtime.

That is a deliberate budget. Every dependency in a data pipeline is a thing that can pin a version, break a rebuild or disappear from a wheel index, and two well-known packages are a cost worth stating out loud.

Where it stops

It cannot tell a page that argues badly from a page that argues well. It ignores headings, images and links, and its syllable count follows TeX break points, which is approximate by nature; the readme says so.

A readability grade is a sort key. It finds the pages that need a person, and the person decides what to do with them.

What to do with the first hundred

Read ten of them before sending the list to anyone. The tail of a catalogue is usually pages nobody wrote on purpose: old service pages, duplicated descriptions, an import that carried the wrong text. Those are fixed by deleting or merging, not by editing.

The remaining ninety need a person, and the report should say which formula failed. A page that fails on sentence length and a page that fails on syllable-heavy words are two different edits.

How to start small

Export a thousand documents, run the scorer, and look at the distribution before choosing a threshold. A catalogue usually has a long tail of pages nobody wrote on purpose, and those are the first hundred on the list.

Related: multilingual readability for how the formula sets differ by language, and content readability for the formulas themselves.