Full-text OCR (NDL OCR-Lite)

Run OCR over already-imported item images after the fact, so their text becomes full-text searchable. ArchiveBase bundles the National Diet Library's (NDL) lightweight OCR models, so you can generate full text on your own host with no extra cloud cost.

The recognized text is written into each item's body (body), which is indexed by the existing site search right away (the Postgres full-text index, and the Elasticsearch index too for sites that use one).

Two models (pick per material)

Choosing the model that matches the material matters — running the wrong one degrades accuracy badly.

flavour	Model	Target
ndl-ocr	NDLOCR-Lite	Modern printed matter (letterpress books, magazines, …)
ndl-kotenocr	NDL Classical OCR-Lite	Kuzushiji / classical works (handwritten, woodblock, …)

The classical model can also emit TEI XML, which flows into the TEI import path (body display + named-entity extraction).

Set a site's default model

Each site can store its default model as settings.ocr.flavour, via the dashboard settings or the API:

curl -X PATCH "$BASE/api/$ACCOUNT/databases/$DB" \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"settings":{"ocr":{"flavour":"ndl-ocr"}}}'

Running OCR

The OCR engine runs in a container separate from the app (services/ndl-ocr). An operator runs it as follows:

# 1) Start the OCR service (runs on CPU, models bundled)
docker compose --profile ocr up -d

# 2) Batch a site (start with a few items to check accuracy)
npm run ocr:db -- --account=<account> --db=<site> --flavour=ndl-ocr --limit=3

# 3) Once it looks good, run the whole set
npm run ocr:db -- --account=<account> --db=<site>

• Idempotent & resumable: items that already have body text are skipped (use --force to re-OCR).
• Dry run: --dry-run recognizes text without writing, reporting character counts only.
• After the run, the recognized words are findable in that site's search box.

For the processing flow and environment variables (switching local vs production), see Using the API and the header comment of scripts/ocr-db.ts.

Annotation (OCR) display styles

Recognized OCR text can be shown as "annotations" overlaid on each frame's image. Readers switch the display style from the Annotations tab on the right of the viewer.

• Boxes — each recognized line is outlined with a rectangle. Hovering (or tapping on touch devices) shows that line's text in a popup.
• Overlay — the recognized text is drawn directly onto its region on the image. Tall regions render vertically, wide regions horizontally (auto-detected). Text colour, background colour, and background opacity are adjustable.

Use Show / Hide to toggle annotations entirely. The reader's chosen style is remembered within the browser session.

Set a site's default display

Each site can store its initial display as settings.annotationDisplay. It is used until the reader changes the style themselves.

curl -X PATCH "$BASE/api/$ACCOUNT/databases/$DB" \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"settings":{"annotationDisplay":{"default":"overlay","overlay":{"textColor":"#111111","backgroundColor":"#ffffff","backgroundOpacity":0.85}}}}'

• default — off (hidden) / boxes / overlay. Defaults to off.
• overlay.textColor / overlay.backgroundColor — #rrggbb colours.
• overlay.backgroundOpacity — a number from 0 to 1 (background opacity).

Opening the recognised text up (DTS)

Recognised text can be served through the text API (DTS), so researchers can pull the full text programmatically or feed it to other tools.

Turn on Serve text over DTS in the site settings (settings.dtsText over the API).

curl -X PATCH "$BASE/api/$ACCOUNT/databases/$DB" \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"settings":{"dtsText":true}}'

• Only public items that carry text are served. Private sites and private items are not.
• An item with no text of its own is still served when its frames carry OCR. Where frames hang under an item, the frame is the citable unit.
• The citable unit is derived automatically: one frame or page for OCR, one utterance (with its playback position) for A/V transcripts, and the whole text as one unit when there is no structural clue.
• Sites ingested as TEI are already served over DTS without this setting.

See Interoperability (for developers).

Caveats

• Accuracy depends on the material. Modern print is good; degraded scans, complex layouts, and kuzushiji leave errors. Treat the output as a base for correction.
• It is compute-heavy. Hundreds to thousands of pages take real time (plan for an overnight batch).
• License: the OCR models and code are CC BY 4.0 (source: National Diet Library). Credit the source when you publish the output.

Work in progress

A one-click "OCR this site" button in the dashboard is in progress. For now the batch command above is the entry point.