Skip to main content

subocr service

Service Overview: SubOCR (https://subocr.drheck.dev/) is an end-to-end Japanese burned-in video subtitle (telop / テロップ) extraction, 漢字かんじ 振ふり仮名がな annotation, offline JMdict + JLPT (N5–N1) morphological dictionary lookup, LLM study-note refinement, and interactive video study platform running on host ul, accelerated by the AMD Radeon AI PRO R9700 (gfx1201, RDNA 4, 32 GB VRAM) GPU.

1. Architecture & Network Topology

SubOCR is deployed as a containerized FastAPI + ONNX Runtime (ROCm 7.2 MIGraphX) service bound to a dedicated IPv6 ULA address on eth77, reverse-proxied through Nginx with Authentik ForwardAuth single sign-on, and paired with the local ollama-rocm (qwen2.5:32b) LLM engine on the same GPU.

$$\text{Video Frame } I(t) \longrightarrow \text{PP-OCRv5 (MIGraphX FP32)} \longrightarrow \text{Qwen2.5-32B (ROCm)} \longrightarrow \text{UniDic + JMdict/JLPT Study View}$$

  • Public URL: https://subocr.drheck.dev/
  • Dedicated ULA IPv6 Address: fd77:7777:5010:0:1337:f1ac:d00d:0c70 (eth77, port 8000)
  • Host Directory: /data/docker/subocr/
  • Databases:
    • Application Database: /data/docker/subocr/data/subocr.db (SQLite 3 in WAL mode; stores jobs, timed cues, readings, translations, and custom edits)
    • Offline Dictionary & JLPT Database: /data/docker/subocr/data/jmdict_jlpt.db (SQLite 3, 63 MB; 218,856 EDRDG JMdict entries, 499,325 indexed kanji/kana forms, and 7,972 JLPT N5–N1 vocabulary entries)
  • Export & Sidecar Storage:
    • Per-job exports: /data/docker/subocr/data/jobs/<job_id>/
    • Automatic sidecar writes next to local videos: /data/language/Japanese/Video/... (mounted read-write at /videos inside the container)
  • Authentication: Authentik ForwardAuth (subocr ProxyProvider, 30-day token validity) via /etc/nginx/sites-available/subocr
  • DNS Resolution:
    • Local Unbound (danknasty, @10.0.0.1): A 10.0.0.100, AAAA fd77:7777:5010::1337:f1ac:d00d:aaaa
    • Public Authoritative DNS (deSEC): AAAA fd77:7777:5010::1337:f1ac:d00d:aaaa

2. Two-Stage GPU Processing & Morphological Dictionary Pipeline

When a job is submitted from https://subocr.drheck.dev/new (either by selecting a local video under /data/language/Japanese/Video, uploading a video file, or providing a YouTube URL fetched via yt-dlp), SubOCR executes a two-stage GPU pipeline on the AMD Radeon AI PRO R9700 (gfx1201) followed by deterministic morphological and dictionary analysis:

Stage 1: ROCm 7.2 MIGraphX Visual Subtitle OCR (PP-OCRv5)

  1. Frame Sampling & Spatial Crop:
    • OpenCV samples the video at a configurable rate $f_s$ (default $2.0\text{ fps}$, or frame interval $\Delta t = 0.5\text{ s}$).
    • To isolate burned-in subtitles and ignore top-of-screen variety-show topic banners by default, frames are cropped to the lower subtitle region $y \in [0.58H, 0.98H]$.
  2. GPU Inference (MIGraphXExecutionProvider FP32):
    • Detection (PP-OCRv5_server_det_infer.onnx) and recognition (japanese_PP-OCRv5_mobile_rec_infer.onnx) run via onnxruntime-rocm with MIGraphXExecutionProvider.
    • Compiled MIGraphX kernel plans (.mxr) are cached in /data/docker/subocr/cache/migraphx so subsequent container restarts bypass ahead-of-time compilation.
    • Precision Note: migraphx_fp16_enable is explicitly set to 0 (FP32) because FP16 causes bounding-box coordinate underflow in PP-OCRv5 detection on gfx1201.
  3. Bilingual Line Classification & Spatial Role Separation:
    • Detected text boxes within each frame are sorted vertically and classified into Japanese (ja_lines) or English (en_lines) based on character script ratios.
    • Within the lower subtitle band, higher-positioned smaller Japanese commentary boxes ($y < 0.66H$) are separated from the primary bottom speaker dialogue ($y \ge 0.66H$) so simultaneous on-screen editor commentary and speaker subtitles are preserved cleanly.
  4. Temporal Deduplication:
    • Consecutive frames whose normalized Japanese string similarity satisfies: $$\text{sim}(A, B) = \frac{2 \cdot |M|}{|A| + |B|} \ge 0.78$$ are merged into a single timed subtitle cue $[t_0, t_1]$, selecting the highest-confidence OCR recognition frame as the canonical raw text (ja_raw) and storing the initial English line (en_raw).

Hardware & ROCm Note: Always keep MIGRAPHX_DISABLE_MIOPEN_FUSION=1 and migraphx_fp16_enable=0 enabled in /data/docker/subocr/docker-compose.yml. Enabling FP16 on gfx1201 degrades bounding-box detection recall in PP-OCRv5.

Stage 2: Contextual LLM Refinement, Kanji-Only Furigana & Offline Dictionary Tagging

  1. Batched Contextual Refinement (qwen2.5:32b on ollama-rocm):
    • Extracted cues are processed in batches of 15 cues against http://172.28.0.1:11434/api/chat (qwen2.5:32b).
    • The LLM receives both the raw Japanese OCR (ja_raw) and any burned-in English subtitle (en_raw), repairing single-character visual OCR confusions (e.g., katakana ニ/二, カ/力, or visually degraded glyphs) while preserving the original spoken phrasing, slang, and dialect.
    • If en_raw is present and accurate, it is cleaned; if missing or garbled, a natural English translation (en_text) is synthesized alongside contextual idiom/slang notes (vocab).
  2. Punctuation & On-Screen Symbol Restoration (restore_raw_punctuation):
    • Post-processing compares ja_raw and ja_text to ensure the LLM never strips on-screen quotation marks ("...", 「...」, 『...』), wave dashes (~, 〜), exclamation/question marks (!, ?), ellipses (…), or counter katakana (何カ国, 6カ国).
    • Example preserved cue: "スペシャルゲスト~!"が<ruby>無<rt>な</rt></ruby>くなり<ruby>皆<rt>みな</rt></ruby>メンバーに
  3. Deterministic Kanji-Only Furigana (fugashi + unidic-lite):
    • Morphological tokenization (fugashi + unidic-lite + jaconv) in generate_kanji_furigana() strips leading, trailing, and internal okurigana and attaches <ruby> annotations strictly above Kanji characters, leaving hiragana and katakana unannotated:
    • Example Render: 世界せかいの違ちがいに6カ国こくが大混乱だいこんらん — それ考かんがえたらすごい日本にほんっぽいっていうか
  4. Exhaustive Per-Word Tokenization, Grammar Categorization & Offline JMdict / JLPT Lookup (analyze_cue_tokens):
    • Every word in every subtitle cue is tokenized by fugashi (MECAB_TAGGER), lemmatized to its dictionary base form (orthBase), and matched against the local read-only /data/docker/subocr/data/jmdict_jlpt.db SQLite database and a curated GRAMMAR_GLOSS_MAP for Japanese particles and colloquial auxiliaries (てる/てん, ちゃう, たら, じゃん, Kansai ねん/やん/へん, etc.).
    • Each token is assigned:
      • JLPT Level Badge (jlpt): N5, N4, N3, N2, N1, or Other
      • Part-of-Speech Filter & Chip Tag (pos_cat / pos_label): Grouped under the 6 品詞 filter buttons (名詞, 動詞, 形容詞, 副詞, 表現, 助詞) and labeled on each vocabulary chip with its specific Japanese grammatical category (see the complete reference table below)
      • English Gloss (meaning): Offline JMdict definition or grammatical function

Reference Table: Japanese Part-of-Speech (品詞) Filter Groups & Vocabulary Chip Tags

Chip Tag
(pos_label)		 Reading
(振ふり仮名がな)		 Filter Button
(品詞)		 English Translation Grammatical Role & Subtitle Examples
名詞 名詞めいし 名詞
(Noun)		 Common Noun General nouns and suru-noun stems (e.g., 世界, 違い, 大混乱)
固有名詞 固有名詞こゆうめいし 名詞
(Noun)		 Proper Noun Places, countries, people, and brand names (e.g., 日本, 関西, フランス)
代名詞 代名詞だいめいし 名詞
(Noun)		 Pronoun Personal and demonstrative pronouns (e.g., 私, 僕, これ, それ, どこ)
助数詞 助数詞じょすうし 名詞
(Noun)		 Counter Numerical counter suffixes (e.g., カ国, 人, 回, つ, 個)
動詞 動詞どうし 動詞
(Verb)		 Verb All main verbs (Godan, Ichidan, Suru, Kuru); displayed as conjugated (dictionary form) when inflected (e.g., 言った (言う), 考えたら (考える))
形容詞 形容詞けいようし 形容詞
(Adjective)		 i-Adjective Inflecting adjectives ending in -i (e.g., すごい, 面白い, 無い)
形容動詞 形容動詞けいようどうし 形容詞
(Adjective)		 na-Adjective Adjectival nouns that take な before nouns (e.g., 好き, 大変, 複雑)
連体詞 連体詞れんたいし 形容詞
(Adjective)		 Pre-noun Adjectival Non-inflecting words that directly modify nouns (e.g., この, その, あんな, いろんな)
副詞 副詞ふくし 副詞
(Adverb)		 Adverb Words modifying verbs, adjectives, or clauses (e.g., ちょっと, めっちゃ, 全然, やっぱり)
表現 表現ひょうげん 表現
(Expression)		 Expression / Idiom Multi-word idioms, set phrases, and contextual slang compounds
感動詞 感動詞かんどうし 表現
(Expression)		 Interjection Exclamations, greetings, and conversational fillers (e.g., えっ, はい, うわ, なるほど)
接続詞 接続詞せつぞくし 表現
(Expression)		 Conjunction Sentence/clause connectors (e.g., でも, だから, そして, ただ)
助詞 助詞じょし 助詞
(Particle)		 Particle (General) General or adverbial/quotative particles (e.g., って, とか, だけ, くらい)
格助詞 格助詞かくじょし 助詞
(Particle)		 Case Particle Marks grammatical case relationships (e.g., が, を, に, で, と, から)
係助詞 係助詞かかりじょし 助詞
(Particle)		 Topic / Binding Particle Marks topic, contrast, or inclusion (e.g., は, も, こそ)
終助詞 終助詞しゅうじょし 助詞
(Particle)		 Sentence-Ending Particle Adds tone, question, or emphasis at clause end (e.g., ね, よ, な, か, じゃん)
接続助詞 接続助詞せつぞくじょし 助詞
(Particle)		 Conjunctive Particle Connects clauses (e.g., て/で, けど, から, ば, のに, し)
助動詞 助動詞じょどうし 助詞
(Particle)		 Auxiliary Verb Inflecting verbal/copular endings and colloquial contractions (e.g., だ/です, た, ない, ます, てる/てん, ちゃう, ねん, やん, へん)
接頭辞 接頭辞せっとうじ 助詞
(Particle)		 Prefix Honorific or intensifying prefixes attached to stems (e.g., お, ご, 大, 超)
接尾辞 接尾辞せつびじ 助詞
(Particle)		 Suffix Meaning-modifying suffixes attached to stems (e.g., さん, たち, っぽい, 的, 目)

3. Web Application Layout & Study Controls

SubOCR uses a sharp-cornered (border-radius: 0) layout organized into three focused pages:

  • Main OCR Library (https://subocr.drheck.dev/):
    • Displays all OCR jobs with status badges, cue counts, video durations, category folders, and a Source Video ↗ link whenever an original video URL (youtube_url) is recorded.
    • Provides direct export and download links (JA .SRT, Bilingual .SRT, WebVTT .VTT, Anki .TSV, Study .MD, JSON, and Save to /data) on each OCR job card.
    • Supports pinning (★), category filtering, searching, sorting (Pinned & Newest, Oldest First, Title (A–Z), Most Cues), inline editing (Organize drawer for Title, Category, and Original Video URL), one-click video recovery (Re-download Video via POST /api/jobs/{job_id}/redownload-video if a local video file is missing on disk), and job deletion (Delete).
  • New Subtitle OCR (https://subocr.drheck.dev/new):
    • Dedicated job submission form for local videos under /data/language/Japanese/Video, direct file uploads, or YouTube URLs, with optional start/end timestamps (start_sec, end_sec), frame sampling FPS, category assignment, and an Original Video URL field across all tabs (plus automatic YouTube ID extraction from -<11char_id>.mp4/.mkv filenames) so source URLs are preserved for backup recovery.
  • Dedicated OCR Study View (https://subocr.drheck.dev/ocr/{job_id}):
    • Resizable Split-Screen Workspace: Full-viewport two-column layout with a 52vw default left video column and an interactive 8px vertical drag handle (#split-resizer) between the left video panel and right subtitle cue panel. Dragging the divider resizes the video player and subtitle list in real time, persists the chosen split width in localStorage (subocr_split_width), and resets to 52vw on double-click.
    • Compact top bar above the HTML5 video player with ← Library, Source Video ↗ link, category badge, and small muted video title (native video player WebVTT overlays are disabled by default so burned-in subtitles remain unobstructed). If the local video file is ever missing on disk, a recovery banner appears above the player with a Re-download from Source URL button (yt-dlp).
    • Clicking any subtitle card seeks the video to that cue's start timestamp, starts playback, and highlights the active card in high-contrast amber/gold (#fbbf24 border/pill and #fef08a text). Each card header displays the cue index, timestamp span, and OCR <confidence>%.
    • Fixed-Width Display & Vocabulary Filter Controls (under the player controls on the left, persisted in localStorage with fixed-width ON/OFF labels that never shift button size):
      • Display Toggles: Japanese Subs, Furigana, English Sub, Vocab, and Original OCR
      • JLPT Level Filters: N5, N4, N3, N2, N1, Other
      • 品詞 (Part-of-Speech) Filters: 名詞 (Noun), 動詞 (Verb), 形容詞 (Adjective), 副詞 (Adverb), 表現 (Expression), 助詞 (Particle / Auxiliary)
    • Universal Pre-Generated Vocabulary & Grammar Pages (https://subocr.drheck.dev/{品詞}/{単語}{読み}.html):
      • Clicking any vocabulary box on a subtitle cue card navigates to a universal, pre-generated reference page for that word, phrase, or grammatical particle (e.g., https://subocr.drheck.dev/代名詞/私わたし.html or https://subocr.drheck.dev/動詞/考えるかんがえる.html).
      • Pages are pre-generated automatically in /data/docker/subocr/data/vocab_pages/{pos_label}/{slug}.html at the end of every video analysis job (and updated whenever a subtitle cue is edited).
      • Each vocabulary page includes:
        1. Full Dictionary Senses & JLPT Level: All matching definitions and readings from jmdict_jlpt.db (218,856 JMdict entries + 7,972 JLPT entries + colloquial/dialect grammar map).
        2. Interactive 300x300px Kanji Stroke Order & kanji.drheck.dev Integration: For any word containing Kanji, embeds an interactive 300x300px KanjiVG + Vivus stroke-order viewer (160x160px per frame in Step-by-Step mode, with compact 5.5px SVG stroke numbers, grid lines, and replay controls) alongside direct links to https://kanji.drheck.dev/?text=...&mode=animated&duration=327&showNumbers=true&showGrid=true&numberSize=6&boxSize=300&nightMode=true.
        3. Complete Verb & i-Adjective Conjugation Tables: For all verbs (五段動詞, 一段動詞, サ変動詞, カ変動詞), suru-nouns, and 形容詞, renders a full 19-row conjugation table (辞書形, ます形, ない形, ません, た形, ました, なかった, て形, なくて・ないで, ている形, 可能形, 意向形, 受身形, 使役形, 使役受身形, 仮定形, たら形, 命令形, 禁止形) with <ruby> furigana and plain kana readings.
        4. Token-Highlighted Cross-Video Subtitle Occurrences & Corpus Examples: Lists every subtitle cue across processed videos where the word appears (clicking any occurrence jumps directly to /ocr/{job_id}?cue={cue_index}), paired with offline EDRDG Tanaka Corpus Japanese-English example sentences (92,940 indexed sentences). Morphological token and <ruby> span matching (fugashi.Tagger) lightly highlights the target word or phrase—including conjugated verb and adjective inflections—using <mark class="ex-hl"> without matching inside <rt> furigana tags.
    • Keyboard Shortcuts (active whenever focus is outside text inputs):
      • Space or K: Play / Pause video without scrolling the page
      • A or ←: Jump to previous subtitle cue (preserving current play/pause state)
      • D or →: Jump to next subtitle cue (preserving current play/pause state)
      • R: Replay current subtitle cue from its start timestamp
      • L: Toggle single-cue A/B loop mode (Loop: ON/OFF)
    • Self-Supporting Per-Cue & Job-Wide Local Re-Analysis (Re-analyze, Edit, and Auto-Repair Cues):
      • One-Click Re-analyze Button (POST /api/cues/{cue_id}/reanalyze): Every subtitle cue card header includes a Re-analyze button next to Edit that re-runs local Radeon AI PRO R9700 GPU LLM refinement (qwen2.5:32b via ollama-docker-proxy.socket) on that single cue with surrounding dialogue context (cue - 1 and cue + 1), quote normalization ("..." -> 「...」), and output sanity validation (is_valid_llm_ja), updating the card, furigana, vocabulary chips, and pre-generated vocabulary pages in place in ~1.5 seconds.
      • Inline Cue Editor & Targeted Recovery Modes: Clicking Edit on any cue card opens an inline editor for Japanese text, Furigana reading, English meaning, and vocabulary notes, with four built-in actions:
        1. Reset to Raw OCR (mode="raw"): Instantly restores ja_raw and re-runs local JMdict + JLPT morphological token and furigana analysis.
        2. Re-analyze Edited JA (AI) (mode="custom_ja"): Preserves your manually corrected Japanese text and uses the local GPU LLM + JMdict to regenerate accurate Furigana, English translation, and vocabulary notes.
        3. Re-OCR Video + AI (mode="reocr"): Re-samples the video frames across [start_sec - 0.2, end_sec + 0.2] at a dense 4.0 fps on the GPU (PP-OCRv5) and runs local LLM refinement.
        4. Save Cue: Saves manual field edits directly.
      • Job-Wide Auto-Repair Cues (POST /api/jobs/{job_id}/repair-cues): Located in the left control panel, scans all cues in the current video for any truncated or invalid LLM outputs and automatically re-analyzes every affected cue on the local GPU.

4. Operations, Maintenance & Backup / Restore

Container Management

Click to expand Docker Compose & Service Management Commands 
# Check container status and logs
$ docker ps --filter name=subocr
$ docker logs -f --tail 100 subocr

# Restart after modifying /data/docker/subocr/app.py or vocab_pages.py
$ docker restart subocr

# Rebuild container image if Dockerfile dependencies change
$ docker compose -f /data/docker/subocr/docker-compose.yml up -d --build

Automated System Backup & Restore

Backup Integration: SubOCR is integrated into the standard host ul backup and restore suite under /home/luke/git/backups/ul/scripts/ and runs automatically during sudo /home/luke/git/backups/ul/scripts/run_all_backups.

  • Backup Script (/home/luke/git/backups/ul/scripts/backup/subocr):
    1. Rotates any existing backup in /data/backups/services/current/subocr/ to /data/backups/services/archive/subocr/.
    2. Refreshes /data/docker/subocr/data/video_sources.json (recording every job's ID, title, category, youtube_url, source_path, and host_path) and executes a non-blocking SQLite WAL checkpoint (PRAGMA wal_checkpoint(TRUNCATE);) on /data/docker/subocr/data/subocr.db.
    3. Creates a compressed archive (subocr_data.tar.xz, ~1.4 MB) of /data/docker/subocr/ including app.py, vocab_pages.py, Dockerfile, docker-compose.yml, static/, data/subocr.db, data/video_sources.json, all pre-generated .html vocabulary/grammar pages (data/vocab_pages/*/*.html), and all exported job subtitle/study files (data/jobs/*/*.srt, .vtt, .anki.tsv, .study.md), while excluding transient model/kernel caches (cache/), the static JMdict reference database (data/jmdict_jlpt.db), and large raw video files (*.mp4, *.mkv, *.webm, *.mov).
    4. Generates 10% Reed-Solomon parity protection archives (subocr_data.tar.xz.par2) via par2 create -r10.
  • Restore Script (/home/luke/git/backups/ul/scripts/restore/subocr):
    1. Verifies archive integrity against the par2 parity data (par2 verify).
    2. Extracts subocr_data.tar.xz into /data/docker/subocr/, restoring the SQLite database, all pre-generated vocabulary HTML pages, and all subtitle exports.
    3. Restarts the subocr container and checks all jobs in subocr.db: if any job's video file is missing on disk and has a recorded youtube_url, automatically re-downloads the video via yt-dlp.
# Run a standalone SubOCR backup
$ sudo /home/luke/git/backups/ul/scripts/backup/subocr

# Restore from the latest backup in /data/backups/services/current/subocr/
$ sudo /home/luke/git/backups/ul/scripts/restore/subocr

# Restore from a specific timestamped backup
$ sudo /home/luke/git/backups/ul/scripts/restore/subocr 2026-10-03_1133.09