Metadata-Version: 2.4
Name: captionforge
Version: 0.2.0
Summary: Local, free automatic video-caption generator using faster-whisper and ffmpeg.
License: MIT
Project-URL: Homepage, https://github.com/Bryandero98/captionforge
Project-URL: Repository, https://github.com/Bryandero98/captionforge
Project-URL: Issues, https://github.com/Bryandero98/captionforge/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: faster-whisper
Requires-Dist: argostranslate
Requires-Dist: fastapi
Requires-Dist: uvicorn
Requires-Dist: python-multipart
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: httpx; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Provides-Extra: build
Requires-Dist: pyinstaller; extra == "build"
Dynamic: license-file

# CaptionForge

**English** | [Español](README.es.md)

Local, free automatic video captions - the CapCut/Kapwing auto-caption
experience, but 100% on your own machine. No watermark, no monthly limit,
no account.

Drag in a video, get back a `.srt` file and/or the video with subtitles
burned in, in a modern social-media style. Transcription runs locally via
[faster-whisper](https://github.com/SYSTRAN/faster-whisper); translation
(to any language, not just English) runs locally via
[argos-translate](https://github.com/argosopentech/argos-translate);
burning subtitles into the video uses [ffmpeg](https://ffmpeg.org/).
Nothing leaves your machine.

## Requirements

- Python 3.10+
- [ffmpeg](https://ffmpeg.org/download.html) on your `PATH`

## Install & run

```sh
pip install -e ".[dev]"
captionforge serve
```

This starts a local server (default `http://127.0.0.1:8420/`) and opens it
in your browser. The **ES / EN** switch in the top-right corner sets the
UI language (remembered for next time via `localStorage`; it defaults to
your browser's own language). Drag in a video, choose a Whisper model size
and optionally a source language / translation target, and click
**Generate captions**.

Once transcription finishes:

- **Download `.srt`, `.vtt`, or `.ass`** directly - the same segments,
  three formats (`.vtt` for a plain HTML `<video><track>`, `.ass` for an
  editor that wants real styling/karaoke tags).
- **Edit captions** - fix a transcription mistake before burning, and drag
  each segment's start/end directly over a waveform to retime it. Words
  faster-whisper transcribed with low confidence are underlined (with a
  subtle highlight) right in the editor, so you know what to double-check
  instead of trusting the transcript blindly.
- **Pick a caption style** (Modern, TikTok bold, YouTube classic, Minimal)
  and turn on **word-by-word karaoke highlighting** for the burn - available
  whenever a segment has per-word timing, whether that's faster-whisper's
  own real timing or the approximate, character-length-based timing this
  app synthesizes after you translate or edit a segment's text (see "Known
  limitations").
- **Burn into video** - a separate, on-demand step from transcription -
  you're never forced to re-encode the whole video just to get the text.

**Recent jobs**, below the main card, remembers your last 10 uploads in
this browser (`localStorage`) with direct re-download links for all four
formats - handy after you've moved on to a new video and the "current
job" indicator above has moved with you.

CaptionForge processes one video at a time by design on the backend - a
second upload while one is running is rejected with a clear error rather
than silently overwritten. The frontend builds on that: **select or drop
several videos at once** and they're queued client-side (name + status:
waiting/running/done/error), uploaded one at a time, automatically moving
to the next file as soon as the current one reaches done or error - so one
bad file in a batch doesn't block the rest. Editing and re-burning are
only offered for the current job (the last one in a queued batch); older
jobs in history are downloads only (see "Known limitations").

## Architecture

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/architecture-dark.svg">
  <img src="docs/architecture-light.svg" alt="Diagram: the browser uploads a video to FastAPI, which hands it to a background pipeline that calls ffmpeg, faster-whisper and argos-translate in turn, writing results to a job directory on disk, while the browser watches progress over a separate SSE stream relayed from JobStore.">
</picture>

The upload request returns in milliseconds - the real work runs as a
background task while the browser watches it happen over a live SSE
stream, not by polling.

## How it's built

- `src/captionforge/srt.py` - pure formatting/assembly for `.srt`, `.vtt`,
  and karaoke-capable `.ass` (per-word `\k` tags whenever `Segment.words` is
  present). `WordTiming.probability` carries faster-whisper's own per-word
  confidence (`None` only for a word this app synthesized itself, never a
  real transcription). `redistribute_word_timings()` is the shared
  approximate-timing heuristic both translation and text-editing use once a
  segment's words no longer match its ORIGINAL per-word timing - see "Known
  limitations" for exactly what it does and doesn't guarantee. Also the
  plain-dict (de)serialization used to persist segments to
  `segments.json`. No I/O.
- `src/captionforge/translate.py` - local translation of already-timed
  segments via argos-translate, decoupled from Whisper (whose own
  `task="translate"` only ever translates into English). The
  ORIGINAL-language per-word timing can't survive a translation (different
  words, count, and often order) - rather than dropping word-level timing
  outright, `redistribute_word_timings()` approximates new timing for the
  translated text.
- `src/captionforge/waveform.py` + `ffmpeg_utils.build_waveform_extract_cmd`
  - downsampled audio amplitude data for the editor's waveform backdrop: a
  single ffmpeg pass decodes+resamples a job's original video to raw 8-bit
  PCM at a low, fixed sample rate, bucketed down to at most 2000 peaks
  before being sent to the browser.
- `src/captionforge/ffmpeg_utils.py` - pure ffmpeg `argv` construction
  (audio extraction, subtitle burning, waveform extraction) - never
  executes anything itself. `STYLE_PRESETS` (modern/tiktok/youtube/minimal)
  is the single source of truth both the plain-SRT `force_style` burn and
  the karaoke `.ass` burn render from.
- `src/captionforge/jobs.py` - an in-memory, thread-safe job state
  machine (`queued -> extracting_audio -> transcribing -> done ->
  burning_subtitles -> burned`, or `error` from anywhere). Holds ONE job
  at a time by design - `segments.json` and the output files persisted to
  disk (not this in-memory store) are what let "recent jobs" history keep
  working after a newer job takes over.
- `src/captionforge/pipeline.py` - orchestrates the above: ffmpeg runs via
  `asyncio.create_subprocess_exec`, Whisper/Argos (blocking, CPU-bound)
  run in a worker thread via `asyncio.to_thread`, so the server stays
  responsive (including the live progress stream) while a video is
  processing. Writes `segments.json` alongside the `.srt`; the karaoke burn
  path builds a `karaoke.ass` from it on demand.
- `src/captionforge/app.py` + `routes/` - the FastAPI layer: upload,
  Server-Sent Events for live progress, the `.srt`/`.vtt`/`.ass`/video
  downloads (with a disk-existence fallback for a job that's no longer the
  one JobStore is tracking - safe because CaptionForge's one-job-at-a-time
  design guarantees any older job already reached a terminal state),
  segment editing (`GET`/`PUT .../segments`, current job only - `PUT`
  accepts `text`, and/or `start`/`end` for a waveform-drag retime),
  `GET .../waveform` (works for any job, current or historical - it only
  ever needs the original video file), and burn (`style`/`karaoke` form
  fields).
- `src/captionforge/static/` - the frontend: one plain HTML/CSS/JS page,
  no build step, no framework. `i18n.js` is a small flat-dictionary
  translator (Spanish/English, `localStorage`-backed) that drives every
  `data-i18n`-tagged element in `index.html`; job stage labels are derived
  client-side from the language-neutral `status` field the API already
  returns, not from the backend's own (Spanish-only) `stage_label` text.
  The segment editor (`app.js`) renders the waveform as a `<canvas>` with
  draggable start/end handles per segment, and underlines any word below a
  confidence threshold using the per-word `probability` the API returns.

`scripts/smoke_test_pipeline.py` exercises the whole transcribe ->
translate -> burn pipeline directly against a real video, no server
involved - the fastest way to sanity-check the core after touching
anything Whisper/ffmpeg/Argos-related.

## Development

```sh
python -m venv .venv
source .venv/Scripts/activate   # or .venv/bin/activate on Linux/macOS
pip install -e ".[dev]"
pytest
```

`tests/fixtures/tiny_test_clip.mp4` is a real ~10s clip (synthesized
speech) used by the live-pipeline tests - not a mock.

## Packaged build (first step)

A deliberately narrow first step toward issue #2's packaged installer -
**not** the full multi-platform, signed, GPU-aware installer described in
"Roadmap" below.

```sh
pip install -e ".[build]"
python scripts/build_installer.py
```

This produces a single `dist/captionforge` executable (`.exe` on Windows)
via PyInstaller, driven by `scripts/captionforge.spec` (the commented,
reproducible build config) from the `scripts/pyinstaller_entrypoint.py`
entry point - the same thing `captionforge serve` does (open the browser,
serve on the default port with the default model), minus argparse, since
a packaged executable has no terminal to pass flags to.

What this covers:

- **One platform at a time** - whichever OS you run the build script on.
  PyInstaller doesn't cross-compile; a Windows machine only ever produces
  a Windows executable, same for Linux/macOS.
- **CPU-only** - bundles whatever faster-whisper/ctranslate2 backend is
  already installed in the build venv, no CUDA/cuDNN.

What this deliberately does **not** cover yet (see issue #2's own
discussion for why each of these is a separate, non-trivial piece of
work):

- **Multiple platforms from one place** - building/distributing
  Windows + macOS + Linux together.
- **Code signing** - the built executable will trigger Windows
  SmartScreen / macOS Gatekeeper warnings on first run.
- **GPU/CUDA build selection** - no per-OS GPU detection or a
  GPU-enabled build variant.
- **Bundling ffmpeg** - the built executable still expects `ffmpeg` on
  `PATH`, exactly like running from source.

## Known limitations

- `argos-translate`'s default `compute_type="auto"` resolves to a quantized
  kernel that silently produces garbage (repetition-loop) output for at
  least one language pair on at least one real CPU - verified live during
  development. `captionforge` forces `float32` (see `translate.py`) to
  avoid this; if you use `argos-translate` directly elsewhere, verify your
  own language pair isn't affected before trusting quantized output.
- The UI language switch is frontend-only. Everyday status text (stage
  labels, generic error framing) is fully bilingual, but the rare
  server-generated error message - an unsupported file format, a job
  conflict, an ffmpeg failure - is still written in Spanish by the backend
  and shown as-is regardless of the selected UI language.
- Editing captions and re-burning are only available for the CURRENT job -
  once a new upload starts, JobStore forgets the old one (by design; see
  jobs.py), so an older job in "recent jobs" offers downloads only. This
  matches the natural flow (transcribe -> optionally edit -> burn) and
  the one-job-at-a-time state machine, which has no path back from BURNED.
- Karaoke highlighting needs word-level timing. A segment translated or
  manually edited gets APPROXIMATE word timing instead of its original
  (real) one: `redistribute_word_timings()` splits the segment's existing
  [start, end) span across the new text's words, proportionally by
  character length - a cheap, honest stand-in for real forced alignment,
  not an acoustically verified one. It is NOT lip-synced: a translated
  sentence's words rarely land where the corresponding sound actually
  occurs, especially for language pairs with very different word order.
  Every word this app synthesizes this way has `probability: null` in the
  segments API response specifically so nothing mistakes it for a real
  transcription confidence score. The karaoke checkbox is simply hidden
  when no segment has any word timing at all (real or approximate).
  Real forced alignment (a wav2vec2-style model, the way
  [WhisperX](https://github.com/m-bain/whisperX) does it) would fix this
  properly, at the cost of a whole new model dependency - out of scope for
  now; see the diarization/voice-separation issues below for the same
  "new heavy ML dependency" trade-off applied to two other features.
- The low-confidence word highlighting in the editor is only as good as
  faster-whisper's own per-word `probability` - a word can be confidently
  wrong (misheard but pronounced clearly) or unconfidently right (correct
  despite noisy audio). Treat the underline as "worth a second look", not
  as a correctness guarantee.
- The waveform editor's drag handles let you shrink or grow a segment
  freely; there's no validation against a NEIGHBORING segment's start/end,
  so it's possible to drag two segments into overlapping (or gapped) time
  ranges. Nothing crashes, but review the result before burning if you make
  a large adjustment.
- "Recent jobs" lives in `localStorage`, so it's private to one browser -
  it does not survive clearing site data and is never shared between
  devices.
- The upload queue lives only in the page's memory - reloading mid-batch
  resumes the single file that was actively uploading (same as any single
  job), but any files still queued behind it are lost; re-select them to
  keep going.
- A job's files (video, `.srt`/`.vtt`/`.ass`, `segments.json`) are deleted
  automatically 7 days after they were last written - each new upload
  prunes anything past that age. An entry can outlive its files in "recent
  jobs" (which has no expiry of its own); its download links just 404 once
  that happens.

## Roadmap

Ideas worth doing eventually, deliberately not started yet:

- **A packaged native installer** (Windows `.exe`, macOS `.dmg`, Linux
  `.AppImage`/`.deb`) so a user doesn't need Python or ffmpeg pre-installed.
  A first, narrow step exists today (see "Packaged build (first step)"
  above: one platform, CPU-only, unsigned, ffmpeg not bundled). What's
  still missing - multi-platform distribution from one place, a bundled
  static ffmpeg per OS, per-OS GPU/CUDA detection, and code-signing (to
  avoid Windows SmartScreen / macOS Gatekeeper warnings) - is comparable
  in effort to building the app itself, which is why it's out of v1 on
  purpose.
- **A hosted version** - CaptionForge needs real CPU (or GPU) for
  Whisper/ffmpeg, so a free-tier host isn't enough for serious use; a paid
  host is the realistic next step if there's ever demand for a
  "no-install-at-all" option. See "Support this project" below.
- **Speaker diarization** ("who said what") via
  [pyannote.audio](https://github.com/pyannote/pyannote-audio) - see
  [issue #4](https://github.com/Bryandero98/captionforge/issues/4) for why
  it's deferred: a second heavy PyTorch-based ML dependency, plus real
  friction from pyannote's gated Hugging Face models (an account + accepted
  terms + a personal access token, unlike faster-whisper's anonymous
  downloads today).
- **Vocal/source separation before transcription** (via
  [Demucs](https://github.com/facebookresearch/demucs)) for noisy or
  music-heavy audio - see
  [issue #5](https://github.com/Bryandero98/captionforge/issues/5) for why
  it's deferred: a third heavy ML dependency, real added runtime cost for
  the common case (clean dialogue) that doesn't need it, and a quality
  trade-off that needs real before/after comparison, not just an
  assumption that separation always helps.
- **Real forced alignment** after a translation or manual text edit (a
  wav2vec2-style model, the way WhisperX does it) - today's approximate,
  character-length-based word-timing redistribution (see "Known
  limitations") is a deliberately cheap stand-in for this, not a
  replacement for it.
- **A real parallel backend queue** (processing more than one video at
  once) - CaptionForge is one-job-at-a-time by design today (see `jobs.py`
  and the frontend's own client-side upload queue, which uploads
  sequentially specifically because the backend can only run one job at a
  time). Worth doing eventually for a multi-core machine, but a genuinely
  bigger change (worker pool, per-job resource limits, a queue that
  survives a server restart) than anything else in this list - no concrete
  plan yet, flagged here only so it isn't mistaken for an oversight.

## Support this project

CaptionForge is free and local by design, and will stay that way. A tip
doesn't unlock anything - it goes toward eventually paying for a real host,
so people who don't want to install anything have a "no-install-at-all"
option too:

- **Ko-fi:** [ko-fi.com/bryandero98](https://ko-fi.com/bryandero98)
- **USDT (TRC20):** `TEG4Kk2qXYMQ4mHNd7dPhSPRyT14CGr2or` - double-check the
  network is set to **TRC20** before sending; a transfer on the wrong
  network can't be recovered.

## Ideas & contributions

Suggestions for what CaptionForge should do next are welcome, not just bug
reports - open an issue with what you'd want, even a rough one. See
[CONTRIBUTING.md](./CONTRIBUTING.md) for how to send a PR.
