How to Transcribe Confidential Audio Offline with Whisper and CTranslate2

Some recordings must never leave the machine they were made on. A government secretariat needed its confidential session recordings turned into text, without handing the audio to a cloud speech-to-text service. So I built a tool that transcribes entirely on-device, with zero cloud dependency.

This post shows how to run Whisper fast enough on an ordinary CPU using CTranslate2, how to design an interface that makes reviewing a transcript quick, and how to package an "offline" promise you can actually verify. Small Python examples at the end show the core pieces.

Background

Why offline

Cloud speech-to-text is a solved problem if you can send the audio. The interesting work is when you cannot. Confidential government session recordings are that case: the secretariat has to turn them into text, but they are not recordings you hand to a third party for transcription, let alone for the third party's training data.

The market for on-device speech is real and has been narrow on tooling. The off-the-shelf desktop apps are either slow, paid subscriptions that still ping a server, or locked to a specific hardware vendor. I wanted a tool that was honest about its architecture: the audio file enters the process, the transcript comes out, nothing crosses the boundary in either direction.

Whisper, CTranslate2 and faster-whisper

Whisper is OpenAI's open-source family of speech recognition models; its reference implementation runs on PyTorch. CTranslate2 is an inference engine for transformer models written in C++, with support for quantization, such as storing weights as 8-bit integers (int8) instead of floating-point numbers. faster-whisper is a Python library that runs Whisper models on CTranslate2.

Running Whisper fast on a CPU

OpenAI's Whisper models are the obvious starting point. The obvious deployment, the reference PyTorch model, is also slow on CPU. CTranslate2 reimplements transformer inference in optimized C++ with quantization, dramatically cutting both memory and latency.

In practice the same audio file that takes a reference Whisper model a minute on CPU runs in a fraction of that with faster-whisper / CTranslate2 at int8 quantization. The quality drop at int8 on the medium and large models is small enough that a human reviewer rarely catches a difference compared to the float16 baseline.

Building an interface for the reviewer

Real-time waveform feedback

The thing that made the tool feel professional instead of academic was not the transcription quality. It was the waveform.

During recording or playback, the UI renders a live waveform driven from the audio buffer. Spikes show where speech is present. Silent gaps are visible. The user can see which slice of the recording is being processed and seek directly to a region. Pairing waveform-driven seeking with the segmented output Whisper produces gave the UI an editing-board feel rather than a black-box-with-a-progress-bar feel.

# waveform.py, buffer-driven peaks
def compute_peaks(samples: np.ndarray, target_buckets: int) -> np.ndarray:
    chunk = max(1, len(samples) // target_buckets)
    chunks = np.array_split(samples[: chunk * target_buckets], target_buckets)
    return np.array([np.abs(c).max() for c in chunks], dtype=np.float32)

Segment-level output, not a wall of text

Whisper returns segments with start and end timestamps. The transcript view in the app respects those boundaries, each segment is its own clickable line. Clicking a segment seeks the audio to its start. Editing a segment edits only that segment's text. The user is never staring at an unbroken paragraph and trying to find where their attention was three minutes ago.

Packaging the offline promise

A tool that claims to be offline cannot ship as a Python script that pulls models from the internet on first run. The packaging had to make the offline promise legible.

Models are downloaded once, into a known local cache directory under the user's data path. The download is an explicit, one-time UI step, not an invisible side-effect.
After the first download, the app runs with the network off. I verify this by literally disabling the network adapter during integration tests on the build machine.
No telemetry. No analytics. No update-check ping. The app does not call out, period. That is the product.

What surprised me

CPU was enough

I designed the architecture assuming I would eventually need to ship a GPU path. For the audiences this tool serves, attorneys with modern laptops and clinicians with workstations, the CPU path is already faster than real-time on the medium-quality model. The GPU path is a future-me problem, not a today-me problem.

The hard part was the editing experience

Transcription accuracy plateaued early. The remaining work was all about giving the human reviewer good tools to fix the mistakes Whisper made, keyboard navigation across segments, undo on edits, fast playback for the segment under the cursor, export to plain text or DOCX. The interface is now most of the value of the tool.

Design decisions

Transcription accuracy plateaued early. Most of the design work went into the reviewer's tools.

The waveform is the main control

A live waveform shows where speech is, where the gaps are, and which slice is being processed. Clicking it seeks there, like an editing board.
One line per segment

Each Whisper segment is its own clickable line. Clicking seeks the audio; editing changes only that segment, never a wall of text.
Keyboard-first review

Keyboard navigation across segments, undo on edits, fast playback of the segment under the cursor, and export to plain text or DOCX.
Make the offline promise visible

The model download is an explicit one-time step in the UI, not a hidden side effect. After that: no telemetry, no analytics, no update ping.

Practical examples

Simplified example. The snippets below are generic Python showing the techniques from this post. They are not the production code.

1. Transcribe a file on CPU with faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel("medium", device="cpu", compute_type="int8")
segments, info = model.transcribe("recording.wav", beam_size=5)

print(f"Detected language: {info.language} ({info.language_probability:.2f})")

for segment in segments:  # transcription runs as you iterate
    print(f"[{segment.start:6.2f} -> {segment.end:6.2f}] {segment.text.strip()}")

Expected output (illustrative):

Detected language: en (0.98)
[  0.00 ->   4.20] Good morning, everyone.
[  4.20 ->   9.86] Let's start with the first item on the agenda.

2. Download the model once, then load it only from disk

# One-time step, triggered explicitly from the UI (needs the network once).
from faster_whisper import download_model

download_model("medium", output_dir="models/whisper-medium")

# Every run after that: offline only.
import os

os.environ["HF_HUB_OFFLINE"] = "1"  # set before importing faster_whisper

from faster_whisper import WhisperModel

model = WhisperModel("models/whisper-medium", device="cpu", compute_type="int8")

Loading from a local folder never needs the network, and HF_HUB_OFFLINE makes any accidental model-hub lookup fail instead of quietly going online.

3. Turn audio samples into waveform peaks

import numpy as np

samples = np.array([0.0, 0.1, -0.8, 0.2, 0.05, -0.02, 0.6, -0.3])
compute_peaks(samples, target_buckets=4)

Expected output, using the compute_peaks() function shown earlier:

array([0.1 , 0.8 , 0.05, 0.6 ], dtype=float32)

The eight samples are split into four buckets of two. Each bar of the waveform is the loudest absolute value in its bucket.

4. Make tests fail if anything touches the network

import socket

import pytest
from faster_whisper import WhisperModel


@pytest.fixture
def no_network(monkeypatch):
    def refuse(*args, **kwargs):
        raise RuntimeError("network access attempted")

    monkeypatch.setattr(socket.socket, "connect", refuse)


def test_transcribes_offline(no_network):
    model = WhisperModel("models/whisper-medium", device="cpu", compute_type="int8")
    segments, _ = model.transcribe("tests/fixtures/short.wav")
    assert list(segments)

This is a cheap check that can run on every build. It complements, rather than replaces, testing with the network adapter switched off.

Common mistakes

Benchmarking the reference PyTorch model on CPU and concluding local transcription is too slow.
Requesting float16 on a CPU. It is meant for GPUs; use int8 on CPU.
Treating segments as a list. It is a generator: nothing is transcribed until you iterate, and you can iterate only once.
Letting the first run download a model silently, so the "offline" app goes online without telling anyone.
Adding telemetry, analytics or an update check "just for now".
Showing the transcript as one wall of text and throwing away the segment timestamps.

Best practices

Run Whisper through faster-whisper with int8 on CPU, and compare against the float16 baseline on your own audio.
Make the model download an explicit, one-time step into a known local directory.
Prove the offline claim by disabling the network during integration tests.
Keep Whisper's segment boundaries in the UI: one clickable line per segment that seeks the audio.
Once accuracy plateaus, spend the effort on review tools: keyboard navigation, undo, segment playback and export.

Conclusion

On-device transcription is practical. Whisper through CTranslate2 at int8 is already faster than real-time on the medium-quality model on a modern CPU, and the quality drop is small enough that a human reviewer rarely notices. Once accuracy plateaued, most of the value moved into the reviewer's tools: the waveform, the segment list and the keyboard.

The takeaway: when the data cannot leave the machine, make the offline promise something users can see and you can test. An explicit model download, no network calls after it, and a test that fails if anything tries.

What I would do next

Speaker diarization. Whisper doesn't separate speakers. A diarization step layered on top would turn the tool from a transcription tool into a meeting tool.
Vocabulary biasing. Domain-specific names, reference numbers, proper nouns, are where on-device transcription still fumbles. A user-managed glossary that reweights the decoder would close most of that gap.
Cross-platform installers. Currently focused on Windows. macOS and Linux installer pipelines are the next unglamorous deliverable.

References and further reading

Need on-device AI integration? I take on offline / on-prem speech, NLP, and inference pipeline work end-to-end.

Start a conversation