STT for an Unsupported Language on iPhone – Part 2

In the first part, I described why I can't simply turn on Apple Intelligence and dictate my journal in Czech, and how I arrived at the Whisper model. Now let's look at the question I should have asked right at the start: where should the intelligence actually run?

Phone or Cloud?

The simplest path to a mood mentor leads through an external service. I send the text to a large language model in the cloud and get a response a moment later. For developers this is convenient, and the models there tend to be more powerful. But a journal is the most personal thing you can write into an app. It contains relationships, worries, health conditions and things you never say out loud to anyone. That is why, from the very beginning, I cared about keeping as much as possible directly on the phone and, where I can't do without a server, having it run within the EU.

Model Directly on the Device

This option has many advantages: entries never leave the phone, and it works even offline and without ongoing API call fees. Unfortunately, it also has drawbacks that are still quite painful: the model takes up hundreds of megabytes that must be downloaded and stored, and the phone has limited memory and performance, which limits the size of the model that can be used.

External Service

Here the advantages, namely that you can use more powerful models and thereby keep the app small, are outweighed by the drawbacks: data leaves the phone, and you depend on connectivity, fees and secure storage of the API key.

Apple Intelligence: A Dead End for an Unsupported Language

For users with English or German, the answer would be easy: use what Apple has already built into the phone. Apple Intelligence runs largely on the device itself, and SpeechTranscriber entirely so. I download nothing and pay nothing. But Czech is not among the supported languages, so I have to take care of it myself.

That doesn't mean I'm giving up on the idea, though. The app simply has to support two paths. If the phone has a supported language, it uses Apple's built-in tools. If not, it downloads its own model and handles things by itself.

So I tried:

• SmolLM2 1.7B (HuggingFace, ~1050 MB) – but inference didn't work: swift-transformers 0.1.24 lacks TextGenerationPipeline
• SmolLM2-360M-Instruct 8bit (CoreML, 388 MB) – produced nonsense on the A15.
• Qwen2.5-0.5B-Instruct 4bit (286 MB) – chosen for its better Czech, but it didn't do well on the A15 either.

So for now, the only options are either to do without analysis of the transcribed voice, or to enable access to an external LLM and run the analysis externally. Users can decide for themselves. For the external analysis, I offer the EU-based LLM Mistral or OpenAI, with users supplying their own key. Or they can use our subscription and get our OpenAI model on our server.

How It Works in Practice

Everything starts on the wrist. I open the app on the watch, dictate an entry and start processing. The watch hands the recording over to the iPhone, which transcribes it to text. Because the model runs in the background, this works even with the phone locked in my pocket.

The transcribed text is combined with data from the watch and calendar. The result is a structured response and an icon that appears in the entry preview. I see it on both the iPhone and the watch.

The First Launch Is Slow, the Next Ones Aren't

On the very first use, the app downloads the model and prepares it for the specific phone. This takes on the order of minutes, and in the meantime the watch shows "Processing". After that, the model loads into memory in five seconds. However, the preparation sometimes has to be repeated, for example after an iOS or app update, and then there is a short wait again.

In the next part: Czech as a "small" language, what doesn't work yet and where the results are heading.