Notes: AI

I've been using Whisper pretty much nonstop for meeting transcription over the past six months, and I've noticed a funny effect: the bigger the model, the better it is at recognizing actual speech — and the more confidently it makes shit up when there is no speech at all.

For example, this often happens at the beginning of a webinar or a meeting while people are still joining. It's quiet, nobody is talking, maybe someone is loudly sipping coffee. And then Whisper suddenly starts spitting out something like this:

Subtitle editor A. Sinetskaya, proofreader A. Yegorova
Subtitle editor A. Sinetskaya, proofreader A. Yegorova
Subtitle editor A. Sinetskaya, proofreader A. Yegorova
Subtitle editor A. Sinetskaya, proofreader A. Yegorova

And worse, it sometimes keeps doing this even after perfectly intelligible speech has started.

This got annoying enough that I finally went down the rabbit hole.

Why does it transcribe silence like that?

Whisper isn't just a sound-to-text recognizer. Under the hood, there's a pretty powerful language model trying to predict the most likely text based on both the audio and the context it has already transcribed. Most of the time, that's exactly what you want: it handles speech, context, names, and complicated phrases remarkably well.

But if the input is silence, noise, or some unintelligible mumbling, the audio stops being very useful. And that's where the model starts guessing: okay, what kind of text could plausibly go here? It was trained on a gigantic dataset of audio with text transcripts scraped together by OpenAI from all over the internet — around 700,000 hours, if I remember correctly.

And I assume plenty of that stuff came from videos with subtitles — and subtitles are messy as hell. During a silent bit, the subtitle track might happily contain the translator's name, an editor credit, a studio name, "thanks for watching", or some other garbage that isn't present in the audio at all.

And that, I suspect, is how Sinetskaya and Yegorova entered the history books. I couldn't figure out what exactly they worked on, but apparently some subtitle credit mentioning them occurred often enough in the training data for Whisper to eventually decide that silence and Yegorova are, statistically speaking, basically the same thing.

Why doesn't this happen only during silence?

Yeah, if it merely produced garbage during pauses, this wouldn't really be a problem. The garbage phrase is known, so you could just filter it out. The nasty part is that a bad transcription poisons the model's context. Oversimplifying a bit: Whisper hears silence, decides it's Yegorova, then feeds “Yegorova” back into itself as context for the next chunk and carries on from there.

The next chunk might also become Yegorova. And the next one. And the next... By the time the speaker finally starts talking, they'd better enunciate REALLY clearly if they want to convince the model that yes, this is in fact human speech and not Yegorova, who has now accumulated twenty chunks' worth of momentum. As a result, Whisper can get stuck in this state for a surprisingly long time.

The problem has been around forever — okay, fine, about four years — and it affects a whole bunch of languages. Russian, incidentally, stands out: while Italians, French, and Spaniards get the boring old "Subtitles created by the Amara.org community", we, shall we say, have our own fandom.

How do you deal with it?

The usual advice is to tune inference: reset context after long pauses, fiddle with no_speech_threshold and condition_on_previous_text, strip silence beforehand using VAD, and so on. I never got around to trying VAD, but I did experiment with the other options, and yes — you can get perfectly decent results for a particular troublesome video.

The problem is that this doesn't really work for my use case. I need to transcribe all kinds of stuff: meeting recordings, lectures, phone calls, whatever. A combination of tricks carefully tuned for one video tends to fall apart on another. Out of pure engineering stubbornness, I even tried building a wrapper along the lines of: "run the video through three different inference setups, analyze all three outputs, then pick whichever one has the lowest Yegorova concentration". It still worked kinda meh.

So I eventually went with a different solution: use a dumber model!

Seriously. It works. At least on my recordings, Whisper Small falls into repetition loops on silence much less often. My guess is that the weaker model simply has a harder time confidently hallucinating a garbage pattern, so when it hears silence, it just outputs nothing.

Oversimplifying again:

• Whisper Small: Uh... I can't hear anything. No idea.
• Whisper Medium: Uh... I can't hear anything... (strains all its parameters) OH! I KNOW! (fanfare starts playing) THAT'S YEGOROVA!

:D Suffering from success. Literally.

The transcription quality with Whisper Small is obviously a bit worse — more mangled words, less coherent phrasing — but that's still better than Sinetskaya eating a third of the recording and Yegorova devouring the rest overall, the result is much more predictable.

Besides, I'm going to throw the transcript into something like ChatGPT afterwards anyway to generate a summary, and for that absolute gigachad, reconstructing the conversation context is barely an inconvenience even if a few chunks of dialogue got mangled or dropped.

The desktop version of Codex has pets — little pixel characters rendered on top of the UI that show what the agent is currently doing. When it's working, the pet walks around; when it's waiting for input, it calls for you; when something goes wrong, it squints, and so on. The general idea is that you kick off some workflow, go do something else, and can still tell at a glance how things are going.

Not exactly a killer feature, but it's fun. I wasn’t particularly into the built-in pets, though, so I started browsing community-made ones and somehow ended up making my own — based on Hatsune Miku's design from Miku'n POP, an ancient little platformer.

Turns out it's not especially complicated. I took the sprites from MikuPet, whose author uses the same Miku'n POP design. The pet format itself is pretty straightforward, and with Codex helping me poke around, there wasn't much reverse-engineering involved. I actually spent more time figuring out who owns which bits of artwork, what the reuse terms are, and all that fun licensing stuff.

You can grab the pet from GitHub. There’s also a screenshot and a detailed rundown of what's under the hood.

Free bonus: the release workflow for a project like this can, with complete justification, be called "Release pet" :)

One thing AI has genuinely made better for developers: tiny side projects actually get finished now.

I'm talking about all those little tools, MVPs, and random ideas that used to go straight into the "maybe someday" pile. You want to build them, sure, but spending three evenings on yet another tiny utility is a pretty hard sell.

Last week, for example, I vibe-coded an Obsidian plugin for keeping note properties tidy. The idea is simple: for each folder, you can define which properties notes are supposed to have and what order they should appear in. The plugin adds missing ones, removes unwanted ones, and puts everything back where it belongs — instead of preserving whatever mess I hastily threw together months ago and never bothered to fix.

There's a more detailed explanation in the GitHub repo if you're curious.

It's a tiny thing, obviously. Mostly just me feeding my inner perfectionist. But I do like things neat, and apparently I'm not alone: seventy people downloaded the plugin in its first three days.

Greetings, fellow people who derive unreasonable satisfaction from tidy metadata :)

Corpus has published the Russian edition of If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares. Both are well-known researchers in AI safety, AI alignment, and the risks of superintelligence.

In the book, the authors explain clearly and step by step why the title isn't hysteria, exaggeration, or clickbait. The point is simple and unpleasant: if anyone on the planet actually builds AGI or ASI using technology anywhere close to what we have now, humanity is screwed.

I hadn't read Soares before, but Yudkowsky has always had this rare combination of deep expertise and the ability to talk about it like a normal human being, without disappearing into galaxy-brain jargon every thirty seconds. So you don't need a specialist background: a lot of work has clearly gone into making the text understandable for people who don't happen to have a PhD in applied math.

Anyway, the book is great. I honestly can't remember the last time I tore through something like this in one sitting. It's available on Ozon, at Chitai-Gorod, and even in certain suspiciously nautical corners of the internet — which I condemn, obviously, but if your black-flag frigate is already parked outside your building, what can I do? :)

I started recording all my meetings back in 2020 — at least, that’s what my own notes say. Video is always more accurate than memory, and clicking "Start Recording" in OBS is the cheapest way to not lose some random-but-important detail.

The downsides are obvious, though: you can't quickly grasp the gist of a meeting from a video, searching through it is basically impossible, it eats disk space like it's bulking season, and it's painfully easy to accidentally capture something private. To partially compensate, I used to jot down key points in bullet form during the meeting, and later either throw them into a task tracker or write a mini-summary for myself: who I met with, what we discussed, what decisions we landed on. If I forgot something or missed details, I'd double-check the video.

But this method isn't perfect either. Even a rough outline steals focus from the actual meeting. And some "oh right, that detail matters" moments only reveal themselves when it's already too late.

So I ended up with a better approach: extract the meeting audio from the video, turn it into text (with a neural net), then turn that transcript into a detailed meeting summary (with another neural net). Yes, it's neural nets all the way down.

The easiest way to rip the audio is with ffmpeg (a command-line utility for working with audio/video). Here's an example (mono, 16 kHz sampling rate + loudness normalization):

ffmpeg.exe -y -i "D:\video.mkv" -vn -ac 1 -ar 16000 -af loudnorm -c:a pcm_s16le "D:\audio.wav"

As for speech-to-text: I experimented with Vosk + recasepunc, but... Yeah, no. Let's just say I'd rather not relive that experience. Meanwhile Boromir Whisper (OpenAI's speech recognition model) installs in the background in about 10 minutes:

py -3.10 -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install setuptools wheel
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install openai-whisper

Example run:

whisper "D:\audio.wav" --model small --language Russian --output_format txt

The result is a plain text transcript you can shove into any chatbot and get a fairly coherent summary. Sure, you still need to proofread it — fix mistakes and hallucinations, rephrase a couple things — but it's still way better than trying to write notes live.

And that’s basically the whole method. The only thing left is writing a simple script so you don’t have to run two commands manually every time. If you’re on Windows and you can't be bothered to vibe-code, you can grab my script and tweak it.

The script finds the first .mkv in its folder, runs it through ffmpeg + Whisper, and saves the result back into the same folder. If you use it: note that it runs via CUDA (CPU works too, just much slower), and it stores downloaded Whisper models not in the default cache, but inside the current Python virtual environment folder.

(if you really want, you can wire up an API call or point it at some local model via LM Studio — but for my personal setup I decided: nope, that's already too much engineering for a lazy win)

Meanwhile, OpenAI opened a branch in Tbilisi Actually, they just sell vapes here, and the logo designer was probably inspired by clouds of steam. But every time I walk by, it still catches my eye :)

Colleagues are actively experimenting with o1-preview from OpenAI: the model turned out to be genuinely interesting. Unfortunately, tasks for it don’t come up too often — regular 4o handles most day-to-day stuff just fine. Bug-finding in code, analyzing medical tests in an unfamiliar language, or trying to recall the name of a childhood book with only a vague recollection of the plot — I can’t easily think of anything it wouldn’t handle.

It’s always fascinating to see how the AI reasons, jokes, and generally tries to act human. I’d say it won’t pass the Turing test just yet, but every now and then, it comes eerily close.

The answer below made me smile. Even leaving philosophy aside — looks like Skynet is off the table! :) Though, I’m afraid the future from the SOMA ending is still possible.

I needed to quickly generate some images for a pet project. I grabbed the first tool at hand — WaifuDiffusion (a clone of StableDiffusion, but trained specifically on anime and manga).

Prompt:

A dimly lit, empty classroom with faintly visible music notes on the chalkboard. A young woman with long blue hair and blue eyes, is standing behind the teacher's desk, packing her things into a small handbag. She looks tired and worried.

Result:

It seems like a great opportunity to talk about how far neural networks have come! The model instantly got the idea and decided to introduce into the scene... Hmm... A wise bearded man? Or perhaps it was suggesting that I should learn to appreciate unexpected surprises more?

I think WaifuDiffusion just wanted to brighten my mood. They say laughter extends life. In any case, I consider the first generation a success! But it definitely needs some calibration :)

When ChatGPT remembers a detail from a conversation for the future, it displays a "Memory updated" badge above its response. Like, got it:

Every time I see it, it makes me smile because it instantly reminds me of The Walking Dead and the meme “Clementine will remember that”. In that game, the characters surrounding the protagonist would remember his decisions, and it would influence their behavior. This included Clementine — a girl the main character saved at the beginning of the game.

In the game, this mechanic was honestly implemented in a rather mediocre way, but when it comes to ChatGPT — it works. The attention to detail is so strong that sometimes I have to remind myself that I’m talking to a language model, not an extraordinarily attentive and very knowledgeable person.