Why Is Video So Hard to Search in the First Place?
Video is hard to search because it's a timeline, not a document. A text file has words you can grep in milliseconds. A video has no built-in index of what happens inside it, only a stream of frames playing over time. To find the three seconds you want in a 40-minute file, the default tool is your own eyeballs plus a scrubber bar.
The common workaround is to transcribe the audio and search that. This helps for interviews and lectures, but it quietly fails on a huge amount of real footage. A transcript only records what was said. It knows nothing about what was shown: the product on the table, the sign in the background, the person laughing, the drone shot of a coastline. Silent B-roll produces a blank transcript. Music-only montages produce a blank transcript. Anything visual is invisible to a word index.
So the real engineering problem isn't "search the words." It's "understand the video, then make that understanding searchable, and tie every piece of it back to a precise second." The rest of this post walks through how we approached each of those steps. For the plain-English version of the concept, see what video search is.
Step 1 — How Do You Decide Which Frames to Analyze?
You analyze a representative subset of frames, because analyzing every frame is wasteful and analyzing too few misses moments. A typical video runs at 24 to 30 frames per second. A 30-minute clip is roughly 54,000 frames. Sending all of them to a vision model would be slow and expensive, and most adjacent frames are nearly identical anyway. The whole game is choosing frames that carry information.
There are two broad strategies, and we lean on a blend of both.
Uniform Sampling
Uniform sampling grabs a frame at a fixed interval, say one frame per second or one every two seconds. It's simple, predictable, and guarantees even coverage across the whole timeline. The downside is that it's blind to content. A fixed interval can land twice on a static talking head and completely skip a fast cut that flashes by in under a second.
Scene-Change Detection
Scene-change detection compares consecutive frames and only keeps a new frame when the picture shifts meaningfully. It measures how much changed, a cut, a new location, a big movement, and samples more densely where the action is. This spends your analysis budget where it matters and stops wasting it on frames that barely differ from the last one.
The Trade-Off, and Keyframes
The trade-off is coverage versus cost. Sample too sparsely and you'll miss a quick moment; sample too densely and you pay for near-duplicate frames that teach the model nothing new. Our approach is to sample at a sensible baseline interval, then increase density around detected scene changes, and select a single keyframe to represent each stable shot. A keyframe is the frame that best summarizes a segment, so one clear image stands in for a run of similar ones. The result is fewer frames sent to the model, with the important moments still represented.
Step 2 — Why a Multimodal Vision-Language Model Instead of a Transcript?
Because a transcript reads audio, and a multimodal model reads the actual picture. A vision-language model takes an image (or a set of frames) and produces a structured description of what's in it: objects, actions, people, settings, and text baked into the frame. That's a fundamentally richer signal than speech-to-text, because it captures the layer of meaning that never reaches the microphone.
Concretely, from a single frame a multimodal model can surface things like:
- Objects: a red car, a coffee cup, a whiteboard, a laptop.
- Actions: running, pouring, pointing, hugging, opening a door.
- People and interactions: how many people, what they're doing, how they react.
- Scene and setting: indoor or outdoor, day or night, a kind of location.
- On-screen text: signs, slide titles, captions, and labels the camera caught.
None of that requires anyone to speak. That's the point. A stunned facial expression, a logo on a wall, a sunset, all of it is searchable when a model reads frames directly. We compare the two approaches head-to-head in SearchByVideo vs transcript search.
What Actually Matters When You Pick a Model
Choosing a multimodal model is a set of trade-offs, not a single "best" answer. In our experience, four factors dominate:
- Accuracy on real footage. Clean benchmark images are easy. Motion blur, low light, odd angles, and busy frames are where models diverge, so we test on messy, real-world clips, not curated samples.
- Latency. Analysis reads a whole video, so per-frame speed compounds. A model that's a second slower per frame turns a two-minute job into a much longer one.
- Cost per frame or per token. This sets the price of analyzing an hour of footage. Small differences multiply across thousands of frames.
- Context window. A larger window lets the model reason over several frames together, which helps it understand an action unfolding rather than judging each still in isolation.
There's no universal winner. The right pick depends on how you weight speed against accuracy against price for the videos your users actually upload.
Step 3 — What Does the Searchable Index Look Like?
The index is a list of structured moments, each one a small record of what was detected, how confident the model was, and when it happened. Instead of storing pixels, we store meaning tied to time. Once that record exists, searching it is a fast text-matching problem, not a video problem, which is why every search after analysis is instant and free.
A single indexed moment looks roughly like this:
{
"timestamp": 132.5,
"concept": "person pouring coffee into a white mug",
"tags": ["coffee", "mug", "kitchen", "pouring"],
"on_screen_text": null,
"confidence": 0.87
}Multiply that across a whole video and you get a dense, machine-readable table of contents. When someone searches "pouring coffee," we match their words against the concept and tags of every stored moment, rank the hits, and return timestamps. No frames are re-read. The expensive understanding step already happened once.
This is also why the index is portable and cheap to query. It's just structured data. You can filter it, sort it, and match against it thousands of times for effectively nothing, because the model was only ever called during analysis. For the user-facing walkthrough of this flow, see how it works.
Step 4 — How Do Detections Map Back to a Precise Second?
Every detection carries the timestamp of the frame it came from, so a match points straight back to a real second in the source video. When the sampler pulls a frame, it records that frame's exact position on the timeline. Whatever the model detects in that frame inherits the same timestamp. Click a result and the player seeks directly there.
The subtlety is that we sampled, so there are gaps. If we analyzed one frame per second, a detection is accurate to within that window, not to the individual millisecond. In practice that's fine, because a person looking for "the toast" wants to land near the toast, not on a specific pixel of it. Where precision matters more, denser sampling around scene changes tightens the window automatically, since that's exactly where we already added extra frames.
Ranking handles the last problem: the same concept often appears in several nearby frames. Rather than dumping every near-duplicate hit on the user, we group detections of the same concept in a time neighborhood and surface the strongest one, ranked by the model's confidence. The top result is usually the clearest, best-framed instance of what you asked for, and the rest stay available if you want them.
Step 5 — How Do You Keep This Affordable?
The core trick is analyze-once, search-forever. The only step that calls an expensive model is analysis, which runs a single time per video. Everything after that, every query, every refinement, every re-sort, runs against the local index and costs essentially nothing. So the pricing model can make search unlimited and free, because search genuinely is cheap once the index exists.
On top of that structural decision, a few tactics keep the one-time analysis cost down:
- Smart sampling. Fewer, better-chosen frames beat brute-force full-frame analysis. Keyframe selection and scene-aware density are cost controls as much as quality controls.
- Batching. Sending frames in groups reduces per-request overhead and uses the model's context more efficiently than one lonely frame at a time.
- Skipping near-duplicates. If two frames are almost identical, analyzing both buys nothing. Dropping the duplicate saves a call with no loss of coverage.
- Right-sizing the model. Not every job needs the largest, slowest model. Matching model capability to the footage keeps spend proportional to value.
The honest summary: cost control isn't one clever hack, it's the compounding effect of sampling fewer frames, reusing the index forever, and never paying twice for the same understanding.
What Doesn't Work Yet, and What This Doesn't Replace
Visual search is powerful, but it isn't magic, and pretending otherwise would be dishonest. A few areas are still genuinely hard.
- Fine visual distinctions. Telling two similar-looking objects apart, or reading a specific brand of a generic item, is unreliable when the frame is small, blurry, or poorly lit.
- Fast micro-moments. A detail that flashes on screen for a few frames can fall between samples. Denser sampling helps but costs more, so there's a real limit.
- Abstract or subjective queries. "The emotional part" or "the funny bit" depends on interpretation the model doesn't share with you. Concrete, describable queries work far better than vibes.
- Precise wording. When you need an exact quote, the audio track is still the right source. Visual understanding tells you what was shown, not the literal phrase someone spoke.
This is why we treat scene search and transcript search as complements, not rivals. Reading frames finds the silent reaction and the background sign; reading audio finds the exact sentence. A system that does both covers more of a video than either alone. For more articles in this series, browse the blog.
Closing Thoughts
Turning a video into something searchable comes down to four honest engineering steps: sample the frames that carry information, read them with a multimodal model that understands pictures and not just speech, store that understanding as structured moments tied to timestamps, and align every match back to a precise second. The analyze-once design is what makes it practical, since the heavy model work happens a single time and every search afterward is cheap. None of it removes the hard trade-offs around sampling density, model choice, and cost, but naming those trade-offs plainly is how you end up with a tool that actually finds the moment you were looking for.