How We Built AI Scene Search: Segments, Tags, Timestamps

Why Is Video So Hard to Search in the First Place?

Video is hard to search because it's a timeline, not a document. A text file has words you can grep in milliseconds. A video has no built-in index of what happens inside it, only a stream of frames playing over time. To find the few seconds you want in a 20-minute file, the default tool is your own eyes plus a scrubber bar.

The common workaround is to transcribe the audio and search that. This helps for interviews and lectures, but it fails on a lot of real footage. A transcript only records what was said. It knows nothing about what was shown: the product on the table, the sign in the background, the person laughing, the drone shot of a coastline. Silent B-roll and music-only montages produce a blank transcript.

So the engineering problem isn't "search the words." It's "describe the video once, make that description searchable, and tie every piece of it back to a point on the timeline." This post explains how SearchByVideo does that today, including the parts that are deliberately simple. For the plain-English version of the concept, see what video search is.

Step 1 — One Model Pass Over the Whole Video

We don't run our own frame sampler or scene-cut detector. Each uploaded video is sent once to Google Gemini 2.5 Flash, a multimodal model that can watch a video file directly, picture and sound together. The request goes through an API provider, and if that provider fails we fall back to Google's own Gemini API so the analysis still completes.

The prompt asks the model to act as a video analyst and cover the entire video with no gaps. For every segment it must return:

• a start and end time;
• a factual description of the people (appearance, clothing, expressions, actions), the objects and any on-screen text, the setting and lighting, and the audio — speech content, sounds and music;
• 5 to 15 short keyword tags in English.

The same prompt also asks the model to check for NSFW content first. If the video is NSFW, the model returns only a refusal flag and no segments, and the analysis is rejected.

Step 2 — Segment Length Depends on Video Length

How finely the video is cut into segments is the most important trade-off in the system, because it sets both cost and timestamp precision. We pick the segment size from the video's duration:

Short clips get dense descriptions; long videos get coarser ones so that the analysis stays within budget. Video length itself is capped at 30 minutes on Pro and 5 minutes without an active subscription, and the analysis request has a hard 120-second server time limit. If the model fails, times out or returns no segments, the 5 credits reserved for the analysis are refunded automatically.

Step 3 — What the Searchable Index Looks Like

The index is simply the list of segments the model returned, saved to our database. A single segment looks roughly like this:

{
  "startTime": "00:00",
  "endTime": "00:03",
  "description": "A woman with short black hair wearing a white sleeveless top is cutting a round cake with a knife on a white table.",
  "tags": ["woman", "short hair", "cake", "cutting", "knife", "table", "celebration"]
}

Because the index is text tied to time, searching it never touches the video again. That is why searching an analyzed video costs no extra credits, and why we can delete the uploaded file after 7 days while search keeps working: the index stays in the database.

Step 4 — Two-Phase Search: Instant Match, Then Meaning

When you type a query, search runs in two phases:

1. Instant match. Your query is matched locally against the tags and descriptions of every segment. If a segment contains what you asked for, it comes back immediately.
2. Semantic fallback. Only if the instant match finds nothing do we ask a language model (DeepSeek) to match the meaning of your query against the segment descriptions — so "someone celebrating" can still find a segment described as "people cheering and raising glasses."

Scores come back in coarse buckets rather than a calibrated probability: on our homepage sample, an exact hit scores about 100, a good hit about 50, and off-topic segments 10–20. The interface hides anything below 40 instead of showing low-confidence guesses, and labels the rest as exact or strong matches.

Clicking a result seeks the player to the start of the matching segment. For a 30-second clip that lands you within a few seconds of the moment; for a 25-minute video it lands you at the start of a segment that can be 30–60 seconds long. We think that's an honest trade: you still skip almost all of the scrubbing, and you know roughly how much to watch.

Step 5 — How the Pricing Follows the Architecture

The only expensive step is the single model pass in Step 1, so that is the only step that costs credits: 5 credits per new analysis, whatever the length within your plan's limit. Every search afterwards reads the stored segments, so searching again costs no extra credits. This is what "analyze once, search as often as you like" means in practice. You can see the plans on the pricing page.

What Doesn't Work Well, and What This Doesn't Replace

Visual search is useful, but it isn't magic, and pretending otherwise would be dishonest.

• Brief moments in long videos. In a long video each segment covers 30–60 seconds, so a detail that flashes on screen briefly can be left out of the description.
• Fine visual distinctions. Telling two similar-looking objects apart, or reading a small brand name, is unreliable when the frame is small, blurry or dark.
• Abstract queries. "The funny bit" depends on interpretation the model doesn't share with you. Concrete, describable queries work far better.
• Exact wording. The model describes speech content, but it doesn't produce a word-for-word transcript. When you need an exact quote, a transcript is still the right tool.
• Clips. SearchByVideo gives you timestamps; it does not cut or export clips.

We compare scene search and transcript search head-to-head in SearchByVideo vs transcript search. For the user-facing walkthrough, see how it works, and for more articles, browse the blog.

Closing Thoughts

SearchByVideo's scene search comes down to a few deliberate choices: one multimodal model pass that describes picture and sound, segment sizes that scale with video length, a text index tied to the timeline, and a two-phase search that tries an instant match before asking a model about meaning. Each choice trades some precision for cost and speed, and we'd rather tell you where those trade-offs sit than promise a frame-perfect search we don't run.