Video analysis: What are my options?

Video analysis covers a range of tasks: describing what happens in a clip, summarizing a long recording, tagging content against a taxonomy, searching a library for a specific moment, and turning footage into vectors that feed a downstream system. Teams evaluating tools for these tasks often start by asking which vendor to pick. A more useful question is whether the model processes video natively or reconstructs it from sampled frames and a transcript.

Video carries meaning through several channels running at once: visuals, motion, dialogue, and on-screen text, all unfolding across a timeline. A model built for video understanding processes these signals together rather than running frame-by-frame image classification and merging the results afterward. This distinction explains why some approaches look strong in a short demo and then degrade once applied to a real archive of unlabeled, variable-length footage. A longer treatment of this problem is available at why video is the hardest data problem in AI.

Key Takeaways

• Video analysis options fall into three broad buckets: manual review, general-purpose multimodal LLMs applied to sampled frames, and video-native foundation models.

• Sampling frames and transcribing audio separately loses signal, since video-native models process visual, audio, dialogue, and on-screen text together across time.

• TwelveLabs splits the problem into Pegasus for video-to-text generation and Marengo for multimodal retrieval and embeddings.

• The right option depends on the output needed: structured text points to Analyze, moment retrieval points to Search, and downstream ML pipelines point to Embed.

• A pilot can start without pre-indexing for analysis tasks, since a video URL or file can return structured output directly.

Option 1: Manual Review and Tagging

The baseline against which every automated option gets measured is a human watching footage and writing tags, summaries, or timestamps by hand. Accuracy can be high, since a trained reviewer catches nuance that automated tools miss. But the approach does not scale. A media archive with tens of thousands of hours, a security operation running continuous camera feeds, or a sports library adding hundreds of games a season cannot rely on manual review for search or classification.

Labeling cost grows with library size, and turnaround for a single clip request can stretch to days. Manual review stays useful for spot-checking output and for building small sets of training data, but it does not function as a production search or analysis pipeline on its own.

Option 2: General-Purpose Multimodal LLMs on Sampled Frames

A common shortcut is to sample frames from a video at fixed intervals, paste them into a general-purpose multimodal LLM, and ask it to describe or caption the sequence. This works for short clips and simple questions. It runs into trouble once the task requires exact timestamps, coverage of a long video in a single call, or reasoning about what happened between the sampled frames rather than at them. Audio and on-screen text can also be dropped by these pipelines unless a separate transcription step is added on top.

Capability comparisons between Pegasus's video-to-text capabilities and general models such as Gemini 3.1 Pro and GPT-5.5 show the difference in concrete terms: differences in maximum single-call video duration, whether output arrives as structured, timestamped segments, and whether the model reasons over defined time windows rather than a loose sequence of frames. For teams building a search or reporting product, these differences surface as missed moments, inconsistent timestamps, and extra engineering effort spent stitching outputs together after the fact.

Option 3: Video-Native Foundation Models

Analyze: Video-to-Text Generation with Pegasus

Pegasus takes a video and returns text shaped to whatever the caller asks for. The output format can be set directly in the prompt: a summary, a chapter list, an incident report, or a JSON schema with named fields. Visuals, audio, and speech are processed together, and the requested structure is generated with time-coded metadata included for downstream indexing.

Pegasus handles videos up to two hours long in a single call, and no pre-indexing step is required before analysis begins, so the workflow amounts to a URL or file going in and structured output coming out. The analyze videos guide walks through common patterns, including content structuring, incident reporting, and generative video question-answering. The Pegasus model documentation covers context window and segmentation behavior in more detail.

Search: Multimodal Retrieval with Marengo

Where Analyze produces text, Search finds moments. Marengo enables multimodal retrieval across visual, audio, dialogue, and on-screen text, letting a query be text, an image, or a combination of both against a single embedding space. A search can target what was said or what appeared on screen, without maintaining separate pipelines for transcript search and visual search.

Note that the hosted Search endpoint currently runs on Marengo 3.0. Marengo 3.5 adds newer capabilities, including native support for more than 36 languages, but per the Marengo 3.5 model documentation, those 3.5 embeddings are not queried through the `/search` endpoint. They are queried in the customer's own retrieval system after export. Teams that need the hosted Search product today should plan around Marengo 3.0's capabilities, and teams that want Marengo 3.5's newer capabilities should plan for self-hosted retrieval over exported embeddings.

Embed: Vectors for Production Pipelines

For teams building their own retrieval, classification, or recommendation systems, Marengo produces 512-dimensional embeddings ready for semantic search and RAG, drawing on visual, audio, dialogue, and on-screen text signals. These embeddings feed retrieval-augmented generation systems, recommendation engines, and anomaly detection pipelines without requiring a team to train a video encoder from scratch.

The create embeddings guide describes the difference between query-time embedding, at-scale batch embedding, and indexed embedding for search, depending on how a pipeline needs to consume vectors. The Marengo 3.5 launch post and the Marengo 3.5 model documentation detail newer capabilities: composed multimodal queries that combine several inputs into one query, an uncertainty signal for flagging low-confidence matches, document inputs alongside video, native support for more than 36 languages, and removal of a hard video and audio duration cap. As noted above, these Marengo 3.5 embeddings are queried in the customer's own system rather than through the hosted `/search` endpoint.

How to Choose Between These Options

The choice among Analyze, Search, and Embed depends on the shape of output a project needs, not on which capability sounds most advanced. If the requirement is a structured report, a chapter list, or a JSON object feeding a downstream system, Analyze fits. If the requirement is locating a specific moment inside a large, unlabeled library, whether described in text or matched against a reference image, Search fits, keeping in mind the Marengo 3.0/3.5 split described above. If the requirement is vectors for a custom pipeline, such as a recommendation model, a RAG system, or an in-house anomaly detector, Embed fits.

Marengo's capability tables list use cases spanning content discovery for media libraries, surveillance and security monitoring, contextual advertising placement, and domain-specific search for sports or government archives. Pegasus output, meanwhile, incorporates on-screen text, jersey numbers, and whiteboard content alongside speech, which matters for sports highlight generation or classroom video review, where the spoken track alone misses part of the story.

Viewing Marengo and Pegasus side by side on one page makes the difference between the two clearer. A tour of video understanding use cases covers how semantic search, classification, clustering, summarization, and question-answering map onto real deployments across industries.

Getting Started

Analyze accepts a video URL or file and returns structured text without a separate indexing step, which shortens the loop between a question and an answer during a pilot. Search and Embed require indexing a library once, after which retrieval and embedding calls run against that index with low latency.

SDKs are available for Python, Node, and REST, and a small pilot indexing a representative sample of a few hundred clips is usually enough to measure search precision, time-to-clip, and API latency before committing to a larger migration across an archive. For teams weighing an RFP, it helps to document what happens on a large unlabeled archive: whether the option under review still returns accurate timestamps, whether it still reads on-screen text, and whether search precision holds once the library grows past a small pilot set. Teams evaluating Search specifically should also confirm which Marengo version a vendor's hosted endpoint runs, since capabilities can differ by version.

FAQs

What Counts as a Video-Native Model Versus a Bolted-Together Approach?

A video-native model processes visual, audio, dialogue, and on-screen text together across a timeline, rather than sampling still frames and captioning them separately. A bolted-together approach usually pairs an image model with a separate transcript, then merges results after the fact, which tends to lose timing precision and drop signals like on-screen text. Reviewing Pegasus's video-to-text capabilities shows how a video-native model handles single-call duration and structured segmentation differently than a general-purpose LLM.

When Should a Team Use Analyze Instead of Search?

Analyze fits when the goal is text output, such as a summary, chapter list, or structured JSON report describing a video. The output format can be set directly in the prompt rather than parsed from free-form text after the fact. Search fits instead when the goal is finding a specific moment inside a larger library.

Does Marengo Support Queries That Combine Text and Images?

Yes. Marengo supports multimodal retrieval across visual, audio, dialogue, and on-screen text, and a query can be text, an image, or a combination of both against one embedding space. A search request can describe what was said or what appeared on screen, with no separate pipeline needed per signal type. This combined-query support is available through the hosted Search endpoint, which currently runs on Marengo 3.0; Marengo 3.5's newer capabilities are queried separately, as described above.

What Can Embeddings from Marengo Be Used for Beyond Search?

The 512-dimensional embeddings ready for semantic search and RAG also feed recommendation engines and anomaly detection pipelines built in-house. Because the vectors already carry visual, audio, dialogue, and on-screen text signals, teams do not need to train a separate video encoder for these downstream tasks.

How Long Can a Video Be for a Single Analyze Call?

Pegasus handles videos up to two hours long in a single call and does not require a pre-indexing step before analysis begins. Details on context window and segmentation behavior are covered in the Pegasus model documentation.

See a Demo

Teams comparing these options can review interactive demos of these capabilities to see search precision, analysis output, and embedding behavior on realistic footage before committing engineering time to an integration.