Video Segmentation API: How to Extract Structured Data From Video

A video segmentation API takes any video and returns every event you care about as timestamped, schema-compliant JSON: no preprocessing pipeline, no training data, no ingestion step.

You write a plain-language description of what to find: "every dunk," "every moment our logo appears," "every speaker change", and the API finds every matching instance across your footage, complete with start/end timestamps and custom field values. The output is schema-compliant JSON ready to feed directly into your downstream systems.

That's the core capability of Pegasus 1.5's Segment feature. This post covers how it works, what you can define, where it outperforms existing approaches, and which teams it's built for.

TL;DR: A video segmentation API takes a plain-language description of what to find in any video and returns every matching moment as timestamped, schema-compliant JSON, in a single API call. Pegasus 1.5 processes videos up to 2 hours long with no ingestion pipeline, no training data, and no preprocessing. Define up to 10 segment types with 20 custom fields each. Output integrates directly with databases, pipelines, and AI agent frameworks. Benchmarks show 30%+ better segmentation accuracy than Gemini 2.0 Pro.

Why existing approaches can't extract structured video data

Most organizations already know they have a video data problem. The harder realization is that the common fixes don't actually solve it. They just push the problem around.

Frame extraction captures fragments, not events

Sampling individual frames misses everything that happens between frames: transitions, actions, sequences. A dunk isn't a frame; it's a 4-second event. A speaker change happens at a precise moment. Frame extraction gives you pixels, not meaning, and certainly not timestamps tied to events.

Transcript search is blind to visual action

Text search works on what's spoken. A logo appearance, a camera cut, a product placement, an athletic play: none of those are in the transcript. The most operationally important moments in most video libraries are never spoken aloud. Transcript search finds words, not events.

Manual tagging doesn't scale

Hiring specialists to review footage and add metadata is expensive, slow, and completely impractical against a growing archive. A team of three annotators working on broadcast sports footage can't keep pace with a single weekend of games. The backlog grows faster than any team can clear it.

General LLMs hit a wall on video

General-purpose models weren't designed for temporal reasoning. They struggle with structured output schemas, can't reliably handle anything over an hour, and aren't built to find specific events: they're built to respond to prompts. Coercing a chat model into a video segmentation workflow produces inconsistent results at best.

Key takeaway

Most enterprises can tell you what videos they have, but they can't tell you what's in them. A purpose-built video segmentation API closes that gap without requiring a pipeline rebuild.

How a video segmentation API works: define, point, get JSON

The workflow has three steps. None of them require model training, data preprocessing, or a custom ingestion pipeline.

Step 1: Define a segment schema

Write a plain-language description of what to find. This is your segment definition: "every time the speaker changes," "every dunk," "every moment our logo appears on screen." You can define up to 10 segment types per request. Each definition supports up to 20 custom output fields, from simple string labels to rich event descriptions.

Step 2: Point it at any video, up to two hours

Pass a URL or file directly to the API. No ingestion pipeline, no preprocessing, no indexing step. Pegasus 1.5 processes up to 2 hours of video in a single pass: more than twice what competing models handle. That covers full-length broadcast footage, long-form events, and feature films without splitting or preprocessing.

Step 3: Get structured, timestamped JSON back

Every matching segment is returned as schema-compliant JSON with start/end timestamps and your custom field values. The output is ready to feed directly into downstream pipelines, databases, or AI agent frameworks.

{
  "id": "scoring_play",
  "segments": [
    {
      "start": "00:08:46",
      "end": "00:08:54",
      "description": "Player #24 dunks on a fast-break pass",
      "event_type": "dunk"
    },
    {
      "start": "00:01:23",
      "end": "00:01:55",
      "description": "Overtime begins, game tied 58-58",
      "event_type": "game_state_change"
    }
  ]
}

{
  "id": "scoring_play",
  "segments": [
    {
      "start": "00:08:46",
      "end": "00:08:54",
      "description": "Player #24 dunks on a fast-break pass",
      "event_type": "dunk"
    },
    {
      "start": "00:01:23",
      "end": "00:01:55",
      "description": "Overtime begins, game tied 58-58",
      "event_type": "game_state_change"
    }
  ]
}

Example output: schema-compliant JSON with start/end timestamps and custom fields.

Schema design: up to 10 definitions, 20 custom fields each

The schema is the most underappreciated part of this workflow. It's what transforms a general-purpose model into a domain-specific extraction tool: without training.

Multiple event types in one request

A single API call can carry up to 10 segment definitions. For sports, that means scoring_play, camera_cut, brand_appearance, and player_spotlight all returned together in one structured response. For content ops, it's topic_change, ad_break, and speaker_intro simultaneously. You're not making separate calls for each event type: everything comes back in one pass.

20 custom fields per segment

Each segment definition supports up to 20 custom output fields. Fields can be simple string labels ("event_type": "dunk") or rich descriptions ("description": "Player #24 dunks on a fast-break pass"). You define the schema to match what your downstream system expects: not what a generic model happens to output.

Zero training data required

Segment definitions are plain language. "Every time the brand logo appears on screen." "Every moment a forklift enters a restricted zone." "Every camera cut to a crowd reaction shot." The model applies your taxonomy zero-shot across every video in your library. You don't need annotated examples to get started. You need a clear description of what matters to you.

On-the-fly processing: no ingestion pipeline needed

Most video AI workflows assume you've already indexed your library. Segment doesn't. There's no ingestion step, no preprocessing, no special upload format.

Pass any publicly accessible URL or a file directly to the API. Pegasus 1.5 handles the rest: processing the full video, extracting segments against your schema, and returning structured output. For teams that want to test against a single video before committing to a full pipeline, this matters: you get value immediately, with one API call, before building anything.

The 2-hour limit is meaningful in practice. Competing models cap out under an hour, which means broadcast footage, long-form events, and feature-length content require splitting, stitching, and deduplication logic on your side. Pegasus 1.5 handles them in a single pass.

Multimodal prompting: find by image, not just text

Segment also supports image-based queries, a capability that opens use cases text prompts can't reach.

Pair a reference image with a text description. Pass a photo of your brand logo alongside "every moment this logo appears on screen." Pass a headshot of an athlete alongside "every play this player is involved in." Pegasus 1.5 finds every matching visual moment, timestamped, without any training or custom detection code.

Where this matters most

Logo placement tracking for sponsor reporting. Athlete detection for performance analytics. Product appearance monitoring for brand safety. These are the use cases that matter most to media and sports workflows, and they're the ones that text-only queries can't reliably handle. Multimodal prompting closes that gap.

How Pegasus 1.5 compares to existing approaches

The choice isn't just Pegasus 1.5 vs. another model: it's Pegasus 1.5 vs. the approaches most teams currently use. Here's how the options stack up on the dimensions that actually matter for structured video data extraction.

Note

The 30%+ better segmentation figure is measured against Gemini 2.0 Pro on TwelveLabs' internal benchmark dataset. "30% better segmentation isn't a benchmark win. It's the difference between metadata you can act on and metadata you have to clean up."

Who uses a video segmentation API, and for what

The capability is domain-agnostic. The taxonomy changes; the workflow doesn't.

Sports analytics

Scoring plays, tactical formations, player actions, camera angles: labeled across an entire season of broadcast footage in a fraction of the time manual review would require. Output feeds directly into performance analytics tools, automated highlight systems, and coaching platforms.

Media and entertainment

Scene detection, speaker changes, B-roll categorization, ad break identification. Content operations teams that once manually tagged every segment of every episode can run the entire back catalog through a single API call and get structured metadata out the other side.

Brand monitoring

Logo appearances, product placements, sponsor visibility: tracked across broadcast, streaming, and social video simultaneously. Pass your brand asset as a reference image and get timestamped placement data for every appearance, across every piece of content.

Content operations and AI agents

Structured video metadata is the input layer for AI agents that need to reason about what's in a video library. Chapter generation, content moderation, archival search, compliance monitoring, all of these downstream workflows start with the same need: timestamped, schema-compliant data about what's on screen, when. Segment provides that foundation.

Enterprise readiness

Pegasus 1.5 ships with a 12-month deprecation policy and enterprise SLA support. The API output integrates directly with databases, downstream pipelines, and AI agent frameworks: no custom reformatting. For teams managing large-scale archives or building production workflows on top of video metadata, that stability matters.

For technical teams: one endpoint, a JSON schema, results. Up to 10 segment definitions and 20 custom fields each. Pass any URL or file directly: no indexing, no ingestion pipeline. Output integrates with databases, pipelines, and AI agent frameworks out of the box.

For business leaders: no new headcount, no custom ML project. Define a schema once: apply it retroactively across decades of footage. Unlock automation across archiving, compliance, content ops, and AI workflows without rebuilding your infrastructure.

Get started

Pegasus 1.5 is built for time-based metadata extraction. To see what it returns against your own footage, use the free Playground: upload a video, define a segment schema, and get timestamped structured output in real time. No setup required.

Try it now:

playground.twelvelabs.io: free Playground, no setup required.

Learn more:

Introducing Pegasus 1.5 : (how it was built, the evaluation dataset, what RL training taught the research team)

Pegasus 1.5 API reference