Building a Sports Highlight Intelligence App with TwelveLabs Marengo, Pegasus, and Jockey

Introduction

The moments that move sports audiences are not in the box score. A scoreboard change records that a point was scored. It does not capture the bench erupting, the crowd holding its breath, the broadcaster's voice cracking, or the player sinking to their knees. Those are the moments that get shared, replayed, and monetized.

The NBA generated more than 5 billion social media views during the 2025 Finals, up 215% from the previous year. That growth is driven by fast, moment-specific short-form content. But the production gap is real: hour-long games require human review to find what matters, and manual workflows cannot scale to meet the distribution pace.

Production-ready sports intelligence requires three things in sequence: semantic discovery across raw footage, structured understanding of selected moments, and an agentic layer that reasons over the game the way a producer would. Finding a goal is a starting point. Understanding the goal's crowd response, the emotional arc, the broadcast context, and which of four different formats it belongs in is what makes it publishable.

This tutorial walks through Sports Jockey, a production-grade sports video intelligence application built with TwelveLabs. The application uses Marengo for semantic search across game footage, Pegasus 1.5 for clip-scoped analysis of selected moments, and Jockey as the agent layer reasoning over the knowledge store. By the end, you will have a clear picture of how these three capabilities connect into a single workflow that turns raw footage into searchable moments, structured editorial context, highlight lanes, entity intelligence, and production-ready outputs.

What the Application Does

The application supports four connected workflows:

1. Semantic search finds plays, reactions, celebrations, logos, fan moments, and atmosphere directly from video using Marengo. This includes text queries, image-based search, and entity-based search for registered players, logos, or any visual reference.

2. Clip analysis with Pegasus 1.5 takes a selected moment and returns structured editorial metadata: description, emotional tone, key action, participants, score context, editorial use, and recommended formats. Pegasus does not re-analyze the full match. It focuses on exactly the segment the producer selected.

3. Highlight lane generation uses Jockey over the Knowledge Store to organize footage into production-ready categories: Best Plays, Emotional Moments, Fan Experience, and Behind the Scenes. Each lane includes timestamped clips with evidence, rationale, and editorial guidance.

4. Live chat with Jockey turns the Knowledge Store into an interactive production assistant. A producer can ask for a specific moment in natural language, refine a reel idea, or explore footage conversationally across multiple turns.

Intelligence Stack Behind Sports with Jockey Application 

Three TwelveLabs capabilities power the workflow, each with a defined role and a defined hand-off point.

Before any search or reasoning can happen, source videos are uploaded to TwelveLabs as assets. Each uploaded asset becomes the canonical video object referenced across the full workflow. The backend stores the returned asset ID and maps it to the registered source video name.

After upload, each asset connects to two TwelveLabs layers: the search index and the Knowledge Store. The index makes footage available for Marengo-powered semantic search and indexed playback. The Knowledge Store gives Jockey a grounded context layer for reasoning over the game.

The sections below break down how each capability fits into the workflow, what it produces, and how its output flows into the next step.

Prerequisites

Build the foundation once and reuse it across every workflow. To set up the application, do clone the code repository.

1. Create a TwelveLabs account and generate an API key.

2. Create a TwelveLabs index with Marengo enabled for semantic video search and record the Index ID.

3. Create the Entity Collection and record the Entity Collection ID in .env.

4. Create a TwelveLabs Knowledge Store for Jockey and record it for .env.

5. Install FFmpeg to generate or export assembled highlight reels.

6. Create backend/.env with the following variables: TWELVELABS_API_KEY, INDEX_ID, ENTITY_COLLECTION_ID, and KNOWLEDGE_STORE_ID.

If you would like to explore the notebook workflow for using Jockey with sports footage, refer to this repository.

Section 1: Search the Game Like a Producer Thinks

The most valuable production moments are often absent from the official game log. A scoreboard change records that a point was scored. It does not capture the bench reaction, the crowd energy, the broadcast replay, or the emotional pause before the next play. Finding those moments requires understanding that spans visuals, audio, motion, atmosphere, and context together.

Marengo is the search layer that makes this possible. Instead of forcing production teams to scrub through footage manually or depend on pre-written tags, the application lets users search game footage the way they naturally think about it: "a late comeback," "a bench reaction," "a fan in the stands after the score."

1.1 - Marengo for Semantic Discovery

The workflow begins by indexing source footage with TwelveLabs. Once the video is available in the index, Marengo can search across the game using natural language.

Marengo returns timestamped results across all indexed footage. Each result includes the time range, video reference, relevance information, and source mapping needed to open the exact moment in the application.

This matters most for the moments that live between the data points. Sponsor boards, player gestures, crowd signs, and emotional reactions never appear in any stats feed, but they are frequently the most shareable content in the footage.

1.2 - Search by Images, Entity, or Logo

Some production moments cannot be put into words. A producer may want footage matching a visual reference: a specific player's face, a jersey number, a sponsor logo, a team badge, or a sideline figure who appears in multiple games.

The application extends search beyond text for these cases. A user uploads an image and the application sends it to TwelveLabs as visual search input, returning footage that matches the reference visually.

The workflow also supports entity-based search. Each entity is created with a name, description, and reference image, then stored inside the entity collection in TwelveLabs for reuse across games. Once registered, the entity is searchable across all indexed footage without recreating the context each time.

This is especially useful in sports media workflows where the important subject is visual rather than textual. The example below shows how the "Rolex" logo is found and tracked across indexed sports footage. Sponsor boards, jersey numbers, and team badges that never appear in any transcript are still fully searchable through this workflow.

Section 2: Analyze Only the Moment That Matters

A timestamp is not enough. A producer needs to understand what happens inside the clip, who appears, why the moment is useful, and which format it belongs in: a highlight reel, a social edit, a recap, or a rights-compliance review.

This section covers how the application uses Pegasus 1.5 to move from a discovered moment to structured, decision-ready video intelligence.

2.1 - Pegasus 1.5 for Clip-Scoped Analyze

Once Marengo surfaces a relevant moment, the application sends only that selected time range to Pegasus 1.5. The full match is not re-analyzed. Pegasus receives exactly the segment the producer selected, which keeps analysis fast, precise, and scoped to the actual production decision.

The backend receives the selected video name, timestamp range, and asset ID before calling Pegasus:

result = twelvelabs_analyze_video({
    "model_name": "pegasus1.5",
    "video": {
        "type": "asset_id",
        "asset_id": asset_id,
    },
    "start_time": analyze_start,
    "end_time": analyze_end,
    "prompt": prompt,
    "temperature": 0.2,
    "response_format": {
        "type": "json_schema",
        "json_schema": SELECTED_CLIP_ANALYSIS_SCHEMA,
    },
    "max_tokens": 3200,
})

Pegasus handles multimodal understanding inside the selected clip with full-context awareness. The application controls the scope, schema, and workflow.

2.2 - Structuring Model Output as Editorial Metadata

The goal is metadata that supports a real cut decision, not a free-form description.

The application asks Pegasus 1.5 to return a structured JSON response instead of unstructured text. Each analyzed clip maps to a consistent editorial object:

SELECTED_CLIP_ANALYSIS_SCHEMA = {
    "type": "object",
    "properties": {
        "description": {"type": "string"},
        "emotional_tone": {"type": "string"},
        "key_action": {"type": "string"},
        "participants": {"type": "array"},
        "moment_types": {"type": "array", "items": {"type": "string"}},
        "tags": {"type": "array", "items": {"type": "string"}},
        "score_context": {"type": "string"},
        "producer_summary": {"type": "string"},
        "story_arc": {"type": "string"},
        "editorial_use": {"type": "string"},
        "recommended_formats": {"type": "array", "items": {"type": "string"}},
        "visual_evidence": {"type": "array", "items": {"type": "string"}},
        "audio_evidence": {"type": "array", "items": {"type": "string"}},
        "transcript_evidence": {"type": "array", "items": {"type": "string"}},
        "confidence": {"type": "number"},
    },
    "required": [
        "description",
        "key_action",
        "producer_summary",
        "editorial_use",
        "confidence",
    ],
}

The fields that drive cut decisions are editorial_use, recommended_formats, and confidence. A producer can compare moments across different clips using the same fields, save useful analysis for later, and reuse the metadata in a downstream reel assembly or review workflow. The visual_evidence and audio_evidence fields document what Pegasus saw and heard, making the rationale auditable rather than opaque.

The application can save approved analysis back into the indexed asset metadata for the corresponding TwelveLabs video ID. This makes clip insights reusable across sessions: the context stays attached to the video asset and reloads on demand, rather than disappearing at the end of the review.

Section 3: Build Highlight Lanes with Jockey

Search finds the right moments. Pegasus explains a selected clip. Jockey takes the next step: it reasons across the prepared game knowledge and organizes the footage into production-ready categories.

This is where the workflow shifts from search and analysis toward something closer to a production assistant. Instead of asking a producer to build every reel manually from a clip list, the application uses Jockey over the Knowledge Store to generate structured highlight categories with timestamped clips, entity intelligence, and editorial context.

3.1 - Grounding Jockey in the Game Knowledge Store

Before Jockey can reason over the footage, the source videos need to be prepared as a knowledge store.

The ingestion flow starts by uploading source videos as TwelveLabs assets. Those assets are added to a TwelveLabs Knowledge Store, where they become searchable, referenceable items for Jockey. During this step, the application creates a new knowledge store if one does not already exist:

def create_knowledge_store(spec):
    metadata = {
        "game_tag": spec["tag"],
        "source": "sports-jockey",
    }

    return twelvelabs_create_knowledge_store(
        name=spec["knowledge_store_name"],
        ingestion_config={
            "enrichment_config": {
                "type": "json_schema",
                "json_schema": SPORTS_HIGHLIGHT_INGESTION_SCHEMA,
            }
        },
        metadata=metadata,
    )

The Knowledge Store is created with an enrichment schema designed for sports contexts, focused on score changes, key plays, emotional moments, fan reactions, and broadcast context. This structured foundation gives Jockey reliable grounding before it is asked to build highlight lanes.

After creating the knowledge store, the backend uploads the source videos, adds the indexed video assets, and waits until the items are ready:

def run_ingestion(payload, progress=None):
    spec = parse_ingestion_payload(payload)

    store_id = spec["knowledge_store_id"]
    if not store_id:
        store = create_knowledge_store(spec)
        store_id = store["_id"]
        progress(f"created knowledge store {store_id}")
    else:
        progress(f"using existing knowledge store {store_id}")

    upload_source_assets(spec["source_videos"], state, progress)
    upload_index_assets(spec["index_videos"], state, progress)
    add_index_items(store_id, spec["index_videos"], state, progress)

    poll_items_until_ready(
        store_id=store_id,
        state=state,
        poll_attempts=spec["poll_attempts"],
        poll_interval_seconds=spec["poll_interval_seconds"],
        progress=progress,
    )

    game_payload = {
        "tag": spec["tag"],
        "label": spec["label"],
        "sport": spec["sport"],
        "knowledge_store_id": store_id,
        "source_videos": [video.name for video in spec["source_videos"]],
        "video_asset_ids": source_asset_ids_for_game(spec["source_videos"], state),
        "video_reference_map": build_video_reference_map(state, spec["index_videos"]),
    }

    register_game(game_payload)

This structure keeps the output usable. Every moment has a timestamp, a reason for being selected, evidence from the footage, and guidance for how it can be used in an edit. The enhanced lanes can include semantic context such as momentum, emotion, celebration, fan energy, and production value.

3.2 - Generating Highlight Categories from the Knowledge Store

Once the footage is available in the Knowledge Store, the application asks Jockey to build highlight lanes directly. The backend sends a structured request to the TwelveLabs /responses API with the game's knowledge_store_id and a JSON schema defining the expected output.

The request is organized around four production categories: Best Plays, Emotional Moments, Fan Experience, and Behind the Scenes.

Every clip in the output includes a timestamp, the reason it was selected, evidence from the footage, and guidance for editorial use. Jockey surfaces semantic context that no stats feed carries: momentum shifts, emotional peaks, crowd energy levels, and production value signals. The recommended_formats field tells a producer whether the clip belongs in a 60-second social cut, a long-form recap, or a vertical-format reel.

3.3 - Tracking Who Appears, When They Matter, and How They Connect

Highlight lanes answer what moments matter. Entity tracking answers who is involved and how those individuals connect across the footage.

The application uses Jockey for entity intelligence through a dedicated tracking workflow. The backend asks Jockey to extract the most important grounded entities from the Knowledge Store: players, teams, coaches, officials, fan groups, benches, and broadcast subjects. Jockey can also return relationships when the footage supports them: a scorer celebrating with teammates, a coach reacting to a play, a crowd responding to a goal, or opponents contesting a call.

For sports rights holders, this workflow directly supports sponsor reporting, player-specific content packages, and compliance review. A producer can pull every frame where a sponsor's logo appears alongside a positive emotional moment, or extract all footage featuring a specific athlete across multiple games, without a manual search pass.

The workflow deliberately returns a focused set of high-confidence entities rather than an exhaustive list. A smaller, higher-quality set is more useful to a production team than a noisy inventory that requires its own curation pass.

When entity tracking output is generated, the application stores it back on the indexed asset metadata. The same intelligence becomes reusable across future sessions without triggering a fresh Jockey request.

Section 4: Talk to the Game with Jockey Chat

Highlight lanes are useful when the workflow knows what structure to generate. Production work is often more open-ended. A producer may want to ask for a specific moment they remember from watching the game live, explore a creative direction without knowing the exact search phrase, or refine a reel idea through iteration.

Live chat with Jockey turns the Knowledge Store into an interactive production assistant. A producer describes their intent. Jockey finds the matching footage.

4.1 - Natural Language Requests, Grounded in Actual Footage

Consider a producer who asks: "Find me the moment right after the lead changed where someone on the bench had the biggest reaction." That is not a keyword query. It requires understanding the game timeline, the score context, the specific location in frame, and the intensity of a physical reaction.

The backend routes chat messages through the Jockey /responses workflow. Each request includes the user message, the registered Knowledge Store, optional source-video scope, and conversation history:

request_body = {
    "model": "jockey1.0",
    "instructions": jockey_chat_instructions(),
    "input": [
        {
            "type": "message",
            "role": "user",
            "content": jockey_chat_prompt(
                game=game,
                message=message,
                limit=limit,
                video_name=video_name,
                include_reel=include_reel,
                conversation_history=conversation_history,
            ),
        }
    ],
    "knowledge_store_id": configured_knowledge_store_id(game),
    "text": {
        "format": {
            "type": "json_schema",
            "name": "jockey_highlight_manifest",
            "schema": JOCKEY_CHAT_SCHEMA,
        }
    },
}

Rather than letting the agent respond with an unstructured paragraph, the application asks Jockey to return a clean, typed object with two parts: a short narrative summary and a list of playable clips. The schema below defines the required structure:

JOCKEY_CHAT_SCHEMA = {
    "type": "object",
    "properties": {
        "narrative_summary": {"type": "string"},
        "clips": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "video_reference": {"type": "string"},
                    "start_time": {"type": "string"},
                    "end_time": {"type": "string"},
                    "moment_type": {"type": "string"},
                    "emotional_intensity": {"type": "string"},
                    "jockey_rationale": {"type": "string"},
                    "confidence": {"type": "number", "minimum": 0.01, "maximum": 1},
                    "highlight_potential": {"type": "number", "minimum": 0, "maximum": 1},
                },
                "required": [
                    "video_reference",
                    "start_time",
                    "end_time",
                    "moment_type",
                    "emotional_intensity",
                    "jockey_rationale",
                    "confidence",
                    "highlight_potential",
                ],
                "additionalProperties": False,
            },
        },
    },
    "required": ["narrative_summary", "clips"],
    "additionalProperties": False,
}

The narrative_summary gives the producer a concise explanation of what Jockey found and why. The clips array turns that explanation into something immediately actionable: a source reference, a start and end time, the moment type, the emotional intensity, the rationale behind the selection, and confidence scores. The same object can power clip previews, reel assembly, save actions, or downstream editorial review.

Jockey Chat supports iteration through conversation history. A producer can open with a broad request, narrow it in the next turn, and request a specific format in the turn after that, without restarting the workflow or losing context.

4.2 - Persisting Production Decisions on Indexed Assets

Sports production workflows often span multiple sessions and multiple team members. A producer who reviews footage on Tuesday should not have to redo the same discovery pass on Thursday. A coordinator handing off to an editor should not have to re-explain which clips were approved.

When a producer finds a strong Pegasus clip analysis or a useful Jockey Chat response, the application saves that result back to the source video's TwelveLabs indexed asset metadata:

In the TwelveLabs integration layer, this is a PATCH request to the indexed asset:

def update_indexed_asset_user_metadata(index_id, indexed_asset_id, user_metadata):
    return request_json(
        "patch",
        indexed_asset_endpoint(index_id, indexed_asset_id),
        {"user_metadata": user_metadata},
    )

Saved results reload on demand: clip ranges, Pegasus analysis, chat decisions, and summaries all come back with the video. For teams working across multiple sessions or sharing footage review across roles, the saved context is persistent and shared rather than siloed per user.

What This Approach Makes Possible

The shift this application demonstrates is not faster highlight generation. It is a different relationship between a production team and its footage.

A full match becomes something a team can ask, understand, and reuse rather than something they have to watch end to end. Marengo finds plays that no stats feed would surface. Pegasus converts a timestamp into a structured editorial decision with auditable evidence. Jockey organizes the game into production-ready lanes and responds to natural language requests with playable, structured results. Persistent metadata keeps useful decisions available across sessions and team members.

For sports rights holders, this translates to fewer hours between footage delivery and publishable output, and more content produced from a single match. A game does not have to yield one recap. It can yield a Best Plays reel, an Emotional Moments cut, a fan-focused social edit, and entity-tracked coverage of a specific player, all from the same footage, the same workflow, and the same knowledge store.

The source code and a live demo are available for teams who want to see the workflow before building.

Resources

• TwelveLabs Pegasus 1.5 overview - Learn how Pegasus 1.5 enables deeper video understanding and structured analysis.

• TwelveLabs Marengo 3.0 overview - Explore how Marengo 3.0 powers multimodal search across video.

• TwelveLabs Jockey Workflow - Reference notebook implementation for the Jockey and Knowledge Store workflow.

• Sports Jockey Intelligence repository - Source code for the sports video intelligence application.

• TwelveLabs Jockey Knowledge Store guide - Follow the guide for creating a Knowledge Store used by Jockey.

• TwelveLabs sample applications - Browse example applications built with TwelveLabs video intelligence.

• Join the TwelveLabs Discord community - Connect with the TwelveLabs community, ask questions, and share what you build.