TLDR

Most streaming platforms have spent years building content libraries and almost no time building tools to navigate them. The result is massive episode catalogs that younger audiences scroll past in favor of TikTok and Reels.

This leads to a predicament as the bottleneck is not content volume, but rather intelligent systems as legacy manual tagging cannot scale to identify every tension-filled glance or designer outfit across thousands of episodes. Basic metadata can also not distinguish a playful argument from a relationship-ending fight, which are the moments younger audience members prioritize in their short-form content.

This tutorial walks through a complete solution built on TwelveLabs Pegasus 1.5, Marengo 3.0, and Jockey 1.0. The output is a personalized, TikTok-style vertical feed that serves profile-ranked clips in under 2 seconds, runs semantic search across indexed episodes, and surfaces cross-season story arcs through natural language queries. 

What you will build: A pipeline that transforms full reality TV episodes into queryable, bite-sized moments. The system matches scenes to user personas based on emotional intensity, tracks cast entities across entire seasons, and generates dynamic feeds with transparent AI reasoning behind every clip selection.

Introduction

Three structural problems prevent streaming platforms from building the kind of experience that actually retains superfans and the younger generation.

1. Scale. Identifying every meaningful moment in a 40+ minute episode, across a library of hundreds of episodes, requires armies of human loggers. Beyond the direct cost, those loggers need full-show context to avoid surfacing spoilers or irrelevant clips out of sequence, which poses IP and platform reputation risks if mistakes are made at pivotal moments in the show.

2. Depth. Traditional metadata are often surface-level tags. A tag like "argument" tells a recommendation engine nothing about whether the scene is playful shade between friends or a relationship-ending confrontation. The distinction determines whether a clip belongs in a Drama Addict's feed or a Romance Fan's. Without that depth, recommendations feel random.

3. Continuity. Single-video analysis, even good single-video analysis, has no memory. It can tag a fight scene but cannot recognize that the same two cast members have been building toward this moment for three episodes. This context is what can differentiate scenes as normal versus one that engages the superfan more with the TV series.

This tutorial builds a hybrid architecture that solves all three at once. The system combines three TwelveLabs models, each solving a distinct layer of the problem:

Figure 1: Explore Tab displaying Personalized Shorts for “Drama Addict”

This architecture and release of new unified foundational multimodal models like Jockey proves that it is now possible to build the editorial intelligence of a full logging team into an API pipeline, and extend it with corpus-level reasoning that no human logger could produce in real time.

Here is a walkthrough of the application in action!

You can also visit the live demo site here: Superfan Vertical Feed

Prerequisites

Before building along, confirm you have the following:

Runtime (Vercel / Next.js)

1. Node.js 20+

2. TwelveLabs API key with:

   1. TL_INDEX_ID: Marengo index for search and HLS playback

   2. TL_KS_ID: Jockey Knowledge Store for cross-episode reasoning

Pre-processing (local Python pipeline)

3. Python 3.10+

4. pip install -r requirements.txt (twelvelabs, requests, python-dotenv)

5. Episodes uploaded as TwelveLabs assets (24-char hex IDs, HLS enabled). Assets can be created in the TwelveLabs Playground under "Assets."

Clone and run:

git clone https://github.com/nathanchess/fox-superfan-feed.git
cd fox-superfan-feed/app
npm install
cp .env.example .env.local
npm

git clone https://github.com/nathanchess/fox-superfan-feed.git
cd fox-superfan-feed/app
npm install
cp .env.example .env.local
npm

Architecture Overview

The platform splits into two distinct phases: offline pre-processing (run once per season or whenever new episodes are added) and runtime serving (real-time API calls to TwelveLabs video models on Vercel).

💡 System Design Note: This is the same pattern enterprise search engines use. To keep feed generation fast for personalized or profile-based requests, computationally expensive work happens offline. Pegasus segments every episode, Jockey enriches the corpus, and the resulting manifest gets committed to the repo. At request time, the app reads pre-built JSON and runs in-memory scoring. No model is re-invoked on the fly for standard feeds unless needed.

Figure 2: Superfan Vertical Feed Engine Architecture (Link)

Each model in the pipeline serves a specific purpose:

Pegasus 1.5 is the primary source of per-clip metadata. The time-based metadata feature lets you extract structured JSON from each 15-30 second segment, including scene description, emotional intensity, primary category, subtags, and key cast members. These fields directly power the ranking algorithm: a clip gets boosted if a favorite cast member appears, if its emotional intensity matches a persona's preference, and if its category aligns with the persona's interests.

Jockey 1.0 solves the one thing Pegasus cannot: relative importance. Two clips might have identical categories, intensities, and cast members. What distinguishes them is how much weight the broader season gives to each moment. A recurring feud between the two leads is more significant than a one-off confrontation, and only corpus-level reasoning can surface that distinction. Jockey runs across the full Knowledge Store, identifies season-defining moments, and applies a jockey_boost flag to the segments within 10-15 seconds of those moments.

Marengo 3.0 handles ad-hoc semantic search. When a user types "awkward dinner party" or "someone storms out" in the Explore tab, Marengo queries the full indexed library across visual, audio, and transcription modalities and returns timestamped hits grounded in actual content, not keyword overlap on a title field.

With this architecture in mind, let’s get started building 🏗️!

Phase 1: Pre-processing Pipeline (Offline)

The /pre-processing folder in the repository handles everything in this phase. The output is a manifest file: data/feed_manifest.json. The Next.js app reads this file at request time. No Python runs on Vercel.

Step 0: Uploading Assets & Creating Jockey Knowledge Store

TwelveLabs uses two distinct storage models, and this pipeline requires both:

1. Indexes: Support Pegasus 1.5 and Marengo 3.0. Used for per-video analysis and semantic search.

2. Knowledge Stores: Support Jockey 1.0. Organized at the corpus level, enabling cross-episode reasoning.

The helper functions for both operations live in pre-processing/knowledge_store.py. The one configuration worth customizing is the Knowledge Store's enrichment schema:

response = requests.post(
    f"{BASE_URL}/knowledge-stores",
    headers=HEADERS,
    json={
        "name": "show-name",
        "ingestion_config": {
            "enrichment_config": {
                "type": "json_schema",
                "json_schema": SUPERFAN_CTV_ENRICHMENT_JSON_SCHEMA,
            }
        },
    },
)

response = requests.post(
    f"{BASE_URL}/knowledge-stores",
    headers=HEADERS,
    json={
        "name": "show-name",
        "ingestion_config": {
            "enrichment_config": {
                "type": "json_schema",
                "json_schema": SUPERFAN_CTV_ENRICHMENT_JSON_SCHEMA,
            }
        },
    },
)

💡Tip: The enrichment configuration is a Jockey-specific feature that acts as a pre-indexing prompt. Rather than loading your full taxonomy into each query (where token budgets are tighter), you define what the model should focus on once, at ingestion time. 

Here is the schema used for this project:

SUPERFAN_CTV_ENRICHMENT_JSON_SCHEMA = {
    "type": "json_schema",
    "json_schema": {
        "type": "object",
        "properties": {
            "primary_scene_category": {
                "type": "string",
                "description": "Dominant moment type for specific scene for vertical-feed taxonomy and profile matching",
            },
            "subtags": {
                "type": "array",
                "description": (
                    "Concrete visual/audio signals of scene for what is happening and can be heard in the scene"
                ),
                "items": {"type": "string"},
                "maxItems": 5,
            },
            "description": {
                "type": "string",
                "description": (
                    "Precise moment summary: who, what happened, tone, body language, "
                    "and dialogue subtext for long-form episodic TV. Should pinpoint exact names and map to people."
                ),
            },
            "emotional_intensity": {
                "type": "string",
                "description": "Salience for feed ranking and profile intensity preference",
                "enum": ["low", "medium", "high", "explosive"],
            },
            "key_figures": {
                "type": "array",
                "description": "On-screen people or named participants; pairs or groups in conflict/alliance when clear. Get specific names and map to people.",
                "items": {"type": "string"},
            },
            "feed_headline": {
                "type": "string",
                "description": "One short headline suitable for a vertical clip card",
            },
            "scene_setting": {
                "type": "string",
                "description": (
                    "Setting type: interview, ensemble_conversation, event, travel, "
                    "home, workplace, competition_stage, other"
                ),
            },
            "cross_episode_significance": {
                "type": "string",
                "description": (
                    "If inferable: recurring storyline, callback, finale/reunion relevance, "
                    "or franchise-defining beat; empty string if none"
                ),
            },
        },
        "required": [
            "primary_scene_category",
            "subtags",
            "description",
            "emotional_intensity",
            "key_figures",
            "feed_headline",
        ]
    }
}

SUPERFAN_CTV_ENRICHMENT_JSON_SCHEMA = {
    "type": "json_schema",
    "json_schema": {
        "type": "object",
        "properties": {
            "primary_scene_category": {
                "type": "string",
                "description": "Dominant moment type for specific scene for vertical-feed taxonomy and profile matching",
            },
            "subtags": {
                "type": "array",
                "description": (
                    "Concrete visual/audio signals of scene for what is happening and can be heard in the scene"
                ),
                "items": {"type": "string"},
                "maxItems": 5,
            },
            "description": {
                "type": "string",
                "description": (
                    "Precise moment summary: who, what happened, tone, body language, "
                    "and dialogue subtext for long-form episodic TV. Should pinpoint exact names and map to people."
                ),
            },
            "emotional_intensity": {
                "type": "string",
                "description": "Salience for feed ranking and profile intensity preference",
                "enum": ["low", "medium", "high", "explosive"],
            },
            "key_figures": {
                "type": "array",
                "description": "On-screen people or named participants; pairs or groups in conflict/alliance when clear. Get specific names and map to people.",
                "items": {"type": "string"},
            },
            "feed_headline": {
                "type": "string",
                "description": "One short headline suitable for a vertical clip card",
            },
            "scene_setting": {
                "type": "string",
                "description": (
                    "Setting type: interview, ensemble_conversation, event, travel, "
                    "home, workplace, competition_stage, other"
                ),
            },
            "cross_episode_significance": {
                "type": "string",
                "description": (
                    "If inferable: recurring storyline, callback, finale/reunion relevance, "
                    "or franchise-defining beat; empty string if none"
                ),
            },
        },
        "required": [
            "primary_scene_category",
            "subtags",
            "description",
            "emotional_intensity",
            "key_figures",
            "feed_headline",
        ]
    }
}

For more on customizing the enrichment configuration, see the official TwelveLabs documentation.

Step 1: Segmentation with Pegasus 1.5 Time-Based Metadata

With content uploaded to both the Jockey Knowledge Store and the TwelveLabs indexes, the next step is to run Pegasus 1.5 against each episode. Most of the implementation lives in ranking.py. The key call uses time_based_metadata mode, which instructs Pegasus to partition the video into 15-30 second windows and emit structured JSON for each:

create_response = client.analyze_async.tasks.create(
    model_name="pegasus1.5",
    video=VideoContext_AssetId(asset_id=asset_id),
    analysis_mode="time_based_metadata",
    temperature=0.2,
    max_tokens=PEGASUS_15_SEGMENTATION_MAX_TOKENS,
    min_segment_duration=SEGMENT_MIN_DURATION_SEC,
    max_segment_duration=SEGMENT_MAX_DURATION_SEC,
    custom_id=custom_id,
    response_format=AsyncResponseFormat(
        type="segment_definitions",
        segment_definitions=[feed_moment_segment_definition()],
    ),
)

create_response = client.analyze_async.tasks.create(
    model_name="pegasus1.5",
    video=VideoContext_AssetId(asset_id=asset_id),
    analysis_mode="time_based_metadata",
    temperature=0.2,
    max_tokens=PEGASUS_15_SEGMENTATION_MAX_TOKENS,
    min_segment_duration=SEGMENT_MIN_DURATION_SEC,
    max_segment_duration=SEGMENT_MAX_DURATION_SEC,
    custom_id=custom_id,
    response_format=AsyncResponseFormat(
        type="segment_definitions",
        segment_definitions=[feed_moment_segment_definition()],
    ),
)

A typical output segment looks like this:

"6a14ddbaddce351fa0ae8952": {
      "asset_id": "6a14ddbaddce351fa0ae8952",
      "task_id": "6a17154aa35bbfb00c5cea12",
      "status": "ready",
      "segments": [
        {
          "segment_id": "6a14ddba_seg_0000",
          "asset_id": "6a14ddbaddce351fa0ae8952",
          "start_sec": 195,
          "end_sec": 225,
          "duration_sec": 30,
          "description": "The housewives arrive at the ultra-pink Trixie Motel in Palm Springs. Drag queen Trixie Mattel greets them in a fluffy mint-green outfit, and the group marvels at the drag-themed decor.",
          "emotional_intensity": 7,
          "explanation": "This segment highlights the glamorous arrival at the Trixie Motel and the housewives' first encounter with Trixie Mattel, which is why it makes for a great clip for fashion and parties fans watching The Real Housewives of Salt Lake City.",
          "feed_headline": "Trixie Mattel Welcomes Housewives to Pink Paradise",
          "key_participants": [
            "Trixie Mattel",
            "Lisa Barlow",
            "Heather Gay",
            "Meredith Marks",
            "Whitney Rose",
            "Angie Smibert"
          ],
          "primary_category": "luxury_fashion",
          "show_name": "The Real Housewives of Salt Lake City",
          "subtags": [
            "fashion_beat",
            "parties_nightlife"
          ]
        },
    }
}

"6a14ddbaddce351fa0ae8952": {
      "asset_id": "6a14ddbaddce351fa0ae8952",
      "task_id": "6a17154aa35bbfb00c5cea12",
      "status": "ready",
      "segments": [
        {
          "segment_id": "6a14ddba_seg_0000",
          "asset_id": "6a14ddbaddce351fa0ae8952",
          "start_sec": 195,
          "end_sec": 225,
          "duration_sec": 30,
          "description": "The housewives arrive at the ultra-pink Trixie Motel in Palm Springs. Drag queen Trixie Mattel greets them in a fluffy mint-green outfit, and the group marvels at the drag-themed decor.",
          "emotional_intensity": 7,
          "explanation": "This segment highlights the glamorous arrival at the Trixie Motel and the housewives' first encounter with Trixie Mattel, which is why it makes for a great clip for fashion and parties fans watching The Real Housewives of Salt Lake City.",
          "feed_headline": "Trixie Mattel Welcomes Housewives to Pink Paradise",
          "key_participants": [
            "Trixie Mattel",
            "Lisa Barlow",
            "Heather Gay",
            "Meredith Marks",
            "Whitney Rose",
            "Angie Smibert"
          ],
          "primary_category": "luxury_fashion",
          "show_name": "The Real Housewives of Salt Lake City",
          "subtags": [
            "fashion_beat",
            "parties_nightlife"
          ]
        },
    }
}

Pegasus returns six fields per segment that feed directly into the ranking algorithm:

1. Primary Category: One of a fixed set of enums (Fights, Fashion, Romance, etc.) for profile matching

2. Subtags: Granular signals within each category (Screaming, Table Flip, Designer Clothes)

3. Emotional Intensity: An integer from 0-10 used for per-persona intensity preference matching

4. Feed Headline: The vertical card title shown in the UI

5. Feed Description: A concise scene summary

6. Key Participants: Cast members visible in the clip, later used by Jockey to build character arcs

This is equivalent to what a human logger produces, but generated in a fraction of the time and at scale. Pegasus reads tone, body language, dialogue, and how all of these attributes shift over the course of a segment. It can distinguish playful shade from a relationship-ending confrontation. Compared against average logging labor costs and Pegasus 1.5 API pricing, the cost reduction per episode is approximately 80%.

If you would like to learn more about Time-Based Metadata API calls I highly recommend you read the documentation and research: Building Pegasus 1.5: From Clip-Based QA to Time-Based Metadata.

Step 2: Enrich with Jockey Cross-Episode Top Moments

Per-clip metadata from Pegasus is sufficient for basic persona matching. The harder problem is distinguishing clips that have identical categories, intensities, and cast members. What separates a fight between incidental co-stars from a feud between the show's central characters is context that no single-video analysis can surface.

Jockey solves this through corpus-level reasoning. Given a full Knowledge Store of indexed episodes, it tracks character relationships, plot progression, and emotional arcs across the entire season. The developer interface for this is a single natural language prompt and a structured JSON output schema:

"""
Cross-episode top moments via Jockey POST /responses (jockey + knowledge store).

Taxonomy enums match Pegasus (PRIMARY_CATEGORIES, SUBTAG_ENUM, intensity 0-10).
"""
body = {
    "model": "jockey1.0",
    "knowledge_store_id": os.environ["TL_KS_ID"],
    "instructions": JOCKEY_INSTRUCTIONS,
    "input": [
        {
            "type": "message",
            "role": "user",
            "content": JOCKEY_USER_PROMPT_TEMPLATE.format(top_n=top_n),
        }
    ],
    "text": {
        "format": {
            "type": "json_schema",
            "name": "superfan_feed",
            "schema": JOCKEY_FEED_SCHEMA,
        }
    },
    "stream": False,
}

response = requests.post(
    f"{BASE_URL}/responses",
    headers={
        "x-api-key": os.environ["TL_API_KEY"],
        "Content-Type": "application/json",
    },
    json=body,
    timeout=600,
)

"""
Cross-episode top moments via Jockey POST /responses (jockey + knowledge store).

Taxonomy enums match Pegasus (PRIMARY_CATEGORIES, SUBTAG_ENUM, intensity 0-10).
"""
body = {
    "model": "jockey1.0",
    "knowledge_store_id": os.environ["TL_KS_ID"],
    "instructions": JOCKEY_INSTRUCTIONS,
    "input": [
        {
            "type": "message",
            "role": "user",
            "content": JOCKEY_USER_PROMPT_TEMPLATE.format(top_n=top_n),
        }
    ],
    "text": {
        "format": {
            "type": "json_schema",
            "name": "superfan_feed",
            "schema": JOCKEY_FEED_SCHEMA,
        }
    },
    "stream": False,
}

response = requests.post(
    f"{BASE_URL}/responses",
    headers={
        "x-api-key": os.environ["TL_API_KEY"],
        "Content-Type": "application/json",
    },
    json=body,
    timeout=600,
)

Run it with:

Jockey returns precise start and end timestamps for season-defining moments, paired with a cross_episode_significance field explaining why each moment matters in the context of the full run:

{
    "asset_id": "...",
    "cross_episode_significance": "Meredith's escalating rage is central to The Real Housewives of Salt Lake City because her controlled persona makes every blowup feel seismic. This shout-heavy sequence matters across episodes because it cements Meredith as someone whose calm can snap into legendary confrontation mode, a rhythm the show keeps returning to.",
    "emotional_intensity": 10,
    "end_sec": 2258,
    "start_sec": 2248
}

{
    "asset_id": "...",
    "cross_episode_significance": "Meredith's escalating rage is central to The Real Housewives of Salt Lake City because her controlled persona makes every blowup feel seismic. This shout-heavy sequence matters across episodes because it cements Meredith as someone whose calm can snap into legendary confrontation mode, a rhythm the show keeps returning to.",
    "emotional_intensity": 10,
    "end_sec": 2258,
    "start_sec": 2248
}

In the next step, the merge script applies a jockey_boost flag to any Pegasus segment whose timestamp window overlaps with these Jockey-identified moments. That boolean becomes a +1.0 weight in the ranking algorithm at query time.

Step 3: Sync the Manifest to the Next.js App

With Pegasus segmentation and Jockey enrichment complete, run:

This script reads data/feed_manifest.json, applies the Jockey boost merges, slims the payload to segments and metadata only, and writes the result to app/data/feed_manifest.json. That file is what the Next.js app reads at request time.

QA Before Proceeding: Before moving to Phase 2, validate the manifest. The following command prints the diversified scroll order for each persona to stdout, letting you inspect categories, intensities, Jockey reasoning, and match scores before the UI is involved:

This is where prompt engineering has the most leverage. If the output does not match your editorial expectations, iterate on the segment definitions in feed_moment_segment_definition() or the enrichment configuration in SUPERFAN_CTV_ENRICHMENT_JSON_SCHEMA before committing the manifest.

Phase 2: Runtime Ranking Engine & Personalization

With app/data/feed_manifest.json synced, the computationally expensive work is done. Phase 2 is entirely within the Next.js application deployed on Vercel.

The app reads the pre-built manifest directly into memory at request time. It calls TwelveLabs APIs live only where dynamic retrieval adds genuine value: ad-hoc Explore searches via Marengo, on-demand Spotlight queries via Jockey, and HLS URL resolution for playback. All ranking decisions are exposed transparently in the UI.

💡 The System Design "Why": Nothing in this phase re-runs Pegasus segmentation or batch Jockey reasoning. Profile feeds target a strict sub-2-second latency. Pre-index offline, rank instantly on query. This is the same pattern enterprise search engines have used for decades, now applied to video.

The Core Feed Scoring & Persona Logic

Before wiring up the Next.js API routes, the ranking math needs a definition. Every clip in the offline manifest is scored against one of three superfan personas:

The score is additive. Category match is the initial gate; every other signal applies a weighted boost:

matchScore = base + Σ(subtagBoosts) + intensityBonus + jockeyBoost

Implementation in app/lib/scoring.ts:

export const SCORE_BASE = 1.0;
export const SCORE_SUBTAG = 0.25;
export const SCORE_INTENSITY = 0.5;
export const SCORE_JOCKEY = 1.0;

export function scoreSegment(segment: Segment, profile: PreferenceProfile): ScoredSegment | null {
  // Gate: If the category doesn't match, drop the clip
  if (!profile.categories.includes(segment.primary_category)) return null;
  
  let score = SCORE_BASE;
  const breakdown: Record<string, number> = { base: SCORE_BASE };

  // Subtag Boosts
  for (const rawTag of segment.subtags ?? []) {
    const tag = normalizeSubtag(rawTag);
    if (profile.subtag_boosts.includes(tag) && SUBTAG_ENUM.has(tag)) {
      score += SCORE_SUBTAG;
      breakdown[`subtag_${tag}`] = SCORE_SUBTAG;
    }
  }

  // Intensity Bonus
  if (intensityMatchesPreference(segment.emotional_intensity, profile.intensity_preference)) {
    score += SCORE_INTENSITY;
    breakdown.intensity = SCORE_INTENSITY;
  }

  // Cross-Episode Jockey Boost
  if (segment.jockey_boost) {
    score += SCORE_JOCKEY;
    breakdown.jockey = SCORE_JOCKEY;
  }

  return { ...segment, match_score: score, score_breakdown: breakdown };
}

export const SCORE_BASE = 1.0;
export const SCORE_SUBTAG = 0.25;
export const SCORE_INTENSITY = 0.5;
export const SCORE_JOCKEY = 1.0;

export function scoreSegment(segment: Segment, profile: PreferenceProfile): ScoredSegment | null {
  // Gate: If the category doesn't match, drop the clip
  if (!profile.categories.includes(segment.primary_category)) return null;
  
  let score = SCORE_BASE;
  const breakdown: Record<string, number> = { base: SCORE_BASE };

  // Subtag Boosts
  for (const rawTag of segment.subtags ?? []) {
    const tag = normalizeSubtag(rawTag);
    if (profile.subtag_boosts.includes(tag) && SUBTAG_ENUM.has(tag)) {
      score += SCORE_SUBTAG;
      breakdown[`subtag_${tag}`] = SCORE_SUBTAG;
    }
  }

  // Intensity Bonus
  if (intensityMatchesPreference(segment.emotional_intensity, profile.intensity_preference)) {
    score += SCORE_INTENSITY;
    breakdown.intensity = SCORE_INTENSITY;
  }

  // Cross-Episode Jockey Boost
  if (segment.jockey_boost) {
    score += SCORE_JOCKEY;
    breakdown.jockey = SCORE_JOCKEY;
  }

  return { ...segment, match_score: score, score_breakdown: breakdown };
}

Why additive rather than a black-box ML model? For ranking algorithms, explainability matters more than marginal ranking accuracy. Product and data teams need to answer "Why did this clip surface?" at a glance. The additive breakdown maps directly to what Pegasus tagged and what Jockey boosted in Phase 1, which is what makes the Reasoning Panel (Step 7) possible. It also allows for fine-tuning of the weights down the line if the needs of your customers change.

Step 4: Enable Semantic Discovery with Marengo 3.0

Profile feeds on the home screen read directly from the bundled manifest with no live API calls. The Explore tab is different. When a superfan types "awkward dinner party" or "someone storms out," the system needs real-time retrieval across the entire indexed library.

The /api/search route posts a multimodal query to Marengo via the TwelveLabs Search API:

const form = new FormData();
form.append("index_id", indexId);
form.append("query_text", query.trim());
form.append("search_options", "visual");
form.append("search_options", "audio");
form.append("search_options", "transcription");
form.append("group_by", "clip");
form.append("operator", "or");
form.append("page_limit", "30");

const res = await fetch(`${config.baseUrl}/search`, {
  method: "POST",
  headers: { "x-api-key": config.apiKey },
  body: form,
});

const form = new FormData();
form.append("index_id", indexId);
form.append("query_text", query.trim());
form.append("search_options", "visual");
form.append("search_options", "audio");
form.append("search_options", "transcription");
form.append("group_by", "clip");
form.append("operator", "or");
form.append("page_limit", "30");

const res = await fetch(`${config.baseUrl}/search`, {
  method: "POST",
  headers: { "x-api-key": config.apiKey },
  body: form,
});

Marengo returns timestamped hits grounded in visual, audio, and transcription content, not keyword overlap on a title field.

Joining search results to the manifest. After Marengo returns hits, the app joins them back to the offline Pegasus manifest using asset_id and timestamp overlap:

const segment = findManifestSegment(
  manifest.segments,
  hit.asset_id,
  hit.start_sec,
  hit.end_sec,
);
const scored = segment ? scoreSegment(segment, preference) : null;

const segment = findManifestSegment(
  manifest.segments,
  hit.asset_id,
  hit.start_sec,
  hit.end_sec,
);
const scored = segment ? scoreSegment(segment, preference) : null;

Results sorted by Marengo's search_rank first, with match_score as a tiebreaker.

This hybrid approach handles two different failure modes. Marengo alone can find "vibes" that Pegasus never explicitly tagged. Pegasus alone provides deterministic metadata (headlines, exact intensity scores) but cannot answer zero-shot queries like "Lisa side-eye at reunion." Together they power an Explore tab that behaves like a genuine superfan search engine rather than a rigid category browser.

Step 5: Rank Feeds with Persona Logic + Diversity Constraints

The home feed (/api/feed) is the core product loop. It loads the manifest once, scores every segment for the active persona, and returns a paginated clip list.

A raw sort by match_score creates a different problem: a Drama Addict's top ten clips by pure score might all be fights_confrontation. To keep the feed bingeable, the app implements diversified round-robin ordering:

1. Bucket segments by primary_category

2. Sort within each bucket by match_score descending

3. Interleave one clip per category per round (Fights → Shade → Emotional → Fights...)

4. Paginate with offset + limit for infinite scroll

Because this route performs only JSON reads and in-memory sorting, latency is in the tens of milliseconds with no live TwelveLabs API call required.

Step 6: Interactive Spotlight with Runtime Jockey

Phase 1 and Phase 2 use Jockey differently, and it is worth being explicit about the distinction:

The Spotlight page (/api/spotlight) is where Jockey answers complex, ad-hoc narrative questions the manifest was never designed to pre-compute. Two modes:

1. Actor Spotlight: "Find all clips featuring Meredith Marks" returns top 8 defining moments, relationship summaries, and narrative arcs

2. Moment Discovery: "How did the Lisa and Meredith friendship fall apart?" returns chronologically ordered story scenes mapped to structural roles (origin, buildup, turning point, aftermath)

Strict JSON schemas ensure the UI can immediately map Jockey's response to the video player:

const jockeyBody = {
  model: "jockey1.0",
  instructions: isActor ? ACTOR_SPOTLIGHT_INSTRUCTIONS : MOMENT_DISCOVERY_INSTRUCTIONS,
  input: [{ type: "message", role: "user", content: userContent }],
  knowledge_store_id: ksId,
  text: {
    format: {
      type: "json_schema",
      name: isActor ? "actor_spotlight" : "moment_discovery",
      schema: isActor ? ACTOR_SPOTLIGHT_SCHEMA : MOMENT_DISCOVERY_SCHEMA,
    },
  },
};

const jockeyBody = {
  model: "jockey1.0",
  instructions: isActor ? ACTOR_SPOTLIGHT_INSTRUCTIONS : MOMENT_DISCOVERY_INSTRUCTIONS,
  input: [{ type: "message", role: "user", content: userContent }],
  knowledge_store_id: ksId,
  text: {
    format: {
      type: "json_schema",
      name: isActor ? "actor_spotlight" : "moment_discovery",
      schema: isActor ? ACTOR_SPOTLIGHT_SCHEMA : MOMENT_DISCOVERY_SCHEMA,
    },
  },
};

Passing a session_id on follow-up requests enables multi-turn refinement. A user can ask "Now show me only moments from the Atlanta trip" and Jockey narrows the response without re-ingesting the corpus.

Step 7: Deliver with Transparent Reasoning in the Feed UI

The final deliverable is not just a video player. It is an explainable ranking surface. Every clip selection exposes its exact rationale through a slide-out Reasoning Panel:

Why This Clip?

• Pegasus: "This segment highlights a screaming confrontation at the reunion dinner... making it a perfect match for your High-Intensity Drama preference."

• Jockey: "Across the full run of the show, this moment mirrors the Season 3 reunion fight between the same cast members and is referenced in two later episodes."

For technical evaluators, a Raw Data view toggles the full Pegasus and Jockey JSON payloads. The Explore tab includes an export function that writes clip metadata to Excel, giving QA and editorial teams an auditable sheet of categories, intensities, match scores, and Jockey reasoning.

Deploy to Vercel

The /app directory is now Vercel-ready! Congratulations on building and following along with this tutorial. In order to push your own version to the web, you must:

1. Push to GitHub. Fork the repository to your own account.

2. Import the project in Vercel. Set the Root Directory to app.

3. Add environment variables:

   • TL_API_KEY (Required)

   • TL_INDEX_ID (Required for Marengo Search and HLS playback)

   • TL_KS_ID (Recommended for Jockey Spotlight)

4. Deploy. npm run build runs automatically.

What runs where after deploy:

When new episodes are added, re-run Phase 1 locally, commit the updated manifest to Git, and redeploy. The episode video files never enter the Git repo or Vercel; they stay in TwelveLabs.

Why This Architecture Matters

This pipeline turns static, expensive episode catalogs into highly queryable, explainable moment inventories. That is the core value proposition for any streaming or broadcasting platform evaluating TwelveLabs: not just faster tagging, but a fundamentally different kind of content intelligence.

Pegasus 1.5 produces per-clip metadata at scale that would require full-time logging staff to generate manually, at roughly 20% of the cost. Marengo 3.0 makes that metadata searchable with natural language queries that no keyword index could match. Jockey 1.0 adds the layer that neither model can provide alone: an understanding of how individual moments fit into the larger story of a season, which is exactly what superfans care about.

The result is a feed engine that does not just recommend content. It surfaces the right moment, in the right context, with a transparent explanation of why.

If you’d like to see more technical blogs like visit our Blogs or if you want custom enterprise video solutions for your company please talk to Sales!

Additional Resources:

• Sign up for Jockey: https://www.twelvelabs.io/jockey

• Source Code: github.com/nathanchess/jockey-superfan-vertical-feed

• Live Demo: jockey-superfan-vertical-feed.vercel.app

• TwelveLabs Playground: playground.twelvelabs.io

• API Reference: https://docs.twelvelabs.io/v1.3/agents/get-started/quickstart

• Architecture Diagram: LucidChart Diagram