Turning Retail CCTV Footage into Actionable Intelligence with TwelveLabs Pegasus and Jockey

Retail video at scale is not just a monitoring problem. It is an intelligence problem.

Retail store cameras continuously record operational details that can shape better decisions from growing checkout lines and customers waiting for assistance to declining shelf stock, replenishment activity, and incidents at the point of sale. The difficulty is not capturing these events but locating and understanding the relevant footage quickly enough to respond. That need is becoming more urgent. According to a 2025 study by the National Retail Federation and Loss Prevention Research Council, surveyed retailers experienced an 18% increase in average annual shoplifting incidents during 2024, while theft-related threats and acts of violence rose by 17%. Retailers already possess the visual evidence, and the challenge is extracting the insight hidden within hours of footage.

Turning retail footage into useful intelligence requires semantic search, structured interpretation of visible activity, and timestamped evidence that supports every finding. Metrics are also more trustworthy when reviewers can trace them directly to the source footage.

This tutorial walks through building Sentinel, an evidence-grounded retail video intelligence application using TwelveLabs. The application uses Knowledge Store Search to find relevant moments across the complete footage library or within a selected video. Pegasus 1.5 transforms the footage into schema-driven analysis for operations, security, customer behavior, and inventory, while Jockey lets retail operators ask questions in natural language and receive answers grounded in playable, timestamped citations. 

Inside the Retail CCTV Intelligence Pipeline

Sentinel transforms retail CCTV footage into structured, evidence-grounded operational intelligence. Pegasus 1.5 analyzes each video across operations, security, customer behavior, and inventory, producing schema-validated findings, KPIs, summaries, and timestamped evidence. Jockey supports natural-language search and questions across indexed footage, while YOLO26m and ByteTrack ground selected Pegasus moments with object detections, anonymous customer and employee tracks, reviewable face or head crops, and calibrated floor-level movement paths.

Every result remains connected to its source moment, allowing users to move directly from an insight to the exact section of footage that supports it. From video ingestion and semantic analysis to KPI review, evidence playback, suspicious-person inspection, customer tracking, inventory observations, and zone-aware movement heatmaps, a detailed walkthrough of the complete workflow is provided below—

You can explore the demo of the application here: https://tl-retail.vercel.app/

Setting Up the Environment

Before building, complete the following prerequisites.

1. Clone the GitHub repository for the backend API endpoint source code.

2. Create a TwelveLabs account and generate an API key. 

3. Create a TwelveLabs Knowledge Store for Jockey and record it for the .env. 

4. Create backend/.env using the variables mentioned in .env.example.

Prepare Knowledge Store for Search and Reasoning

A checkout queue, a customer waiting for assistance, and a shelf being replenished may appear across different recordings of footage. Isolated recordings force investigators to repeatedly locate and review.

The workflow begins by organizing the footage into a shared intelligence layer. Each uploaded recording becomes a TwelveLabs asset that represents the source video and its playback resources. The asset is then added to a knowledge store as an item, where TwelveLabs processes it for semantic retrieval and grounded reasoning.

Creating a Knowledge Store for Retail Video Intelligence

A Knowledge Store defines the collection of footage available for search and agentic reasoning. TwelveLabs combines the uploaded media with derived temporal context, semantic representations, and embeddings that make the collection accessible through natural-language requests.

A Knowledge Store can use the default ingestion behavior or additional instructions describing which information should receive greater attention during processing. The knowledge store creation configuration provided below is built on retail-focused guidance covering checkout activity, customer assistance, shelf conditions, replenishment, and observable safety events. 

The following snippet creates the Knowledge Store used to organize and process the retail footage.

import os

from twelvelabs import (

    EnrichmentConfig_Description,

    IngestionConfig,

    TwelveLabs,

)

client = TwelveLabs(

    api_key=os.environ["TWELVELABS_API_KEY"],

)

knowledge_store = client.knowledge_stores.create(

    name="Retail Video Intelligence",

    description=(

        "Retail footage for semantic search, agentic reasoning, "

        "and evidence-grounded analysis."

    ),

    ingestion_config=IngestionConfig(

        enrichment_config=EnrichmentConfig_Description(

            description=(

                "Prioritize observable retail activity, including checkout queues, "

                "customer assistance, shelf conditions, replenishment activity, "

                "and visible safety or security events."

            )

        )

    ),

    metadata={

        "domain": "retail",

        "source": "retail-video-intelligence",

    },

)

print(f"Knowledge Store ID: {knowledge_store.id}")

The returned Knowledge Store ID should be assigned to KNOWLEDGE_STORE_ID in the backend environment variable. The application uses this identifier when adding ready video assets as Knowledge Store items and when sending requests to Knowledge Store Search and Jockey, ensuring that ingestion, retrieval, and agentic reasoning operate over the same retail footage collection.

Turning the Retail Footage into TwelveLabs Assets and Knowledge Store Items

A newly uploaded retail store footage first becomes a TwelveLabs asset, which represents the original video and its playback resources. That asset is then added to the Knowledge Store as an item.

def add_asset_to_knowledge_store(

    self,

    knowledge_store_id: str,

    asset_id: str,

    metadata: dict[str, Any] | None = None,

) -> KnowledgeStoreItemRef:

    item_metadata = {

        str(key): str(value)

        for key, value in (metadata or {}).items()

        if value is not None

    }

    item_response = self.sdk_client.knowledge_store_items.create(

        knowledge_store_id,

        asset_id=asset_id,

        asset_type="video",

        metadata=item_metadata,

    )

    return KnowledgeStoreItemRef(

        id=item_response.id,

        asset_id=item_response.asset_id or asset_id,

        status=item_response.status or "processing",

        raw=item_response.model_dump(

            mode="json",

            by_alias=True,

            exclude_none=True,

        ),

    )

The application checks the file size before choosing direct or multipart upload. The direct-upload path creates the asset with HLS playback and thumbnail. An asset ID that becomes the stable reference for playback, Pegasus analysis, and persistent metadata.

After the asset and its preview resources are ready, the application creates a Knowledge Store item from that asset as defined below. The application coordinates the complete transition from uploaded asset to ready Knowledge Store item inside finish ingestion workflow. The Knowledge Store item turns that video into searchable and conversational knowledge.

Find Any Critical Retail Moment with Knowledge Store Search

Retail footage is valuable only when the right moment can be found, as it hides inside hours of recordings. A growing queue, an unattended counter, a shelf gap, or an unusual register interaction moment may last only a few seconds, yet locating it manually can take far longer.

Knowledge Store Search turns that review process into a simpler text query lookup. An operator describes the event they want to find, and the application searches content from the configured knowledge store. The search can cover the entire footage library of the knowledge store or focus on the selected store video.

Finding Retail Events Without Manual Tags or Timeline Scrubbing

Knowledge Store Search allows an operator to describe the critical moment in natural language search instead of scrubbing through the complete timeline.

The application is designed to support configurable search options, allowing users to search either across all video footage in the knowledge store or within footage from a specific retail store. With the entire library search, no scenario is supplied, and the configured knowledge store is searched across all video items. With the selection of the particular video, the backend uses the scenario identifier to resolve the knowledge_store_item_id associated with the video currently open in the application. That item ID becomes an explicit search filter.

The below core method performs the TwelveLabs Knowledge Store Search.

backend/app/services/twelvelabs_service.py (Line 476)

def search_knowledge_store(

    self,

    knowledge_store_id: str,

    query: str,

    *,

    item_id: str | None = None,

    modalities: list[str] | None = None,

    page_size: int = 10,

    group_by: Literal["none", "item"] = "none",

) -> dict[str, Any]:

    selected_modalities = modalities or ["visual", "audio"]

    search_filter = SearchKnowledgeStoreFilter(

        asset_type=AssetTypeFilter(eq="video"),

        item_id=ItemIdFilter(eq=item_id) if item_id else None,

    )

    search_response = self.sdk_client.knowledge_stores.search(

        knowledge_store_id,

        query=KnowledgeStoreSearchQuery(text=query),

        filter=search_filter,

        search_options=SearchKnowledgeStoreOptions(

            video=VideoSearchOptions(

                modalities=selected_modalities,

            )

        ),

        group_by=group_by,

        page_size=page_size,

        include_metadata=True,

    )

    return search_response.model_dump(mode="json",by_alias=True,

       exclude_none=True,)

Knowledge Store Search locates matching clips and returns their source items and time ranges with the ranking of the most relevant to the query.

The example below searches for “find unusual or suspicious activity” within the selected retail video. The returned results highlight the moments whose visual content most closely matches the operator’s request.

Each result provides a timestamped entry point for investigation and can be opened directly in the video player. The ranked matches turn a broad security question into a focused evidence trail, allowing the operator to move directly to the strongest moments and assess their significance in context.

Turn Footage into Retail Signals with Pegasus 1.5

Retail footage becomes valuable when it can take a team from "something happened” to “this is the operational impact, and here is the exact footage that supports it.”

TwelveLabs Pegasus 1.5 interprets the activity visible in the video. The backend converts those observations into measurable retail outcomes and actionable insights. Finally, the processed evidence is attached to the TwelveLabs asset metadata so it can be restored later.

For every ready video asset, the backend runs four separate Pegasus 1.5 analysis tasks in sequence. Each task uses its own retail prompt and JSON Schema for operations, security, customer flow, or inventory. This produces focused results instead of one broad summary attempting to represent every type of store activity at once.

Defining What Matters within Retail Analysis Schemas

Pegasus 1.5 returns structured JSON that conforms to the schema supplied through response_format. The schema defines the expected observations, field types, permitted values, and supporting evidence, enabling the backend to construct consistent retail metrics, timelines, alerts, and timestamped findings.

The backend supplies the category’s prompt and response schema when creating each analysis task.

backend/app/services/twelvelabs_service.py  (Line 614)

created_task = self.sdk_client.analyze_async.tasks.create(

    video=VideoContext_AssetId(asset_id=asset_id),

    model_name="pegasus1.5",

    custom_id=custom_id,

    prompt_v_2=AnalyzePromptV2(input_text=prompt_text),

    temperature=0.2,

    max_tokens=max_tokens,

    response_format=AsyncResponseFormat(

        type="json_schema",

        json_schema=build_pegasus_response_schema(response_schema),

    ),

)

The prompt_text contains the category-specific evidence rules and configured store zones. The response_format defines the structure Pegasus must follow, giving the backend predictable fields that can be validated before they are presented in the interface.

Operations

The operations schema captures checkout lanes and their customer observations, entries, counter waits, unattended-register intervals, merchandise-recovery events, and other visible store activity. It accepts people_count only when stable anonymous customer identifiers support a defensible count.

The backend turns these observations into metric receipts, checkout-lane summaries, service alerts, and a chronological operations timeline. As observed from the above operation view, it connects calculated performance signals with the checkout activity and timestamped events behind them.

Security

The security schema records directly observable incidents with their time ranges, zones, visibility, confidence, threat levels, threat types, rationales, and supporting indicators.

Its indicator families are suspicious_observation, concealment_tooling, and intrusive_action. This structure allows the interface to show the visible evidence supporting an assessment instead of presenting only a risk label.

In this example above, five distinct concealment events are identified and organized into a chronological timeline. Each incident includes its timestamp, store zone, threat classification, and directly observed signals such as merchandise examination and concealment. Reviewers can select any incident to inspect the corresponding moment in the footage.

Customer

The customer-flow schema returns a chronological narrative and continuous anonymous customer visits to configured store zones. Each visit records its start and end times, visible activity, confidence, visibility, and a clip-local person identifier only when continuity can be maintained reliably.

The Customer view turns Pegasus-supported observations into a chronological account of activity across configured store zones. In this example above, each checkout-counter visit records when the visitor appeared, how long the visit remained observable, whether it overlapped with another visit, and the quality of visibility. Moment thumbnails link every observation to the corresponding footage, helping operators examine customer flow, dwell patterns, and periods of concurrent demand.

Inventory

The inventory task returns timestamped shelf observations. These identify the shelf, visible stock state, change from an earlier observation, product description, confidence, and visibility. The allowed stock states are healthy, low_stock, empty, and needs_restock, while changes are recorded as unchanged, depleted, or recovered.

Every schema is also designed to capture the analysis_quality result with a status of sufficient, partial, or insufficient. Concrete limitations record conditions such as occlusion, interrupted continuity, an unsuitable camera angle, or limited shelf visibility. When the footage cannot support a category, the prompts instruct Pegasus to return an empty collection and explain the limitation.

Grounded Pegasus Evidence with the Yolo26 and ByteTrack

Pegasus explains what happened and when, while the localization layer shows where the supporting subject or object appears within the frame. For each cited moment, YOLO26m detects supported object classes, and ByteTrack connects consecutive detections using window-scoped track IDs.

Localization is limited to bounded evidence windows received from the Pegasus instead of processing the entire video for efficient usage. The application consolidates duplicate windows, caches completed results, and displays only detections that meet its 85% threshold. This produces precise visual overlays and review candidates while keeping every localization tied to the original Pegasus finding.

Employee and Customer Tracking

Within each Pegasus-selected window, YOLO26m detects visible people and ByteTrack links those detections across consecutive frames. The application then classifies anonymous tracks as 'customer' or 'employee' by calibrated or automatically learned staff-side zones and Pegasus context describing the event being reviewed.

Customer Face Evidence

For customer analysis, the application searches only within customer-related evidence windows returned by Pegasus. YOLO26m and ByteTrack first locate and follow the relevant person tracks; the localizer then extracts the best reviewable face or head crop based on visibility, detection confidence, and image quality.

Staff-side tracks, weak matches, repeated candidates, and ambiguous customer versus employee observations are filtered out. The resulting portrait, therefore, acts as a timestamp-linked customer review candidate

Suspicious Person Review

When Pegasus identifies a security incident, its cited time range becomes the search boundary for suspicious person localization. The system detects visible person tracks, associates their movement with ByteTrack, and selects the strongest face or head candidate from the relevant moment.

Customer Movement Tracking Heatmap

Each eligible customer detection contributes a visible foot anchor taken near the bottom centre of its bounding box. When the camera has a valid four-point floor calibration, a homography projects that anchor from normalized camera coordinates onto a measured floor plane.

The byteTrack connects these projected anchors into short, anonymous movement paths. The application can then aggregate them into a zone-aware heatmap showing where customers moved or concentrated during the Pegasus-selected evidence windows. The heatmap represents movement only within the limited evidence supplied by Pegasus. This provides the spatial insights of the customer movement through a monitored zone and traffic density within the calibrated floor area.

Ask the Store with Jockey

Retail investigations are often less predictable. An operator may notice a long wait, an unusual register interaction, or a shelf issue and need to explore what happened through several related questions.

Jockey adds this agentic layer to the knowledge store. The application sends the operator’s question, focuses the request on the selected video item, and retains the returned session ID for follow-up questions. When Jockey provides timestamped references, the application converts them into playable evidence.

Ask Questions the Way Retail Operators Actually Speak

Operators can ask questions such as “What caused the longest customer wait?” “Where did service slow down?” or “Was there any unusual activity near the register?” They do not need to know the underlying schema fields or construct separate searches for every part of the event.

The following snippet captures the core Jockey request used by the application.

backend/app/services/twelvelabs_service.py (Line 684)

def ask_jockey(

    self,

    knowledge_store_id: str,

    query: str,

    session_id: str | None = None,

    item_id: str | None = None,

) -> AskAnswer:

    scoped_query = f"{{{{sel:0}}}} {query}" if item_id else query

    request: dict[str, Any] = {

        "knowledge_store_id": knowledge_store_id,

        "input": [

            ResponseInputItem(

                type="message",

                role="user",

                content=scoped_query,

            )

        ],

        "session_id": session_id,

    }

    if item_id:

        request["selections"] = [

            ResponseSelection(kind="item", id=item_id)

        ]

    response = self.sdk_client.responses.create(**request)

    answer = " ".join(

        part.text.strip()

        for output in response.output or []

        for part in output.content or []

        if part.text and part.text.strip()

    )

    answer, citations = self.extract_inline_video_references(

        answer or "Live answer returned no text."

    )

    return AskAnswer(

        answer=answer,

        citations=citations,

        session_id=response.session_id or session_id,

        live=True,

    )

When a video item is available, reference that selection inside the question. This focuses Jockey on the footage open in the application. The selection acts as strong prompt guidance rather than a strict access boundary.

The first response returns a session_id, which the frontend sends with the next request. An operator can ask where service slowed down and then follow with “What happened immediately before that?” without repeating the original context. The session is cleared when another video is selected.

Returned moment references are normalized into citations containing start and end times. The interface presents them as clickable evidence and seeks the video player to the corresponding point in the footage. If Jockey does not return a specific moment, the application labels the answer with a scenario-level reference instead of presenting an unsupported timestamp as precise evidence.

What This Approach Makes Possible

This approach transforms disconnected retail footages into a searchable and explainable operational intelligence layer. Teams can locate critical moments using natural language search, convert footage into structured retail signals with Pegasus 1.5, and investigate events conversationally through Jockey without manually reviewing hours of video.

Combining semantic understanding with YOLO26m and ByteTrack-based localization ensures that each insight remains directly linked to timestamped, visually reviewable evidence. Operators can examine checkout delays, customer movement, shelf conditions, service gaps, and suspicious activity. The result is a unified workflow that transforms retail video into measurable performance signals, verifiable evidence, and faster operational intelligence.

Resources

• Sentinel Retail CCTV Intelligence source code - Explore the complete implementation of the retail video search, structured Pegasus analysis, Jockey Q&A, and evidence-localization workflow.

• Sentinel Retail CCTV Intelligence App - Explore the complete retail video intelligence workflow in the live application.

• TwelveLabs Pegasus 1.5 overview - Learn how Pegasus 1.5 enables deeper video understanding and structured analysis.

• TwelveLabs Jockey Knowledge Store guide - Follow the guide for creating a Knowledge Store used by Jockey.

• TwelveLabs sample applications - Browse example applications built with TwelveLabs video intelligence.

• Join the TwelveLabs Discord community - Connect with the TwelveLabs community, ask questions, and share what you build.