Introduction

Ever watched a show and thought, “Where can I buy that?” Until now, streaming platforms have struggled with clunky solutions—manual product tagging, static overlays, and checkout flows that break the moment.

⭐️Check out the demo here!

That’s where Shoppable Video comes in. This demo app, built with the Twelve Labs API, shows how you can turn any video into an interactive shopping experience. It automatically detects products in the scene, generates context-aware descriptions, and even lets viewers check out without pausing playback. Think TikTok Shop vibes—only optimized for long-form streaming content!

Prerequisites

• Sign up for the Twelve Labs Playground and generate your API key 

• Create an index and upload at least one video you’d like to apply to the application 

• Clone the project from GitHub and configure environment (Follow README > Quick Start)

App Demo

Loom Link

How It Works

The app works by following a simple flow:

1. Fetch Videos – GET videos API: The app pulls the list of videos from your default Twelve Labs index and selects the most recent one by default.

2. Fetch Video Details – GET video API: For the selected video, the app retrieves full details including the HLS playback URL and any existing product metadata.

3. Analyzing Video – ANALYZE API: If no product metadata exists, the app calls the Twelve Labs Analyze API with a custom prompt to detect products and generate contextual descriptions.

4. Saving Metadata – PUT video API: The analysis results are saved back to the video via a PUT request, so they can be retrieved later without re-analyzing.

5. Rendering in the UI: The app displays product markers on top of the video only when items appear on screen, with detailed information shown in the sidebar and direct links to shop at Amazon.

Code Walkthrough

1 - Fetching Videos – GET videos API

The app first fetches the list of videos from your default Twelve Labs index. This allows the UI to show available videos and select the most recent one by default.

1-1. Client: Fetching the video list

In page.tsx, the loadVideos function calls /api/videos with the index ID. It retrieves up to 50 videos and automatically selects the latest one.

page.tsx (line 28 - 49)

// Load videos from TwelveLabs index
 const loadVideos = useCallback(async () => {
   const defaultIndexId = process.env.NEXT_PUBLIC_DEFAULT_INDEX_ID;
   if (!defaultIndexId) {
     console.error('Default index ID not configured');
     return;
   }


   setIsLoadingVideos(true);
   try {
     const response = await fetch(`/api/videos?index_id=${defaultIndexId}&limit=50`);
     if (!response.ok) {
       throw new Error(`Failed to fetch videos: ${response.statusText}`);
     }


     const data = await response.json();
     setVideos(data.data || []);


     // Select the most recent video by default (first in the list since API returns newest first)
     if (data.data && data.data.length > 0) {
       setSelectedVideoId(data.data[0]._id);
     }

1-2. Server: /api/videos

This is a simple Next.js API route that proxies to Twelve Labs’ GET/v1.3/indexes/{indexId}/videos. See full implementation in the GitHub repo.

⭐️Check out details here for Twelve Labs’ GET videos API

2 - Fetching Video Details – GET video API

After a video is selected, the app fetches its details, including HLS playback URL and any existing product metadata.

2-1. Client: Fetching video details

When a video is selected, loadVideoDetail is called. It hits the /api/videos/[videoId] route with the index ID, retrieves the video details including the HLS playback URL, and checks whether product metadata already exists.

page.tsx (line 59 - 90)

// Load video detail when a video is selected
 const loadVideoDetail = useCallback(async (videoId: string) => {
   const defaultIndexId = process.env.NEXT_PUBLIC_DEFAULT_INDEX_ID;
   if (!defaultIndexId || !videoId) {
     return;
   }


   setIsLoadingVideoDetail(true);


// ...


   try {
     const response = await fetch(`/api/videos/${videoId}?indexId=${defaultIndexId}`);
     if (!response.ok) {
       throw new Error(`Failed to fetch video detail: ${response.statusText}`);
     }


     const data = await response.json();
     setVideoDetail(data);


     // Check if custom metadata exists and generate if needed BEFORE setting video URL
     await checkAndGenerateMetadata(videoId, defaultIndexId, data);


     // Set video URL from HLS data if available (AFTER metadata check)
     if (data.hls?.video_url) {
       setVideoUrl(data.hls.video_url);
     }     


// ...

2-2. Server: /api/videos/[videoId]

This route proxies to Twelve Labs’
GET /v1.3/indexes/{indexId}/videos/{videoId}. Full implementation in the GitHub repo.

⭐️Check out details here for Twelve Labs’ GET video API

3 - Analyzing Video – ANALYZE API

The app only calls the Analyze API when no usable product metadata exists for the selected video.

3-1. Client: Trigger analysis (page.tsx)

In `checkAndGenerateMetadata`, the client first checks `videoData.user_metadata`. If nothing is usable, it calls /api/analyze?videoId=... and parses the JSON array of products.

page.tsx (line 103 - 147)

// Check and generate metadata if needed
 const checkAndGenerateMetadata = useCallback(async (videoId: string, indexId: string, videoData: VideoDetail, forceReanalyze = false) => {
   // 1) Use existing metadata if present (unless forceReanalyze)
   if (!forceReanalyze && videoData.user_metadata && Object.keys(videoData.user_metadata).length > 0) {
     if (videoData.user_metadata.products) {
       let existingProducts;
       try {
         // Parse products if it's stored as JSON string
         if (typeof videoData.user_metadata.products === 'string') {
           existingProducts = JSON.parse(videoData.user_metadata.products);
         } else {
           existingProducts = videoData.user_metadata.products;
         }


         setProducts(existingProducts);


// ...


     setIsAnalyzingVideo(false); // Stop analyzing when using existing metadata
     return;
   }


   // 2) Otherwise, analyze now
   setIsAnalyzingVideo(true);


   try {
     const analyzeResponse = await fetch(`/api/analyze?videoId=${videoId}${forceReanalyze ? '&forceReanalyze=true' : ''}`);


         // ...


     const analyzeData = await analyzeResponse.json();

3-2. Server: /api/analyze

/api/analyze route calls the Twelve Labs Analyze API. 

⭐️Check out details here for Twelve Labs’ Analyze API

This API requires a carefully designed prompt so that the model returns structured, machine-readable results.

📌Prompt Design

The prompt is crafted to ensure the API response is complete and easy to parse:

• timeline: [start, end] in seconds; drives when markers are shown on screen.

• brand / product_name: Used for product labels and Amazon search queries.

• location: [x%, y%, width%, height%] as percentages on a 1920×1080 frame; resolution-independent coordinates.

• price: Adds purchase context if visible or inferable.

• description: Short contextual summary (voiceover, subtitle, scene).

We also enforce:

• Output must be a valid JSON array only (no markdown, no prose).

• If multiple products appear at once, list them separately.

See below for the server code including the actual prompt text. 

api/analyze/route.ts

export async function GET(req: Request) {
   const { searchParams } = new URL(req.url);
   const videoId = searchParams.get("videoId");
   const forceReanalyze = searchParams.get("forceReanalyze") === "true";


   const prompt = `
  List all the products shown in the video with the following details:


- timeline: [start_time, end_time] in seconds
- brand: brand name
- product_name: full product name
- location: [x%, y%, width%, height%] — percentage values relative to a 16:9 aspect ratio video player
- price: price if shown or mentioned. If you cannot find price information directly from the video, use external search or your knowledge
- description: describe what is said or shown about the product


⚠️ If multiple products appear in the same scene, list them separately with their own location coordinates.


Respond with a valid JSON array only, no markdown formatting:


[
 {
   "timeline": [start, end],
   "brand": "brand_name",
   "product_name": "product_name",
   "location": [x%, y%, width%, height%],
   "price": "price_info",
   "description": "product_description"
 }
]
`
// ...


   const url = `${TWELVELABS_API_BASE_URL}/analyze`;


   const requestBody = {
       prompt: prompt,
       video_id: videoId,
       stream: false
   };


   const options = {
       method: "POST",
       headers: {
           "Content-Type": "application/json",
           "x-api-key": API_KEY,
       },
       body: JSON.stringify(requestBody)
   };


   try {
     const response = await fetch(url, options);
         // ...


     // Return the complete data object instead of just data.data
     return NextResponse.json(data, { status: 200 });
   } 


         // ...


}

Example (responseText) of expected output:

{
"Id":"5cef3150-dc0a-42e1-8ae5-ba8aab536206",
"data":"[\n  {\n    \
"timeline\": [7, 8],\n    \
"brand\": \"Google\",\n    \
"product_name\": \"Chromecast\",\n    \
"location\": [50, 50, 100, 100],\n    \
"price\": \"$35\",\n    \
"description\": \"Everything you love now on your TV.\"\n  }\n]",
"finish_reason":"stop","usage":{"output_tokens"

* Note: data is a stringified JSON array so you need to JSON.parse it on the client.

4 - Saving Metadata – PUT video API

Once the analysis response is parsed, the results are saved back to the video. This ensures future requests can load products directly without re-analyzing.

4-1. Client: Saving product metadata

The client sends a PUT request to /api/videos/saveMetadata, which proxies to the Twelve Labs API.

Page.tsx (165 - 182)

// Save the generated metadata
       const saveRequestBody = {
         videoId: videoId,
         indexId: indexId,
         metadata: {
           products: products,
           analyzed_at: new Date().toISOString(),
           reanalyzed: forceReanalyze
         }
       };


       const saveResponse = await fetch('/api/videos/saveMetadata', {
         method: 'PUT',
         headers: {
           'Content-Type': 'application/json',
         },
         body: JSON.stringify(saveRequestBody)
       });

4-2. Server: /api/videos/saveMetadata

The server route (/api/videos/saveMetadata) is a thin wrapper that calls PUT /v1.3/indexes/{indexId}/videos/{videoId} on the Twelve Labs API. Full implementation is available in the GitHub repo.

⭐️Check out details here for Twelve Labs’ PUT video API

5 - Rendering in the UI 

With product metadata saved and available, the app renders product markers and details dynamically.

5-1. Video Player: Product markers

In ProductVideoPlayer.tsx, markers are drawn on top of the video only when products are visible in the current timeline.

ProductVideoPlayer.tsx (line 199 -225)

{/* Product Markers Overlay */}
<div className="absolute inset-0 pointer-events-none">
       {visibleProducts.map(product => {
         const position = calculateMarkerPosition(product);
         return (
           <button
             key={product.product_name}
             className="product-marker absolute rounded-full bg-black bg-opacity-70 flex items-center justify-center cursor-pointer pointer-events-auto animate-pulse-slow transition-all duration-300 ease-in-out z-10"
             style={{
               left: position.left,
               top: position.top,
               width: '48px', 
               height: '48px', 
               transform: 'translate(-50%, -50%)'
             }}
             onClick={(e) => {
               e.stopPropagation();
               console.log('Product marker clicked:', product.product_name);
               onProductSelect(product);
             }}
             aria-label={`View ${product.product_name} details`}
           >
             <ShoppingBag className="text-white" style={{ fontSize: '24px' }} />
           </button>
         );
       })}
     </div>

5-2. Sidebar: Product details

In ProductDetailSidebar.tsx, each detected product is listed with its description and a “Shop at Amazon” button.

ProductDetailSideba.tsx (line 279 - 311)

                 onClick={() => {
                   if (shouldEnableShopButton) {


                             // ...


                     // Create search query
                     let searchQuery;
                     if (shouldFilterBrand) {
                       // Use only product name if brand is filtered
                       searchQuery = product.product_name;
                     } else {
                       // Use brand + product name if brand is valid
                       searchQuery = `${product.brand} ${product.product_name}`;
                     }


                     const q = encodeURIComponent(searchQuery);
                     window.open(`https://www.amazon.com/s?k=${q}`, '_blank');
                   }
                 }}

👉 Together, these components provide a seamless “shop-the-look” experience:

• Markers appear on-screen only during product timelines.

• Sidebar displays product name, description, brand, and a link to buy.

Conclusion

In this tutorial, we built a shoppable video demo using the Twelve Labs API. We walked through fetching videos, analyzing them with a custom prompt, saving metadata, and rendering product details in the UI.  

This simple flow shows how AI-powered video understanding can turn any video into an interactive shopping experience. Check out the GitHub repo to explore further and adapt it for your own projects!