Qencode and TwelveLabs: Bringing Video Intelligence Into the Media Pipeline

1. Introduction

Video infrastructure has historically focused on making video playable, portable, and cost efficient. That work is still essential. Teams need to ingest files, transcode them into the right formats, compress them without sacrificing quality, deliver them globally, and make them available across players, devices, storage systems, and content management workflows.

But once a video is processed, a harder question remains: what is actually inside it?

For many content operations teams, answering that question still depends on people. Editors scrub through footage to find the right moment. Trust and safety teams review content manually. Media teams create tags, summaries, and metadata by hand. Developers build separate workflows for transcripts, object detection, search, classification, and policy enforcement. The result is a pipeline that can move video efficiently, but cannot fully understand it.

That gap matters because modern video workflows are no longer just about delivery. They are about discovery, automation, personalization, compliance, and reuse. A video file should not only be encoded for playback. It should also be transformed into structured intelligence that downstream systems can act on.

That is where Qencode and TwelveLabs come together.

• Qencode provides the video infrastructure layer: transcoding, compression, storage, delivery, live streaming, player workflows, and other services teams need to process video at scale. 

• TwelveLabs provides the video understanding layer: models that can interpret visual content, speech, sound, text, actions, objects, scenes, and context across time.

Together, the integration turns a standard media processing job into an automated content operations workflow. As a video moves through Qencode, developers can add TwelveLabs-powered intelligence outputs such as descriptions, categories, moderation results, custom prompt responses, and natural language search indexes. The output is not just another playable rendition. It is machine-readable context that can feed CMS metadata, review queues, search experiences, highlight tools, recommendation systems, ad workflows, and internal AI applications.

For technical teams, the value is simple: keep the video pipeline you already use, then add understanding as a configurable output.

2. About Qencode

Qencode is a full stack video platform that gives teams everything they need to build and scale amazing video experiences. The platform covers the entire pipeline: transcoding, live streaming, storage, content delivery, player, analytics, automatic subtitles, video intelligence, smart thumbnails, smart video cropping, forensic watermarking, AI-powered video detection, and more.

At the core is AI optimization that works with existing codecs, including AV1. It reduces file sizes by 60% without quality loss, produces 3x faster load times and lowers both storage and CDN spend in the same workflow.

Video Intelligence is the layer that sits between processing and downstream applications. It is where Qencode integrates with TwelveLabs, so any file moving through the Qencode pipeline can come out the other side with structured understanding attached.

3. About TwelveLabs

TwelveLabs is a video understanding platform built to help developers and enterprises make video content searchable, analyzable, and actionable. Instead of treating video as a flat media file or relying only on metadata, transcripts, or image frames, TwelveLabs models analyze video across multiple modalities, including visuals, speech, sounds, on-screen text, objects, actions, and temporal context.

The platform exposes these capabilities through APIs that map to common developer needs:

1. Search lets users find exact moments in video using natural language or image-based queries. A developer can search for scenes like “a person opening a red car door,” “crowd cheering after a goal,” or “the moment someone mentions the refund policy” and receive timestamped results.

2. Analyze generates text from video. This includes summaries, chapters, highlights, titles, topics, hashtags, answers to questions, and custom outputs based on prompts. This is useful for metadata generation, editorial workflows, accessibility, content review, and downstream LLM applications.

3. Embed creates multimodal embeddings that represent the meaning of video, audio, image, and text inputs. These embeddings can power semantic search, recommendations, clustering, classification, retrieval workflows, and custom AI applications.

Two TwelveLabs model families power the Qencode integration:

1. Pegasus is the generative video understanding model. It is designed for video-to-text tasks such as descriptions, summaries, question answering, custom extraction, and prompt-driven analysis. In Qencode, Pegasus powers outputs such as descriptions and Custom mode, where users can provide natural language instructions for the model.

2. Marengo is the multimodal search and embedding model. It is designed to make video searchable across visual, audio, image, and text signals. In Qencode, Marengo powers Search mode, which lets users query processed videos and retrieve timestamped moments ranked by relevance.

The key advantage for developers is that TwelveLabs abstracts away much of the complexity of building a video intelligence stack from scratch. Instead of stitching together separate speech-to-text, OCR, object detection, classification, moderation, vector search, and prompt orchestration systems, teams can use a video-first model layer built specifically for understanding what happens inside video over time.

Paired with Qencode, these capabilities become available directly inside the media processing pipeline. That means video intelligence is not a separate offline workflow. It becomes part of the same operational path that already handles source files, transcoding, compression, storage, and delivery.

4. The Integration: How It Works

The TwelveLabs Video Intelligence integration is built directly into both the Qencode UI and the Qencode API as another output format on a transcoding job.

In the API, you add a video_intelligence format to your start_encode2 call and pick a mode. That is the whole pattern:

{
  "query": {
    "encoder_version": "2",
    "format": [
      {
        "output": "video_intelligence",
        "mode": "description"
      }
    ],
    "source": "https://your-bucket/source_video.mp4"
  }
}

One job replaces two pipelines. The same start_encode2 call that transcodes the file also runs the intelligence job. Source URL, destination, auth, and job state are shared. You skip the work of triggering a second vendor after transcode completes, reconciling failures across two systems, and stitching outputs back together in your own code.

In the Qencode portal, the same options live on the Transcode Media page. Select Video Intelligence as your output format, pick a mode, and submit.

Five modes are available:

1 - Description

Generates a written summary of the video. Use it for catalog enrichment, accessibility metadata, or feeding downstream LLM workflows that need context about a clip without re-watching it. Returns a single description string.

2 - Categorization

Returns a list of tags based on video content. Ships with 47 default categories covering Comedy, Music, Sports, News, Education, How-To, Technology, Travel, and more. Pass a categories array to override the defaults with your own taxonomy, useful when you need tags that match your existing CMS schema.

3 - Content Moderation

Returns a boolean violates flag and a list of reasons. Defaults cover 23 violation types including Graphic Violence, Hate Speech, Adult Nudity, Misinformation, and Scam. Pass a violation_reasons array to scope moderation to your platform's policy language.

4 - Custom (Pegasus)

A prompt-driven mode for use cases the predefined outputs do not cover. Pass any natural language prompt and get a generated answer back. Pulled from the docs:

{
  "prompt": "Extract the license plate number on the car."
}

Response: "The license plate number on the red car is AM-84865."

This removes the friction from customers having to decide which model maps to which use case, learn each model's parameters, and brainstorm and develop specific prompts. Collapsing this into five specific modes allows the solution to map cleanly onto the things M&E teams actually do. All of this is enabled without requiring customers to trade flexibility for simplicity. Categories can be overridden with a custom categories array, moderation can be scoped with a custom violation_reasons list, and the Custom mode exposes raw Pegasus prompts for anything the predefined modes don't cover.

Useful for sports highlight detection, ad break placement, brand safety checks against your own guidelines, or anything you would normally write a custom classifier for.

5 - Search (Marengo)

Indexes the video so you can query it later in natural language or with an image. Returns timestamped clip ranges with relevance ranks. Configure what part of the video to search through search_options:

• visual: visual content

• audio: non-speech audio

• transcription: spoken audio only

Use search_rank_threshold to cap how many results come back. Pass prompt for a text query, media_prompt for an image URL (JPEG or PNG, 128x128 minimum, up to 5 MB).

Outputs land as JSON in whatever destination you specify, alongside your transcoded video. Set result_name to control the filename, and destination.url to control the path. By default each mode writes to its own file: description.json, categorization.json, moderation.json, custom.json, search.json.

Upload limits vary by mode. Description, Categorization, Moderation, and Custom accept files up to 2 GB and 2 hours. Search accepts up to 4 GB and 4 hours. All modes support FFmpeg-compatible formats between 360x360 and 5184x2160. Source URLs for Video Intelligence jobs must be HTTP or HTTPS.

Because the file is already moving through Qencode, intelligence runs on content that is already part of your storage, CDN, and player workflow. There is a caveat for combined jobs. If your job includes both video intelligence and other Qencode processing, the source gets downloaded twice, once by Qencode and once by TwelveLabs. Add multiple video intelligence modes and the egress multiplies, since each mode runs as a separate TwelveLabs job internally. Optimization is planned for a future release.

Demo: https://vimeo.com/1181227821/53185ca75e

5. What You Can Build

Once video understanding becomes part of the processing pipeline, developers can build content operations workflows that are difficult to scale manually. The Qencode and TwelveLabs integration is especially useful when teams need to process large video libraries, enrich media assets, reduce manual review, or make video searchable for internal and external applications.

Here are several patterns developers can build with the integration.

Automated metadata enrichment

Use Description and Categorization modes to generate useful metadata as soon as a file is processed. A media team can attach descriptions, categories, and keywords to each asset without waiting for manual tagging. A CMS can use that output to improve discovery, organize libraries, populate asset pages, or route content into the right collection.

For example, a sports archive could automatically tag content by sport, event type, player activity, or highlight category. An e-learning platform could classify lessons by topic, subject, difficulty, or format. A marketing team could generate short descriptions for campaign videos before they are published.

Brand safety and policy review

Use Content Moderation mode to evaluate whether a video violates a set of content guidelines. The default policy categories can support common safety checks, while custom violation reasons let teams align the model with their own trust and safety rules.

This can turn moderation from a fully manual workflow into a triage system. Safe content can move forward automatically, risky content can be routed to human reviewers, and violations can be logged with structured reasons. For platforms handling user-generated content, ads, creator submissions, or partner media, this provides a practical way to add intelligence before publication or distribution.

Custom extraction with Pegasus

Custom mode is useful when predefined outputs are not specific enough. Developers can pass a prompt and ask Pegasus to extract, summarize, transform, or reason over the video.

This opens the door to workflows such as:

• Extracting product mentions from a livestream

• Identifying ad break opportunities in long-form content

• Generating social copy from a finished video

• Pulling key moments from interviews or webinars

• Detecting whether required visual elements appear in a compliance recording

• Creating structured summaries for internal review

The important shift is that a developer can express the task in natural language rather than building a bespoke computer vision pipeline for each use case.

Searchable video libraries with Marengo

Search mode makes processed videos queryable. Instead of relying only on filenames, tags, or transcripts, teams can search across the actual content of the video and retrieve timestamped segments.

A developer could build a search experience where users ask for “shots of a city skyline at night,” “clips where the speaker discusses pricing,” or “moments with applause after a product announcement.” The result is a set of relevant time ranges that can be previewed, clipped, reviewed, or fed into another application.

This is useful for media archives, sports footage, training libraries, support recordings, entertainment catalogs, and any workflow where the value is hidden inside long-form video.

Content operations agents

The integration also creates a foundation for AI agents that operate on video libraries. Once Qencode processes the media and TwelveLabs produces structured outputs, an application can combine those outputs with business rules, LLMs, vector databases, CMS systems, or review tools.

For example, a content operations agent could:

1. Detect the category of a new upload

2. Generate a description and suggested metadata

3. Check the video against moderation rules

4. Index the video for natural language search

5. Route the asset to the right team or publishing workflow

6. Create a review task only when the output requires human attention

This turns video processing from a passive infrastructure step into an active automation layer.

The broader takeaway is that Qencode handles the media pipeline and TwelveLabs adds the intelligence layer. Developers do not need to build a separate video AI stack, move assets through disconnected systems, or manually coordinate model outputs with media processing jobs. They can start with one familiar transcoding workflow, add video understanding as an output, and build from there.

6. Getting Started

The integration is live on the Qencode platform today. Full tutorial documentation is here: docs.qencode.com/tutorials/transcoding/video-intelligence

1. Sign up at cloud.qencode.com or sign into your existing account

2. Submit a transcoding job with video_intelligence as the output format

3. Pick your mode and any optional parameters

4. Pull the JSON output from your destination once the job completes

That JSON can then power whatever comes next: metadata fields in a CMS, a moderation decision in a review queue, a search result in a video portal, a highlight candidate in an editing tool, or context for an LLM-based application.

Going from "we want video understanding" to a working job is one format block on a transcoding call you already make. The alternative is a multi-week project: model evaluation, taxonomy design, prompt engineering, output schema definition, pipeline orchestration, storage handoff, error handling, monitoring. Qencode collapses that into a configuration change.

If you are building content operations on top of video, the combination of Qencode processing and TwelveLabs understanding gives you one pipeline from raw upload to structured, searchable, moderated output. Start a job today.