JD.com Open-Sources Real-Time Video AI Model — Deployable on a Single RTX 4090

JD.com has open-sourced JoyAI-VL-Interaction, a real-time video AI model that can run on a single RTX 4090. Unlike traditional multimodal models that wait for user queries, this model continuously watches video streams and proactively decides when to speak — making it ideal for surveillance, elderly care, live translation, and smart glasses scenarios.

💡 What You Will Learn

JD.com has open-sourced JoyAI-VL-Interaction, a real-time video AI model that can run on a single RTX 4090. Unlike traditional multimodal models that wait for user queries, this model continuously wat

📜 Table of Contents

• Three Breakthroughs in Real-Time Video AI
• Benchmark Data
• Open Source Stack
• Deployment Guide: One RTX 4090 Is Enough
• Personal Take

Three Breakthroughs in Real-Time Video AI

1. Proactive judgment, not passive answering. Traditional models wait for user questions before processing frames. JoyAI-VL-Interaction continuously watches the video stream, making its own judgment every second: should I speak, stay silent, or escalate to the backend? Tell it "alert me when the referee shows a red card," and it watches silently until the moment happens — no need to ask "what just happened."

2. Real-time response, not post-hoc summarization. Standard video understanding records first, analyzes later. But for security alerts, live translation, cooking guidance — even a few seconds delay changes everything. JoyAI-VL-Interaction detects changes and responds in sub-second latency.

3. Agent delegation: foreground and backend work separately. The foreground model continuously watches the video and decides when to speak. When it needs code generation, tool calls, or complex reasoning, it delegates to the backend LLM. The foreground provides presence; the backend handles heavy lifting.

Benchmark Data

JD tested 58 real-world scenarios in blind evaluation against Doubao (ByteDance) and Google Gemini:

Notably, in surveillance alert scenarios, it scored 100% against both competitors. This makes sense — Doubao and Gemini are fundamentally turn-based products (you ask, they answer), while JoyAI-VL-Interaction's interactive capability is built into the model itself.

Open Source Stack

JD released the full technology stack: model weights + 4 million time-aligned interaction data points + training recipes + deployable system. Supports camera, livestream, and surveillance feeds. ASR, TTS, backend models, and visualization UI are all pluggable. Apache 2.0 license, vLLM-Omni day-0 support.

Deployment Guide: One RTX 4090 Is Enough

• GPU minimum: RTX 3090 (24GB), recommended: RTX 4090 (24GB)
• VRAM: ~10GB with FP8 quantization, ~17GB with BF16 full precision
• System RAM: 16GB minimum, 32GB+ recommended
• Storage: 20GB minimum, 50GB+ for multi-model
• OS: Linux (CUDA 12.x + driver 535+)

FP8 quantization is the key trick: vLLM supports online FP8, dropping main model VRAM from 16.8GB to 9.9GB (-41%). A single RTX 3090 (24GB) handles it easily — even an RTX 4070 (12GB) might work. Caveat: FP8 is ~19% slower than BF16 for single-stream inference due to dynamic scaling overhead.

Minimal deploy needs only 2 components (model inference + WebUI). Full deploy adds ASR, TTS, and backend Agent (recommended 3 GPUs).

Quick start:

git clone https://github.com/jd-opensource/JoyAI-VL-Interaction.git
cd JoyAI-VL-Interaction
# Install dependencies, then start
# Open https://127.0.0.1:8099 and connect a camera

Personal Take

JD quietly built a complete video AI product line in 4 months: March (JoyAI-LLM Flash), April (JoyAI-Image-Edit), June 3 (JoyAI-Echo), June 20 (JoyAI-VL-Interaction). JD has unique advantages: warehousing, logistics, livestreaming, customer service — the world's largest physical operations network generating real-time visual data that feeds the model's training loop.

This 8B model won't challenge Doubao or Gemini on every front. But the path is clear: build advantage in vertical scenarios through "presence," then use scenario data to feed back into model capability. For developers, this may be the most accessible real-time interaction framework available — no massive compute required, just a single RTX 4090 + vLLM.