Evaluating generative video models remains an open problem. Reference-based metrics such as Structural Similarity Index Measure (SSIM) and Peak Signal to Noise Ratio (PSNR) reward pixel fidelity over semantic correctness, while Fréchet Video Distance (FVD) favors distributional textures over physical plausibility. Binary Visual Question Answering (VQA) based benchmarks like VBench 2.0 are prone to yes-bias and rely on low-resolution auditors that miss temporal failures. Moreover, their prompts target a single dimension at a time, multiplying the number of videos required while still not guaranteeing reliable results.
WorldJen addresses these limitations directly. Binary VQA is replaced with Likert-scale questionnaires graded by a VLM that receives frames at native video resolution. Video generation costs are addressed by using adversarially curated prompts that are designed to exercise up to 16 quality dimensions simultaneously. The framework is built around two interlocking contributions. First, a blind human preference study is conducted, accumulating 2,696 pairwise annotations from 7 annotators with 100% pair coverage over 50 of the curated prompts × 6 state-of-the-art video models. A mean inter-annotator agreement of 66.9% is achieved and the study establishes a human ground-truth Bradley–Terry (BT) rating with a three-tier structure. Second, a VLM-as-judge evaluation engine using prompt-specific, dimension-specific Likert questionnaires (10 questions per dimension, 47,160 scored responses) judges the videos and reproduces the human-established three-tier BT rating structure independently. The VLM achieves a Spearman ρ̂ = 1.000, p = 0.0014 that is interpreted as tier agreement with the human results. Three focused ablation studies validate the robustness of the VLM evaluation framework.
How naturally and steadily does the video move?
Does the video obey the rules of our world?
Does the output follow what the prompt asked for?
How pleasing and well-crafted is the video?
Upload one generated MP4 and score it across WorldJen dimensions.
Playground is an experimental system simulating the paper for easy accessibility; stability and correctness issues are possible. If you encounter any issue please report to us at research@moonmath.ai.
| Dimension and model columns | Veo 3.1 Fast | Kling v2.6 Pro | Wan v2.2 A14B | LTX-2 | Hunyuan v1.5 | Wan 2.1 1.3B | Insights | |
|---|---|---|---|---|---|---|---|---|
| Motion & stability | Subject Consistency | 4.59 | 4.59 | 4.23 | 4.12 | 4.52 | 3.94 | |
| Scene Consistency | 4.82 | 4.73 | 4.34 | 4.65 | 4.61 | 4.53 | ||
| Motion Smoothness | 4.24 | 4.47 | 4.00 | 4.13 | 4.13 | 3.60 | ||
| Temporal Flickering | 4.74 | 4.64 | 4.35 | 4.54 | 4.61 | 4.21 | ||
| Inertial Consistency | 3.18 | 3.31 | 3.27 | 3.07 | 2.72 | 3.01 |
Physics remains the hardest gap
Lowest averages concentrate in the highlighted physics rows.
|
|
| Logic & physics | Physical Mechanics | 3.45 | 3.12 | 3.09 | 3.08 | 2.73 | 2.97 | |
| Object Permanence | 4.41 | 4.40 | 4.07 | 4.09 | 4.30 | 3.70 | ||
| Human Fidelity | 4.15 | 3.67 | 3.71 | 3.81 | 3.61 | 2.94 | ||
| Dynamic Degree | 4.22 | 4.16 | 4.13 | 4.25 | 3.91 | 3.79 | ||
| Instruction adherence | Semantic Adherence | 4.53 | 4.39 | 4.32 | 4.05 | 3.90 | 4.02 | |
| Spatial Relationship | 4.38 | 4.43 | 4.14 | 4.18 | 4.09 | 3.93 | ||
| Semantic Drift | 4.83 | 4.78 | 4.73 | 4.65 | 4.68 | 4.61 | ||
| Aesthetic quality | Composition & Framing | 4.68 | 4.66 | 4.50 | 4.62 | 4.58 | 4.31 |
Aesthetic dimensions cluster higher
Group D skews toward higher Likert bins across models.
|
| Lighting & Volumetric | 4.00 | 4.14 | 3.88 | 4.08 | 3.47 | 3.67 | ||
| Color Harmony | 4.86 | 4.84 | 4.75 | 4.73 | 4.70 | 4.59 | ||
| Structural Gestalt | 3.98 | 3.90 | 3.59 | 3.60 | 3.61 | 3.24 | ||
Each line is one model’s human BT, WorldJen BT, and VBench Quality rank. WorldJen matches humans (ρ̂ = 1.00, p = 0.0014); VBench Quality ρ̂ = 0.60 (p = 0.21, NS at n = 6).
Correlation with Human Eval: WorldJen (ρ = 1.00) vs VBench (ρ = 0.60)
Case study
@misc{inbasekar2026worldjen,
title = {{WorldJen}: An End-to-End Multi-Dimensional Benchmark for Generative Video Models},
author = {Inbasekar, Karthik and Rom, Guy and Shlomovits, Omer},
year = {2026},
howpublished = {\url{https://worldjen.moonmath.ai}}
}