HABench is a benchmark tool designed to systematically leverage MLLMs across all dimensions relevant to video generation assessment in generative models. By incorporating few-shot scoring and chain-of-query techniques, HA-Video-Bench provides a structured, scalable approach to generated video evaluation.

⭐Overview | 📒Leaderboard | 🤗HumanAlignment | 🛠️Installation | 🗃️Preparation | ⚡Instructions | 🚀Usage | 📭Citation | 📝Literature

• Overview
• Leaderboard
• Installation
• HumanAlignment
• Installation
  • Installation Requirements
  • Pip Installation
  • Download From Huggingface
• Praparation
• Instruction
• Usage
  • Standard Mode
  • Custom Mode
• Citation
• Literature

Notes:

• Higher scores indicate better performance.
• The best score in each dimension is highlighted in bold.

Notes:

• Higher scores indicate better performance.
• The best score in each dimension is highlighted in bold.

• Python >= 3.8
• OpenAI API access Update your OpenAI API keys in config.json:

  {
      "GPT4o_API_KEY": "your-api-key",
      "GPT4o_BASE_URL": "your-base-url",
      "GPT4o_mini_API_KEY": "your-mini-api-key",
      "GPT4o_mini_BASE_URL": "your-mini-base-url"
  }

• Install with pip

  pip install HAbench

• Install with git clone

  git clone https://github.com/yourusername/Video-Bench.git
  cd Video-Bench
  pip install -r requirements.txt

wget https://huggingface.co/Video-Bench/Video-Bench -O ./pytorch_model.bin

or

curl -L https://huggingface.co/Video-Bench/Video-Bench -o ./pytorch_model.bin

Please organize your data according to the following data structure:

# Data Structure
/Video-Bench/data/
├── color/                           # 'color' dimension videos
│   ├── cogvideox5b/
│   │   ├── A red bird_0.mp4
│   │   ├── A red bird_1.mp4
│   │   └── ...
│   ├── lavie/
│   │   ├── A red bird_0.mp4
│   │   ├── A red bird_1.mp4
│   │   └── ...
│   ├── pika/
│   │   └── ...
│   └── ...
│
├── object_class/                    # 'object_class' dimension videos
│   ├── cogvideox5b/
│   │   ├── A train_0.mp4
│   │   ├── A train_1.mp4
│   │   └── ...
│   ├── lavie/
│   │   └── ...
│   └── ...
│
├── scene/                           # 'scene' dimension videos
│   ├── cogvideox5b/
│   │   ├── Botanical garden_0.mp4
│   │   ├── Botanical garden_1.mp4
│   │   └── ...
│   └── ...
│
├── action/                          # 'action' 'temporal_consistency' 'motion_effects' dimension videos
│   ├── cogvideox5b/
│   │   ├── A person is marching_0.mp4
│   │   ├── A person is marching_1.mp4
│   │   └── ...
│   └── ...
│
└── video-text consistency/             # 'video-text consistency' 'imaging_quality' 'aesthetic_quality' dimension videos
    ├── cogvideox5b/
    │   ├── Close up of grapes on a rotating table._0.mp4
    │   └── ...
    ├── lavie/
    │   └── ...
    ├── pika/
    │   └── ...
    └── ...

Video-Bench provides comprehensive evaluation across multiple dimensions of video generation quality. Each dimension is assessed using a specific scoring scale to ensure accurate and meaningful evaluation.

Video-Bench supports two modes: standard mode and custom input mode. Video-Bench only supports assessments of the following dimensions: 'aesthetic_quality', 'imaging_quality','temporal_consistency', 'motion_effects','color', 'object_class', 'scene', 'action', 'video-text consistency'

This evaluation mode assesses videos generated by various video generation models using the prompt suite defined in our HAbench_full.json. Video-Bench supports evaluating videos produced by our preselected seven generation models as well as any additional models specified by the user.

To evaluate videos, simply specify the models to be tested in the --models parameter. For example, if you want to evaluate videos under modelname1 and modelname2, use the following commands with --models modelname1 modelname2

python evaluate.py \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --config_path ./config.json \
 --models modelname1 modelname2

or

HAbench \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --config_path ./config.json \
 --models modelname1 modelname2

This mode allows users to evaluate videos generated from prompts that are not included in the Video-Bench prompt suite.

You can provide prompts in two ways:

1. Single prompt: Use --prompt "your customized prompt" to specify a single prompt.
2. Multiple prompts: Create a JSON file and use --prompt_file $json_path. Create a JSON file containing your prompts and use --prompt_file $json_path to load them. The JSON file can follow this format:

{
    0: "prompt1",
    1: "prompt2",
    ...
}

python evaluate.py \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --mode custom_static \
 --config_path ./config.json \
 --models modelname1 modelname2

or

HAbench \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --mode custom_static \
 --config_path ./config.json \
 --models modelname1 modelname2

python evaluate.py \
 --dimension $DIMENSION \ 
 --videos_path ./data/ \
 --mode custom_nonstatic \
 --config_path ./config.json \
 --models modelname1 modelname2

or

HAbench \
 --dimension $DIMENSION \
 --videos_path ./data/ \
 --mode custom_nonstatic \
 --config_path ./config.json \
 --models modelname1 modelname2

If you use our dataset, code or find Video-Bench useful, please cite our paper in your work as:

@article{ni2023content,
  title={Video-Bench: Human Preference Aligned Video Generation Benchmark},
  author={Han, Hui and Li, Siyuan and Chen, Jiaqi and Yuan, Yiwen and Wu, Yuling and Leong, Chak Tou and Du, Hanwen and Fu, Junchen and Li, Youhua and Zhang, Jie and Zhang, Chi and Li, Li-jia and Ni, Yongxin},
  journal={arXiv preprint arXiv:xxx},
  year={2024}
}