EgoProceVQA

🔔News

[2026.7]: Paper accepted to ACM MM 2026.

Abstract

Most daily activities are inherently procedural. However, existing evaluations for egocentric video understanding seldom address procedural understanding and largely overlook complex key-step-level reasoning under the widely used video question answering (VQA) paradigm for MLLMs. Such capabilities are crucial for building procedural AI assistants deployable on wearable devices. To bridge this gap, we introduce the Egocentric Procedural Understanding VQA task (EgoProceVQA), which systematically evaluates egocentric procedural reasoning abilities of current MLLMs and agents through six types of key-step-centric questions. Furthermore, we develop EgoProceGen, a data generation platform that efficiently constructs QA data tailored to different question types. Based on this platform, we build a benchmark with 3,600 questions, four common procedural scenarios, and 31 everyday procedural tasks. Evaluations on EgoProceVQA show that existing MLLMs and agents still have substantial room for improvement in procedural understanding. Therefore, we further propose EgoProceAgent, a self-skill-exploration agentic framework. We design a generic tool library for procedural understanding and a standardized sub-skill library shared across tools and models, enabling self-exploration without ground-truth supervision. By exploring how to compose and select sub-skills, the agent discovers effective skill strategies for diverse problems, and attains state-of-the-art performance among open-source models on multiple tasks. Together, our benchmark, generation platform, and agentic framework establish a unified foundation for EgoProceVQA. We will release our benchmark data and source code.

Figure 1: Overview of EgoProceVQA. (a) demonstrates that we uniquely introduce a benchmark for key-step-level VQA across six question types. (b) shows that EgoProceAgent autonomously explores and constructs its own skills, and subsequently leverages these skills for tool invocation and reasoning. (c) shows our outstanding performance.

EgoProceVQA

Figure 4: Statistics of the six task types and four scenarios in EgoProceVQA.

Figure 5: Statistics of 31 procedural task types in EgoProceVQA.

Six Key-Step-Centric Task Types

EgoProceVQA decomposes procedural understanding into six progressively structured, key-step-centric dimensions that move beyond generic perception and probe whether a model can reason about critical procedural units under increasing difficulty:

Task 1 — Key Step Recognition (KSR). Input: a single clip. Question: "What is the key action performed in this clip?" Answer: one of four options (A–D), the correct choice being the ground-truth key step description. This task provides the perceptual grounding foundation upon which all higher-order reasoning rests.
Task 2 — Procedural Sequence Reasoning (PSR). Input: a clip plus the full procedure context P. Question: "Given the shown clip as the current step, what are the immediately preceding and following steps?" Answer: one of four combined (prev, next) option pairs. This task examines commonsense causal chains, short-horizon temporal prediction, and text–visual alignment.
Task 3 — Key Step Grounding (KSG). Input: a composite clip spanning 3–5 consecutive key steps, accompanied by a numbered step list. Question: "At what time interval does key step Z occur?" Answer: an open time interval (t_s, t_e) in seconds. This task evaluates key-step identification and temporal grounding in an open-ended setting.
Task 4 — Key Step Ordering (KSO). Input: three clips in randomised order, plus procedure context P. Question: "Arrange the clips into the correct chronological order according to the procedure context." Answer: one of four permutation options. This requires inferring the ordering of relevant key steps within a segment based on the procedure text.
Task 5 — Key Step Missing (KSM). Input: two clips representing steps 1 and 3 of a three-step sequence. Question: "Which step is missing from the provided video?" Answer: one of four options; the correct answer is step 2's description. The model must infer the gap from visual content and procedural context alone.
Task 6 — Task Completeness (TC). Input: a single clip and the task name. Question: "Is the task completed after this step?" Answer: binary choice (Completed / Not completed). This mirrors the completion judgment required by proactive assistants.

Figure 2: Overview of EgoProceVQA. For each task type, we present a visualized example to facilitate understanding. At the center of the figure, we show a complete key-step sequence for preparing coffee, around which all examples except KSR are constructed. Our task design remains tightly focused on the key-step sequence of the procedural task.

Data Construction

We collect egocentric procedural videos from four real-world activity domains: CaptainCook4D (24 kitchen recipes), EPIC-Tent (outdoor tent setup), Assembly101 (indoor toy car assembly), and EgoOops (tabletop crafting). Proportional random sampling based on the original scale of each dataset yields four scenarios: Cooking (1,800 QA pairs, 632 clips), Outdoor Tent Assembly (600 QA pairs, 124 clips), Toy Car Assembly (600 QA pairs, 341 clips), and Handicraft Activities (600 QA pairs, 175 clips).

EgoProceGen: Automatic Data Generation Platform

EgoProceGen integrates a dual-pipeline generation mechanism to produce task instructions and associated QA pairs across all six evaluation dimensions:

LLM-Assisted Semantic Generation (Tasks 1, 2). An LLM-in-the-loop mechanism generates semantically confounded distractors according to three confusion strategies: action confusion (same object, different action), object confusion (same action, different object), and temporal confusion (plausible steps drawn from adjacent positions in the canonical procedure). To prevent hallucination, metadata fields are forcibly rewritten after parsing the model's JSON output, and all generated items are reviewed by three human annotators.
Rule-Based Structural Generation (Tasks 3–6). Tasks whose ground truth can be deterministically derived from temporal metadata are handled by a fully deterministic pipeline that requires no LLM involvement, ensuring complete reproducibility and eliminating model-induced variance.

Evaluation Metrics

EgoProceVQA adopts a closed-set multiple-choice format for five of the six tasks, enabling deterministic, reproducible evaluation without human or LLM judges. Each question targets a precisely defined procedural event with semantically proximate distractors, preventing models from exploiting surface-level cues. We choose Accuracy for multiple-choice tasks (1, 2, 4, 5, 6) and tIoU for the temporal grounding task (3), ensuring fair and objective evaluation.

Key Contributions

EgoProceVQA: a new task for systematically evaluating egocentric procedural understanding in MLLMs and agents through six progressively structured, key-step-centric subtasks.
EgoProceGen: an automatic data generation platform that enables scalable, controllable, and transferable construction of procedural QA across diverse activity scenarios.
Large-scale benchmark: 3,600 egocentric video clips and associated QA-pairs across four daily scenarios and 31 task types, together with an evaluation protocol for fine-grained diagnosis of procedural reasoning abilities.
EgoProceAgent: a strong training-free agentic baseline that performs self-skill-exploration, demonstrating that current systems still face substantial challenges in key-step-level procedural understanding while attaining state-of-the-art open-source performance on multiple tasks.

Self-Skill-Exploration Agent

Procedural understanding often requires different reasoning strategies for different question types, much like humans selectively apply different problem-solving skills depending on the task. EgoProceAgent is a training-free agentic framework that performs self-skill-exploration over a procedural-understanding tool library. It runs in two separated phases:

Phase I — Self-Skill-Exploration (Offline): a powerful LLM functions as the central "brain". It first categorizes all questions and their option types into sub-types, then performs self-skill-exploration of the optimal skill strategy over all questions, and ultimately distills a generalizable skill corresponding to each sub-type.
Phase II — Skill Execution (Online): each incoming question is classified into its sub-type, and the corresponding distilled skill strategy is executed by the video-LLM M_exec, sequentially invoking the required sub-skills and tools. If the primary strategy yields no answer, the fallback strategy is automatically attempted.

Figure 3: Overview workflow of EgoProceAgent. Skill1(·) represents the answers of Skill 1. Based on the tool usage defined for each sub-skill, our self-skill-exploration proceeds in three main stages: it first learns to solve "easy questions" with generated skill strategies, then leverages these solutions as references to tackle "hard questions", and finally distills the optimal strategy for this class of problems.

Tool Library

EgoProceAgent maintains a lazily-initialized shared tool library of four specialized tools:

Grounding DINO: open-vocabulary object detection conditioned on text queries.
CLIP: a shared image–text embedding space enabling zero-shot visual-semantic matching, used for option scoring and clip boundary detection.
Web Search WikiHow: Web Knowledge Retrieval that queries WikiHow for structured step-by-step procedure descriptions.
Temporal Segmentation: generates structured time-reference anchors based on the question.

Sub-Skill Decomposition

We decompose the tool calling into 12 atomic sub-skills (tool functions) organized in a three-layer architecture. A skill strategy π = (s₁, s₂, …, s_L) is an ordered sequence of sub-skill IDs, where each sub-skill's output feeds into subsequent sub-skills via a shared context dictionary.

Input Layer (I): extracts structured signals from the question and options.
- I₁ — Option Discriminative Analysis
- I₂ — Procedural Knowledge Retrieval
- I₃ — Question Structure Parsing
Perception Layer (P): processes video frames using external tools.
- P₁ — CLIP Video Segmentation
- P₂ — CLIP Text-Visual Scoring
- P₃ — GroundingDINO Object Detection
- P₄ — Temporal Reference Generation
Decision Layer (D): produces the final answer.
- D₁ — VLM Step Identification
- D₂ — Evidence-Augmented VLM Answer
- D₃ — Text Reasoning
- D₄ — Fuzzy Text Matching
- D₅ — Set Elimination

Four-Pass Self-Skill-Exploration

We determine the optimal strategy for each question without ground-truth labels via a consistency-based four-pass self-exploration protocol:

Pass 0 — Sub-Type Classification. Questions are batched and assigned concise categorical labels by the planning model; near-duplicate labels are merged via edit-distance matching.
Pass 1 — Dual-Strategy Exploration. For each discovered type, two candidate strategies are proposed under type-specific constraints and executed by M_exec. Samples where the two answers agree are marked as "<easy question>"; disagreements become "<hard question>".
Pass 2 — Reference-Guided Consolidation. Hard questions are re-planned using up to 5 easy-question reference strategies from the same discovered type.
Pass 3 — Strategy Distillation. A primary and secondary strategy pair per type is distilled via frequency-based voting followed by LLM verification, yielding the final strategy card.

Main Results on EgoProceVQA

Comparison on EgoProceVQA. Avg represents the average of all accuracy metrics except KSG. ≥20s / <20s denote samples' accuracy (excl. KSG) with duration ≥ / < 20 seconds. ≥20s(G) / <20s(G) denote samples' tIoU (KSG) with the same duration split. Red marks the best 7B/8B model under 8-frame input; Purple marks the best 7B/8B model under 16-frame input.

Model	Frame	KSR	PSR	KSG	KSO	KSM	TC	≥20s (G)	<20s (G)	≥20s	<20s	Avg
Random	-	25.0	25.0	-	25.0	25.0	50.0	-	-	-	-	30.0
Human	-	91.0	91.0	0.85	93.0	96.0	90.0	-	-	-	-	92.2
Closed-source
GPT-4o	8	59.3	44.3	0.27	25.7	52.3	51.5	0.24	0.36	46.3	47.2	46.6
GPT-5.1	8	66.5	47.5	0.34	26.5	55.2	62.5	0.32	0.39	53.3	49.0	51.6
Gemini-3-Flash	8	62.7	41.2	0.34	24.3	69.7	58.8	0.32	0.42	49.8	53.8	51.3
Qwen3.5-Plus (Commercial)	8	76.0	86.3	0.32	38.2	59.5	58.5	0.40	0.27	67.4	57.8	63.7
Qwen3.5-Plus (Commercial)	16	76.5	89.2	0.28	37.5	63.0	59.0	0.37	0.23	70.6	56.2	65.0
Open-source
LLaVA-OneVision	8	52.5	39.2	0.10	26.8	11.2	52.3	0.07	0.18	36.3	36.6	36.4
LLaVA-OneVision	16	52.0	39.0	0.10	26.2	9.2	52.7	0.07	0.18	35.8	35.8	35.8
Vinci	8	41.2	40.0	0.20	27.2	23.0	52.5	0.18	0.26	35.8	38.3	36.8
Vinci	16	41.7	38.5	0.20	25.0	23.0	52.5	0.18	0.26	35.5	37.2	36.1
EgoGPT	8	49.5	38.3	0.10	26.7	4.3	53.7	0.07	0.17	33.9	35.4	34.5
EgoGPT	16	49.5	38.3	0.08	26.5	5.2	52.2	0.05	0.15	33.9	35.0	34.3
Video-LLaMA2	8	34.2	40.5	0.17	26.8	42.8	52.5	0.14	0.24	38.6	40.6	39.4
Video-LLaMA2	16	34.3	40.7	0.16	25.7	41.7	52.5	0.13	0.25	38.9	39.1	39.0
Video-LLaVA	8	29.5	27.7	0.15	21.3	23.8	49.8	0.12	0.24	27.8	34.6	30.4
Qwen2-VL-7B	8	55.2	43.8	0.22	24.8	38.7	52.5	0.20	0.26	41.1	46.0	43.0
Qwen2-VL-7B	16	57.0	43.0	0.22	24.3	42.0	53.0	0.20	0.27	41.7	47.2	43.9
Qwen2.5-VL-7B	8	44.3	39.5	0.20	22.2	42.2	53.5	0.18	0.25	39.1	42.2	40.3
Qwen2.5-VL-7B	16	51.2	40.2	0.18	24.0	45.8	53.3	0.15	0.24	42.6	43.5	42.9
Qwen3-VL-8B	8	59.0	45.7	0.24	27.2	25.3	52.5	0.20	0.34	42.1	41.7	41.9
Qwen3-VL-8B	16	55.0	47.8	0.24	27.8	24.7	52.8	0.20	0.34	41.4	42.1	41.6
InternVL3-38B	8	54.2	47.6	0.15	28.7	51.5	61.4	0.37	0.25	66.1	57.0	48.7
InternVL3-38B	16	62.8	49.3	0.13	26.3	56.3	61.8	0.39	0.25	69.5	56.4	51.3
Ours (Qwen2.5-VL-7B)	8	49.2 ↑11.1%	72.8 ↑84.3%	0.29 ↑45.0%	27.8 ↑25.2%	43.2 ↑2.4%	53.2	0.27	0.37	52.0	44.9	49.2 ↑22.1%
Ours (Qwen2.5-VL-7B)	16	54.5 ↑6.4%	81.2 ↑102.0%	0.29 ↑61.1%	27.8 ↑15.8%	48.2 ↑5.2%	53.3	0.27	0.36	55.5	49.0	53.0 ↑23.5%

Table 1: Random means results of randomly selected. Red = best 7B/8B model in 8-frame input; Purple = best 7B/8B model in 16-frame input.

Generalization: EgoProceL

Generalization evaluation on EgoProceL (PC Assembly & PC Disassembly). Our method performs self-skill-exploration over the defined question types under a QA framework, outperforming several visual clustering baselines.

Method	PC Assembly F1	PC Assembly IoU	PC Disassembly F1	PC Disassembly IoU
Random	15.1	7.2	15.3	7.1
Uniform	17.4	8.9	18.1	9.1
CnC [2]	25.1	12.8	27.0	14.8
GPL-2D [1]	24.0	12.6	27.4	15.9
UG-I3D [1]	22.0	11.7	24.2	13.8
GPL-w BG [1]	27.6	14.4	26.9	15.0
GPL-w/o BG [1]	27.5	15.2	26.7	15.2
OPEL [8]	33.7	17.9	32.2	16.9
Ours	39.2	21.8	40.5	19.2

Table 2: Performances on EgoProceL under 8-frame input.

Ablation Study (8-frame)

Ablation experiments validating the contributions of sub-skill configuration and skill exploration, compared against three levels of Chain-of-Thought (CoT) prompting.

Variant	KSR	PSR	KSG	KSO	KSM	TC
Baseline	44.3	39.5	0.20	22.2	42.2	53.5
CoT Level 1	31.8	26.8	0.06	20.7	27.2	50.5
CoT Level 2	27.8	26.2	0.18	21.3	26.5	49.0
CoT Level 3	42.8	31.2	0.20	21.7	29.0	51.0
w/o Sub-Skill	46.0	33.7	0.29	22.5	38.7	53.2
w/o Skill-Exploration	29.3	29.8	0.05	16.7	39.3	39.0
Ours	49.2	72.8	0.29	27.8	43.2	53.2

Table 3: Results of ablation experiment under 8-frame input.

Benchmark Comparison

Benchmark	Year	Scale	Mod.	View	Step Ann.	Temp. Order	Task Comp.	Crit. Step	Multi-task	Primary Task
AssistQ [39]	2022	100 vid, 531 QA	V+T	E	×	×	×	×	×	Instruction-following VQA
EgoTaskQA [17]	2022	2K vid, 40K QA	V+T	E	✓	×	×	×	×	Goal & State QA
EgoSchema [27]	2023	250 hrs, 5063 QA	V+T	E	×	×	×	×	×	Long-form VQA
EgoPlan-Bench [4]	2023	—, 4939 QA	V+T	E	✓	×	×	×	×	Procedural Planning
Ego4D Goal-Step [28]	2023	430 hrs, 48K seg	V+T	E	✓	×	×	×	×	Step Prediction
EgoThink [6]	2024	750 QA, 12 tasks	I+T	E	×	×	×	×	✓	Egocentric General Eval
OpenEQA [26]	2024	180+ env, 1636 QA	V+T	E	×	×	×	×	✓	Embodied QA
MM-Ego [40]	2024	7M QA	V+T	E	×	×	×	×	✓	Egocentric General Eval
VidEgoThink [5]	2024	195 vid, 600 QA	V+T	E	×	×	×	×	✓	Egocentric General Eval
EgoTextVQA [45]	2025	1.5K vid, 7K QA	V+T	E	×	×	×	×	×	Scene-text VQA
ProMQA [14]	2025	384 vid, 401 QA	V+T	E	✓	×	×	×	×	LLM-judge open-ended QA
EgoProceVQA (Ours)	2026	3600 QA (6×600)	V+T	E	✓	✓	✓	✓	✓	Key-step-level Procedural VQA

Table 4: Comparison of egocentric video benchmarks. Step Ann. = step-level annotation; Temp. Order = temporal step-order reasoning; Task Comp. = task completeness understanding; Crit. Step = critical step identification; Multi-task = multiple distinct reasoning task types. V = video; I = image; T = text; E = egocentric. Bold ✓ marks capabilities unique to EgoProceVQA.

Benchmark Statistics

Scenario	Source Dataset	# Video Clips	# QA Pairs
Cooking	CaptainCook4D [29]	632	1,800
Outdoor Tent Assembly	EPIC-Tent [9]	124	600
Toy Car Assembly	Assembly101 [34]	341	600
Handicraft Activities	EgoOops [12]	175	600
Total (4 scenarios, 31 tasks)	—	1,272	3,600

Table 5: Statistics of EgoProceVQA across four procedural scenarios and 31 task types.

Visual Analysis

Figure 6: Visual analysis for each type of evaluation task. Here, skills come from the self-exploration outcomes.

Record Setting

Figure 7: Overview of Real-world Demonstration. (a) shows the data collection device Meta Quest 3. (b) shows the data collection process. (c) shows the three simple procedural tasks we designed. (d) shows our platform built on Gradio.

BibTeX


@misc{li2026egoprocevqanovelegocentricprocedural,
      title={EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent}, 
      author={Junlong Li and Junxi Li and Yuxiang Yang and Wenbin Zou and Lap-Pui Chau and Yi Wang},
      year={2026},
      eprint={2607.13792},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.13792}, 
}

Acknowledgments

The research work described in this paper was conducted in the JC STEM Lab of Machine Learning and Computer Vision funded by The Hong Kong Jockey Club Charities Trust. This research received partially support from the Global STEM Professorship Scheme from the Hong Kong Special Administrative Region.

EgoProceVQA

A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent

🔔News

Abstract

EgoProceVQA Benchmark

Six Key-Step-Centric Task Types

Data Construction

EgoProceGen: Automatic Data Generation Platform

Evaluation Metrics

Key Contributions

EgoProceAgent Framework

Self-Skill-Exploration Agent

Tool Library

Sub-Skill Decomposition

Four-Pass Self-Skill-Exploration

Leaderboard

Main Results on EgoProceVQA

Generalization: EgoProceL

Ablation Study (8-frame)

Benchmark Comparison

Benchmark Statistics

Visual Analysis

Demo Video

Record Setting

BibTeX

Acknowledgments