Project 11 · Local VLM evaluation

Industrial video timeline extraction

I tested two local Qwen3-VL models on a 70-second assembly clip, then compared their timestamped event logs with human annotations.

Evidence: 70 frames at 1 FPS
Hardware: RTX 3090 Ti, 24 GB
Evaluation: Exact label + tIoU ≥ 0.30
Interface: Vanilla JavaScript

A companion LinkedIn write-up will be linked here after final review.

Interactive comparison

Review the timeline

Select a run, play the clip, or click an event to inspect its timing and evidence.

At this frame

Human: none Qwen: none

Human reference vs. model prediction

Timeline

Matched Missed or mislocalized

Video position

00:00.0 01:10.0

SELECT
AN EVENT

Inspection

Click any timeline block to seek the video.

The selected event’s timing, status, evidence, and temporal IoU appear here.

tIoUn/a

Current analysis

Results

The strict score measures distinct, correctly localized instances. Visual review shows stronger action recognition than that score alone suggests.

Strict · 4B BF16 2 / 7 matched

F1 0.36 · mean tIoU 0.41 · median boundary error 4.26 s

Strict · 8B INT8 3 / 7 matched

F1 0.55 · mean tIoU 0.51 · median boundary error 2.27 s

Independent coverage 3 / 7 · 4 / 7

4B and 8B coverage when one prediction may overlap multiple reference intervals.

Action types · both runs 4 / 5 recognized

Both found camera setup, gloves, tool pickup, and fastening. Both missed the connection label.

Recognition was better than segmentation

Both models recovered 80% of the action categories, but they were much less reliable at separating repeated events and matching human boundaries. The useful result is a coarse workflow timeline, not a precise event log.

Several misses need context

Camera setup was semantically plausible but too short or invalid. The 4B tool pickup aligned with visible handling within roughly one sampled frame of the annotation. These are temporal disagreements, not unsupported labels.

Fastening exposed the scoring tradeoff

Each model returned one 48–69 s fastening phase. It overlaps both human intervals at tIoU 0.430 and 0.302, but one-to-one matching can credit only one. That is useful for indexing, but insufficient for counting operations.

The connection event was a real miss

Neither model selected “connect leg and feet.” The footage shows metal parts being joined, but the dataset label depends on product-specific object names. This is both a model miss and an ontology challenge.

Experiment notes

Method and limits

Evaluation

I sampled 70 frames at 1 FPS, supplied a fixed action vocabulary, and required structured JSON intervals. Predictions match one-to-one only when the labels agree and temporal IoU reaches 0.30.

Two useful views

Strict matching asks whether each event was recovered as a separate instance. Independent coverage asks whether relevant activity appeared in the right part of the video. The second view adds context without replacing the reproducible strict score.

Limits of the comparison

Human boundaries and class names involve judgment, while 1 FPS sampling cannot support millisecond precision. Model confidence values are not calibrated. Model size and weight precision also changed together, so the 4B-versus-8B result is not a precision-controlled benchmark.

Practical use

The current system can index long recordings, draft timelines, retrieve evidence, and accelerate annotation with a person reviewing the output. It is not suitable by itself for safety enforcement, closed-loop control, compliance evidence, or exact operation counting.

Why local processing

Industrial video can contain proprietary processes, workers, and restricted facilities. Local analysis keeps raw footage near its source and can export only structured events or reviewed evidence. This test ran offline on an RTX 3090 Ti; it was not an embedded-edge or real-time deployment.

Video source

Processed 70-second excerpt from the IndEgo dataset by Fraunhofer IPK, licensed under CC BY 4.0. The source clip was trimmed to the annotated interval and re-encoded for this case study.

Reference

Glossary

VLM: A vision-language model that reasons across images or video and text.
Qwen3-VL: Alibaba’s open-weight vision-language model family used for both runs.
4B / 8B: Approximate parameter counts: four billion and eight billion model parameters.
BF16: A 16-bit floating-point format used for the 4B model weights and compute.
INT8 / FP16: The 8B run used 8-bit weights with 16-bit floating-point compute to fit local GPU memory.
Temporal IoU: The overlap between predicted and reference time intervals divided by their combined duration.
1 FPS: One sampled frame per second, giving the model 70 frames from the 70-second clip.
Precision / recall / F1: Precision measures correct predictions, recall measures recovered reference events, and F1 balances both.
Boundary error: The absolute timing error at the start and end of matched events.