Computer vision · 2026
Multi-view pedestrian detection with partial labels
Keep multi-camera people detectors accurate when most ground-plane labels are missing, then prove it on simulated crowds and a live Unity demo.
MVDet MODA 22.7 → 81.5 with 60 % of labels dropped (Wildtrack)
- Problem
- Multi-view detectors (MVDet, SHOT, MVDeTr, EarlyBird, TrackTacular) need every person labelled on the ground plane in every frame. Drop 60 % of those labels and MVDet falls from 86.9 to 22.7 MODA on Wildtrack, MVDeTr to 0.
- Approach
- An Ablation Loss (AABL) for BEV heatmaps that stops punishing confident predictions on unlabelled background, plus detector pseudo-labels. Two Unity-simulated crowd datasets with perfect calibration to test transfer, and a server + browser + Unity demo that runs every checkpoint live.
- Result
- At 60 % dropped labels: MVDet 22.7 → 81.5, SHOT 4.9 → 66.7, MVDeTr 0.0 → 76.1 MODA on Wildtrack, across all five architectures. Simulated sets reproduce real crowd behaviour that MultiviewX lacks (38 % standing, 54 % in groups on the plaza, against 0 % and 8 %). Live demo at 2.3 Hz on a 6 GB laptop GPU.
What I did
The code is private while the work is under review for SAC 2027 and will be released afterwards. The numbers below are copied from the experiment spreadsheet, the training logs and the demo UI.
- Loss function, with the team. Co-designed the Ablation Loss (AABL) for ground-plane heatmaps, ran the threshold ablation grid on MVDet and SHOT, and wired the loss and detector pseudo-labels into all five codebases so every architecture could be compared under the same dropped-label protocol.
- Two simulated datasets. Built the Unity 2018 + Python pipeline that turns 1,256 PersonX characters into MultiviewX-format datasets with analytic calibration (median reprojection error under 0.001 px), then designed a social crowd model so the data has the standing, grouping and speed statistics of a real square, which MultiviewX does not.
- Demo system. One Python server and browser UI that runs the thesis checkpoints on datasets, on a live Unity crowd streamed over TCP, or on any set of videos, with a Kalman tracker, per-frame MODA scoring, a custom CUDA deformable-attention build and a Docker image.
Problem
Multi-view pedestrian detection fuses several calibrated cameras into one bird’s-eye-view (BEV) occupancy map, so people hidden in one view are still found by another. The catch is supervision: a 7-camera, 400-frame set such as Wildtrack means 2,800 images and over 40,000 boxes, and every ground-plane position must be labelled or the model learns that a visible person is “background”.
We simulate that failure by dropping 20 %, 45 % and 60 % of the training annotations and measuring MODA (multiple-object detection accuracy) on the untouched test split. The baselines collapse, and the transformer-based MVDeTr collapses first because its default threshold never fires.
| MODA, labels dropped → | 0 % | 20 % | 45 % | 60 % |
|---|---|---|---|---|
| MVDet, Wildtrack | 86.9 | 81.0 | 59.2 | 22.7 |
| SHOT, Wildtrack | 92.1 | 84.7 | 26.9 | 4.9 |
| MVDeTr (thr 0.6), Wildtrack | 91.9 | 17.1 | 0.0 | 0.0 |
| EarlyBird, Wildtrack | 90.5 | 85.7 | 57.9 | 31.8 |
| MVDet, MultiviewX | 82.9 | 68.7 | 30.8 | 11.1 |
| SHOT, MultiviewX | 89.4 | 80.7 | 51.0 | 20.0 |
| MVDeTr (thr 0.6), MultiviewX | 93.4 | 60.1 | 2.8 | 0.0 |
| EarlyBird, MultiviewX | 94.9 | 89.9 | 68.5 | 57.6 |
Precision stays near 99 % in every one of these runs. The models are not hallucinating people; they are being taught to ignore the real ones that nobody labelled.
Pipeline
N synchronised cameras. Wildtrack (7 cams, 12 × 36 m) or MultiviewX-layout scenes (6 cams, 25 × 16 m). Calibration from the dataset, from the Unity rig, or estimated with no intrinsics at all (CalibFree: YOLO + MapAnything + ground-plane fit).
Per-view CNN. ResNet-18 features per camera, the same backbone across MVDet, SHOT, MVDeTr, EarlyBird and TrackTacular so that only the fusion and the loss differ.
Perspective projection to BEV. Features are warped onto the ground plane through each camera homography (SHOT adds height-stacked homographies, MVDeTr a deformable transformer).
BEV occupancy heatmap. A 2.5 cm grid (1000 × 640 for MultiviewX). Soft Gaussian targets are built from the labelled foot positions.
AABL + pseudo-label loss. Positive and easy-negative pixels keep plain MSE. Background pixels where the model is already confident are "confusing" and get a down-weighted or mirrored term instead of a full background penalty. An offline detector (YOLO26 or Detic) adds soft pseudo-positives, suppressed within 1 m of real labels.
MODA / MODP evaluation. Hungarian matching at 0.5 m against the full test labels, identical for every model and for the demo server.
The loss
A missing label looks exactly like background to a standard heatmap loss, so the gradient pulls a correct detection towards zero. AABL splits the heatmap into three regions after building the soft ground truth:
- Positive pixels, where the soft target is at least
pos_thr: standard MSE to the target. - Easy negative pixels, background where the prediction is low: standard MSE to the target.
- Confusing pixels, background where the detached prediction is already at least
confuse_thr: either pull them towards 1 (mirror mode, treating them as potential positives) or keep the MSE but scale it bybeta.
The three sums are divided by the total pixel count, so the loss stays on the same scale as the original and the schedules did not need retuning. The assignment uses a detached prediction, so the model cannot shrink the loss by moving pixels between regions. An earlier focal-style variant had exactly that bug: with a mirrored positive term, predicting “person everywhere” drove the loss to zero, and the optimiser found it within an epoch. The fix, which we kept, is that recalibration only ever scales a penalty down, never replaces it with a term that vanishes at p = 1.
We swept pos_thr ∈ 0.5 × confuse_thr ∈ 0.6 on MVDet and SHOT, then transferred the best settings (β = 0.1, confuse 0.3 for MSE-based models; β = 0.05, confuse 0.4 for MVDeTr) to the other architectures unchanged.
Results
MODA with the same dropped-label splits, all five architectures, before and after adding AABL. Precision stays above 93 % in every AABL row, so the gains come from recall.
| MODA at 20 / 45 / 60 % dropped | Wildtrack, no AABL | Wildtrack, AABL | MultiviewX, no AABL | MultiviewX, AABL |
|---|---|---|---|---|
| MVDet | 81.0 / 59.2 / 22.7 | 88.6 / 84.7 / 81.5 | 68.7 / 30.8 / 11.1 | 79.1 / 68.7 / 55.5 |
| SHOT | 84.7 / 26.9 / 4.9 | 89.6 / 86.6 / 66.7 | 80.7 / 51.0 / 20.0 | 88.3 / 83.5 / 76.0 |
| MVDeTr (thr 0.6) | 17.1 / 0.0 / 0.0 | 89.4 / 88.0 / 76.1 | 60.1 / 2.8 / 0.0 | 92.6 / 91.9 / 88.8 |
| EarlyBird | 85.7 / 57.9 / 31.8 | 88.6 / 86.0 / 76.3 | 89.9 / 68.5 / 57.6 | 94.4 / 91.4 / 87.2 |
| TrackTacular (SegNet) | 85.9 / 43.3 / 6.8 | 91.2 / 89.9 / 83.8 | 85.8 / 59.6 / 50.1 | 95.2 / 94.1 / 91.8 |
Adding the pseudo-label term on top of AABL helps most where AABL alone still lags, at 60 % dropped labels:
| MODA at 60 % dropped | Wildtrack: none → AABL → AABL + pseudo | MultiviewX: none → AABL → AABL + pseudo |
|---|---|---|
| MVDet | 22.7 → 81.5 → 83.0 | 11.1 → 55.5 → 56.0 |
| SHOT | 4.9 → 66.7 → 68.6 | 20.0 → 76.0 → 75.5 |
| MVDeTr (thr 0.6) | 0.0 → 76.1 → 88.1 | 0.0 → 88.8 → 91.0 |
| EarlyBird | 31.8 → 76.3 → 85.0 | 57.6 → 87.2 → 90.2 |
| TrackTacular (MVDet) | 8.1 → 82.1 → 84.1 | 23.8 → 89.1 → 90.9 |
The fully-unsupervised setting (no labels, pseudo-labels only) is still open: MVDeTr reaches 62.8 MODA on Wildtrack at 20 % but drops to 6.0 at 60 %, so the label-free direction is the next experiment rather than a result.
Training
Loss per epoch
MODA (%) per epoch
Show as table
| Epoch | Train loss | Val loss | MODA (%) |
|---|---|---|---|
| 1 | 2.497 | 0.41 | 73 |
| 2 | 1.167 | 0.247 | 83.3 |
| 3 | 0.981 | 0.233 | 82.7 |
| 4 | 0.881 | 0.168 | 88.1 |
| 5 | 0.798 | 0.145 | 89.5 |
| 6 | 0.715 | 0.113 | 90.8 |
| 7 | 0.666 | 0.093 | 91.8 |
| 8 | 0.609 | 0.088 | 92.5 |
| 9 | 0.559 | 0.084 | 92.4 |
| 10 | 0.531 | 0.088 | 92.5 |
The run converges in eight epochs and ends at 92.5 MODA, 98.6 precision and 93.8 recall. The pseudo-label term stays small (0.05 of the total loss) by design: its job is to keep confident detector responses from being treated as background, not to replace the labels.
Two simulated datasets
MultiviewX is itself synthetic, but its crowd walks at one speed, almost never stops and almost never groups. Real squares look nothing like that. I built a pipeline that renders MultiviewX-format data from Unity so we could generate test scenes with real-like behaviour, perfect labels and zero annotation cost.

| Plaza (main set) | City square | |
|---|---|---|
| Frames / cameras | 400 at 2 fps / 6 × 1920 × 1080 | 400 at 2 fps / 6 × 1920 × 1080 |
| People per frame (mean) | 41.3 | 38.8 |
| Annotated identities | 214 | 192 |
| Cameras covering a location | 4.38 (MultiviewX: 4.41) | 4.38 |
| Calibration | analytic from the Unity cameras, median reprojection 0.0001–0.0008 px | same |
| Scene | POLYGON city pack rebuilt as a square: café terrace, clock column, hot-dog cart, benches | Japanese Otaku City asset, placed by an automatic height-map search |
| Behaviour | walk alone or in groups, stop and chat, avoid each other, route around obstacles, queue at two stalls, sit (35 % sit 20–60 s when a seat is free); unlabelled background pedestrians | walk alone or in groups, stop and chat, avoid each other; 100 unlabelled background pedestrians |


The behaviour statistics come from the same script run on the annotations of all three sets (tools/behaviour/behaviour_stats.py, re-run on 26 Oct 2026). The plaza lands on Wildtrack’s numbers; MultiviewX does not.
| Crowd statistic (from labels, 2 fps) | Wildtrack (real) | MultiviewX | Plaza (mine) |
|---|---|---|---|
| Stationary (< 0.3 m/s) | 37 % | 0 % | 38 % |
| Walking speed, p10–p90 | 0.58–1.64 m/s | 1.15–1.21 m/s | 0.57–1.67 m/s |
| In a group | 46 % | 8 % | 54 % |
| Standing in a group | 21 % | 0 % | 26 % |
| Walking alone | 38 % | 92 % | 35 % |
| Nearest neighbour, median | 1.01 m | 1.44 m | 0.96 m |
| Body overlaps per 100 person-frames | 0.46 | 2.33 | 0.28 |
How the pipeline works, in order:
- Characters. 1,266 PersonX prefabs are converted to Humanoid rigs by script (1,256 valid), given one shared walk/idle blend tree, and speed-matched so feet do not slide.
- Cameras. The six MultiviewX cameras are rebuilt from the original calibration (2.2 m high, 15° down, 94° horizontal FOV) and verified against Unity’s own projection to under 0.2 px. The rig never moves; the environment is moved and rotated around it, so calibration and the BEV grid are identical across every dataset.
- Scene placement. A height map of the city asset is scanned over 72 rotations to find flat, unobstructed spots where no camera ends up indoors; candidates are rendered and picked by eye.
- Crowd. A seeded steering model with personal speeds, group formation, chatting stops, avoidance within 1 m and NavMesh routing around obstacles. The same code runs in Play mode, in the Editor preview and in batch generation, so what you see is what gets rendered.
- Render and convert. Unity renders 6 × 1920 × 1080 per frame (about 13 minutes and 5 GB per set). A Python script writes the calibration XML,
rectangles.pom(3.84 million rectangles),annotations_positions, and a BEV preview for checking.



What the sets revealed. Running the MultiviewX-trained MVDeTr + AABL checkpoints on the 40 test frames of the city-square set without any retraining costs about 17 MODA, almost all of it recall:
| MVDeTr + AABL trained on MultiviewX with | MultiviewX test MODA | City-square test MODA | Precision | Recall |
|---|---|---|---|---|
| 20 % dropped | 93.0 | 75.9 | 96.6 | 78.7 |
| 45 % dropped | 92.5 | 73.6 | 94.8 | 77.8 |
| 60 % dropped | 91.2 | 74.1 | 96.6 | 76.8 |
Precision holds despite 100 unlabelled background pedestrians, and all three checkpoints degrade by the same amount, so the partial-label training is not what breaks. The gap is scene and crowd behaviour (standing groups, denser crowds, a new city), which is exactly the gap the sets were built to expose, and the number our next experiments have to beat.
Demo system
The demo exists so that a reviewer can verify any claim on one screen: which checkpoint, its logged test MODA, this frame’s TP / FP / FN, the latency and which calibration was used. Detections come from the checkpoints only; 2-D boxes are projections of a 1.8 m person through the calibration, never model outputs.
Architecture. A Python server (PyTorch, fp16) with a checkpoint registry that scans the repo for MultiviewDetector.pth files, including inside zips, and reads dataset, drop ratio, thresholds and the logged test MODA from each run’s log.txt. Backends for MVDet and MVDeTr, with a custom-built CUDA deformable-attention op that runs 2.2–2.4× faster than the PyTorch fallback at identical output (MODA 90.23 on 40 Wildtrack test frames, heatmap difference 4e-5). A constant-velocity Kalman tracker with Hungarian matching (1 m gate) gives IDs and velocities. The browser UI is plain HTML, CSS and JavaScript.

Three input modes.
- Dataset. Browse Wildtrack, MultiviewX or the simulated sets, run the real checkpoint on the GPU and score the frame against ground truth: GT rings, matched, false-positive and missed colours, per-frame MODA.
- Live · Unity. The Unity project renders six cameras into one 3 × 2 atlas with
AsyncGPUReadback(no JPEG encoding, no GPU stall) and streams it over raw TCP with the GT feet and the rig poses. The server builds the MultiviewX calibration from the rig (1.12 px error against Unity’s own projection) and runs the checkpoint; the tracker’s velocities let both the browser and Unity draw every person atposition + velocity × elapsed, so dots move with people between detections. Pause freezes Unity (timeScale = 0) and the browser on the same frame, from either window. - Scene · Video. Any N videos (per-video sync offsets, sample rate, time span) or an image folder, with a calibration folder, calibration-free estimation (48 frames → YOLO + MapAnything → ground-plane fit → foot refinement, about 5 minutes), or no calibration at all. A BEV editor drags, rotates and mirrors the checkpoint’s fixed grid over the camera footprints.

Interaction. Click a person in any camera, on a node or on the map to highlight them in every camera; Ctrl-click to select up to six. Click an empty map point to project a 1.8 m probe box into every camera and see whether someone really stands there. Follow one track with a trail. Two layouts: a grid, or “Surround”, which draws each camera at its calibrated position around the BEV with its field-of-view edges.





Run it.
# clone, then install torch (CUDA), torchvision, kornia, numpy, Pillow, scipy, opencv-python
.\demo_multiview\build_deform_attn.ps1 # optional: CUDA 12.8 + VS 2022 Build Tools
.\demo_multiview\run_demo.ps1 # http://127.0.0.1:8765, Live tab at /#live
# or without Unity: replay a recorded sequence into the Live tab
python demo_multiview\live_replay.py F:\PersonX\MultiviewX_Otaku_opt1_social_v6 --start 360 --loop
# or: docker compose up -d (image with code + 20 checkpoints, datasets mounted, runs on a 6 GB GPU)Measured on an RTX 4050 Laptop (6 GB), GPU shared with Unity, 60 s runs:
| Configuration | Detection rate | Unity fps | Track error p90 | MODA |
|---|---|---|---|---|
| default (2.5 captures/s) | 2.3 Hz | 12–14 | 16–17 cm | 82–83 % |
| smoother Unity (3 ms GPU gap, 10 captures/s) | 1.5 Hz | 18 | 16 cm | 82.5 % |
| fastest detection (10 captures/s) | 3.5 Hz | 7 | 21 cm | 79.5 % |
Median track error is 6–7 cm in all three because positions are extrapolated with velocity. Video playback decodes seven Wildtrack streams at exactly 10 samples/s while the detector runs at 3.25 Hz on the newest sample; the first version managed 0.3 samples/s because eight requests were seeking the same H.264 file.
Challenges and lessons
- A loss that could be gamed. The first AABL variant let the optimiser zero the loss by predicting foreground everywhere. We now only ever scale penalties down and always threshold on a detached prediction. Lesson: check what the global minimum of a new loss is before training on it.
- A silent no-op. SHOT’s model called plain MSE inside
forward, so--loss brlchanged nothing in its early runs. Found by comparing logs that should have differed; those runs were discarded and the README now states it. - Coordinate conventions. MultiviewX’s calibration is mirrored (all projected points have negative depth). Rebuilding the rig in Unity meant flipping one axis and proving it with a depth check and a 0.2 px reprojection test, then keeping that test in the pipeline.
- Numbers that look fine while images are wrong. Drawing foot dots and 3-D boxes on the rendered frames early caught every large bug (wrong screen-point API in batch mode, foot sliding, a 1:10 scale city) that the summary statistics missed.
- GPU budget. Unity and PyTorch share 6 GB. Reading cameras back asynchronously, sending one atlas instead of six JPEGs, and copying with
memcpyinstead of Unity 2018’s per-elementNativeArray.CopyTo(350 ms for 22 MB) is what made 2.3 Hz possible.
Credits
Capstone group GFA26AI02 at FPT University: Lai Le Dinh Duc, Ho Dang Manh, Nguyen Nhut An and me, supervised by Dr. Dao Duy Son. Built on the public MVDet, SHOT, MVDeTr, EarlyBird and TrackTacular codebases, the Wildtrack and MultiviewX datasets (MultiviewX images above: hou-yz/MultiviewX, CC BY-SA 4.0), PersonX characters, and the Japanese Otaku City and POLYGON city assets.