GSR: fix the evaluator's prediction path and land the TrackLab staging tooling - #25
Open
AtomScott wants to merge 5 commits into
Open
GSR: fix the evaluator's prediction path and land the TrackLab staging tooling#25AtomScott wants to merge 5 commits into
AtomScott wants to merge 5 commits into
Conversation
…ion file
Scoring the first real GSR predictions raised three errors in sequence. All three lived in
`soccertrack_records_to_gs`, the flat-records -> GameState converter used for predictions:
1. bbox_pitch carried only x/y_bottom_middle. With EVAL_SPACE='pitch' the upstream scorer
reads six keys unconditionally (x/y_bottom_left, _middle, _right) -> KeyError.
2. image_id was an int. The scorer's unmatched-id branch does `len(image_id) == 10`
-> TypeError: object of type 'int' has no len().
3. predictions numbered frames 0-based while the ground truth uses string ids
("3000001" is 1-based frame 1), so no prediction ever matched a ground-truth frame.
tests/test_gs_hota_identity.py could not catch any of this: because the released ground truth
is already GameState, that test symlinks one file in as both ground truth and prediction and
never calls the converter. It verified the scorer's configuration, not this code path.
Changes:
* emit all six bbox_pitch keys. The three points are degenerate in the released GT (left ==
middle == right), so replicating the middle across all three matches how the GT is built.
* `image_id_for_frame` maps a 0-based frame index to the GT's string id, defaulting to the
released convention, with `image_id_mapper_from_gt` to derive it from a GT file instead of
trusting the default. Measured 100% coverage: 67,625 of 67,625 prediction frames for
128057's first half.
* reconcile _SN_CATEGORIES with the released categories block (1 player, 2 goalkeeper,
3 referee, 4 ball). It previously said 4=other, 5=ball, which disagreed with every real
file. Pitch-space GS-HOTA keys off attributes.role, so no score changed.
Adds tests/test_gs_hota_prediction_path.py, which re-encodes real ground truth into the flat
prediction layout, pushes it back through the converter and scores it. A perfect prediction can
only reach 1.0 if the pitch keys, category_id and id mapping are all correct.
test_gs_hota_prediction_path: HOTA=1.000000 LocA=1.000000, 6600 pred dets = 6600 GT dets
test_gs_hota_identity: HOTA=1.000000 (unchanged)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eproducible Getting TrackLab to run on a 45-minute half needed staging work that existed only in a scratchpad. This lands it, with the reasoning, so it does not have to be rediscovered. scripts/gsr/ stage_soccernetgs.py extract img1/*.jpg and repair the label metadata make_calibrated_keypoints.py generate the calibrated keypoint sets the release omits stage_prefix_sequences.py nested-prefix sequences for the length study, zero extra disk score_length_sweep.py score each run on its own length AND on a shared window plot_length_sweep.py accuracy / runtime / memory vs length README.md the reproduction sequence and the measured costs Why each is needed: * TrackLab reads frames from <seq>/img1/000001.jpg, never from video, so a half must be extracted (~110 GB at -q:v 2, matching the 1.67 MB/frame of the pre-existing staged clip). * Staging also repairs three defects in the released labels: width/height say 3840x1504 when the frames are 4096x1080; info.id is "1" in all 20 files and info.name collides between the two halves of a match; and the GT annotates ~25 more frames than the video contains, so seq_length must be clamped or TrackLab requests JPEGs that cannot exist. * <match>_calibrated_keypoints.json ships for 117093 ONLY, and manual_calib_distorted needs a distorted and a calibrated set, so no other match could run. The calibrated set is just an intermediate space (TPS maps distorted->calibrated, H_pc maps pitch->calibrated, both fit from the same points, so the canvas cancels); make_calibrated_keypoints.py proves this by regenerating 117093's and comparing the full image->pitch chain to the shipped one, getting 0.0000 m over 725 on-pitch grid points. It still normalises the canvas, because a raw balance=0 undistortion puts 128057's control points at x in [-7973, 11234] for a 4096-wide image and the TPS then extrapolates wildly. 132831 picks up data_corrections/ automatically. * The prefix sequences symlink img1, so a 30-minute prefix costs a label file and no frames. The README records the measured per-module costs, the two non-obvious flags the run cannot work without (use_rich=False or the log stays empty; num_cores=12 for a 2.05x speedup over the shipped num_cores=1), and the attribute ablation showing jersey numbers cost 20.9 GS-HOTA points and team classification 12.5, while detection recall is a healthy 72%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One file, no server, no sibling assets: plots and videos are embedded as data URIs, with videos
transcoded down first (a 4096-wide 30 s clip is ~7 MB and base64 inflates by a third). Full
resolution originals stay on disk and are listed by path.
Re-runnable at any point. Every section degrades gracefully when its inputs are absent, so a
report generated mid-experiment is still valid, just shorter -- which matters when the thing
being reported on is a sequence of overnight GPU runs.
python scripts/gsr/make_report.py --work <scratchpad> --out report.html
Sections: the length decomposition (what sequence length does and does not break), the length
sweep, the attribute ablation, the minute-0 vs minute-1 diagnosis, GTA's tracklet bookkeeping,
runtime and memory, the code defects found and fixed, and the open questions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ging for all 20 halves
The label-to-imagery offset is NOT uniform. It falls into two clusters, roughly 0 and roughly 25
frames, and varies by HALF rather than by match:
offset 0 : 117092 1st, 117093 2nd, 118575 1st+2nd, 118577 1st, 118578 2nd
offset 25 : the other fourteen
Assuming a single value would have misaligned six halves by a second in one direction or
fourteen in the other. Two independent methods now agree on every half:
* arithmetic: (End - Start) * fps / 1000 from <match>_padding_info.csv, minus the video's
frame count. This is what stage_soccernetgs.py applies.
* measurement: scripts/gsr/measure_frame_offset.py, which asks for each candidate lag whether
the ground-truth boxes land on players or on empty grass. A pitch is overwhelmingly green
and players are not, so box occupancy peaks at the correct lag. No detector, no prior
staging, and it never consults the evaluation metric.
Agreement on 18 of 18 halves where the measurement had signal. For 132831 and 132877 the
occupancy curve is flat (gain 0.001-0.004, z below 2.6) so the estimator is uninformative and
the arithmetic is applied there WITHOUT independent confirmation -- noted rather than hidden.
An earlier worry that padding_info's declared frame count disagrees with the ground truth's own
seq_length by up to 251 frames turned out not to affect the offset: that surplus is annotation
past the end of the video, which staging trims. The measurement is what settled it.
measure_frame_offset.py carries a --validate mode that checks it against the two halves whose
offset was established independently, because three earlier versions of the estimator failed:
a background-subtraction centroid returned 67 against a known 25 (at kickoff players stand
still, so a median background contains them and subtraction erases them), and seeking with
cap.set(CAP_PROP_POS_FRAMES) injected an unknown offset of its own, not being frame-accurate on
these files. Do not remove --validate.
Labels re-staged for all 20 halves. Frames are untouched: the defect is purely which
annotations attach to which frame, so no re-extraction was needed.
Also adds a 2-minute prefix to stage_prefix_sequences.py, and an experiments + offsets section
to the HTML report.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nk by confidence
Two defects of the same shape: both make an evaluator or loader unusable on real data, and
neither could be caught by the identity tests already in the tree.
1. _parse_gsr could not read a single released file
It did `records = json.loads(...)` then iterated the result as a list. Every released GSR
file is a JSON OBJECT (SoccerNet GameState), so iteration yielded the top-level key strings
and `r["image_id"]` raised "TypeError: string indices must be integers" on all 20 files.
Both layouts are now accepted, dispatching on the parsed type rather than on a filename:
GameState objects (released ground truth; string image_id with a 1-based numeric suffix,
dict bboxes, pitch position in bbox_pitch's bottom-middle keys) and flat record lists (what
predictions use; 0-based integer image_id, sequence bboxes).
Frame.image_id keeps whichever convention the file uses -- 1-based for GameState, 0-based
for flat records -- rather than silently reconciling them, because the offset between
annotations and video is a per-half property that has to be MEASURED. Assuming it cost this
project 18 GS-HOTA points once already; see scripts/gsr/measure_frame_offset.py.
2. The BAS evaluator threw away confidence ranking
`_ap_tolerant` states its contract in a comment -- "callers: sort by confidence desc" -- and
its own caller `_map_per_class` then sorted by `(half, t_ms)`, by TIME, discarding it. Event
had no score field at all. So the metric was not average precision but a time-ordered
precision-recall traversal: a detector gained nothing from ranking its output well and was
not penalised for emitting a flood of low-confidence spots. Every BAS number the project
ever produced would have been wrong in that way.
Event gains `score`, _parse_bas reads it from "score" or "confidence", and _map_per_class
ranks by descending score with a deterministic time fallback for ties.
Neither identity test could have caught either defect: scoring ground truth against itself
makes every prediction a true positive, so AP is 1.0 in any order, and no test pointed the GSR
loader at real data. The two new tests are built to discriminate:
tests/test_gsr_loader.py fails with TypeError against the old loader; asserts on real
data (22 entities in every frame, roles player/goalkeeper only,
1-based first frame, pitch coords in range)
tests/test_bas_map_ranking.py AP 1.0000 with false positives ranked last vs 0.5000 ranked
first. Equal scores mean ranking is being ignored.
All five tests pass:
test_bas_map_identity, test_bas_map_ranking, test_gsr_loader,
test_gs_hota_prediction_path, test_gs_hota_identity
Also adds docs/experiment-design-bas.md: the BAS plan, the resolved time-alignment question
(position is absolute from match start, not per-half as format-bas.md claims), why a
ball-centred crop from ground truth is an ORACLE and must be labelled as one, and five open
questions for Atom.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things, both discovered while getting the GSR baseline to actually run end to end.
1. The GS-HOTA prediction path could not score a real prediction file
Three errors in sequence, all in
soccertrack_records_to_gs(the flat-records → GameStateconverter used for predictions):
bbox_pitchcarried onlyx/y_bottom_middleEVAL_SPACE='pitch'→KeyErrorimage_idwas anintlen(image_id) == 10→TypeError"3000001"= 1-based frame 1)Why the existing test missed all three: the released GSR ground truth is already
SoccerNet GameState, so
tests/test_gs_hota_identity.pysymlinks one file in as both groundtruth and prediction and never calls the converter. It verified the scorer's config, not this
code path.
tests/test_gs_hota_prediction_path.pycloses that gap — it re-encodes real ground truth intothe flat prediction layout, pushes it back through the converter, and scores it:
Also reconciles
_SN_CATEGORIESwith the released categories block (it said4=other, 5=ball;every real file says
4=ball). No score changes — pitch-space GS-HOTA keys offattributes.role.2.
scripts/gsr/— the staging needed to run TrackLab on a 45-minute halfTrackLab reads frames from
<seq>/img1/000001.jpg, never from video. Staging also repairsthree defects in the released labels:
width/heightsay3840x1504when the frames are4096x1080;info.idis"1"in all 20 files andinfo.namecollides between halves;and the GT annotates ~25 more frames than the video has.
<match>_calibrated_keypoints.jsonships for 117093 only, so no other match could run atall. The generator is validated by regenerating 117093's set and comparing the full
image→pitch chain against the shipped one: 0.0000 m over 725 on-pitch grid points.
Context that is not in this PR
A quadratic-time defect in TrackLab's
merge_dataframeshad to be fixed to make a full halffeasible — per-batch cost grew linearly (1.02 s/batch at batch 0 → 3.74 s by batch 2353),
making total cost quadratic in sequence length. That fix lives in the external tracklab tree.
It was validated by running a full 750-frame pipeline before and after: GS-HOTA 29.135% both
times, metric summary files byte-identical, only float noise at 3.9e-11 m differing.
First real numbers (30 s clip, match 117093)
Detection and pitch projection are good — 72% recall. Jersey numbers cost 20.9 points,
team classification 12.5. AssA 87.60 is a selection effect, not an association result.
🤖 Generated with Claude Code