adirik/taco_resized/taco_analysis - Storage Bucket

TACO Calibration Analysis

Quality analysis of camera calibration and object/hand pose annotations for the TACO dataset.

Summary

We projected ground-truth object meshes and hand skeletons onto all 12 allocentric camera views for every sequence (2114 total) and manually reviewed them for alignment issues.

Results:

• 1758 sequences marked as good
• 352 sequences marked as bad (misaligned object/hand overlays)
• 4 sequences marked as bad_hand_pose

Key findings

1. The camera rig is static across all 2120 sequences (same 12 cameras, <10mm position std across most dates)
2. Misalignment is systematic — the same cameras tend to have issues across sequences, suggesting the calibration has a consistent bias rather than random noise
3. Object poses may contribute to misalignment in some sequences (mesh overlay doesn't match the actual object in the image)

Files

File	Description
annotations.json	Per-sequence annotations: good, bad, bad_camera, bad_hand_pose, with optional bad_cams list
overlays/	Pre-rendered overlay images (2114 JPEGs, ~600MB) — 3x4 grid of all 12 cameras with mesh + hand skeleton projections
taco_calib_reviewer_web.py	Web-based review tool (Flask) for browsing and annotating sequences
prerender_overlays.py	Script to regenerate overlay images

Interactive Reviewer

Browse all sequences with mesh and hand overlays. Filter by annotation status to review problematic cases.

Setup

# Install dependencies (flask is the only extra)
pip install flask

# Download this folder
huggingface-cli download mzhobro/taco_dataset --include "taco_analysis/*" --repo-type dataset --local-dir ./taco_analysis_data

# Start the reviewer (adjust paths in the script if needed)
python taco_analysis/taco_calib_reviewer_web.py --port 8501

Then open http://localhost:8501 in your browser.

Controls

Key / Action	Description
Arrow Right / D	Next sequence
Arrow Left / A	Previous sequence
G	Mark as good
B	Mark as bad
O	Mark as bad object pose
H	Mark as bad hand pose
U	Unmark
N	Next unannotated
Click camera	Toggle camera as problematic

Filtering

Use the checkboxes in the toolbar to filter sequences by status. Check bad + bad_hand and click Apply to browse only problematic cases. Click Show All to return to the full list.

Annotation Format

{
  "(brush, brush, bowl)/20230919_036": {
    "status": "bad",
    "bad_cams": ["22139914"]
  },
  "(cut, spoon, plate)/20231027_099": {
    "status": "good",
    "bad_cams": []
  }
}

Regenerating Overlays

The overlay images show frame 0 of each sequence with object meshes (blue=tool, orange=target) and hand skeletons (green=left, yellow=right) projected using the provided camera calibration.

python prerender_overlays.py

Requires the TACO dataset with Allocentric_Camera_Parameters, Marker_Removed_Allocentric_RGB_Videos, Object_Poses, Hand_Poses_3D, and object_models_released.