Leela Yanamaddi
November 19, 2026

A playable video can still be bad training data. Check files and usage rights first, task visibility and timing second, and annotations third. Block failed clips before they reach the next stage.
A release-ready dataset needs checks at four levels:
Set task-specific limits before review. Rework clips only when the missing quality can be restored; reject those with missing information or unresolved critical failures.
Video Dataset QA: Four Gates Before Training
Start with the file itself. If it cannot be decoded cleanly, stop.
Check every file against the source manifest for size, SHA-256, container readability, and full-file decode. Quarantine files with checksum mismatches, then compare them with the source or backup before retrying the transfer. Decode every frame - not just the first few seconds - and save error counts and failing-frame locations. A matching checksum confirms file identity, not that the content is usable.
Use FFprobe to extract the codec, resolution, frame rate, bitrate, duration, and stream inventory. Compare the results with the versioned task requirements, including required audio or sensor streams. Check timestamps for gaps, reversals, and repeats. Repair recoverable transfer or container faults. Reject files with unrecoverable corruption, missing required streams, or source resolution below the task minimum. Use stream copy when re-encoding isn't needed; transcoding can reduce quality.
Once technical checks pass, confirm that the clip can be used and assigned to a dataset split safely.
Require these metadata fields: a clip ID that isn't shared with another clip; collection batch; capture date and time with time zone; task ID; camera or sensor ID; operator or site, when permitted; camera settings; environment or collection conditions; duration; and parent recording or session ID. Use controlled values for task type, camera model, lighting, and indoor/outdoor setting.
Link each asset to its processing history and training-use rights record. Include permitted uses, restrictions, retention terms, and proof of authorization. Publicly available does not mean approved for training. Automated checks confirm only that records exist; a privacy or legal owner must resolve unclear permissions. Block clips with unresolved faces, voices, badges, license plates, screens, or other identifying information if the project lacks a lawful basis or an approved de-identification process.
Use SHA-256 to find exact duplicates, then normalized-media or perceptual hashes to find re-encoded and near duplicates. Review uncertain matches instead of deleting them automatically. Group related clips before assigning training, validation, and test splits. For intentional repeats, record the reason, parent clip ID, and split-placement rule.
Log every decision before the clip moves forward. For each check, store the run ID, tool and rule versions, timestamp, observed value, and evidence location.
Quarantine is a hold, not a pass. Exclude held clips from annotation and training manifests, assign an owner and due date, and release only versioned allowlisted IDs. Preserve originals. Link repaired derivatives to their originals, give them new IDs and checksums, and rerun automated checks.
| Automated check | Evidence | Configured threshold | Severity | Owner | Disposition |
|---|---|---|---|---|---|
| Manifest checksum | Expected and received SHA-256 | Exact match required | Critical | Ingestion owner | Pass/Rework |
| Full decode | Decoder log and failing-frame index | Zero unrecoverable decode errors | Critical | Media pipeline owner | Pass/Rework/Reject |
| Container and codec | Probe JSON | Approved formats/codecs only | High | Ingestion owner | Pass/Rework/Reject |
| Resolution and frame rate | Stream metadata | Task-specific minimums | High | Dataset engineer | Pass/Rework |
| Duration and timestamps | Duration plus gap/jump report | Within task range; no unexplained discontinuities | High | Dataset engineer | Pass/Rework/Reject |
| Required streams | Stream inventory | All required streams present | Critical | Sensor-data owner | Pass/Reject |
| Required metadata | Structured manifest fields | No missing mandatory fields | High | Collection owner | Pass/Rework |
| Rights and consent | License/consent IDs and permitted-use code | Training use explicitly allowed | Critical | Legal/privacy owner | Pass/Quarantine/Reject |
| Privacy | Screening or redaction record | No unresolved prohibited exposure | Critical | Privacy reviewer | Pass/Rework/Reject |
| Exact duplicates | File hash | Duplicate grouped or dispositioned | Medium | Data curator | Pass/Group/Reject |
| Near duplicates | Perceptual or embedding similarity report | Project-specific similarity threshold | Medium | Data curator | Group/Review/Reject |
After technical QA, check task usability before annotation. Once a clip passes file and rights checks, screen it for blur, focus loss, clipping, glare, noise, compression artifacts, and motion. Check frame-level scores - not just clip averages - and flag defects during grasp, contact, insertion, or release. Set thresholds for each task: brief blur during a noncritical transition may be acceptable, but blur during a critical action should trigger review or rejection. Human reviewers must confirm that the hand, object, contact point, target, and workspace stay visible. Review every borderline flag and a random sample of automated passes.
Confirm that video timestamps are monotonic, frame intervals stay within the permitted range, and video, action, and robot-state streams align within the project’s tolerance. At 30 fps, one frame is about 33 ms - even that offset may exceed tolerance. Set the limit based on motion speed and contact requirements. Check that calibration matches the camera, lens, robot, and session. For metric 3D tasks, inspect calibration drift before approving footage.
Once synchronization and calibration pass, check the task protocol. Require the prescribed start condition, action sequence, and visible end state. Check camera position, viewpoint, object identity, tool use, workspace boundaries, and interruptions. A correct outcome does not prove the clip is usable: off-camera help, repositioning, or a missing release can invalidate it. Keep failed attempts only when the spec permits them and reviewers can reliably label both cause and outcome.
Audit accepted footage across sites, operators, objects, actions, viewpoints, and difficulty - not just by clip count. Pair random sampling with targeted review of new contributors, difficult actions, unusual lighting, and heavily flagged batches. Record visibility, task completion, and protocol deviations separately so recurring collection problems stay visible.
For each failure, record the clip ID, defect code, affected frame range, measured value, applicable threshold, reviewer, and disposition. Allow trimming only to remove setup or idle footage without losing the required start condition, action sequence, or end-state evidence. Allow redaction only if hands, objects, targets, and relevant robot state remain visible. Before accepting reprocessed footage, compare it with the original to confirm that no task evidence changed. Repeated protocol failures should trigger batch quarantine, contributor retraining, and review of earlier submissions - not isolated clip fixes.
Use the same disposition language for every failure.
| Defect | Evidence and task-specific rejection threshold | Review owner | Disposition |
|---|---|---|---|
| Critical-action blur or poor focus | Sharpness falls below the task threshold during grasp, contact, or release; the action cannot be identified | Domain reviewer | Recollect; reprocess only if the original evidence remains clear |
| Exposure clipping or glare | Clipping or glare hides the object, contact point, or target | QA reviewer | Reprocess if recoverable; otherwise recollect |
| Critical occlusion | A required hand, object, or target is hidden, and available views cannot establish the action | Domain reviewer | Reject or recollect |
| Timing or calibration error | Stream offset exceeds the task limit, or calibration is invalid for the required geometry | Data engineer / capture operations | Fix only if alignment or geometry can be restored; otherwise recollect |
| Incomplete demonstration | A required start condition, step, or end-state evidence is missing | Domain reviewer | Reject the success annotation; keep only as a labeled failure when permitted |
| Repeated protocol violations | Assistance not allowed by the protocol, scene resets, camera movement, or wrong-object use recur | QA lead / operations | Quarantine batch; investigate scope, retrain contributor, and reject affected episodes |
A clip can pass video QA and still fail annotation QA. Release labels only after automated checks and human audits confirm they meet the annotation spec. Start with schema checks, then move to human review.
Catch structural errors before reviewers spend time checking label meaning. Run parsing and schema checks on every annotation file. Check required fields, approved classes, valid coordinates, timestamp ranges, and action spans with correctly ordered start and end times. Each track ID must identify only one object within a clip and stay consistent across that object's frames. Flag missing required labels and broken track references, and report the exact sample, field, and blocking error.
Human reviewers should compare annotations against the video to check class accuracy, complete action boundaries, box or mask placement, and track continuity. Review interpolated frames, occlusions, exits, and re-entries - not just keyframes. Set separate limits for boundary error, box or mask overlap, and identity consistency before review begins. Frame-level overlap alone cannot confirm that a track follows the correct object.
After the structure passes, use a gold set to audit timing and labeling decisions.
Maintain a versioned gold set with expert-approved labels and decision notes. Double-label 5% to 10% of samples during routine review, along with new annotators’ first batches.
Measure categorical agreement, temporal boundary error, box or mask overlap, and identity consistency separately. Report sample sizes and results for each class.
Pair random audits with targeted reviews of rare classes, ambiguous actions, heavy occlusion, and recurring defects. Send disagreements that affect release to an adjudicator - don't average them into a single score.
Give each failure a specific correction path rather than sending everything to a generic rework queue. Log rework and adjudication rates, annotation version, reviewer, correction time, root cause, and guideline changes. After corrections, rerun validation and review the affected labels again. When schemas or guidelines change, identify the affected subset. If the change has high impact, review every affected sample.
| Defect type | Detection method | Metric or tolerance | Owner | Disposition |
|---|---|---|---|---|
| Invalid schema, field, or timestamp | Automated parser, schema, and range checks | 100% schema validity; timestamps within clip duration | Data engineer | Block; repair and revalidate |
| Wrong class or missing event | Human audit against video and specification | Correct classes; no missing required events | Annotation lead | Rework; fully reannotate affected samples if systematic |
| Incorrect action boundary | Gold comparison and human audit | Versioned start/end tolerance | Annotation lead | Adjust; adjudicate disputed boundaries |
| Poor box or mask placement | Geometry checks and visual review | Task-specific IoU or Dice threshold | Annotation lead | Relabel |
| Identity switch or fragmented track | Continuity checks and frame-level review | Defined gap limit; zero unexplained switches in critical clips | Tracking QA owner | Repair; block unresolved identity errors |
| Unclear occlusion or re-entry | Targeted human review | Versioned visibility and identity rules | Subject-matter expert | Adjudicate; clarify guidance |
| Missing required evidence | Human review | Evidence sufficient to establish the required label | Reviewer | Recollect or exclude |
Turn the file, task, and annotation check results into a final release decision.
Every clip needs linked evidence. The dataset must also meet agreed coverage quotas before release. These quotas cover classes, environments, viewpoints, outcomes, and difficult samples. Per-file checks are still required before releasing the dataset.
Use the rules below to turn QA findings into an accept, rework, or reject decision.
Use four machine-readable release statuses: accepted, accepted_with_limitation, rework_required, and rejected.
Any limitation must be noncritical, compatible with training, and approved by the dataset owner. Record the clip or batch ID, failed requirement, reason, risk assessment, intended use, compensating control, and review date.
Reject clips with unrecoverable corruption, missing provenance, insufficient visibility, missing permission, or unresolved annotation or sync failures. Corrections must preserve history, with failed checks and any dependent checks rerun.
Apply project-specific tolerances to this release gate; there is no universal defect rate. Use the table as a one-page review checklist.
| Gate | Owner | Evidence | Threshold | Status | Exception approval | Release decision |
|---|---|---|---|---|---|---|
| Clip: file integrity | Data engineer | Decode, stream, checksum reports | Full decode; required streams and specifications met | [Status] | No critical exceptions | [Approve/Block] |
| Clip: metadata and lineage | Data operations | Schema report; source and transformation IDs | Required fields valid; every output traceable | [Status] | Noncritical only | [Approve/Block] |
| Clip: consent and rights | Rights owner | Permission and license records | 0 missing required permissions | [Status] | None | [Approve/Block] |
| Clip: visibility and task evidence | Capture QA | Review timecodes; task rubric | Required steps visible; task and outcome verified | [Status] | Noncritical only | [Approve/Block] |
| Clip: synchronization | Technical QA | Timestamp and calibration reports | Project tolerance met | [Status] | None | [Approve/Block] |
| Clip: annotations | Annotation lead | Validation and audit reports | Required labels complete; accuracy limits met | [Status] | Noncritical only | [Approve/Block] |
| Dataset: scope and coverage | Dataset owner | Coverage matrix; duplicate report | Mandatory class, scenario, and difficult-sample quotas met | [Status] | Approved deviations only | [Approve/Block] |
| Dataset: export and splits | Release engineer | Loader results, manifest, hashes, split audit | Counts reconcile; 100% parse and hash match; 0 prohibited split overlap | [Status] | None | [Approve/Block] |
Before signing, verify source, annotation, and export links, including the exact clip and annotation versions used. Block release if critical defects exceed limits or systematic errors remain unresolved. Keep versioned reports, exceptions, and approvals from both the QA owner and the dataset or task owner. Add a rights review where needed.
Record the decision and file the evidence. Feed these rules into collection planning to catch recurring defects before the next batch.
Before collecting data, define pass, repair, and reject criteria in the task statement of work. Cover task validity and completeness, maximum sensor synchronization drift, image quality and visibility percentages, and annotation agreement targets with double review.
Use a 100–200-episode pilot to calibrate these limits. For each batch, run automated checks on all data and manually spot-check representative samples. Resolve annotation disagreements before release. Reject samples that fail critical requirements for task completeness, synchronization, visibility, or labels.
Run a 100–200-episode pilot against your written acceptance thresholds. For each task and failure reason, track first-pass rejection, rework success, final rejection, and replacement acceptance. A rejection rate above about 15% usually points to a process-wide or specification problem.
Treat technical quality and task usefulness as separate checks. Test candidate data subsets against a trusted holdout, downstream robot performance, or both. Don’t judge the data by visual cleanliness alone.
Increase human review when audits show inconsistent labeling, poorly defined category boundaries, or unclear labels. Do the same when training reveals gaps in object, scene, or task-stage coverage, or when batches involve changes in operators, hardware, or task protocols. Prioritize clips with rare actions, long-tail objects, or complex multi-step tasks.
If the rejection rate exceeds 15%, expand review to check for systematic collection problems or gaps in operator training. Don’t treat these failures as cleanup alone.