Blog

The Hidden Costs of Cheap Training Data: Rework, Rejection Rates, and Re-Collection

Leela Yanamaddi

Leela Yanamaddi
November 17, 2026

The Hidden Costs of Cheap Training Data: Rework, Rejection Rates, and Re-Collection

Cheap training data can cost more once rejected episodes and repair work enter the budget. In a hypothetical $4,000 order, a 15% rejection rate leaves 850 of 1,000 episodes accepted - making the purchase cost $4.71 per accepted episode, before review, fixes, or replacements.

The right comparison is simple: total buyer cost ÷ final accepted episodes. That includes QA, annotation fixes, integration, and buyer-paid re-collection - not just the supplier’s quote.

To limit repeat spending:

  • Set clear acceptance rules before collection starts.
  • Fix labels only when the raw data supports the change; replace recordings with missing required data.
  • Run a 50–100 episode pilot and track rejection rates, correction hours, and replacement results.
  • Put review deadlines, replacement charges, and supplier duties in writing.

The goal is to buy data that passes review and fits the training task - not just files that arrive.

Why Robotics Demonstrations Fail Quality Checks

A file can play without errors and still fail training review because it lacks the task evidence the model needs. Reviewers must be able to verify the required actions. Wrong labels are fixable; missing evidence is not. Low-cost deliveries add hidden labor when reviewers have to sort usable episodes from those that need replacement. Common problems include blocked views, missing steps, and corrupted sensor data.

Blocked Views, Missing Steps, and Sensor Errors

In warehouse pick-and-place footage, use the specified camera views and sensor streams to check whether the arm hides the grasp and whether recording continues through placement and release. Frozen, dropped, or corrupted video can erase the critical action. Check raw video against robot states and commands; misalignment can make actions appear in the wrong order.

Use capture records to restore missing calibration, frames, and units. Keep technical measurements in their specified units, including meters or radians. Check outcome labels against the final state, and mark collisions, near-miss events, and emergency stops. Then decide whether the error can be fixed or the episode needs to be recorded again.

Decide What to Correct, Reject, or Re-Collect

Apply this distinction during review: Reject means excluding the current episode. Re-collect means recording a replacement if that task coverage is still needed.

Error category Correct when Reject or re-collect when
Task identity The task, object, and scene are clear in the raw data; fix the label. Reject ambiguous tasks; re-collect missing objects, scenes, or required variants.
Episode boundaries The full task is present; locate boundaries using timestamps and events. Reject incomplete recordings; re-collect required steps.
Action visibility Required evidence exists in an accepted view; fix the annotation. Reject episodes with required actions hidden in all accepted views; re-collect needed coverage.
Sensor integrity Minor metadata or annotation errors can be fixed without changing measurements. Reject episodes with missing key video, required streams, or unrecoverable synchronization; re-collect as needed.
Calibration and units Existing metadata supports correcting labels or mappings. Reject episodes if missing calibration or units cannot be restored; re-collect needed coverage.
Outcome The final state is visible enough to support the correct label. Reject unresolved outcomes; re-collect missing placement, stabilization, or failure resolution.
Safety events The event is observable; correct its category and severity. Reject episodes when missing evidence prevents resolving a possible hazard; re-collect needed coverage.

The difference between correction and replacement determines the true cost of each accepted episode.

Failed attempts can still be useful when the spec calls for failure coverage. Save corrections as new versions without overwriting raw evidence, and log annotation work separately from replacement needs. Count corrected and replacement episodes as accepted only after they pass review. Track those labels and correction costs for the full cost calculation.

Calculate the Full Cost per Accepted Episode

Training Data Costs: Quote vs. Cost per Accepted Episode

Training Data Costs: Quote vs. Cost per Accepted Episode

Once acceptance rules are set, use yield to calculate total buyer cost - and cost per accepted episode.

Cost per accepted episode = total buyer cost ÷ final accepted episodes. Include the purchase price, QA review, rework labor, rejected-episode handling, buyer-paid replacements, replacement review, integration, and any material storage or preprocessing costs. Count the initial invoice once. Rejected footage remains part of that purchase unless the supplier provides a documented refund or credit.

Compare Quotes With a Total-Cost Worksheet

Compare suppliers using the same acceptance criteria. The goal is to compare total cost, not just the price on the quote.

Hypothetical worksheet; not market prices or industry averages. Compare cost per accepted episode under identical rules: complete task coverage, visible critical actions, synchronized sensor streams, and valid metadata.

Assumptions: Each supplier delivers 1,000 initial episodes toward a target of 1,000 accepted episodes. QA review takes 0.10 hour per episode at $30/hour, correction labor takes 0.25 hour at $35/hour, replacement collection costs $24 per episode, replacement review takes 0.10 hour per replacement at $30/hour, and integration takes 20 hours at $100/hour. All replacements pass review. Taxes, shipping, storage, and compute are excluded.

Cost item Supplier A Supplier B
Purchase price $20,000 $28,000
QA review 100 hours × $30 = $3,000 100 hours × $30 = $3,000
Correction rate and labor 15% × 1,000 × 0.25 hour × $35 = $1,312.50 5% × 1,000 × 0.25 hour × $35 = $437.50
Episodes rejected after rework 100 30
Replacement collection 100 × $24 = $2,400 30 × $24 = $720
Replacement review 10 hours × $30 = $300 3 hours × $30 = $90
Integration $2,000 $2,000
Final accepted episodes 1,000 1,000
Total buyer cost $29,012.50 $34,247.50
Cost per accepted episode $29.01 $34.25

Supplier A costs less here. But the quote alone doesn't show the full buyer cost or the cost per accepted episode. Include correction and replacement charges only when the buyer pays them.

Treat the rejection rate as a cost multiplier, not just a quality metric.

Compare Rejection Rates and Replacement Yield

For this yield comparison, 5% and 20% refer to rejection after rework - not failure at first review. Post-rework rejection determines how many usable episodes the initial purchase provides.

Apply accepted initial episodes = delivered episodes × (1 − rejection rate): 1,000 × 0.95 = 950, versus 1,000 × 0.80 = 800. Add accepted replacements separately. If replacements fail at 10%, round up to 56 deliveries in the 5% scenario and 223 deliveries in the 20% scenario. These are planning assumptions, not yield guarantees.

Include every buyer-funded replacement delivery and review in the cost. Count only accepted replacements in the denominator.

Record replacement turnaround in business days and link it to the affected training milestone. Exclude delay costs unless your team uses an approved cost-of-delay model.

Reduce Repeat Spending With Acceptance Tests and Pilots

Set Measurable Episode Acceptance Criteria

Once you know how episodes fail, turn those failures into measurable rules before collection starts. This helps prevent rework later. Build an acceptance checklist with five fields: criterion, measurement method, project threshold, decision owner, and failure disposition.

Cover task completeness, view coverage, sensor sync, time-alignment tolerance, calibration, units, labels, safety events, file integrity, metadata, and usage rights. Set numerical limits where they apply. Calling data “synchronized” doesn't make it a testable requirement.

Allow corrections only when the underlying evidence supports them. If evidence is missing, reject or re-collect the episode. Maintain a controlled exclusion log that records the episode ID, task, site, operator, object, collection date, failure reason, reviewer, disposition, replacement ID, and final resolution.

Test a Pilot and Track Rejection Rates

Before scaling, run a 50–100 episode pilot to check how well the rules reject bad data. Include expected tasks, objects, settings, operators, sensor setups, occlusions, recovery cases, and annotation depth. Apply production acceptance rules and test the actual ingestion path: metadata parsing, calibration, coordinate-frame interpretation, and time alignment. Scale only after the batch meets agreed thresholds and you've addressed recurring defects.

Track four rates separately: first-pass rejection, rework success, final rejection, and replacement acceptance. Report both counts and percentages, and keep open reviews separate. Break results down by task, site, operator, object, failure reason, sensor setup, and collection date. Link defects to correction hours and replacement charges. Then use the pilot's rejection rate and measured costs to estimate the actual cost per accepted episode before ordering at scale.

Assign Correction and Replacement Responsibilities

Define review windows, defect rules, rework and replacement deadlines, rejection limits, and whether replacements come at no extra charge. Require raw data, annotations, metadata, calibration files, checksums, quality reports, and collection logs in a machine-readable format. Supplier-funded replacements must still pass the buyer's independent acceptance review.

Require the supplier to document collection quality, defect rates, turnaround times, and acceptance terms in the contract so those requirements can be enforced.

Conclusion: Buy Accepted Episodes, Not Delivered Files

The lowest quote often isn’t the lowest-cost source once you count QA, rework, and re-collection. What matters is cost per accepted episode. Count every buyer cost - not just the quoted price.

Set acceptance criteria and run a pilot before buying at scale. A lower quote saves money only if the accepted data meets your task, safety, privacy, licensing, and schedule requirements. Buy on measured yield, not headline price.

Start with the task spec, acceptance rules, and delivery format.

FAQs

When does rework cost more than re-collection?

Rework costs more than re-collection when the labor and time spent salvaging a defective sample cost more than recording it again.

Choose re-collection when hands leave the frame, motion blur hides contact, required steps are missing, or privacy fixes would remove evidence needed to assess the task. Reject and replace samples that can’t support a defensible label instead of spending operator, technician, and station time on unreliable fixes.

How reliable is a small pilot’s rejection rate?

A small pilot’s rejection rate is a planning assumption, not proof of production quality. Test 100–200 samples - or 20–50 hours - against acceptance thresholds defined in writing before testing.

One pilot can’t show that results will hold up over time. Audit samples that represent different operators and difficulty levels, then check the results against a gold set of adjudicated examples. If rejection exceeds approximately 15%, treat it as a systemic issue and revise the process.

Can rejected episodes create gaps in task coverage?

Yes. Rejected episodes can leave large gaps in coverage of required object sizes, lighting conditions, or recovery behaviors. Relabeling can’t fix missing data. A batch can fail coverage requirements even when every episode is technically complete.

Track the number of accepted episodes for each required condition. More data from already-covered scenarios won’t fill those gaps. When episodes are rejected for missing requirements, you’ll often need to collect replacement data targeting those conditions.

Related Blog Posts