The Hidden Costs of Cheap Training Data: Rework, Rejection Rates, and Re-Collection
Leela Yanamaddi
October 24, 2026
I’d compare training data quotes by cost per accepted sample - not price per file. In this article’s example, a $40 quote becomes $53.75 per usable demonstration after corrections, extra QA, and a 20% rejection rate.
Before buying, I’d check four things:
- What passes: Define task, coverage, sensor, label, consent, and file requirements.
- What needs fixing: Separate repairable errors from recordings that need replacement.
- What you’ll pay: Include review, rework, re-collection, and internal labor; track delay costs separately.
- Who pays for failures: Test 100–200 pilot samples, then put correction deadlines and replacement terms in writing.
My rule: <u>tie payment to accepted data</u>. A low quote only helps if the output is ready for your model.
2. Data Defects That Lead to Rework and Re-Collection
Cheap data gets expensive when defects turn delivered samples into rework, quarantine, or replacement.
Incorrect Labels and Unusable Footage
Some defects can be repaired. Others require new recordings.
Relabel a failed grasp only when the recording clearly shows the outcome. Fix action boundaries only when a shared rule exists. Mark hidden outcomes as unknown.
Replace wearable-camera footage when hands leave the frame or motion blur hides contact. Review video, motion, and force streams together: each can look valid while referring to a different moment. Fix timing only when reliable timestamps or alignment references support the change. Otherwise, reject the episode.
The table separates defects you can repair from those that require replacement or quarantine.
| Defect | Action | Requirement and cost consequence |
|---|---|---|
| Failed grasp labeled successful | Repair | A visible, unambiguous outcome; relabeling and a second review |
| Inconsistent action boundaries | Repair | Relevant frames and a shared boundary rule; timestamp edits and cross-episode checks |
| Hands out of frame or blurred contact | Replace | A replacement recording and another QA pass |
| Sensor streams out of sync | Repair / Replace | Reliable alignment evidence and revalidation for repair; otherwise, replacement |
| Missing object variant or failure condition | Replace | Targeted re-collection to fill the coverage gap |
| Missing consent or consent that does not cover this use | Quarantine | Quarantine until verified; replacement if permission cannot be established |
| Unredacted personal information | Repair / Replace | Approved redaction that preserves task evidence; re-review or replacement |
| Broken manifest or incomplete package | Repair / Replace | Repackaging and validation if source files and relationships are intact; otherwise, replacement |
Gaps in Coverage, Consent, and Delivery Records
Coverage gaps aren't label errors. Relabeling alone cannot fix missing coverage, and a batch can fail even when every episode is technically complete. Track accepted counts by required condition - object size, operator, lighting, clutter, or recovery behavior - not just total deliveries. More recordings of conditions already covered cannot replace targeted re-collection.
Missing records call for a different response. You may be able to restore an operator identifier or collection date from verified source records. But missing consent isn't a metadata typo. Quarantine affected episodes until authorization and permitted use are established. Review faces, badges, screens, and shipping labels for privacy issues. Approved redaction works only when it preserves task-critical evidence.
Packaging defects create integration work even when the recordings are sound. Before delivery, validate matching episode IDs, required modalities, timestamps, calibration files, annotation references, and checksums. Rebuild a broken manifest when the source files are intact. Reject an episode if a required stream is lost or cannot be reliably linked to the task. Track repair, quarantine, and replacement separately so you can price each failure mode.
3. Calculate the Total Cost of Accepted Data
Training Data Costs: From $40 Quoted to $53.75 Accepted
After reviewing defects, price the output you accept - not just the volume delivered.
Cost per accepted demonstration = total attributable cost ÷ accepted demonstrations. Start with the quoted price. Then add only costs outside its scope: QA, corrections, replacement collection, ingestion, and delivery. Mark each charge as included, excluded, or uncertain so you don’t count it twice.
Track internal engineering, QA, and project-management hours at fully loaded rates. Report schedule impact separately.
Worked Example: From $40 to $53.75 per Demonstration
Illustrative figures: Of 1,000 delivered demonstrations, 800 ultimately pass: 600 without correction and 200 after correction. The remaining 200 are rejected. Assume the $40 quote includes collection and annotation, while the charges below are additional.
| Cost item | Calculation | Amount |
|---|---|---|
| Initial collection and annotation | 1,000 demonstrations × $40 | $40,000 |
| Corrections to accepted demonstrations | 200 demonstrations × $5 | $1,000 |
| Extra QA and delivery | Stated incremental cost | $2,000 |
| Total spending | $40,000 + $1,000 + $2,000 | $43,000 |
| Accepted demonstrations | Final accepted output | 800 |
| Cost per accepted demonstration | $43,000 ÷ 800 | $53.75 |
Divide spending by delivered volume, and you get $43.00 per demonstration. But each accepted demonstration costs $53.75. Before calling that a fully loaded unit cost, add any buyer-side ingestion or labor costs.
Account for Replacements and Delivery Delays
Buying 200 replacements at $40 each adds $8,000; $500 in review lifts total spending to $51,500. If all replacements pass, the batch reaches 1,000 accepted demonstrations at $51.50 each; if they fail, keep their cost in the total and exclude them from accepted output.
Track planned and actual dates for collection, review, ingestion, and acceptance. Use your own staffing assumptions to estimate the cost of delays - not a generic surcharge.
For illustration, a 10-business-day delay blocking two engineers for three hours daily at $125 per hour represents $7,500 in lost engineering time.
Report this separately as a schedule-impact estimate, not a supplier charge. Don’t count those same hours again as internal labor.
Test Different Acceptance Rates
Acceptance rate = accepted demonstrations ÷ delivered demonstrations × 100 for a given batch and review stage.
Track first-pass acceptance separately from acceptance after correction. In the worked example, those rates are 60% and 80%, respectively. Lower acceptance can also mean more correction costs, replacement orders, and delays.
Use the same calculation to see how acceptance rates affect unit cost.
The hypothetical scenarios below hold initial volume at 1,000 demonstrations and spending at $43,000, including $1,000 in corrections and $2,000 in extra QA and delivery; they exclude replacements and are not industry benchmarks.
| Acceptance after correction | Accepted demonstrations | Total spending | Cost per accepted demonstration |
|---|---|---|---|
| 70% | 700 | $43,000 | $61.43 |
| 80% | 800 | $43,000 | $53.75 |
| 90% | 900 | $43,000 | $47.78 |
4. Set Quality Gates and Contract Terms Before Collection
Turn the cost-per-accepted-sample math into enforceable contract terms. Spell out who pays for avoidable rework, rejected samples, and repeat collection so those costs don’t fall on the buyer.
Define What Passes, Needs Repair, or Gets Rejected
Write acceptance criteria into the statement of work. Name the final reviewer and require evidence for every pass or rejection. Allow repairs only when the underlying evidence is still valid. Reject samples when a fix would require guessing or collecting new data. Use thresholds like these to draw that line.
These thresholds are examples to adapt before signing, not universal standards.
| Criterion | Pass threshold | Validation method | Repairable defect | Automatic rejection condition |
|---|---|---|---|---|
| Task validity | Follows the protocol and reaches the defined outcome; evidence supports the success/failure label | Reviewer checklist and outcome verification | Missing phase label or unclear borderline outcome | Wrong task, unsafe action, failed reset, or fabricated success |
| Coverage | 100% of required task, object, site, operator, and outcome quotas, unless substitutions are allowed | Manifest and quota report | Underrepresented noncritical category that can be recollected | Missing required category or material shortfall |
| Hand and workspace visibility | Required hands, tools, objects, and interaction area visible in at least 95% of task frames; every critical interaction visible | Frame sampling or automated visibility check | Short occlusion that does not affect the labeled action | Critical grasp, contact, or workspace event is hidden |
| Image quality | Resolution, focus, exposure, and motion blur meet the written minimum | Automated quality metrics and human review | Cropping, trimming, or metadata correction | Persistent blur, unusable exposure, damaged file, or obstructed lens |
| Sensor synchronization | Streams align within the project tolerance | Timestamp audit and synchronization test | Correctable timestamp offset with intact raw streams | Missing stream, irrecoverable drift, or excessive dropped frames |
| Annotation accuracy | At least 90% shared-label agreement; Cohen’s kappa above 0.80 for action labels; boundaries within ±2 frames at 30 fps, where appropriate | Blind reannotation, adjudication, and kappa or F1 measurement | Label correction or boundary adjustment | Systematic error, missing required labels, or unresolved ambiguity |
| Consent | Valid permission and permitted-use records for 100% of applicable samples | Permission audit and manifest reconciliation | Missing permitted-use record | Missing consent, uncertain rights, or unverifiable origin |
| Provenance | Source, collector, session, and processing history recorded for every sample | Record-to-manifest reconciliation | Missing optional descriptive metadata | Unverifiable origin |
| Delivery structure | 100% schema compliance, matching checksums, distinct IDs, and required metadata | Automated validation and test import | Schema or filename correction | Unreadable files, duplicate IDs, missing core metadata, or incompatible format |
For synchronization, a project might require alignment within 10–50 milliseconds, depending on model needs.
Use a Pilot to Prevent Repeat Defects
Review 100–200 pilot samples against the specifications. Include hard cases: rare classes, occlusion, and multiple operators. Check defects and quotas - not just the overall pass rate - to see whether the same failures recur. Have annotators independently label an overlapping subset, resolve disagreements, and save the adjudicated examples as a gold set before scaling.
Put response rules in the contract:
For example, require corrections within five business days, allow no more than two resubmissions, and require supplier-funded replacement collection for supplier-caused rejection.
Set a numeric defect ceiling before collection starts. Pause affected collection if consent is missing, critical visibility failures recur, synchronization falls outside tolerance, or pilot acceptance falls below the contract minimum. Quarantine affected samples. Resume only after a corrective-action plan, a passing mini-batch, and approval.
Compare Quotes by Cost per Usable Sample
Score every supplier with the same worksheet, acceptance rules, and coverage requirements. Require supplier-funded recollection for supplier-caused unusable footage and missing consent. Treat recoverable annotation errors and incomplete metadata as included corrections. Record buyer-requested scope changes separately.
Tie payment to accepted samples or accepted milestones - not raw uploads.
| Worksheet item | What to record before approval |
|---|---|
| Headline price | Price per defined delivery unit; whether QA and transfer are included |
| Expected acceptance rate | Pilot first-pass and post-correction results, including results by required category |
| Rework policy | Included corrections, responsible party, deadline, and resubmission limit |
| Replacement policy | Who funds recollection and the replacement delivery deadline |
| Coverage assurance | Mandatory quotas, permitted substitutions, and responsibility for shortfalls |
| QA evidence | Sample-level defect logs, gold-set results, synchronization checks, consent records, and correction history |
| Accepted data delivery date | Date accepted files must pass import; delay notification and recovery-plan requirements |
| Internal review costs | Buyer QA, engineering, and project-management hours at fully loaded rates |
| Effective cost per usable sample | All attributable invoiced and internal costs divided by accepted output; identify exclusions and keep schedule-impact estimates separate |
5. Conclusion: Buy Accepted Coverage, Not Raw Volume
A low quote stops being cheap when you add rework, replacement collection, and delays.
Set buying targets using cost per accepted, usable sample. Use pilot results to estimate usable yield, rework rate, and QA effort. Budget reserves for corrections and coverage gaps, and keep schedule-impact estimates separate.
Let those pilot results guide the contract terms. For Evlo.ai managed collection, define required coverage, formats, acceptance tests, and responsibility for replacements before collection starts. Managed collection still needs proof that the data fits the model.
Buy coverage your model can use - not just collected volume.
FAQs
How can I budget for rejection rates before a pilot?
Before the pilot, budget around an assumed acceptance rate, set measurable acceptance thresholds, and record reasons for rejection.
During the pilot, estimate the acceptance rate using 10–20 hours per scenario or about 30–50 episodes per task variant. Calculate:
Cost per accepted demo = total session cost (including rework and rejection handling) ÷ accepted demos
If the rejection rate exceeds about 15%, treat it as a process or collection problem and revise the process or collection approach accordingly.
How do I know whether my pilot reflects production quality?
Set written acceptance thresholds and clear exit criteria before labeling begins. Then score your pilot using quality checks that mirror production conditions.
Log why episodes are rejected. Audit both individual episodes and the dataset as a whole, accepting only episodes that meet requirements for completeness, synchronization, schema loadability, and annotation accuracy.
Test candidate data slices on a trusted holdout to verify gains in task success and recovery - not just footage that looks clean.
How can I verify accepted data improves my model?
Verify improvement with held-out physical tests - not data volume. Start with 30 to 50 pilot episodes per task variant to set a baseline. Then compare task success, recovery success, operator interventions, and completion time against a robot-only baseline.
Scale data collection only when trials show that the new data addresses a measured failure. Version datasets and run regression tests to check that corrections and additions maintain or improve model behavior.