Rotor Town

Learning from feedback · Benchmark results

Your decisions.
Better with feedback.

Support tickets. Invoice exceptions. Security alerts.

See how offline feedback tuning compared with context-informed Jev.

21,600 synthetic cases3 business workflows
Research previewTuned results use exported feedback, evaluated outside hosted Rotor.This approach isn’t available in hosted serving yet.

Customer service

96.9%

Offline tuning accuracy, not hosted

88.7% of cases answered automatically

Automatic answer accuracy

Offline tuning96.9%
Jev + context89.4%

Jev automated 82.8% of cases.

Invoice processing

96.5%

Offline tuning accuracy, not hosted

87.8% of cases answered automatically

Automatic answer accuracy

Offline tuning96.5%
Jev + context94.7%

Jev automated 78.5% of cases.

Security triage

98.3%

Offline tuning accuracy, not hosted

95.7% of cases answered automatically

Automatic answer accuracy

Offline tuning98.3%
Jev + context94.9%

Jev automated 81.7% of cases.

Accuracy measures automatic answers. Remaining cases needed human review.

What does that look like?

Illustrative synthetic cases from the evaluation.

Customer service

A small issue. A long wait.

Impact
Cosmetic
Waiting
107 hours
Repeat contacts
7
Workaround
Reliable
Tuned decisionRoute to a specialistMatched the expected routing.

Birch team · Standard channel

Invoice processing

An invoice with missing evidence.

Evidence
Missing
Delivered
97%
Previous clean invoices
25
Dispute
None
Tuned decisionKeep standard handlingMatched the expected routing.

Atlas team · Routing decision, not payment approval.

Security triage

Routine activity. Unusual signals.

Activity
18.1× normal
Corroborating signals
6
Authorization
Absent
Status
Ongoing
Tuned decisionRoute to a specialistMatched the expected routing.

Birch team · Standard channel

Selected correct examples; aggregate scores include every test case.

Your team’s decisions become useful feedback.

  1. 01Start with examples

    Use historical decisions or collect reviewer corrections.

  2. 02Learn the patterns

    Capture preferences that instructions alone might miss.

  3. 03Measure on new cases

    Track accuracy and review rates together.

Explore Rotor
Full results, methods, and limitations

Synthetic cases do not establish customer accuracy.

CatBoost ran outside Rotor’s hosted serving pipeline.

These structured tests did not use Looking Glass.

Native Rotor classifiers lost all three overall comparisons.

Errors and human review

Counts below are per 1,000 incoming cases.

Accuracy includes only automatically answered cases.

Human review means checking the routing decision.

Customer service

7,200 cases · 1,200 independent families
MethodWrong automaticHuman reviewAutomatic accuracy
Rotor classifierLearned from feedback360 feedback examples99.6205.387.5%
Rotor export + CatBoostExternal training workflow1,080 feedback examples27.9113.296.9%
Jev + customer context32 similar examples1,080 feedback examples87.8171.989.4%

Rotor classifierWorse tradeoff

Rotor export + CatBoostImproved tradeoff

Invoice processing

7,200 cases · 1,200 independent families
MethodWrong automaticHuman reviewAutomatic accuracy
Rotor classifierLearned from feedback360 feedback examples93.8123.589.3%
Rotor export + CatBoostExternal training workflow1,080 feedback examples31.1122.496.5%
Jev + customer contextSummary plus 32 matching examples1,080 feedback examples41.7215.494.7%

Rotor classifierWorse tradeoff

Rotor export + CatBoostImproved tradeoff

Security triage

7,200 cases · 1,200 independent families
MethodWrong automaticHuman reviewAutomatic accuracy
Rotor classifierLearned from feedback1,080 feedback examples80.4227.989.6%
Rotor export + CatBoostExternal training workflow1,080 feedback examples16.042.898.3%
Jev + customer context32 similar examples1,080 feedback examples41.8182.694.9%

Rotor classifierWorse tradeoff

Rotor export + CatBoostImproved tradeoff

Which preferences were learnable?

Each workflow included three simulated product teams.

Preferences included accumulated signals, exceptions, and combinations of conditions.

See every policy comparison
WorkflowPolicy typeRotor classifierExport + CatBoost
Customer serviceSignals that add upAutomation unprovenImproved tradeoff
Customer serviceRules with exceptionsWorse tradeoffImproved tradeoff
Customer serviceConditions that work togetherWorse tradeoffImproved tradeoff
Invoice processingSignals that add upAutomation unprovenImproved tradeoff
Invoice processingRules with exceptionsWorse tradeoffImproved tradeoff
Invoice processingConditions that work togetherWorse tradeoffImproved tradeoff
Security triageSignals that add upAutomation unprovenImproved tradeoff
Security triageRules with exceptionsWorse tradeoffImproved tradeoff
Security triageConditions that work togetherWorse tradeoffImproved tradeoff

Feedback you can reuse

  1. Capture inputs and reviewer corrections.
  2. Preserve sources, groups, and evaluation splits.
  3. Export supervised examples or chat training records.

We verified imports and reviewer judgments through hosted exports.

Unresolved disagreement does not become an invented answer.

How we compared

An error counted as 1; a review as 0.2.

These are illustrative costs, not measured financial losses.

Improvement required lower cost and at least 80% automation.

Both requirements had to clear adjusted uncertainty bounds.

Methods and uncertainty

Per workflow: 1,080 feedback examples available.

Calibration: 720. Method selection: 360.

Inputs were structured fields with synthetic reviewer decisions.

All methods used separate calibration examples.

Related cases stayed together during evaluation.

We froze methods before generating fresh test cases.

20,000 bootstrap samples; correction across 24 comparisons.

Positive ranges favor the learned method over Jev.

Ranges describe this synthetic distribution and these fixed models.

Cost improvement · percentage points
WorkflowMethodAdjusted rangeAutomation range
Customer serviceRotor classifier-3.41 to -0.2477.2%–81.7%
Customer serviceRotor export + CatBoost6.03 to 8.3687.2%–90.1%
Invoice processingRotor classifier-4.73 to -2.0486.4%–88.9%
Invoice processingRotor export + CatBoost2.03 to 3.8486.3%–89.2%
Security triageRotor classifier-6.09 to -3.3674.7%–79.8%
Security triageRotor export + CatBoost4.44 to 6.3794.8%–96.5%

Rotor selected linear classifiers for all three workflows.

CatBoost combined 200 small decision trees.

Jev: jev-1.13.0. External learner: CatBoost 1.2.10.

Jev received the best development-selected context setup.

We did not compare against Jev plus external training.

No end-to-end latency or total delivery savings are claimed.

Two timed-out requests were repeated; original billing is unknown.

Download aggregate results

Customer service reference · Invoice reference · Security reference

Domains follow vendor examples, not measured market popularity.

Our independently authored cases differ from TypeSafe’s benchmark.