Skip to main content
The examples on this page use Northpine, a fictional outdoor-gear shop whose refund-exception decision grants, escalates or denies refund requests outside its standard return policy. How Moonbounce works introduces it.
An evaluation is Moonbounce’s answer for an item (Evaluate content). Feedback goes the other way: it says what the right answer was. Moonbounce stores every evaluation as it was and never rewrites it, so feedback sits beside the evaluation. Previews test changes against it and suggestions learn from it. This page follows the owner’s feedback on Northpine’s ticket C, then covers each part of a submission, who can send it, and what it counts for.

​
The smallest submission

Ticket C is the tent that arrived two weeks late. It scored 0.540 and landed on escalate. Northpine’s owner knows it should have been grant, so they say so:
The API answers 201 with the submission’s id and the evaluation it was bound to:
A submission has four parts. about says which evaluation you’re judging. claims holds one or more statements about it. source says who is speaking. reason says why, in your own words, and can be up to 4,000 characters. The reason matters more than it looks: when Moonbounce writes a suggestion, it reads the reasons on the cases it learns from. This reason is how the loop finds out that late deliveries are missing from the spec.

​
What the feedback is about

about takes one of two forms:
  • {"evaluation_id": "evl_..."} names the exact answer you’re judging. Prefer it whenever you kept the id from the evaluate response.
  • {"decision_id": "refund-exception", "item_id": "t-001"} means the item’s latest primary evaluation in your key’s environment, at the moment the feedback arrives. A primary evaluation is one your app made through POST /v1/evaluate. Seed examples never count, so feedback by item can’t land on a seed: for an item that has only seed evaluations, the 404 says so and names the seed’s evaluation_id, which you can address directly (a decisions:feedback-only key gets the plain message). The response sets resolved_from_item: true and tells you which evaluation_id it picked.
An evaluation in another environment is invisible to your key. Both forms answer 404 when they find nothing. A verdict claim (below) needs evaluation_id: it judges one specific answer, so a lookup by item is refused with a 400 on about.

​
Claims

Each claim is one statement about the evaluation, and its kind says which. Moonbounce checks every claim against the evaluation’s own spec and cut points, the ones that were sent with that call.

​
outcome: the right outcome

This is the claim to send most often. It names the outcome the item should have had, and that outcome must be one of the evaluation’s own: below or one of its cuts. For Northpine that is deny, escalate or grant. Case matters, and a wrong name gets a 400 that lists the outcomes that exist:

​
target: whether the target statement is true

The target is the statement the score estimates. For Northpine it is “The refund exception should be granted.” A target claim says whether that statement holds for this item. It needs a spec that declares a target.

​
Middle outcomes say nothing about the target

Outcome and target claims are linked, but only at the ends of the scale. Northpine’s outcomes run from deny (below 0.38) to grant (from 0.60):
  • grant is the highest outcome, so it implies the target is true: the refund should be granted.
  • deny is the lowest outcome, so it implies the target is false.
  • escalate sits in the middle and implies nothing. “A person should look at this” doesn’t say whether the refund should end up granted.
A submission whose outcome and target contradict each other is refused. For example, deny with target: true gets 400 naming the later of the two claims (outcome "deny" implies the target is false, but the target claim says true). escalate with either target value is accepted.

​
verdict: approve or reject our answer

approve means the outcome Moonbounce returned was right. reject means it was wrong. A verdict needs about.evaluation_id and an evaluation whose outcome was actually shown to someone. That means a primary evaluation that decided: it has a score and an outcome. An evaluation that abstained (it has no score) only carries a fallback outcome, and a seed evaluation was never shown, so both refuse verdicts. When a decision has only two outcomes, reject tells Moonbounce the right answer. With three or more, it only says “not this one”. A rejected escalate on ticket C doesn’t say whether the answer was grant or deny. Previews can only replay a case whose agreed answer names an outcome, so if you know the right outcome, send an outcome claim instead.

​
feature: the right value of one feature

A feature claim gives the right value of one feature, either correcting the engine or confirming it. The feature must name a boolean feature of the evaluation’s spec, and a 400 lists the ones that exist. Both uses matter on ticket C. The engine answered 0.95 for outside_window and 0.05 for defective, and both answers are right: the request is past the window, and nothing is broken. Confirming them (outside_window: true, defective: false) tells a suggestion that neither feature’s answer was wrong, so the spec must be missing one. Without the confirmations, a suggestion could blame either of them, since firmly flipping either one would turn ticket C into grant. A correction looks the same, with the value the engine got wrong: had the engine answered 0.8 for item_used on a tent that was never set up, you would send item_used: false. A feature claim refines a case but doesn’t decide one. It tells a suggestion which features were right or wrong in a case that already has an agreed outcome. On its own it sets no right answer for the item. A feature claim is stored against the feature as it was evaluated, identified by the feature’s node hash (see Versions and hashes). Renaming the feature keeps the claim. Changing the prompt’s wording, its own guidance or criteria, or the spec’s guidance gives the feature a new hash, and the claim stops applying: an answer judged against old wording says nothing about new wording.

​
input_broken: the content can’t be judged

Send this when the content itself was the problem. problem is one of truncated, wrong_language, bad_extraction, unreadable or other. Previews and suggestions set the evaluation aside, whatever else was said about it.

​
action: what your app did

An action claim records what your app actually did with the outcome, in your own words (up to 128 characters), with an optional logic_version for your own code. It describes what happened, not what was right, so it never sets an answer. Use it to keep Moonbounce’s record of the item next to what your system did.

​
Rules for one submission

A submission holds 1 to 65 claims. Within one submission:
  • at most one outcome, one target and one verdict claim, and one claim per feature;
  • an outcome and a target must not contradict each other at the ends of the scale.
A claim that breaks a rule, or that the evaluation can’t take, gets a 400 whose param names it (claims.<i> or a field inside it) and whose message says what is allowed. Nothing in a refused submission is stored. You can also set unsure: true when the person judging isn’t confident. It is stored with the submission and included in the export.

​
Sources

source says who is speaking. That decides what the feedback counts for. A key with decisions:develop can send every source. A key that is missing the scope a source needs gets 403 INSUFFICIENT_SCOPE.
  • owner is the decision owner’s judgment: yours, or that of someone you trust to decide. All owner feedback is one voice, whoever sends it. The latest owner submission with an outcome, target or verdict claim decides the case and replaces earlier owner judgments. Owner feedback can’t name an actor (403), because the owner is you.
  • Reviewers answer on the Review page or through the review API. Each reviewer gets one vote, and a tie leaves the case contested, which previews leave out.
  • Seeds are the labeled starting examples described below. They decide a case only when no owner or reviewer has judged it.
  • app_native is your app’s own feedback about the outcome it acted on, for example a user approved or reverted it.
  • app_resolution is how your own process finally settled the case, for example an appeal.
The owner outranks reviewers, and reviewers outrank seeds. The two app_ sources are weak feedback: they never decide a case. They can name an actor, your own id for whoever produced the feedback, which is recorded under your key’s namespace.

​
What weak feedback still does

app_native and app_resolution never decide a case’s right answer, but they aren’t ignored:
  • They can fix a case’s split. The first feedback a case gets decides whether that case can ever be held out to test suggestions (see Train and holdout). If a native verdict arrives before your review, the case goes to training for good.
  • Their reasons reach the suggestion writer. When another source has decided a case, the reasons from every live submission on it, weak ones included, are shown alongside it.
  • input_broken from any source sets the evaluation aside.

​
What decides a case

For each evaluation, Moonbounce works out one agreed answer from its live feedback. The owner decides if they have judged it. Otherwise the reviewers’ majority decides, and the seed decides only if nobody else has. Previews replay a case only when it is a primary or seed evaluation, its content is stored, and its agreed answer names one of the preview’s outcomes. A case with only a target claim, or only a rejected verdict among three outcomes, is counted as left out and not replayed.

​
Train and holdout

Moonbounce keeps some cases back so it can test a suggestion on cases the suggestion writer never saw. Each case is assigned once, at its first feedback, and keeps that assignment:
  • owner or reviewer first: about 70% go to train and 30% to holdout, chosen by a hash, so the same case always lands on the same side;
  • app_native or app_resolution first: train;
  • a seed: seed, used for fitting only.
A case is an item together with any items whose text is the same once case, digits and id-like tokens are ignored, so a templated message sent twice with different order numbers is one case, not two. Cases are per environment: the same item or text in test and in production is two cases, each with its own split, so test feedback never fixes or burns a production case. A held-out case becomes burned, which means training from then on, when a preview lists it individually, when it merges with a near-duplicate case, or when its text arrives later as a seed example. After any of those, the case can no longer be trusted as a test of something that never saw it.

​
Corrections

Send a correction as a new submission that names the one it replaces. Here Northpine’s owner resends ticket C’s feedback with the two feature confirmations described above:
The response carries a new id, fbk_0199f2d1-7e8f-7a90-b1c2-d3e4f5a6b7c8. From then on that submission is ticket C’s owner feedback, and the first one shows superseded_by with the new id. supersedes must name a submission by the same actor, on the same evaluation, that nothing has replaced yet. Anything else is a 409. For feedback sent with an API key, the same actor means the same key, plus the same actor value when you named one. Only the latest submission in a chain counts. The earlier one stays readable and shows superseded_by. Replacing owner, reviewer or seed feedback needs decisions:develop. A decisions:feedback key can replace only its own app_native and app_resolution submissions. Nobody replaces someone else’s judgment. A different person adds their own submission, and the sources above decide which one counts.

​
Reading feedback back

GET /v1/feedback/{feedback_id} returns any submission in your key’s environment as stored: one from POST /v1/feedback, a seed example’s, or a review answer. It shows the claims as they were sent, plus source, channel (the surface it came through, set by the server), actor, supersedes, superseded_by and the case’s split. A reviewer’s answer on a spot check also carries after_reveal: false on the blind answer, true on a later change of it (Review), except while its judgment is withheld. A replaced answer keeps the reason_pending it had: true there means its reason never arrived before it was replaced, so read the latest answer in the chain. Ticket C’s case landed in train, so its feedback reads back in full. A submission on a held-out case reads back without its judgment, even to its author. Here is the owner’s feedback on another Northpine ticket, one that landed in holdout:
withheld: "holdout" means the claims, reason and unsure flag are kept back. A later preview tests a suggestion on held-out cases, and that test means something only if neither the suggestion writer nor the agent asking for it has read their answers. You still see that the submission was recorded and how many claims it holds. The judgment comes back once the case is burned. withheld: "unassigned" is the same for an item that has no split yet.

​
Seeds

A new decision has nothing to test a change against. Seeds fix that: about a dozen labeled examples you send once, so previews have cases to replay from day one. POST /v1/decisions/{decision_id}/seeds takes the version (an inline spec, or the structure hash of a registered one, as on evaluate), weights, outcomes and the examples. Here are three of Northpine’s twelve, sent to POST /v1/decisions/refund-exception/seeds:
The hash is the structure hash of Northpine’s spec. Sending "spec" with the contents of moonbounce/refund-exception.json instead works too, and registers it. The decision comes from the path, so the body has no decision_id. Each example takes the same subject, context, summary and metadata fields as an evaluate request, plus claims, reason and unsure. Examples can’t carry a verdict, since nothing was shown yet. Outcome claims need outcomes in the request, because a claim is checked against the cut points. Each example runs through the real engine as an evaluation in seed mode. The response lists each one with its evaluation_id, feedback_id, outcome, score and agreed: true when every checked outcome, target and feature claim agrees (feature answers use p ≥ 0.5). A single feature conflict makes it false. The top-level agreed counts results whose agreed is true. Seed-01 would come back escalate with agreed: false. That’s the carrier-delay gap showing up before any traffic. agreed is null when nothing could be checked: the evaluation abstained, or the example carries only a target claim and landed on a middle outcome. conflicts lists each feature claim the engine’s answer disagrees with. Each result also carries cost: {engine_calls, cache_hits, input_tokens, usd_micros} with the same fields as a preview. engine_calls counts upstream engine calls on evaluate and seeds; one call can answer several features (Jev compiles them). On previews it counts features sent to engines instead. The summary’s cost sums those fields across the batch, using each evaluation’s recorded usage (including its rounded money amount). usd_micros is millionths of a US dollar. Only previews use the feature cache today; evaluate and seeds report cache_hits: 0. getEvaluation returns the same per-example cost when a usage row exists; otherwise it omits cost. Fresh seed responses always include both levels of cost. A replay of a snapshot stored before costs were reported omits cost from both the summary and every result until those snapshots expire, within 24 hours after deployment; replayed cost is the original spend, not a new charge. Suggestion generation’s own cost isn’t in per-request cost; the usage report shows it under other_spend. Things to know about seeds:
  • All or nothing. Every example’s content and claims are checked before any engine call, and a 400 names the example (examples.<i>...). If any example’s engine call fails, the batch is refused with 502 ENGINE_FAILED, details.items names every failed example, and nothing is saved. A batch can hold up to 200 examples. A batch too big to evaluate in one request is a 422 telling you to send smaller batches.
  • Never traffic. Seed evaluations are real evaluations, but feedback by item never lands on them, the review queue never offers them, and they never count as a version serving traffic.
  • The weakest evidence. Seed cases are used for fitting and are never held out. A preview tested only on seeds carries the Starting examples label, the weakest there is (Suggestions and previews).
  • One key per batch. One Idempotency-Key covers the whole batch, so sending it again replays the first result without calling an engine.

​
Exporting feedback

GET /v1/decisions/{decision_id}/feedback returns the raw feedback on a decision, oldest first and paged with cursor. Each submission comes with the evaluation it judged and its split. It needs decisions:develop, is HTTP only (not an MCP tool), and never lists held-out feedback: those submissions are only counted in withheld on the first page.

​
Retries

POST /v1/feedback and the seeds endpoint honor Idempotency-Key the way evaluate does: retry with the same key and you get the first result with Idempotent-Replayed: true, never a second submission. Who sent it is part of the request: for an API key, that is the key itself plus any actor you name. So the same Idempotency-Key and body sent through a different API key, a rotated one included, or with a different actor, is a 422 rather than a replay.

​
Next

Sending feedback for every item doesn’t scale. Review shows how Moonbounce picks the few items a day worth your judgment, including the blind spot checks that measure how often it agrees with you.