Review

How Moonbounce picks a few items a day for you to judge, and how blind spot checks measure agreement.

The examples on this page use one decision, refund-exception, from an online store’s support app: should a refund request outside the store’s standard return policy be granted, escalated to a person, or denied? How Moonbounce works introduces it.

Feedback showed what you can tell Moonbounce about an evaluation and what each kind of feedback counts for. Review is how Moonbounce asks you for it: a short daily queue of the items it most needs a person to judge, plus random spot checks that measure how often it agrees with you.

Why review

A busy decision makes more evaluations than anyone can label, and most of them teach nothing: a ticket whose grant_score is 0.05 and lands on deny is rarely news. Review spends your attention where it pays. Each day Moonbounce picks a few items that sit near a decision line, that hinge on a shaky answer, or that look unlike anything reviewed so far. It mixes in a random sample so it can say how often it’s right without fooling itself.

Where you review

The Review page is part of the web app at app.moonbounce.ai/review. It works on a phone (one item at a time, one column, the buttons at the bottom within reach of your thumb) and uses the extra room on a wide screen, and it works without JavaScript. Sign in with your Moonbounce account. The page needs no API key.

On a phone the page shows less by default and keeps everything one tap away, with no JavaScript. The “Why items are here:” line becomes a closed Why these items? that opens to the same list. Rows on Done in a narrow list leave out the words after the badge (such as “was Remove”); open the item to read them. On an item, “Why you’re seeing this” shows only the reason and a few words (such as “Borderline: near a cutoff”), and tapping it opens the item’s own sentence. The muted note about what your answer does and the hint about Change, Not sure and Skip sit behind How answering works, just above the answer buttons, and the “Close to the line” pill is left out because the reason already says Borderline. Wide screens show all of it inline.

Opening /review shows your inbox. The header reads “N to review · M done today (UTC)”, where today starts at 00:00 UTC, so it differs from the all-time count on the Done tab, above Continue reviewing and the To review and Done tabs. Their counts span decisions. The decision list narrows the inbox to one decision: All, then every decision with items on the current tab, in the order they were created, each with its count for the tab. On a wide screen it is a sidebar headed “Decisions · to review” or “Decisions · done”, naming the tab those counts are for, beside the list that scrolls on its own however many decisions you have; on a phone it is one menu above the list that names the current filter, such as “All decisions ▾”. Number keys 1–9 pick its first nine decisions. Each tab shows 50 rows. More loads the next 50 on To review.

To review rows line up in columns, named in a muted header row: the title, the decision (hidden while you are filtered to one), why the item was picked, and Moonbounce’s call. A line above the list, “Why items are here:”, says what each reason on the page means: Borderline (value near a cutoff), Low confidence (Moonbounce was unsure), Unusual (unlike items reviewed so far) and Spot check (picked at random, Moonbounce’s call hidden until you answer). On a phone the title takes two lines and the rest follows on one line. A decision whose result is a value rather than an outcome shows it, such as “Value 0.51”. Spot checks hide the call. Titles use the first line of a text state, else the first text value of an object state, or “Review item” if content wasn’t stored. Your integration’s summary (the text your system sent with the item) appears on the item page, saved reviews and reveals with a plug icon and “Note from your integration”, never as the title; inbox rows leave it out. Unanswered spot checks hide it; summaries that repeat the title are omitted. Continue reviewing opens the first open item, or you can choose a row. It also appears on the Done tab while anything is open. Choosing a decision keeps the run in it until it’s empty.

Each answer takes you to the next open item. A spot check pauses to reveal Moonbounce’s answer, then Next, Skip or adding a reason moves on. When nothing remains, you return to the inbox. An item’s Back to inbox link keeps your decision and tab, and “N left in total” (“N left in ” when you filter) shows how many open items remain, including that one. Empty tabs say “All caught up” or “Nothing yet”; an empty decision says “Nothing left in this decision”.

Done keeps your history, newest first, 50 at a time, with Older for the next page and Back to newest on every page after the first. Each row shows the title, the decision, one answer cell, when you answered, and Undo while it is still open. The answer cell is an icon and your result, then how it compares with Moonbounce’s call: “Confirmed Escalate”, “Changed to Allow · was Remove”, “Not sure”, “Skipped”, “Expired”, “Withdrawn”, “Spot check: same call · Allow” or “Spot check: different call · you Allow, Moonbounce Remove”. A decision whose result is a value is answered question by question, so its row reads “Confirmed answers” or “Corrected answers · value was 0.51”. Opening a done item shows the full comparison in two labelled cells: a person icon beside “You” and the Moonbounce symbol beside “Moonbounce”. Opening a done item shows the content and saved result without answer buttons, and offers Change answer for an answer (see below) and says “Saved.” for Not sure; a skipped, expired or withdrawn item says nothing more than its label, since nothing was saved. Skipped, expired and unsure spot checks still hide Moonbounce’s call.

Change your answer

You can change an answer at any time. Open the item from Done and choose Change answer: the same controls as an open item come back, with Moonbounce’s call in view (Confirm, Change to, or the form with a reason, which is required when you disagree with Moonbounce’s outcome; Confirm answers and Correct answers on a value result). Sending it saves a new answer that replaces your last one, keeps the original in the history, and shows “Answer changed.” Done then shows your new answer. Not sure says “Saved.”, and withdrawn, skipped and expired items show only their label: there is no answer to change.

On a spot check, your first answer was given blind, so it still counts toward Moonbounce’s accuracy; the page says so under Change answer. Your change counts as your answer everywhere else: it is your vote on the item and what suggestions learn from. The export marks which answer was blind and which came after the reveal.

Decision dots cycle through blue, purple, red and green in the team’s decision creation order. They appear in inbox rows and the decision list, not on the item page, which names the decision in words. Filtering or paging does not change the colour.

Each item shows, at a glance:

• the decision’s name (its id in sentence case) and the same content-based item title used in the inbox. While items from more than one version are open, it flags the version: “older version”, or the version itself, as in “structure 4f2a”;
• “Why you’re seeing this:” with an icon, the reason word (Borderline, Low confidence, Unusual or Spot check) and its meaning on the same line: “close to where the call flips” (with the value’s distance from the nearest band cut, such as “0.46 is 0.01 above the Escalate cut at 0.45”, only where the value may be shown; “just above” or “just below” when the shown numbers are equal), “one of Moonbounce’s answers was uncertain”, “unlike the items reviewed so far” or “picked at random” (on a spot check, “; Moonbounce’s answer stays hidden until you answer”). The tooltip and Details give the longer reason;
• the state Moonbounce judged, flat under a Content heading: a text state as paragraphs, an object as its top-level keys with their values (text as text, anything else as compact JSON), an array element by element;
• what you’re being asked, with equal-weight outline buttons that name what they record;
• on a normal item whose result is a bands, the value it cut on a 0 to 1 bar, with a tick at each cut, a marker with the value above it, and each outcome’s range under its segment. Blue segments follow outcome order, with darker blue at the last outcome. Screen readers get a sentence that gives the value, its range and every other outcome’s range. The marker gains a light-grey outline near a cut; colour never replaces the words. “Close to the line” adds an icon on a blue tint when the value is within 0.1 of a cut.

One tap away:

• Context, the attribute values your app sent with the call, in a small “Attributes” list, with a chevron that turns when it opens;
• Details, a disclosure with a chevron, with the longer selection reason, version and supporting answers. Each question reads Yes or No, with “unsure” between 0.35 and 0.65 (or “abstained” and “failed” in words). These answers stay hidden on spot checks until you answer. The version line is a link, “How Moonbounce decided (structure 4f2a)”, to a page that explains the structure in words: each question in authored order with its context and the state keys it reads, the attributes, then each aggregation with its inputs (weights, a logistic’s bias, band cuts as “Deny below 0.38 · Escalate from 0.38 · Grant from 0.72”, rules as “When … then …”), and which one is the result. Where Moonbounce’s call may be shown it adds what Moonbounce answered for this item (Yes, No or No answer) and which rule, case or band fired. The full structure hash sits under a collapsed “Version details”. On a spot check that link appears once you have answered.

The header links Review, Team and API keys, with Sign out. The page follows your system’s light or dark appearance. Rows use hairlines; saved results pair icons and words with restrained tints. The answer bar stays at the bottom on phones and follows the content on wide screens.

The Review page shows items from your Production traffic. Test traffic never reaches it.

Keyboard shortcuts

Shortcuts enhance the existing links, buttons and forms; reviewing still works without JavaScript. Press ? or Cmd/Ctrl + / to open a modal sheet of the shortcuts available on the current page. There is no on-screen shortcut button. Esc closes it and restores focus. Only the primary answer buttons (Confirm, Change and the outcome buttons) show a key hint, and only on wide, pointer-driven screens; the hints are hidden on touch devices and below 48rem. The sheet always shows its keys, at any size, and is the complete reference: Not sure, Skip, Details and the tab switch have shortcuts but no hint.

Context	Keys
Inbox	J/K or ↓/↑ focus the next/previous visible row; Enter opens that row, or Continue reviewing when no row is focused. T switches To review/Done. 0 selects all decisions; 1–9 select the first nine decisions in the decision list.
Moonbounce decided	C confirms; X opens Change.
Pick an outcome	1–9 pick the corresponding outcome button on spot checks and undecided items.
Any item	Z triggers the first visible Undo. U records Not sure and S skips when those controls exist. D toggles Details. X opens Answer in detail when offered. Esc closes the sheet first, and otherwise follows Back to inbox.
Spot-check reveal	Enter or J follows Next; on a mismatch it follows Skip to leave the reason unanswered (S also skips).
Detail or reason form	Cmd/Ctrl + Enter clicks the existing submit button, including while typing, with the same validation.

Inputs, textareas, selects and editable content suppress shortcuts except form submission. Other modified keys are ignored; Shift is allowed, and Cmd/Ctrl + / opens the sheet outside typing. Native Enter activation still works on focused links and buttons. The sheet groups the available actions under Navigate, Answer and Item, and offers no action absent from the page. On the inbox, Enter is listed as “Open row, or continue reviewing”, and as “Open row” alone on Done when there is nothing to continue.

What can enter the queue

An evaluation can be queued when all of these hold:

• it is a primary evaluation, one your app made through POST /v1/evaluate, in the environment being reviewed;
• it ran on the decision’s live version, the structure of its newest primary evaluation;
• its content (the state) is stored, so there is something to show;
• it is recent;
• no person has judged it yet: no owner feedback, no review, and it isn’t a seed example;
• it isn’t already in anyone’s queue.

Feedback from your app (app_native or app_resolution) doesn’t keep an evaluation out. A client that reports a verdict on every call still gets reviews and spot checks.

How items are picked

Each time the queue is listed, Moonbounce tops it up from the pool of eligible evaluations, decision by decision:

About a quarter of the slots are spot checks, drawn at random from the whole pool. The rest are active picks, chosen one at a time by how much a person’s answer would teach:

• Close to a cut point. When the result is a bands, a value near one of its cuts is where a small error changes the outcome.
• Unsure on a question that could change the outcome. A hedged answer counts only if it matters. Moonbounce sets the answer to yes, then to no, keeping the other answers and the call’s attribute values as they are, and runs your aggregations again. If the outcome differs between the two, the question can decide this item, and how unsure the engine was about it counts. A decision whose result is a value has no outcome to change, so this never applies to it.
• Unlike what’s been reviewed. An item whose answers and state text (the state’s text values, whatever its shape) resemble cases a person already judged, or items already queued, teaches less than a new kind of case.

Picks continue until today’s slots are used or nothing worth asking is left. The budget is a ceiling, not a quota: on a quiet day you get fewer items.

Tags

Every item carries tags, all the reasons it was picked, and one reason to display:

Tag	Shown as	Example reason
near_cut	Borderline	Value 0.46, close to the escalate line
uncertain	Low confidence	The ‘item_used’ answer is unsure (0.60) and could change the outcome
novel	Unusual	Unlike the cases reviewed so far
spot_check	Spot check	Our answer stays hidden until you answer.

When an active item has several tags, the one displayed is near_cut, then novel, then uncertain.

Spot checks are blind

A spot check shows no prediction until you answer. The card also holds back the note you sent with the item until you answer, then shows it inline. The API leaves both prediction and summary off the unanswered item, and nothing else on it comes from the answers, the value or the outcome: the reason text is fixed, and every outcome the result can produce is offered as an equal choice in the decision’s own order. Under its question, the page says “Your answer is feedback for Moonbounce; it doesn’t act on the request itself.”, which names no call.

Blindness is what makes the agreement number honest. Once you’ve seen “Moonbounce said Escalate”, you’re more likely to agree with it. Active items are chosen where Moonbounce is most likely to be wrong, so they can’t measure agreement either. Spot checks are a random sample answered blind, so the share where you and Moonbounce agree estimates how often it’s right on your traffic.

The daily budget and expiry

Each reviewer gets a daily budget: 12 items per decision and 25 in total, counted per UTC day, with slots shared round-robin across your decisions. Listing the queue again the same day adds nothing once the budget is spent.

An item you don’t answer expires three days after it was queued. An item also expires early when its evaluation gets judged some other way, for example owner feedback through the API. A spot check you never opened can come back once. Any other expired item doesn’t come back.

Answering

Your answers are stored as reviewer feedback under your own user, one vote per reviewer. Owner feedback through POST /v1/feedback still outranks reviews (Sources).

On a normal item, the page asks “Moonbounce decided: Escalate. Do you agree?” and adds a muted line, “Your answer is feedback on Moonbounce’s call; it doesn’t act on the request itself.”, because confirming Escalate records your answer and does not escalate anything:

• Confirm Escalate sends expected_outcome: "escalate", Moonbounce’s own outcome.
• Change to Grant (one other outcome), Change to Grant or Deny… (two) or Change to another outcome… (three or more) opens a panel. “What should it have been?” offers the other outcomes. Picking one sends an expected_outcome claim, and picking none sends verdict: reject (“not this one”). With only two outcomes, the other one is sent. “Why?” asks for a reason, which is required whenever Moonbounce decided an outcome. “Which answers were wrong?” lets you flip any question’s answer with “Wrong?”, which sends an expected_answer claim with the opposite value. “The content itself is broken” sends input_broken.

A disagreement is a rejected verdict or an expected_outcome other than the one Moonbounce decided. A disagreement needs a reason, because the reason is what tells a suggestion what the structure is missing. Question corrections alone don’t count as disagreeing.

On a spot check, the page asks “What’s the right call?” with one button per outcome, which sends expected_outcome. You can also answer each question yes or no under “Also answer the N questions behind this call” (“the question” when there is one). After you answer, the page reveals Moonbounce’s answer: a heading that says “Same call.” or “Different call.” once, then two marked cells, “You: Grant” and “Moonbounce: Escalate”. If the two differ, it asks “Why? This helps improve future decisions.” Your answer is already saved, and the reason is added to it. A spot check never sends a verdict, because you haven’t seen what you’d be approving.

When Moonbounce couldn’t decide (it abstained; any abstain outcome is a fallback, not a call), the page asks “What’s the right call?” with one button per outcome. Each sends that expected_outcome, and no reason is needed.

When the result is a value, not an outcome, the page reads “Moonbounce computed 0.51” and you check it through its answers: Confirm answers sends an expected_answer for each question Moonbounce answered, as it answered, and Correct answers… opens the panel with Yes and No per question. A spot check asks each question directly. A value result takes no expected_outcome or verdict. In every case “Answer in detail” opens the panel for answer corrections or broken content.

Not sure and Skip both take the item out of your queue for good:

• Not sure appears in history as “Not sure” (“You: Not sure” on the saved item): you could not tell, rather than Moonbounce being uncertain. It records a submission with no claims, marked unsure. It counts as a person’s touch: the item isn’t offered again, and its train or holdout assignment is fixed.
• Skip records nothing.

Undo an answer

For 10 minutes after answering, you can withdraw your own answer with Undo. The next page shows a small status such as “You confirmed Allow on ‘Sleeping bag…’ · Undo”. It disappears when you navigate or the window closes. Undo is also available on recent Done rows, in a settled item’s view and during a spot-check reveal, even before adding a reason. Press Z to trigger the first visible Undo.

After Undo, the page says “Undone. Your answer was withdrawn.” You stay where you were reviewing. The item stays in Done as Withdrawn and is not offered again. A spot check keeps the prediction and note you already saw. Its answer no longer contributes to feedback, agreement or review counts. Corrections and changes do not restart the window, and Undo after a change withdraws the answer along with its changes; Not sure, skips and expired items have no answer to undo. All Undo controls work as ordinary form posts without JavaScript.

Ticket D, reviewed

Ticket D is a $95 headlamp, 34 days after delivery. The customer switched it on once at home to check it, then put it back in the box, and has had one refund before. The engine answers outside_window 0.95, item_used 0.60, defective 0.04, repeat_refunder 0.20 and high_value 0.01, so D’s grant_score is 0.456 and handling lands on escalate.

1. It’s picked as near_cut and uncertain. 0.456 is close to the escalate line at 0.38. The engine was also unsure whether the headlamp counts as used (item_used at 0.60), and that question matters here: setting it to no gives 0.576, still escalate, and setting it to yes gives 0.376, which is deny. repeat_refunder could change the outcome too, but the engine was surer of it, so the reason names item_used.
2. The owner opens it on their phone. The card shows the decision’s name, the ticket, and “Moonbounce decided: Escalate. Do you agree?”. “Why you’re seeing this: Borderline” sits above the content. The value and the band cuts are visible beneath Moonbounce’s call. Details holds the longer reason and each question’s answer.
3. They change it. They tap “Change to Grant or Deny…”, pick Grant under “What should it have been?” and write the reason: “Switching it on once to check it is trying it, not using it.” Under “Which answers were wrong?” they tap “Wrong?” on item_used.
4. It’s stored as a review with an expected_outcome claim of grant, an expected_answer claim of item_used: false and that reason, and the case gets its train or holdout assignment. The answer claim points a suggestion straight at item_used: if more cases like this one come in, the fix is better wording for that question (Suggestions and previews).

The review API

The Review page is built on public endpoints, so you can build your own review screen on them. Every endpoint needs decisions:develop. Every POST honors Idempotency-Key.

Endpoint	What it does
GET /v1/review/items	Takes status: open (the default) tops up the queue and lists open items, oldest batch first and in rank order within it; done lists this reviewer’s settled items, newest first, without filling the queue. Settled items include a handled summary of the reviewer’s answer, skip or expiry. Also takes decision_id, limit (up to 50), cursor and counts. Returns items, next_cursor, has_more and remaining_today. With counts=true, the first page (no cursor) of either state also returns counts: your open, done and done_today totals (today starts at 00:00 UTC) and by_decision, each decision’s open and done, its newest open item’s version (newest_open_version) and how many versions are open (open_versions), in creation order. counts ignores decision_id.
GET /v1/review/items/{review_item_id}	One item in any state: questions in authored order, content (state and attribute_values, or null when not stored), outcome_choices, a bands result’s cuts and below, version.structure_hash, and once it may be shown, prediction (status, value, outcome, via, each question’s answers and every aggregation’s value).
GET /v1/review/items/{review_item_id}/configuration	The structure the item was decided with: the full structure_hash and the structure as authored, with its questions (instructions, criteria, reads), attributes, aggregations and result. Never the item’s answers, aggregation values, value, outcome or summary. A spot check returns 409 until you answer it, and stays available after a withdrawal. SDK: getReviewItemConfiguration; MCP: get_review_item_configuration.
POST /v1/review/items/{review_item_id}/shown	Records that the item was put in front of the reviewer. Sets shown_at the first time.
POST /v1/review/items/{review_item_id}/submit	Records the answer as claims, with an optional reason and unsure. Returns feedback_id and the item with its prediction.
POST /v1/review/items/{review_item_id}/skip	{"reason": "unsure"} or {"reason": "skip"}.
POST /v1/review/items/{review_item_id}/reason	Adds the reason to a blind answer that disagreed without one. Once only.
POST /v1/review/items/{review_item_id}/withdraw	Takes { "feedback_id": "fbk_…" }, the current head from handled.feedback_id. Withdraws your own answer within 10 minutes of the original answer. Returns environment and the item with handled.status: "withdrawn". SDK: withdrawReview; MCP: withdraw_review.
POST /v1/review/items/{review_item_id}/change	Takes { "feedback_id": "fbk_…", "claims": [...], "reason": "…" }: the current head from handled.feedback_id and claims as on submit. Changes your own earlier answer at any time; the new answer replaces your last one and the original is kept. Returns environment and the item, whose handled describes the new answer, with the new feedback_id and changed_at. SDK: changeReview; MCP: change_review.

handled always describes your latest answer, compared with Moonbounce’s call: the decided outcome, or on a value result Moonbounce’s answers (a yes when P(true) is at least 0.5). An answer that matches reads confirmed, one that doesn’t (or only corrects answers) reads changed, and one with nothing decided to compare reads answered. A spot check’s first answer reads answered with matched when there was something to compare. at stays the time you first answered; a changed answer adds changed_at and reads confirmed or changed, even on a spot check, because you made it with the call in view.

A change is judged as an answer to a revealed call: a verdict is allowed on a spot check too, and disagreeing with Moonbounce’s decided outcome without a reason is a 400 with param: "reason". It returns 409 for a stale head, or an item that is withdrawn, skipped, unsure, expired or unanswered. Idempotency works as for the other review actions: the same Idempotency-Key replays, changed input returns 422 and a redacted result 410. Foreign reviewer or environment items return 404.

Withdrawal returns 409 for an expired window, stale head, or an unanswered, skipped, expired or already withdrawn item. A retry with the same Idempotency-Key replays even after the window; changed input returns 422 and a redacted result returns 410. Foreign reviewer or environment items return 404.

Each item includes decision_index, its zero-based position among the organization’s decisions ordered by (created_at, decision_id). The registry spans environments. Older replay responses can omit it.

Listing an item doesn’t count as showing it. Only shown does, so call it when the item is on screen unless shown_at is already recorded. That tells Moonbounce which items were actually seen. An item that was shown and then expired is never queued again.

The queue is scoped to the reviewer and the key’s environment. With an API key, the reviewer is the key itself, so each key has its own queue and budget, and a new key starts a new one. Another reviewer’s item is a 404. A test key reviews Test traffic.

Submitting follows the claim rules from Feedback (expected_answer, expected_outcome, verdict and input_broken; a value result takes no expected_outcome or verdict), with two more:

• no verdict on a blind item, or on an evaluation that decided nothing;
• on a non-blind item, a disagreement without a reason is a 400 with param: "reason". On a blind item it is accepted and the reason stays pending until you send it to .../reason.

An item that is no longer open (answered, skipped or expired) is a 409, unless you’re replaying the same Idempotency-Key. Here is ticket D answered through the API:

{
  "claims": [
    { "kind": "expected_outcome", "value": "grant" },
    { "kind": "expected_answer", "question": "item_used", "value": false }
  ],
  "reason": "Switching it on once to check it is trying it, not using it."
}

The response carries the new feedback_id and the item. On a spot check, that item is how you reveal Moonbounce’s answer, since prediction is now filled in.

Loop status

GET /v1/decisions/{decision_id}/loop reports where the decision’s feedback loop stands in your key’s environment. A test key’s loop status counts test reviews only and never lists an accepted suggestion:

{
  "environment": "production",
  "decision_id": "refund-exception",
  "accepted": [],
  "reviews": 18,
  "random_lane": { "n": 6, "agree": 4, "nonresponse": 0.25, "changed_after_reveal": 1 },
  "spot_check": null,
  "stop_check": { "reviews": 18, "threshold": 40, "any_holdout_win": false }
}

• reviews counts the decision’s live owner and reviewer submissions. Replaced and withdrawn ones don’t count.
• random_lane covers the spot checks. n is how many were answered. agree is how many times the reviewer’s own blind answer matched Moonbounce’s outcome. nonresponse is the share of spot checks that were skipped, expired or withdrawn instead of answered. Later feedback on the same item, even the owner’s or the reviewer’s own change, never replaces the blind answer. changed_after_reveal counts the answered spot checks whose reviewer changed their answer afterwards. An evaluation that abstained never counts as agreeing.
• spot_check compares that agreement on Production reviews before and after a suggestion was accepted. It is split by the version that served each review, not by date, and is null until a suggestion has been accepted.
• stop_check says when to rethink rather than keep reviewing. After threshold reviews with no suggestion beating the current version on held-out cases, more reviews probably won’t produce a better structure.
• accepted lists accepted suggestions with the evidence label they were proposed with (Suggestions and previews).

Here, four of six blind answers matched: Moonbounce agreed with the owner on two thirds of a random sample. That’s the number a carrier-delay question should raise.

Next

Reviews like ticket D’s, and feedback like the owner’s on ticket C, are what suggestions learn from. Suggestions and previews shows how Moonbounce turns them into a proposed change and how far to trust its preview.