Review

How Moonbounce picks a few items a day for you to judge, and how blind spot checks measure agreement.

The examples on this page use Northpine, a fictional outdoor-gear shop whose refund-exception decision grants, escalates or denies refund requests outside its standard return policy. How Moonbounce works introduces it.

Feedback showed what you can tell Moonbounce about an evaluation and what each kind of feedback counts for. Review is how Moonbounce asks you for it: a short daily queue of the items it most needs a person to judge, plus random spot checks that measure how often it agrees with you.

Why review

A busy decision makes more evaluations than anyone can label, and most of them teach nothing: a ticket that scores 0.05 and lands on deny is rarely news. Review spends your attention where it pays. Each day Moonbounce picks a few items that sit near a decision line, that hinge on a shaky answer, or that look unlike anything reviewed so far. It mixes in a random sample so it can say how often it’s right without fooling itself.

Where you review

The Review page is part of the web app at app.moonbounce.ai/review. It works on a phone (one item at a time, one column, the buttons at the bottom within reach of your thumb) and uses the extra room on a wide screen, and it works without JavaScript. Sign in with your Moonbounce account. The page needs no API key.

On a phone the page shows less by default and keeps everything one tap away, with no JavaScript. The “Why items are here:” line becomes a closed Why these items? that opens to the same list. Rows in a narrow list leave out the “Statement: …” line and, on Done, the words after the badge (such as “was Remove”); open the item to read them. On an item, “Why you’re seeing this” shows only the reason and a few words (such as “Borderline: near a cutoff”), and tapping it opens the item’s own sentence. The muted note about what your answer does and the hint about Change, Not sure and Skip sit behind How answering works, just above the answer buttons, and the “Close to the line” pill is left out because the reason already says Borderline. Wide screens show all of it inline.

Opening /review shows your inbox. The header reads “N to review · M done today (UTC)”, where today starts at 00:00 UTC, so it differs from the all-time count on the Done tab, above Continue reviewing and the To review and Done tabs. Their counts span decisions. The decision list narrows the inbox to one decision: All, then every decision with items on the current tab, in the order they were created, each with its count for the tab. On a wide screen it is a sidebar headed “Decisions · to review” or “Decisions · done”, naming the tab those counts are for, beside the list that scrolls on its own however many decisions you have; on a phone it is one menu above the list that names the current filter, such as “All decisions ▾”. Number keys 1–9 pick its first nine decisions. Each tab shows 50 rows. More loads the next 50 on To review.

To review rows line up in columns, named in a muted header row: the title, the decision (hidden while you are filtered to one), why the item was picked, and Moonbounce’s call. A line above the list, “Why items are here:”, says what each reason on the page means: Borderline (score near a cutoff), Low confidence (Moonbounce was unsure), Unusual (unlike items reviewed so far) and Spot check (picked at random, Moonbounce’s call hidden until you answer). A decision without outcomes shows its statement under each title, “Statement: …”, so its score has something to be true of. On a phone the title takes two lines and the rest follows on one line. Decisions without outcomes show a score, such as “Score 0.51”, without inferring True or False. Spot checks hide the call. Titles use the content’s first line, or “Review item” if content is absent. Sensitive items and subject parts use “Sensitive content”. Your integration’s summary (the text your system sent with the item) appears on the item page, saved reviews and reveals with a plug icon and “Note from your integration”, never as the title; inbox rows leave it out. Unanswered spot checks hide it; summaries that repeat the subject are omitted. Continue reviewing opens the first open item, or you can choose a row. It also appears on the Done tab while anything is open. Choosing a decision keeps the run in it until it’s empty.

Each answer takes you to the next open item. A spot check pauses to reveal Moonbounce’s answer, then Next, Skip or adding a reason moves on. When nothing remains, you return to the inbox. An item’s Back to inbox link keeps your decision and tab, and “N left in total” (“N left in ” when you filter) shows how many open items remain, including that one. Empty tabs say “All caught up” or “Nothing yet”; an empty decision says “Nothing left in this decision”.

Done keeps your history, newest first, 50 at a time, with Older for the next page and Back to newest on every page after the first. Each row shows the title, the decision, one answer cell, when you answered, and Undo while it is still open. The answer cell is an icon and your result, then how it compares with Moonbounce’s call: “Confirmed Escalate”, “Changed to Allow · was Remove”, “Not sure”, “Skipped”, “Expired”, “Withdrawn”, “Spot check: same call · Allow” or “Spot check: different call · you Allow, Moonbounce Remove”. A decision without outcomes has no call to confirm or change, so an answer reads “You answered False · Moonbounce 0.92 (higher means true)”, Moonbounce’s score and the way it reads, never a True or False inferred from it. Opening a done item shows the full comparison in two labelled cells: a person icon beside “You” and the Moonbounce symbol beside “Moonbounce”, with “Moonbounce: Score 0.51 (higher means true)” for a decision without outcomes, and the statement it asked about. Opening a done item shows the content and saved result without answer buttons, and offers Change answer for an answer (see below) and says “Saved.” for Not sure; a skipped, expired or withdrawn item says nothing more than its label, since nothing was saved. Skipped, expired and unsure spot checks still hide Moonbounce’s call.

Change your answer

You can change an answer at any time. Open the item from Done and choose Change answer: the same controls as an open item come back, with Moonbounce’s call in view (Confirm, Change to, or the form with a reason, which is required when you disagree with Moonbounce). Sending it saves a new answer that replaces your last one, keeps the original in the history, and shows “Answer changed.” Done then shows your new answer. Not sure says “Saved.”, and withdrawn, skipped and expired items show only their label: there is no answer to change.

On a spot check, your first answer was given blind, so it still counts toward Moonbounce’s accuracy; the page says so under Change answer. Your change counts as your answer everywhere else: it is your vote on the item and what suggestions learn from. The export marks which answer was blind and which came after the reveal.

Decision dots cycle through blue, purple, red and green in the team’s decision creation order. They appear in inbox rows and the decision list, not on the item page, which names the decision in words. Filtering or paging does not change the colour.

Each item shows, at a glance:

• the decision’s title and the same content-based item title used in the inbox. While items from more than one version are open, it flags the version: “older version”, or the version itself, as in “structure 4f2a, weights 1b07”;
• “Why you’re seeing this:” with an icon, the reason word (Borderline, Low confidence, Unusual or Spot check) and its meaning on the same line: “close to where the call flips” (with the score’s distance from the nearest cut, such as “0.46 is 0.01 above the Escalate cut at 0.45”, only where the score may be shown; “just above” or “just below” when the shown numbers are equal), “one of Moonbounce’s answers was uncertain”, “unlike the items reviewed so far” or “picked at random” (on a spot check, “; Moonbounce’s answer stays hidden until you answer”). The tooltip and Details give the longer reason;
• the content Moonbounce judged, flat under a Content heading. A short one-line content is the heading itself and is not repeated; other parts still follow, and any context stays under Context. Parts you marked sensitive stay blurred until you tap them;
• the target statement, only for a yes/no statement decision (no outcomes), because that is what you are asked about. Once you have answered it is labelled “Statement:”. A decision with outcomes doesn’t show it, since its outcomes already say what it is;
• what you’re being asked, with equal-weight outline buttons that name what they record;
• on a normal item, Moonbounce’s score on a 0 to 1 bar, with a tick at each cut, a marker with the score above it, and each outcome’s range under its segment. Blue segments follow outcome order, with darker blue at the last outcome. Screen readers get a sentence that gives the score, its range and every other outcome’s range. The marker gains a light-grey outline near a cut; colour never replaces the words. “Close to the line” adds an icon on a blue tint when the score is within 0.1 of a cut.

One tap away:

• Context, the context parts you sent. It reads “Context:” followed by the part names, with a chevron that turns when it opens;
• Details, a disclosure with a chevron, with the longer selection reason, version and supporting answers. Each feature reads Yes or No, with “unsure” between 0.35 and 0.65. Muted text explains which outcome a yes pushes toward. These answers stay hidden on spot checks until you answer. The version line is a link, “How Moonbounce decided (structure 4f2a, weights 1b07)”, to a page that opens with this item’s result (“This item: score 0.46, so Escalate”) and a sentence on how the answers become a score, then lists the outcome scale in words and each question with its answer type, guidance, what Yes and No mean, its weight (“raises the score” or “lowers the score”; a question inside a nested combine says “Weighted inside” and the combine’s name) and what Moonbounce answered for this item (Yes, No or No answer). The full structure and weights versions sit under a collapsed “Version details”. On a spot check that link appears once you have answered.

The header links Review, Team and API keys, with Sign out. The page follows your system’s light or dark appearance. Rows use hairlines; saved results pair icons and words with restrained tints. The answer bar stays at the bottom on phones and follows the content on wide screens.

The Review page shows items from your Production traffic. Test traffic never reaches it.

Keyboard shortcuts

Shortcuts enhance the existing links, buttons and forms; reviewing still works without JavaScript. Press ? or Cmd/Ctrl + / to open a modal sheet of the shortcuts available on the current page. There is no on-screen shortcut button. Esc closes it and restores focus. Only the primary answer buttons (Confirm, Change, the outcome buttons and Yes or No) show a key hint, and only on wide, pointer-driven screens; the hints are hidden on touch devices and below 48rem. The sheet always shows its keys, at any size, and is the complete reference: Not sure, Skip, Details and the tab switch have shortcuts but no hint.

Context	Keys
Inbox	J/K or ↓/↑ focus the next/previous visible row; Enter opens that row, or Continue reviewing when no row is focused. T switches To review/Done. 0 selects all decisions; 1–9 select the first nine decisions in the decision list.
Moonbounce decided	C confirms; X opens Change.
Pick an outcome	1–9 pick the corresponding outcome button on spot checks and undecided items.
Yes or no	Y/N choose the statement-only Yes/No buttons.
Any item	Z triggers the first visible Undo. U records Not sure and S skips when those controls exist. D toggles Details. X opens Answer in detail when offered. Esc closes the sheet first, and otherwise follows Back to inbox.
Spot-check reveal	Enter or J follows Next; on a mismatch it follows Skip to leave the reason unanswered (S also skips).
Detail or reason form	Cmd/Ctrl + Enter clicks the existing submit button, including while typing, with the same validation.

Inputs, textareas, selects and editable content suppress shortcuts except form submission. Other modified keys are ignored; Shift is allowed, and Cmd/Ctrl + / opens the sheet outside typing. Native Enter activation still works on focused links and buttons. The sheet groups the available actions under Navigate, Answer and Item, and offers no action absent from the page. On the inbox, Enter is listed as “Open row, or continue reviewing”, and as “Open row” alone on Done when there is nothing to continue. There are no True/False keys: a statement decision uses Y/N.

What can enter the queue

An evaluation can be queued when all of these hold:

• it is a primary evaluation, one your app made through POST /v1/evaluate, in the environment being reviewed;
• it ran on the decision’s live version, the structure and weights of its newest primary evaluation;
• its content is stored, so there is something to show;
• it is recent;
• no person has judged it yet: no owner feedback, no review, and it isn’t a seed example;
• it isn’t already in anyone’s queue.

Feedback from your app (app_native or app_resolution) doesn’t keep an evaluation out. A client that reports a verdict on every call still gets reviews and spot checks.

How items are picked

Each time the queue is listed, Moonbounce tops it up from the pool of eligible evaluations, decision by decision:

About a quarter of the slots are spot checks, drawn at random from the whole pool. The rest are active picks, chosen one at a time by how much a person’s answer would teach:

• Close to a cut point. A score near one of your cut points is where a small error changes the outcome.
• Unsure on a feature that could change the outcome. A hedged answer counts only if it matters. Moonbounce sets the feature to yes, then to no, keeping the other answers as they are. If the outcome differs between the two, the feature can decide this item, and how unsure the engine was about it counts.
• Unlike what’s been reviewed. An item whose answers and text resemble cases a person already judged, or items already queued, teaches less than a new kind of case.

Picks continue until today’s slots are used or nothing worth asking is left. The budget is a ceiling, not a quota: on a quiet day you get fewer items.

Tags

Every item carries tags, all the reasons it was picked, and one reason to display:

Tag	Shown as	Example reason
near_cut	Borderline	Score 0.54, close to the grant line
uncertain	Low confidence	The ‘item_used’ answer is unsure (0.60) and could change the outcome
novel	Unusual	Unlike the cases reviewed so far
spot_check	Spot check	Our answer stays hidden until you answer.

When an active item has several tags, the one displayed is near_cut, then novel, then uncertain.

Spot checks are blind

A spot check shows no prediction until you answer. The card also holds back the note you sent with the item until you answer, then shows it inline. The API leaves both prediction and summary off the unanswered item, and nothing else on it comes from the score or the outcome: the reason text is fixed, and every outcome is offered as an equal choice in the decision’s own order. Under its question, the page says “Your answer is feedback for Moonbounce; it doesn’t act on the request itself.”, which names no call.

Blindness is what makes the agreement number honest. Once you’ve seen “Moonbounce said Escalate”, you’re more likely to agree with it. Active items are chosen where Moonbounce is most likely to be wrong, so they can’t measure agreement either. Spot checks are a random sample answered blind, so the share where you and Moonbounce agree estimates how often it’s right on your traffic.

The daily budget and expiry

Each reviewer gets a daily budget: 12 items per decision and 25 in total, counted per UTC day, with slots shared round-robin across your decisions. Listing the queue again the same day adds nothing once the budget is spent.

An item you don’t answer expires three days after it was queued. An item also expires early when its evaluation gets judged some other way, for example owner feedback through the API. A spot check you never opened can come back once. Any other expired item doesn’t come back.

Answering

Your answers are stored as reviewer feedback under your own user, one vote per reviewer. Owner feedback through POST /v1/feedback still outranks reviews (Sources).

On a normal item, the page asks “Moonbounce decided: Escalate. Do you agree?” and adds a muted line, “Your answer is feedback on Moonbounce’s call; it doesn’t act on the request itself.”, because confirming Escalate records your answer and does not escalate anything:

• Confirm Escalate sends verdict: approve.
• Change to Grant (one other outcome), Change to Grant or Deny… (two) or Change to another outcome… (three or more) opens a panel. “What should it have been?” offers the other outcomes. Picking one sends an outcome claim, and picking none sends verdict: reject (“not this one”). With only two outcomes, the other one is sent. “Why?” asks for a reason, which is required whenever Moonbounce gave an outcome. “Which answers were wrong?” lets you flip any prompt’s answer with “Wrong?”, which sends a feature claim with the opposite value. “The content itself is broken” sends input_broken.

A disagreement is a rejected verdict, an outcome other than the one Moonbounce gave, or a target answer that contradicts its lowest or highest outcome. A disagreement needs a reason, because the reason is what tells a suggestion what the spec is missing. Feature corrections alone don’t count as disagreeing.

On a spot check, the page asks “What’s the right call?” with one button per outcome. You can also answer each prompt yes or no under “Also answer the N questions behind this call” (“the question” when there is one). After you answer, the page reveals Moonbounce’s answer: a heading that says “Same call.” or “Different call.” once, then two marked cells, “You: Grant” and “Moonbounce: Escalate”. If the two differ, it asks “Why? This helps improve future decisions.” Your answer is already saved, and the reason is added to it. A spot check never sends a verdict, because you haven’t seen what you’d be approving.

When Moonbounce couldn’t decide (it abstained), the page asks “What’s the right call?” with one button per outcome. Each sends that outcome claim, and no reason is needed. On a normal item of a decision with no outcomes, it asks “Is this statement true?” with Yes and No, which send a target claim. In both cases “Answer in detail” opens the panel for answer corrections or broken content.

Not sure and Skip both take the item out of your queue for good:

• Not sure appears in history as “Not sure” (“You: Not sure” on the saved item): you could not tell, rather than Moonbounce being uncertain. It records a submission with no claims, marked unsure. It counts as a person’s touch: the item isn’t offered again, and its train or holdout assignment is fixed.
• Skip records nothing.

Undo an answer

For 10 minutes after answering, you can withdraw your own answer with Undo. The next page shows a small status such as “You confirmed Allow on ‘Sleeping bag…’ · Undo”. It disappears when you navigate or the window closes. Undo is also available on recent Done rows, in a settled item’s view and during a spot-check reveal, even before adding a reason. Press Z to trigger the first visible Undo.

After Undo, the page says “Undone. Your answer was withdrawn.” You stay where you were reviewing. The item stays in Done as Withdrawn and is not offered again. A spot check keeps the prediction and note you already saw. Its answer no longer contributes to feedback, agreement or review counts. Corrections and changes do not restart the window, and Undo after a change withdraws the answer along with its changes; Not sure, skips and expired items have no answer to undo. All Undo controls work as ordinary form posts without JavaScript.

Ticket D, reviewed

Ticket D is a $95 headlamp, 34 days after delivery. The customer switched it on once at home to check it, then put it back in the box. The engine answers outside_window 0.95, item_used 0.60, defective 0.04, repeat_refunder 0.03 and high_value 0.01, so D scores 0.490 and lands on escalate.

1. It’s picked as near_cut and uncertain. 0.490 is close to the grant line at 0.60. The engine was also unsure whether the headlamp counts as used (item_used at 0.60), and that feature matters here: setting it to no gives 0.61, which is grant, and setting it to yes gives 0.41, which is escalate. Other features could change the outcome too (outside_window at 0 would give 0.68), but their answers are confident, so they add little doubt.
2. The owner opens it on their phone. The card shows the decision’s name, the ticket, and “Moonbounce decided: Escalate. Do you agree?”. “Why you’re seeing this: Borderline” sits above the content. The score and named cuts are visible beneath Moonbounce’s call. Details holds the longer reason and each prompt’s answer.
3. They change it. They tap “Change to Grant or Deny…”, pick Grant under “What should it have been?” and write the reason: “Switching it on once to check it is trying it, not using it.” Under “Which answers were wrong?” they tap “Wrong?” on item_used.
4. It’s stored as a review with an outcome claim of grant, a feature claim of item_used: false and that reason, and the case gets its train or holdout assignment. The feature claim points a suggestion straight at item_used: if more cases like this one come in, the fix is better wording for that prompt (Suggestions and previews).

The review API

The Review page is built on public endpoints, so you can build your own review screen on them. Every endpoint needs decisions:develop. Every POST honors Idempotency-Key.

Endpoint	What it does
GET /v1/review/items	Takes state: open (the default) tops up the queue and lists open items, oldest batch first and in rank order within it; done lists this reviewer’s settled items, newest first, without filling the queue. Settled items include a handled summary of the reviewer’s answer, skip or expiry. Also takes decision_id, limit (up to 50), cursor and counts. Returns items, next_cursor and remaining_today. With counts=true, the first page (no cursor) of either state also returns counts: your open, done and done_today totals (today starts at 00:00 UTC) and by_decision, each decision’s open and done, its newest open item’s version (newest_open_version) and how many versions are open (open_versions), in creation order. counts ignores decision_id.
GET /v1/review/items/{item_id}	One item in any state.
GET /v1/review/items/{item_id}/configuration	The structure and weights the item was decided with: full structure_hash and weights_hash, each feature’s name, type, prompt, guidance and criteria in authored order with its weight in the score’s combine (or via, the nested combine a feature is weighted inside), and the below and cuts outcomes, lowest first. Never the item’s answers, score, outcome or summary. A spot check returns 409 until you answer it, and stays available after a withdrawal. SDK: getReviewItemConfiguration; MCP: get_review_item_configuration.
POST /v1/review/items/{item_id}/shown	Records that the item was put in front of the reviewer. Sets shown_at the first time.
POST /v1/review/items/{item_id}/submit	Records the answer as claims, with an optional reason and unsure. Returns feedback_id and the item with its prediction.
POST /v1/review/items/{item_id}/skip	{"reason": "unsure"} or {"reason": "skip"}.
POST /v1/review/items/{item_id}/reason	Adds the reason to a blind answer that disagreed without one. Once only.
POST /v1/review/items/{item_id}/withdraw	Takes { "feedback_id": "fbk_…" }, the current head from handled.feedback_id. Withdraws your own answer within 10 minutes of the original answer. Returns environment and the item with handled.state: "withdrawn". SDK: withdrawReview; MCP: withdraw_review.
POST /v1/review/items/{item_id}/change	Takes { "feedback_id": "fbk_…", "claims": [...], "reason": "…" }: the current head from handled.feedback_id and claims as on submit. Changes your own earlier answer at any time; the new answer replaces your last one and the original is kept. Returns environment and the item, whose handled describes the new answer, with the new feedback_id and changed_at. SDK: changeReview; MCP: change_review.

handled always describes your latest answer. at stays the time you first answered; a changed answer adds changed_at and reads confirmed or changed against Moonbounce’s call, even on a spot check, because you made it with the call in view.

A change is judged as an answer to a revealed call: a verdict is allowed on a spot check too, and disagreeing with Moonbounce’s outcome without a reason is a 400 with param: "reason". It returns 409 for a stale head, or an item that is withdrawn, skipped, unsure, expired or unanswered. Idempotency works as for the other review actions: the same Idempotency-Key replays, changed input returns 422 and a redacted result 410. Foreign reviewer or environment items return 404.

Withdrawal returns 409 for an expired window, stale head, or an unanswered, skipped, expired or already withdrawn item. A retry with the same Idempotency-Key replays even after the window; changed input returns 422 and a redacted result returns 410. Foreign reviewer or environment items return 404.

Each item includes decision_index, its zero-based position among the organization’s decisions ordered by (created_at, decision_id). The registry spans environments. Older replay responses can omit it.

Listing an item doesn’t count as showing it. Only shown does, so call it when the item is on screen unless shown_at is already recorded. That tells Moonbounce which items were actually seen. An item that was shown and then expired is never queued again.

The queue is scoped to the reviewer and the key’s environment. With an API key, the reviewer is the key itself, so each key has its own queue and budget, and a new key starts a new one. Another reviewer’s item is a 404. A test key reviews Test traffic.

Submitting follows the claim rules from Feedback, with three more:

• no action claims;
• no verdict on a blind item, or on an evaluation that abstained;
• on a non-blind item, a disagreement without a reason is a 400 with param: "reason". On a blind item it is accepted and the reason stays pending until you send it to .../reason.

An item that is no longer open (answered, skipped or expired) is a 409, unless you’re replaying the same Idempotency-Key. Here is ticket D answered through the API:

{
  "claims": [
    { "kind": "outcome", "value": "grant" },
    { "kind": "feature", "feature": "item_used", "value": false }
  ],
  "reason": "Switching it on once to check it is trying it, not using it."
}

The response carries the new feedback_id and the item. On a spot check, that item is how you reveal Moonbounce’s answer, since prediction is now filled in.

Loop status

GET /v1/decisions/{decision_id}/loop reports where the decision’s feedback loop stands in your key’s environment. A test key’s loop status counts test reviews only and never lists an accepted suggestion:

{
  "decision_id": "refund-exception",
  "accepted": [],
  "reviews": 18,
  "random_lane": { "n": 6, "agree": 4, "nonresponse": 0.25, "changed_after_reveal": 1 },
  "spot_check": null,
  "stop_check": { "reviews": 18, "threshold": 40, "any_holdout_win": false }
}

• reviews counts the decision’s live owner and reviewer submissions. Replaced and withdrawn ones don’t count.
• random_lane covers the spot checks. n is how many were answered. agree is how many times the reviewer’s own blind answer matched Moonbounce’s outcome. nonresponse is the share of spot checks that were skipped, expired or withdrawn instead of answered. Later feedback on the same item, even the owner’s or the reviewer’s own change, never replaces the blind answer. changed_after_reveal counts the answered spot checks whose reviewer changed their answer afterwards. An evaluation that abstained never counts as agreeing.
• spot_check compares that agreement on Production reviews before and after a suggestion was accepted. It is split by the version that served each review, not by date, and is null until a suggestion has been accepted.
• stop_check says when to rethink rather than keep reviewing. After threshold reviews with no suggestion beating the current version on held-out cases, more reviews probably won’t produce a better spec.
• accepted lists accepted suggestions with the evidence label they were proposed with (Suggestions and previews).

Here, four of six blind answers matched: Moonbounce agreed with Northpine’s owner on two thirds of a random sample. That’s the number a carrier-delay feature should raise.

Next

Reviews like ticket D’s, and feedback like the owner’s on ticket C, are what suggestions learn from. Suggestions and previews shows how Moonbounce turns them into a proposed change and how far to trust its preview.