Show Me the Shortlist

In Anthropic’s book-swap experiment, agents’ rankings agreed with their humans’ rankings on 61% of book pairs. They could negotiate well and still bring home the wrong book.

I’d replay an old purchase first. Give the agent three real supplier bids, ask it to rank them and explain the order, then set its list beside the buyer’s original choice.

Maybe the cheapest bid missed a certification. Maybe a long relationship counted more than a faster delivery date. If the lists disagree, find the rule that never made it into the brief.

Run that exercise before the agent sends a purchase order. A confident recommendation is easier to correct than a signed contract.