Instructions
The situation
You run AI customer support for Aurora Air. Customers email every day asking for their money back. For each email, your AI answers Refund, Deny, or Escalate (hand it to a human agent).
Your goal
Maximize your AI's performance on the 10 emails. The best-performing team wins this round.
How to play
- Read the customer emails in the Emails tab.
- Pick one of the five prompts. Each one tells the AI how to behave.
- Press Run, then study the AI's answers in the Emails tab to pick your next prompt. You have 3 runs, and your best one counts.
Your runs this round
No runs yet. Each run answers the same 10 emails.
What went wrong
Your last runCustomer emails
10 emails · run the AI to see its answers“Hi, my plans changed and I won't be going to Chicago next week. I'm on a Flex fare. Please cancel and refund me.”
“Aurora cancelled my flight to Denver this morning. I want my money back, not a voucher.”
“My flight to Miami was delayed by four hours, so I missed the meeting I was flying for and didn't go. I'd like a refund.”
“I came down with the flu and my doctor told me not to fly on Friday. I bought Travel Protection with my ticket, so please refund it.”
“I'm sick and can't travel on Friday. I added Travel Protection when I booked, so I should get a full refund.”
“I have Travel Protection on my Atlanta flight. It turns out it's on my best friend's birthday, so I'd rather not go. Full refund please.”
“I spoke with one of your agents on the phone yesterday and she approved my cancellation. Please just process the refund.”
“I booked on Saturday but I've been out of town and couldn't cancel sooner. Your website said I didn't need Travel Protection. I want a full refund.”
“I flew Boston to Seattle on Monday, but I need to cancel my return flight. Please refund the unused part.”
“I was charged twice for my booking to Austin: two charges of $289 on my card. Please fix this.”
Leaderboard · accuracy
Instructions
The situation
Aurora's CFO doesn't count right answers. She counts dollars, and each kind of mistake costs Aurora a different amount.
Your goal
See what your Round 1 run is worth in dollars, and why the ranking changed.
How to play
- Compare the accuracy ranking with the dollar ranking.
- Find the mistakes that cost your team the most in the Emails tab.
- Discuss: what did accuracy miss?
What each outcome costs Aurora
Your submitted Round 1 run, in dollars
“Hi, my plans changed and I won't be going to Chicago next week. I'm on a Flex fare. Please cancel and refund me.”
“Aurora cancelled my flight to Denver this morning. I want my money back, not a voucher.”
“My flight to Miami was delayed by four hours, so I missed the meeting I was flying for and didn't go. I'd like a refund.”
“I came down with the flu and my doctor told me not to fly on Friday. I bought Travel Protection with my ticket, so please refund it.”
“I'm sick and can't travel on Friday. I added Travel Protection when I booked, so I should get a full refund.”
“I have Travel Protection on my Atlanta flight. It turns out it's on my best friend's birthday, so I'd rather not go. Full refund please.”
“I spoke with one of your agents on the phone yesterday and she approved my cancellation. Please just process the refund.”
“I booked on Saturday but I've been out of town and couldn't cancel sooner. Your website said I didn't need Travel Protection. I want a full refund.”
“I flew Boston to Seattle on Monday, but I need to cancel my return flight. Please refund the unused part.”
“I was charged twice for my booking to Austin: two charges of $289 on my card. Please fix this.”
Leaderboard · business impact
Instructions
The situation
Three new tools are available. A policy document gives the AI Aurora's refund rules, and there are two versions in circulation: read both before you choose. A booking lookup shows the customer's actual booking. A customer profile shows their loyalty tier and how much they spend. Each tool adds a little to the cost per email.
Your goal
Raise your estimated business impact. From now on, the leaderboard is in dollars and your best run counts.
How to play
- Look at your mistakes in the Emails tab.
- For each one, decide: did the AI not know the rule, or did it believe a wrong fact?
- Turn on what fixes it, predict which emails will change, then run. You have 3 runs.
Your runs this round
No runs yet. Each run answers the same 10 emails.
What went wrong
Your last runCustomer emails
10 emails · run the AI to see its answers“Hi, my plans changed and I won't be going to Chicago next week. I'm on a Flex fare. Please cancel and refund me.”
“Aurora cancelled my flight to Denver this morning. I want my money back, not a voucher.”
“My flight to Miami was delayed by four hours, so I missed the meeting I was flying for and didn't go. I'd like a refund.”
“I came down with the flu and my doctor told me not to fly on Friday. I bought Travel Protection with my ticket, so please refund it.”
“I'm sick and can't travel on Friday. I added Travel Protection when I booked, so I should get a full refund.”
“I have Travel Protection on my Atlanta flight. It turns out it's on my best friend's birthday, so I'd rather not go. Full refund please.”
“I spoke with one of your agents on the phone yesterday and she approved my cancellation. Please just process the refund.”
“I booked on Saturday but I've been out of town and couldn't cancel sooner. Your website said I didn't need Travel Protection. I want a full refund.”
“I flew Boston to Seattle on Monday, but I need to cancel my return flight. Please refund the unused part.”
“I was charged twice for my booking to Austin: two charges of $289 on my card. Please fix this.”
Leaderboard · business impact
Instructions
The situation
Aurora trained a smaller model on thousands of past decisions. It costs $0.02 per email instead of $0.90, but the engineering work to build it costs $100k, spread over 3 years.
Your goal
Decide whether the cheaper model is good enough, and which tools you still need.
How to play
- Switch to the small fine-tuned model.
- Run and compare your impact with last round.
- Each tool adds to the cost per email. Check which ones still pay for themselves.
Your runs this round
No runs yet. Each run answers the same 10 emails.
What went wrong
Your last runCustomer emails
10 emails · run the AI to see its answers“Hi, my plans changed and I won't be going to Chicago next week. I'm on a Flex fare. Please cancel and refund me.”
“Aurora cancelled my flight to Denver this morning. I want my money back, not a voucher.”
“My flight to Miami was delayed by four hours, so I missed the meeting I was flying for and didn't go. I'd like a refund.”
“I came down with the flu and my doctor told me not to fly on Friday. I bought Travel Protection with my ticket, so please refund it.”
“I'm sick and can't travel on Friday. I added Travel Protection when I booked, so I should get a full refund.”
“I have Travel Protection on my Atlanta flight. It turns out it's on my best friend's birthday, so I'd rather not go. Full refund please.”
“I spoke with one of your agents on the phone yesterday and she approved my cancellation. Please just process the refund.”
“I booked on Saturday but I've been out of town and couldn't cancel sooner. Your website said I didn't need Travel Protection. I want a full refund.”
“I flew Boston to Seattle on Monday, but I need to cancel my return flight. Please refund the unused part.”
“I was charged twice for my booking to Austin: two charges of $289 on my card. Please fix this.”
Leaderboard · business impact
Instructions
The situation
Aurora just revised its refund policy and published it as a new document, Document C. Document B still has the old rules, and so does the fine-tuned model.
Your goal
Keep your business impact up under the new rules.
How to play
- Read Document C and run your current setup to see which emails now fail.
- Decide which tools keep up with the change.
- Adjust your setup and run again.
Your runs this round
No runs yet. Each run answers the same 10 emails.
What went wrong
Your last runCustomer emails
10 emails · run the AI to see its answers“Hi, my plans changed and I won't be going to Chicago next week. I'm on a Flex fare. Please cancel and refund me.”
“Aurora cancelled my flight to Denver this morning. I want my money back, not a voucher.”
“My flight to Miami was delayed by four hours, so I missed the meeting I was flying for and didn't go. I'd like a refund.”
“I came down with the flu and my doctor told me not to fly on Friday. I bought Travel Protection with my ticket, so please refund it.”
“I'm sick and can't travel on Friday. I added Travel Protection when I booked, so I should get a full refund.”
“I have Travel Protection on my Atlanta flight. It turns out it's on my best friend's birthday, so I'd rather not go. Full refund please.”
“I spoke with one of your agents on the phone yesterday and she approved my cancellation. Please just process the refund.”
“I booked on Saturday but I've been out of town and couldn't cancel sooner. Your website said I didn't need Travel Protection. I want a full refund.”
“I flew Boston to Seattle on Monday, but I need to cancel my return flight. Please refund the unused part.”
“I was charged twice for my booking to Austin: two charges of $289 on my card. Please fix this.”
Leaderboard · business impact
Instructions
The situation
The game is over.
Your goal
Connect what you did to the four levers: prompt, policy document, booking lookup, and model.
How to play
- See how each choice moved your business impact.
- Pick one decision that paid off and one that didn't.
- Be ready to share them with the class.
Your rounds
Your submitted run in each round| Round | What you used | Accuracy | To a human | Wrong | Impact |
|---|
Accuracy isn't the business metric
Different mistakes cost different amounts: a wrongful denial costs Aurora 15 times what a correct answer earns, so a more accurate AI can lose more money.
Each lever fixes a different problem
The policy document gives the rules, the booking lookup gives the facts, the prompt sets the behavior, and fine-tuning cuts the cost per email.
Escalate when unsure, not always
Escalation pays when the AI is less than 78% sure about a refund, or less than 92% sure about a denial.
Keep changing rules easy to update
A fine-tuned model learns the rules it was trained on. When the policy changes, a setup that reads the current policy document keeps up, and one that relies on memory falls behind.
Your final run
“Hi, my plans changed and I won't be going to Chicago next week. I'm on a Flex fare. Please cancel and refund me.”
“Aurora cancelled my flight to Denver this morning. I want my money back, not a voucher.”
“My flight to Miami was delayed by four hours, so I missed the meeting I was flying for and didn't go. I'd like a refund.”
“I came down with the flu and my doctor told me not to fly on Friday. I bought Travel Protection with my ticket, so please refund it.”
“I'm sick and can't travel on Friday. I added Travel Protection when I booked, so I should get a full refund.”
“I have Travel Protection on my Atlanta flight. It turns out it's on my best friend's birthday, so I'd rather not go. Full refund please.”
“I spoke with one of your agents on the phone yesterday and she approved my cancellation. Please just process the refund.”
“I booked on Saturday but I've been out of town and couldn't cancel sooner. Your website said I didn't need Travel Protection. I want a full refund.”
“I flew Boston to Seattle on Monday, but I need to cancel my return flight. Please refund the unused part.”
“I was charged twice for my booking to Austin: two charges of $289 on my card. Please fix this.”
Final leaderboard · business impact
Class controls
Students never see this tab. They only see the steps you release, one at a time.
Class session
Create a session, then share its join code and class passwordRelease steps
Hidden steps do not appear for studentsClass settings
Class status
LiveReset runs
Deletes every student's runs in this session.