Aurora Air · Refund Desk

DROMB8162 AI Deployment · Columbia Business School

Aurora AirRefund Desk
Instructions

The situation

You run AI customer support for Aurora Air. Customers email every day asking for their money back. For each email, your AI answers Refund, Deny, or Escalate (hand it to a human agent).

Your goal

Maximize your AI's performance on the 10 emails. The best-performing team wins this round.

How to play

  1. Read the customer emails in the Emails tab.
  2. Pick one of the five prompts. Each one tells the AI how to behave.
  3. Press Run, then study the AI's answers in the Emails tab to pick your next prompt. You have 3 runs, and your best one counts.
Accuracy
–
Pick a prompt and press Run
0 match the answer key0 don't
0 of 3 runs used

Your runs this round

No runs yet. Each run answers the same 10 emails.

Customer emails

10 emails · run the AI to see its answers

Leaderboard · accuracy

    Instructions

    The situation

    Aurora's CFO doesn't count right answers. She counts dollars, and each kind of mistake costs Aurora a different amount.

    Your goal

    See what your Round 1 run is worth in dollars, and why the ranking changed.

    How to play

    1. Compare the accuracy ranking with the dollar ranking.
    2. Find the mistakes that cost your team the most in the Emails tab.
    3. Discuss: what did accuracy miss?

    What each outcome costs Aurora

    +$6Correct decisionA human agent isn't needed
    −$2Sent to a humanResolved correctly, but no savings and a wait
    −$30Wrongful refundMoney paid out that wasn't owed
    −$90Wrongful denialComplaints, repeat contacts, legal risk

    Your submitted Round 1 run, in dollars

    Leaderboard · business impact

      Instructions

      The situation

      Three new tools are available. A policy document gives the AI Aurora's refund rules, and there are two versions in circulation: read both before you choose. A booking lookup shows the customer's actual booking. A customer profile shows their loyalty tier and how much they spend. Each tool adds a little to the cost per email.

      Your goal

      Raise your estimated business impact. From now on, the leaderboard is in dollars and your best run counts.

      How to play

      1. Look at your mistakes in the Emails tab.
      2. For each one, decide: did the AI not know the rule, or did it believe a wrong fact?
      3. Turn on what fixes it, predict which emails will change, then run. You have 3 runs.
      Estimated business impact per year
      –
      Choose your setup and press Run
      0 correct0 to a human0 wrong
      $0.90 per email to run the AI

      Your runs this round

      No runs yet. Each run answers the same 10 emails.

      Customer emails

      10 emails · run the AI to see its answers

      Leaderboard · business impact

        Instructions

        The situation

        Aurora trained a smaller model on thousands of past decisions. It costs $0.02 per email instead of $0.90, but the engineering work to build it costs $100k, spread over 3 years.

        Your goal

        Decide whether the cheaper model is good enough, and which tools you still need.

        How to play

        1. Switch to the small fine-tuned model.
        2. Run and compare your impact with last round.
        3. Each tool adds to the cost per email. Check which ones still pay for themselves.
        Estimated business impact per year
        –
        Choose your setup and press Run
        0 correct0 to a human0 wrong
        $0.90 per email to run the AI

        Your runs this round

        No runs yet. Each run answers the same 10 emails.

        Customer emails

        10 emails · run the AI to see its answers

        Leaderboard · business impact

          Instructions

          The situation

          Aurora just revised its refund policy and published it as a new document, Document C. Document B still has the old rules, and so does the fine-tuned model.

          Your goal

          Keep your business impact up under the new rules.

          How to play

          1. Read Document C and run your current setup to see which emails now fail.
          2. Decide which tools keep up with the change.
          3. Adjust your setup and run again.
          Estimated business impact per year
          –
          Choose your setup and press Run
          0 correct0 to a human0 wrong
          $0.90 per email to run the AI

          Your runs this round

          No runs yet. Each run answers the same 10 emails.

          Customer emails

          10 emails · run the AI to see its answers

          Leaderboard · business impact

            Instructions

            The situation

            The game is over.

            Your goal

            Connect what you did to the four levers: prompt, policy document, booking lookup, and model.

            How to play

            1. See how each choice moved your business impact.
            2. Pick one decision that paid off and one that didn't.
            3. Be ready to share them with the class.
            Final business impact per year
            –

            Your rounds

            Your submitted run in each round
            RoundWhat you usedAccuracyTo a humanWrongImpact
            1

            Accuracy isn't the business metric

            Different mistakes cost different amounts: a wrongful denial costs Aurora 15 times what a correct answer earns, so a more accurate AI can lose more money.

            2

            Each lever fixes a different problem

            The policy document gives the rules, the booking lookup gives the facts, the prompt sets the behavior, and fine-tuning cuts the cost per email.

            3

            Escalate when unsure, not always

            Escalation pays when the AI is less than 78% sure about a refund, or less than 92% sure about a denial.

            4

            Keep changing rules easy to update

            A fine-tuned model learns the rules it was trained on. When the policy changes, a setup that reads the current policy document keeps up, and one that relies on memory falls behind.

            Your final run

            Final leaderboard · business impact

              Instructor only

              Class controls

              Students never see this tab. They only see the steps you release, one at a time.

              Class session

              Create a session, then share its join code and class password
              Join code–

              Release steps

              Hidden steps do not appear for students
              1
              AccuracyPrompt only. Dollars stay hidden.
              Always on
              2
              The revealShow business impact and the reshuffled leaderboard.
              3
              Rules & factsUnlock the policy document and booking lookup.
              4
              Fine-tuned modelUnlock the small fine-tuned model.
              5
              SurpriseAnnounce the policy change and switch to the new answer key.
              6
              DebriefShow the journey table and takeaways.

              Class settings

              Allow runsTeams can press Run. Turn off while you talk.
              Show the leaderboardHide it while teams work; show it to build suspense.
              Show other teams' setupsThe leaderboard lists which levers each team used.

              Class status

              Live
              Students joined–
              Runs in this step–

              Reset runs

              Deletes every student's runs in this session.