GitHub Models evaluations tooling now supports multiple variables with any names, not just {{input}}!

Previously, the tooling only recognized a single variable named {{input}}. Prompts using variables like {{experience}} or {{city}} would fail to run, even though they worked perfectly in the prompt editor.

Evaluators are like tests to make sure your AI is doing what you expect. They’re like unit tests for AI outputs. This helps you catch mistakes early and build more reliable AI features.

The evaluations tooling (toggle to ‘compare’ from the prompt editor) now:

  • Detects all variables inside {{ }}, no matter what they’re called.
  • Supports prompts with multiple variables, not just one.
  • Behaves consistently with the prompt editor.

Why this matters

Many real-world prompts rely on multiple variables. Here are some examples of the kinds of prompts you can now test more effectively:

  • Travel or shopping assistants using {{city}}, {{intent}}, and {{budget}} to tailor recommendations.
  • Email and content generators with {{tone}}, {{audience}}, and {{length}} to test brand consistency.
  • Game dialogue engines that respond to {{character_trait}}, {{quest_stage}}, and {{player_choice}} to keep interactions dynamic.

How to try it out

Write a prompt using any number of {{variable}} placeholders and paste it into the comparison view. You’ll be prompted to fill in each variable before running an evaluation.

Or, add them in your .prompt.yml file (see formatting here) system or user prompt to automate the process of evaluating with multiple variables in the future.

Start building AI apps with GitHub Models today

GitHub Models and all our AI development tooling are available now to all GitHub users in public preview. This includes prompt editing and lightweight evaluations. Try our tools out by enabling them in your repository or organization, or learn more in our documentation.

Help us shape what’s next

We’re just getting started, and your feedback helps guide our roadmap. Join the community discussion to share your thoughts and connect with other developers building the future of AI on GitHub.