Detester is a .NET library that enables you to write deterministic tests for AI-powered applications. It provides a fluent builder API for testing AI responses, ensuring consistency and reliability in your AI integrations.
- Fluent Builder API: Chain multiple prompts and assertions in a readable, intuitive way
- Any AI Provider Support: Works with any
IChatClientimplementation (OpenAI, Azure OpenAI, Ollama, etc.) - Model Instructions: Set system messages to guide model behavior and responses
- Response Validation: Assert that AI responses contain expected keywords or text
- Function/Tool Call Verification: Verify that AI models call the correct functions with expected parameters
- JSON Response Validation: Deserialize and validate JSON responses from AI models with type-safe validation
- Method Chaining: Combine multiple prompts and assertions in a single test flow
- Extensible: Build on Microsoft.Extensions.AI abstractions for maximum flexibility
dotnet add package DetesterFor OpenAI or Azure OpenAI support, also install:
dotnet add package Microsoft.Extensions.AI.OpenAI
dotnet add package Azure.AI.OpenAICheck out the Wiki page for a detailed documentation
using Azure.AI.OpenAI;
using Detester;
using Microsoft.Extensions.AI;
using System.ClientModel;
// Create Azure OpenAI client and wrap it as IChatClient
var azureClient = new AzureOpenAIClient(
new Uri("https://your-resource.openai.azure.com"),
new ApiKeyCredential("your-azure-api-key"));
var chatClient = azureClient.GetChatClient("gpt-4-deployment").AsIChatClient();
// Create a builder with the chat client
var builder = DetesterFactory.Create(chatClient);
// Execute a test
await builder
.WithPrompt("Explain quantum computing in simple terms")
.ShouldContainResponse("quantum")
.AssertAsync();Detester works with any IChatClient implementation from Microsoft.Extensions.AI:
using Detester;
using Microsoft.Extensions.AI;
// Use any IChatClient implementation (OpenAI, Azure OpenAI, Ollama, custom, etc.)
IChatClient chatClient = // your chat client implementation
var builder = DetesterFactory.Create(chatClient);
await builder
.WithPrompt("Test prompt")
.ShouldContainResponse("expected text")
.AssertAsync();Set custom instructions (system messages) to guide the model's behavior:
await builder
.WithInstruction("You are a helpful assistant that provides concise answers.")
.WithPrompt("What is machine learning?")
.ShouldContainResponse("algorithm")
.AssertAsync();Test conversational flows by chaining multiple prompts:
await builder
.WithPrompt("Hello, I need help with coding")
.WithPrompt("Can you explain what a variable is?")
.ShouldContainResponse("variable")
.AssertAsync();Use OrShouldContainResponse to create flexible response validation where at least one of the alternatives must match:
await builder
.WithPrompt("What is the capital of France?")
.ShouldContainResponse("capital")
.OrShouldContainResponse("city")
.OrShouldContainResponse("Paris")
.AssertAsync();In this example, the test passes if the response contains "capital" OR "city" OR "Paris". You can chain multiple OR conditions, and the test will pass if any one of them is found in the response.
Detester supports validating JSON responses from AI models by deserializing them to C# types and optionally validating the deserialized objects. This is useful for testing structured outputs from language models.
Verify that the response can be deserialized to a specific type:
public class User
{
public string? FirstName { get; set; }
public string? LastName { get; set; }
public int Age { get; set; }
public DateTime JoinDate { get; set; }
}
await builder
.WithPrompt("Who is the last user joined?")
.ShouldHaveJsonOfType<User>(new JsonSerializerOptions { PropertyNameCaseInsensitive = true })
.AssertAsync();Add custom validation logic to verify the deserialized object:
await builder
.WithPrompt("Who is the last user joined?")
.ShouldHaveJsonOfType<User>(
new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
user => user.Age > 30 && user.FirstName!.Contains("Jo"))
.AssertAsync();Combine multiple validations:
await builder
.WithPrompt("Get user details")
.ShouldContainResponse("Joe") // Text assertion
.ShouldHaveJsonOfType<User>(
new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
user => user.Age > 18) // JSON validation
.ShouldHaveJsonOfType<User>(
new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
user => user.LastName == "Doe") // Additional JSON validation
.AssertAsync();Note:
- The JSON validation uses
System.Text.Jsonfor deserialization - Deserialization exceptions are caught and wrapped in
DetesterExceptionwith helpful error messages - If validation fails, the test throws
DetesterExceptionwith details about what went wrong - For case-insensitive property name matching, use
JsonSerializerOptions { PropertyNameCaseInsensitive = true }
Detester supports verifying that AI models call the correct functions/tools with expected parameters. This is useful for testing AI applications that use function calling capabilities.
Verify that a specific function is called:
await builder
.WithPrompt("What's the weather in Paris?")
.ShouldCallFunction("get_weather")
.AssertAsync();Check that functions are called with the correct parameters:
await builder
.WithPrompt("What's the weather in Paris in celsius?")
.ShouldCallFunctionWithParameters("get_weather",
new Dictionary<string, object?>
{
{ "location", "Paris" },
{ "units", "celsius" }
})
.AssertAsync();Verify multiple function calls in a single response:
await builder
.WithPrompt("Compare the weather in Paris and London")
.ShouldCallFunction("get_weather")
.ShouldCallFunction("get_weather")
.AssertAsync();Combine function call verification with text response assertions:
await builder
.WithPrompt("What's the capital of France?")
.ShouldCallFunction("get_capital")
.ShouldContainResponse("Paris")
.AssertAsync();For more detailed information and examples, see the Function Calling Guide.
Detester throws DetesterException when:
- No prompts are provided before execution
- Expected text is not found in the response
- None of the OR alternatives are found in the response
- Expected function call is not found
- JSON deserialization fails or validation fails
Detester throws InvalidOperationException when:
OrShouldContainResponseis called without a prior assertion
Example:
try
{
await builder
.WithPrompt("What is AI?")
.ShouldContainResponse("impossible text that won't appear")
.AssertAsync();
}
catch (DetesterException ex)
{
Console.WriteLine($"Test failed: {ex.Message}");
}Contributions are welcome! Please feel free to submit a Pull Request.
This project is licensed under the MIT License.
Built on top of Microsoft.Extensions.AI for seamless integration with AI services.