Skip to main content

​
Variant Assignment

Variant assignment is the process of deciding which Variant a specific user or request should be routed to for a given Experiment.
  • Mechanism: AgentTest’s backend assigns variants primarily based on the trafficPercent you define for each variant within an active experiment. When your application calls the assignment endpoint, the system uses this distribution to pick a variant.
  • Endpoint: This is handled by the POST /experiment/:slug/assign endpoint. Your application sends a userId and optional context, and AgentTest responds with the key and payload of the assigned variant.
  • Stateless Backend: The core assignment logic in the backend is stateless. It doesn’t store a persistent record of which user was assigned to which variant over time (unless you explicitly log this information via the logging endpoint). Each call to /assign is typically independent.
  • User Consistency: If you need a user to consistently see the same variant across multiple requests or sessions, your client-side application or SDK will need to handle this. This often involves caching the assigned variant’s key and payload for a specific userId for a certain duration or until the session ends.
The goal of assignment is to expose different segments of your users to different variants in a controlled manner.

​
Result Logging

After a variant has been assigned and your AI agent or prompt workflow has executed using the variant’s payload, it’s crucial to log the outcome.
  • Purpose: Logging captures the data needed to compare the performance of different variants. Without logs, you can’t measure which variant is better.
  • Endpoint: This is done using the POST /experiment/:slug/log endpoint.
  • What is Logged (a “Result”): A log entry, often referred to as a “Result,” typically stores:
    • The variantKey that was used.
    • The input provided to the agent/workflow (e.g., user query).
    • The output generated by the agent/workflow (e.g., LLM response).
    • Custom metrics you define (e.g., latency in milliseconds, token count, user rating, conversion event).
    • Any relevant context (e.g., session ID, environment).
  • Importance: These logs form the dataset for your A/B test analysis. By collecting inputs, outputs, and metrics for each variant, you can compare their performance on key indicators, understand user interactions, and ultimately make data-driven decisions about which prompts, models, or configurations are most effective.
The AgentTest dashboard (coming soon) will use these logged results to provide visualizations and statistical analysis of your experiments.
⌘I