​
Roadmap

AgentTest is an actively developing open-source project, and we have exciting plans for its future! Our goal is to make it the go-to A/B testing and analytics platform for AI agents and prompt workflows. Below are some of the key features and improvements we are planning. This list is not exhaustive and may evolve based on community feedback and technological advancements.

​
Key Planned Features

  • Comprehensive SDKs:
    • Dedicated SDKs for Python and JavaScript/TypeScript.
    • Specific integrations for AI frameworks like LangChain and LangGraph to make A/B testing seamless within those ecosystems.
  • Interactive Dashboard:
    • A web-based dashboard for creating and managing experiments.
    • Visualization of key metrics and statistical analysis of experiment results.
    • Controls for starting, pausing, and stopping experiments.
    • User interface for configuring variants and traffic distribution.
  • OpenTelemetry (OTel) Support:
    • Integration with OpenTelemetry for enhanced observability and tracing of AI agent behavior across different variants. This will allow you to see detailed traces and correlate them with your A/B test data.
  • Webhook Events:
    • Support for configurable webhooks that can be triggered by events such as:
      • New variant assignment.
      • New result/log entry.
    • This will enable real-time integration with other systems or custom alerting.
  • Custom Metric Definitions & Leaderboards:
    • Ability to define custom metrics more formally within the platform.
    • Leaderboards to easily compare the performance of variants across multiple metrics and experiments.
  • Advanced Experimentation Features:
    • Support for more complex experiment types (e.g., multi-armed bandits, sequential testing) in the long term.
    • More granular targeting and segmentation options.

​
Community Driven

The AgentTest roadmap is influenced by the needs of its users. We encourage you to:
  • Suggest Features: Let us know what features would be most valuable to you by opening an issue on our GitHub repository.
  • Contribute: We welcome contributions to the project, whether it’s code, documentation, or feedback.
Stay tuned for updates, and thank you for being part of the AgentTest community!