> For the complete documentation index, see [llms.txt](https://docs.qyrus.com/aiverse/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.qyrus.com/aiverse/bottesting.md).

# Bot Testing

![Banner](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-a0868feab51603752f43b973b5ccfce5c5ead4b1%2Fbot_testing.png?alt=media)

Bot Testing is your ultimate conversational API sandbox! It's an advanced testing engine where a dynamic LLM acts as a real-world user to roleplay, stress-test, and evaluate your chatbot APIs under realistic pressure. 🤖

Simply connect your bot's endpoint, and the platform automatically spins up diverse user personas, multilingual typos, and intent coverages to push your chatbot to its absolute limits. 🚀

Gain total visibility and release-ready confidence with real-time execution streams and automatic safety signal grading, making manual chatbot validation a breeze! 🌬️

***

## 📋 Prerequisites

Before using the Bot Testing harness, ensure you have set up the following details:

### 1. Bot Connection Details (The Hook)

* **API Endpoint & Headers**: You must provide your chatbot's request URL and any required authorization headers or API keys.
* **Body Template**: A JSON schema showing how to send messages to your bot. The harness will automatically discover your conversation history structure and map placeholders like `{{question}}` and `{{history}}`.

### 2. RAG Evaluation Pre-requisites (If Grounding is Enabled)

* **Citations Key**: If RAG evaluation is turned on, your chatbot's response payload must include retrieved source chunks or documents (usually returned under a key like `citations` or `context`). This is required for the LLM judge to verify Faithfulness and RAG quality.

### 3. Test Data & Dynamic Context (Orders, Transactions, Accounts)

* **Randomized Mock Data (Default)**: If no specific data is provided, the simulator dynamically invents realistic mock details (fake names, transaction IDs, order numbers) to keep the conversation flowing.
* **Real Test Data (User-Defined)**: If you need the bot to query real system-of-record databases during the test, you must supply valid, active test records (like a real `order_id` or `account_number`) directly in your scenario descriptions or custom persona stylings.

***

## ✅ Step-by-Step Guide

### 📁 Step 1: Manage or Create Projects

You can either select an existing project or click on **+ Create Project** to start a new one. Provide the required project name and description.

![Bot Gym Projects Dashboard](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-65c66c2555b02a38d046e7ad6419c25c327a68ea%2Fbot_testing_image3.png?alt=media)

![Create New Project Modal](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-e2c3578207b28722666d408bc8aecbd14519bc6f%2Fbot_testing_image4.png?alt=media)

### 🚀 Step 2: Create a New Gym

Inside your selected project, click on **+ Create First Gym** (or **+ Create Gym** at the top right) to configure a target testing environment for your bot.

![No Test Gyms Page](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-3adf7e1eaeb9209c5db66c887fc9082270730b7a%2Fbot_testing_image5.png?alt=media)

### 📝 Step 3: Define Gym Basics

Provide the **Test Gym Name** (which can be any name you prefer) and add a detailed explanation/description of your chatbot in the **Application Description / About** text box. This gives context to the generated personas.

![Gym Basics Setup](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-8c1903f8fef4f2a5c14aaf3145998661c9380c3b%2Fbot_testing_image6.png?alt=media)

### 🔌 Step 4: Configure Bot API

Define how Pulse calls your chatbot:

1. Select the **HTTP Method** (e.g., `POST`).
2. Enter the **Bot API Endpoint URL**.
3. Add any required **Headers** (Key/Value pairs) if supported.
4. Set the **Bot Response Key Path** (the dot-separated key in the response JSON containing the bot message, e.g., `answer`).
5. Design the **Body Payload Template (JSON)** containing placeholders such as `{{question}}` where test queries should be injected. and `{{bot_response}}` for the bot\_response to maintain history of conversation

![Bot API Configuration](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-c56babaf4457a466b02ed21dc05a5a2592c63cc3%2Fbot_testing_image7.png?alt=media)

### 🎯 Step 5: Add Intents

Provide the intents that your bot supports. Click **+ Add Intent** and enter:

* **Intent Name**
* **Description** of the intent/behavior you want to test.
* **Expected Outcome** (the criteria the bot should satisfy).

![Intents Configuration](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-da8ebe9a358ff5c97d3e7865019a10d4a377022f%2Fbot_testing_image8.png?alt=media)

### ⚙️ Step 6: Configure Advanced Settings

Tune the evaluation settings:

* **Max Turns**: Specify the maximum number of conversation turns made in each test session.
* **Judge Model**: Select the LLM model to be used for evaluating the chatbot/RAG responses.
* **RAG Integration**: Toggle **On** if your bot uses RAG and provides citations. Specify the **RAG Citations Key Path** to extract citations from the bot response.
* **Supported Locales**: Add any other languages your chatbot supports.
* **Check Usage**: Toggle this on to monitor token usage limits for gym runs by specifying the model and estimating token parameters.

![Advanced Settings Configuration](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-e786f73fcdef2b5b43e49b26a874c64cfb74de64%2Fbot_testing_image9.png?alt=media)

### 🛡️ Step 7: Establish Guardrails

Provide any high-priority **Guardrails & Safety Guidelines** to outline compliance rules and domain safety boundaries (e.g., *“The response should be only related to bank domain”*).

![Guardrails Configuration](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-dc588baf5f0a6213ac14c74462a15dfe14a74195%2Fbot_testing_image10.png?alt=media)

### 🔍 Step 8: Review and Launch

Review the configured sections. Click **Edit** on any block to make adjustments. Once everything looks correct, click **Save & Generate Personas** to save your configurations and automatically generate 3 testing personas/scenarios.

![Review & Launch Config](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-3ed5b8cc3e51fd2c82ce61edcdec39029e7ee0b3%2Fbot_testing_image11.png?alt=media)

### 🎭 Step 9: Scenarios Dashboard

Once the 3 scenarios/personas are generated, they are displayed on the dashboard under the **Scenarios** tab. This view includes the **Scenario Name**, **Style/Tone & Description** of that persona, **Base Signals** (test vectors) to be tested, and the targeted **Intents** for each scenario.

![Scenarios Dashboard](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-82b0cfffdbd3b7f52c787acd48c6a4a2c0257a2a%2Fbot_testing_image12.png?alt=media)

### ✍️ Step 10: Editing Scenarios

You can manually customize any scenario. Click on the edit icon next to a scenario to open the **Edit Persona Details** panel, where you can modify the persona's details (such as identity name, behavior description, and style/tone) and click **Save Persona Changes**.

![Edit Persona Details](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-e229fed1ad5ddb69af1120c17bef420002ed6394%2Fbot_testing_image14.png?alt=media)

### ➕ Step 11: Generating More Scenarios

You can expand your test coverage by generating additional scenarios. Click the **Generate More** button (shown on the Scenarios list). In the **Generate More Scenarios** panel, you will see a list of unique available personas. Check the scenarios you wish to add (e.g. up to 10) and click **Generate** to add them to your scenario dashboard list.

![Generate More Scenarios Panel](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-2a5ec221a0c91bbe8d98a335e21b6992b1cad30c%2Fbot_testing_image15.png?alt=media)

![Updated Scenarios List](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-59d1ee2a4f2b830c92e9c930fbe1bf47ccf5dff1%2Fbot_testing_image16.png?alt=media)

### ⚙️ Step 12: Gym Configuration & Re-generating

If you need to change your gym configurations, click on the **Config** tab. Note that if you modify basics like intents or application description, you will need to regenerate scenarios. Click the **Update & Re-Generate Personas** button, which will discard the older scenarios and generate new ones reflecting your changes.

![Gym Configuration Overview](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-bf779beed8180d89086c91559c3835c39081db48%2Fbot_testing_image13.png?alt=media)

### 🚀 Step 13: Running the Test Pipeline

To execute your tests, click the green **Run** button. In the **Run Test Pipeline** side panel, specify a **Run name**, select the personas you wish to execute, and click **Execute Run**.

![Run Test Pipeline Panel](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-48adaedc7c6e01b83643d6eea85d6152e7470abb%2Fbot_testing_image17.png?alt=media)

### 📊 Step 14: Accessing Reports

Once the execution starts, navigate to the **Reports** tab to monitor its status. When the status changes to `COMPLETED`, click the **View Report** button.

![Reports Tab Execution Run](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-cad0bcd1e1368a3ffd2f21cb0ba004c04cd166db%2Fbot_testing_image18.png?alt=media)

### 📈 Step 15: Overall Summary Report

The execution report contains two main tabs: **Summary** and **Scenarios**. Under the **Summary** tab, you will find:

* **Overall Grade**: Graded from **A+** (best bot) to **D** (indicating the bot is not fit for production yet), along with a detailed reasoning statement.
* **Pass/Fail Breakdown**: Shows the count of passed and failed intents and scenarios.
* **Evaluation Metrics**:
  * **Hallucination (%)**: Measures how much of the answer contains information that is not supported by the retrieved documents (lower is better).
  * **Faithfulness (%)**: Measures how accurately the answer stays grounded in and reflects the retrieved documents without inventing facts (higher is better).
  * **Relevance (%)**: Measures how well the answer addresses the user's question using the retrieved information (higher is better).
  * **Coverage (%) (also called Completeness)**: Measures how much of the important information from the retrieved documents needed to answer the question is included in the response (higher is better).
* **Run Activity**: Visualizes conversation turns, altered inputs (perturbations like typos, slang, or abbreviations added to test robustness), turns with invented details, and any judge evaluation errors.
* **Signal Alerts & Severity Breakdown**: Displays validation alerts which can show severity categories: **P0\_1 Blocker**, **P1\_High**, **P2\_MEDIUM**, and **P3\_LOW**.

![Execution Run Summary Report](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-d695ad5b9b07705420dbf40b821e5af02a5ef3dc%2Fbot_testing_image19.png?alt=media)

### 🗂️ Step 16: Scenarios Report Tab

Switch to the **Scenarios** tab to drill down into the results for individual scenarios. You can apply filters (All, Failed, Passed, Inconclusive, Invented details, Altered inputs) to focus your analysis. Selecting a scenario displays a summary panel showing its passed/failed intents, faithfulness %, relevance %, and completeness %.

![Scenarios Report Filtering and Summary](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-0bf30c6fd0ce3a541b22ac826b8bf61da6a31101%2Fbot_testing_image20.png?alt=media)

### 💬 Step 17: Conversation Turns Details

Inside the selected scenario, switch to the **Turns** tab to see step-by-step chat messages between the user persona and the bot.

* It shows the input and bot response for each turn.
* Any perturbations (e.g., typos, punctuation, informal shortening) are highlighted with labels (e.g., `typo`, `punctuation`, `informal shortening`) showing what changed.
* You can view turn-level metrics (tokens, faithfulness, relevance, hallucination flags) and the specific reasons/evidence why the judge scored the turn that way (including unsupported claims/invented details).
* The **Diagnostic Alert Log** at the bottom lists critical warnings, severity, evidence, and recommendations.

![Conversation Turn Details](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-919073870d64f185cb60700a9defe50755d585b1%2Fbot_testing_image21.png?alt=media)

![Diagnostic Alert Log](https://3321078861-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGs5aM0ieFmNKqLU6gqpk%2Fuploads%2Fgit-blob-dea777bc5128992d6d39fd12f3e0df37cb11f97b%2Fbot_testing_image22.png?alt=media)

***

## ❓ Q/A – Frequently Asked Questions (Bot Testing)

**Q: What kind of chatbots does the platform support?**\
**A:** The platform supports any HTTP POST-based chatbot API, ensuring compatibility with custom LLM applications, RAG search engines, and standard conversational agents.

**Q: Can I change the bot configuration after starting a simulation, and are results saved?**\
**A:** Bot configurations cannot be modified post-execution. However, all simulation runs and full turn-by-turn trajectories are saved automatically under the run details database and secure S3 storage.

**Q: How does the platform handle chatbot authentication (e.g., API keys, bearer tokens)?**\
**A:** Use the bot API headers configuration to securely define API keys, authorization tokens, or custom headers that need to be injected into requests during the simulation.

**Q: Can the platform manage large numbers of test personas and complex scenarios?**\
**A:** Yes. The platform concurrent-limits execution to 5 personas at a time. If you have many scenarios, the worker queues them up and executes them in parallel chunks seamlessly.

**Q: How can I quickly understand a failed test?**\
**A:** Every run includes a complete turn-by-turn trajectory showing the exact user query, bot reply, triggered compliance signals (like loops or crashes), and the LLM judge's reasoning.

**Q: What are "Signals" in Bot Testing?**\
**A:** Signals are automated validation checks (e.g., checking for RAG hallucinations, duplicate repeating answers, or endpoint connection timeouts) that identify conversational bugs and safety issues during a turn.

**Q: What is stochastic multilingual testing?**\
**A:** It allows the simulator to randomly select a locale from your configured list (e.g. English, Spanish, Hinglish) on each turn, enabling comprehensive multi-language robustness testing.

**Q: Does the platform store actual chatbot conversation data?**\
**A:** Yes. The platform saves the full turn-by-turn trajectory document securely in S3 for auditing, debugging, and historical performance tracking.

**Q: What does the final Letter Grade (e.g., A, B, C) mean in the report?**\
**A:** It represents an LLM-calculated release readiness grade based on overall faithfulness, response relevance, and whether any critical blocker signals were triggered.

***

We’re excited for you to experience the power of Bot Testing! 🌟