Test Reports
Reports
The reports are located in the top right corner of the screen. Clicking the button will take you to the reports splash page.
The splash page shows the user the current metrics for tests ran.
Project summary lists all of the test suites test scripts and top users in a project.
The range of metrics can be changed with the graph time selection.
Running Tests
All of the executions that are currently running will be displayed in the “Running Tests” tab. There is a legend on the right most section of the screen that displays how many tests are currently running and how many have encountered an error in starting an execution.
Click on the “Refresh” button to manually fetch the latest status of any running tests. Or set the dropdown of “Refresh Time” to automatically update the execution statuses based on the selected time.
The flow of execution status is below:
The Test Queued status will have one of the following information after it:
Up Next - this should be the next test that is executed
A number - how many executions are ahead of the test in the device queue
NA - queue information for the test is not availalbe at the moment
Details of Running Tests
App Name: app file name, app activity or bundle id used to run the execution
Test Script Name: name of script/suite that is being executed
Device: name of selected device used for execution
Device Type: displays whether the selected device is shared or dedicated.
Version: selected device version
Start Date Time: execution start time
Status: current state of execution
Actions: actions available for an execution
Actions
Live Preview
Click on the “Live Preview” button to see the stream from the device as the test is being executed along with the description of each executed step and its Pass/Fail status.
Once the execution is finished in the Live Preview page, the screen will be routed to the “Test Results” tab.
Stop Execution
Click on the “Stop” button to open the modal below and click the “Stop Run” button to terminate an execution or click the “Cancel” button to continue the test.
Error in Test
An execution can have this status if one of the following happened:
Problem with creating a session
Previous status was Allocating Device
Problem with updating results
Previous status was Uploading Results
Error with queueing test to device
Previous status was Queueing in Progress
Hover over the “Error in Test” status to learn what went wrong.
Click on the “Retry” button to try the execution again.
Test Results
Displays all the completed tests from the last 6 months and an overview of each execution.
Overview of Test Results
Icon: Depending on the configuration used to run the execution, there might be an icon to signify that configuration before the app name column of each execution. There are 3 categories of icons:
Test Configuration
Scheduled Test Type
Plugin Type
App Name: app file name, app activity or bundle id used to run the execution
Test Name: name of script/suite that is being executed
Start Date Time: execution start time
Duration: how long an execution took
Device: name of selected device used for execution
Device Type: displays whether the selected device is shared or dedicated.
Status: displays result of an execution
Pass
Fail
Cancelled
Actions: actions available for an execution
Icons
Test Configuration
Healer icon will appear if an execution was run with healer option enabled
Email icon will appear if option to email report was selected for execution
Scheduled Test Type
Execution was scheduled only once.
Execution was scheduled daily.
Execution was scheduled on a weekly basis.
Plugin Type
Execution was triggered from a plugin like Jenkins, Azure or Qyrus CLI
Test Result Actions
Click on the ellipsis in the “Actions” column to get list of available actions for a test.
View Script/ View Suite
If the execution type was script, then the “View Script” option will appear and clicking it will route to the steps page of that script. However, if it was a suite execution, “View Suite” option would be present and clicking it will route to the scripts page of that suite.
Play Video
Plays the video recording of the execution.
Email
Sends the test report to the emails specified in the modal that opens.
Download Report
Downloads the zip report of the test.
Retry
Runs the execution again
Create Jira Ticket
Creates a Jira ticket using the any linked Jira configurations for the project with the report as an attachment.
Delete Report
Opens a modal to delete a report or cancel the deletion.
Click on any row of the reports table to view more details about the execution.
Detailed Execution Report
The page displays the complete report of an execution. This page can be broken down into 3 main sections:
Metadata
Test Report
Performance
Metadata
Test Case Scenarios
Displays the total number of scripts executed and breaks down how many passed/failed.
Report Actions
Plays recording of execution
Downloads a zip report of execution
Sends execution report as an email
Environment Name
Displays the environment used for global variables during execution
Date and Time
Start and end times of execution along with how long it took to complete
Device Info
Details about device used for execution
Device Name
Device Type (Android/iOS)
Version
Test Report
The report is broken down by each script in the execution and the name of the script is displayed in the “Test Scenario” column.
Test Report Details
Test Scenario: name of script in the execution
Steps: step number
Step Description: description of step. Might include the error message for failed steps
Expected Result: expected result for a verifier action step
Actual Result: actual result of a step. Might be only available for verifier and “Create Variable” actions
Device Type: displays whether the selected device is shared or dedicated.
Status: displays result of a step
Pass
Fail
Execution Time: displays how long a script took to execute
View Screenshot: clicking on the button opens a screenshot of the device after the step was executed
View Script: clicking on the button routes to the steps page of that script. This option is currently only available for script executions.
Performance
Displays the following device metrics during the execution:
Battery Statistics
Battery Status i.e. charging
Battery Charge
Power consumed by app
CPU Utilization
Memory Consumption
Network Usage Statistics
Hover over any of the keys in the legends of the graphs to view only the graph for that key.
*** Note: This feature is in beta so it might contain some incorrect data. Also, this feature is currently only available for Android devices. ***
Last updated