Tests Basics in Kraken CI

Introduction

Testing in Kraken CI is much more organized that it is in e.g. Jenkins. Testing is planned in branches and high level view on results is also presented for branches. Then, it is possible to dig deeper into runs, jobs and particular test case results. More details about all this can be found in the following sections.

Branch Results

The Branch Results page presents high-level information about the flows
and runs of a given branch. It organizes flows in rows. The top row
contains the latest flow. Each flow row contains a list of boxes. Each
box contains information about a run of a given stage. The order of
boxes with runs is chronological according to their start time.

Run Box

The color of the run box indicates the final status of the run:

  • red: one or more jobs finished with an error
  • yellow: no errors in jobs but some test results failed or there are
    issues discovered in the code
  • green: all jobs finished without error and there are no failures in
    tests and no issues in the code
  • data about execution: total number of scheduled jobs, number of
    erred jobs, etc.
  • summary of testing: pass ratio, total, regressions and fixes counts, etc.
  • issues summary: total number of discovered issues, number of new issues

Test Case Results Table

A job that performs testing reports test results. The test results are
visible on a Run Results page, in Test Results tab. They are
presented in a table.

  • the result status
  • a name of the test case
  • job name and ID
  • age, instability and relevancy
  • values
  • execution environment: system, agent group and actual agent and config

Filtering Test Case Results

It is possible to filter test results by various attributes:

  • by Results Status ie. Passed, Failed, etc.
  • by Result Change ie. Fixes, Regressions, New and No
    Changes
    ; the change is determined between the current result and
    the previous result from previous run
  • by Age where age is a number of previous runs with the same result
    as in the current run; so e.g. age 0 means that in the previous run test
    case had a different result than it has now (so these are regressions
    and fixes); age 1 means that the current run and the previous run has the
    same result but the result before the previous one is different
  • New is a shortcut to set Age between 0 and 0, ie. show changes
    that happened in the last run (fixes + regressions)
  • by Instability where instability is a count of result status
    changes in the last 10 runs; if this is 0 then in the last 10 runs test
    case has the same result (it was pretty stable); if this is 10 then
    it means that in every run of the last 10 runs the status was
    changing compared to the previous one (it was very unstable)
  • by Test Case name
  • by Job name of ID
  • New is a shortcut to set Age between 0 and 0, ie. show changes
    that happend in last run (fixes + regressions)
  • by Instability where instability is a count of result status
    changes in last 10 runs; if this is 0 then in the last 10 runs test
    case has the same result (it was pretty stable); if this is 10 then
    it means that in every run of the last 10 runs the status was
    changing comparing to previous one (it was very unstable)
  • by Test Case name
  • by Job name of ID

Sorting Test Case Results

Test case results can be sorted. This can be done using several columns:

  • name
  • result
  • change
  • age
  • instability
  • relevancy
  • 0 is initial
  • +1 for not passed
  • +1 for not root caused problem
  • +1 for failure
  • +1 for instability <= 3
  • +1 for age < 5
  • +1 for regression

Comments

In the test case results table, it is possible to add sticky comments
to test case results. When a new failure is spotted, a comment
stating, e.g. the reason for the failure or link to a bug in a bug
tracker may be added.

  • new
  • investigating
  • bug in product
  • bug in test

Test Case History

Each test case has its history of results. In the table with test case
results a link under the test case name leads to a page dedicated to the given
test case. This page presents a table with the historical results of the
test case.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store