Tests Basics in Kraken CI

Introduction

Branch Results

Run Box

  • red: one or more jobs finished with an error
  • yellow: no errors in jobs but some test results failed or there are
    issues discovered in the code
  • green: all jobs finished without error and there are no failures in
    tests and no issues in the code
  • data about execution: total number of scheduled jobs, number of
    erred jobs, etc.
  • summary of testing: pass ratio, total, regressions and fixes counts, etc.
  • issues summary: total number of discovered issues, number of new issues

Test Case Results Table

  • the result status
  • a name of the test case
  • job name and ID
  • age, instability and relevancy
  • values
  • execution environment: system, agent group and actual agent and config

Filtering Test Case Results

  • by Results Status ie. Passed, Failed, etc.
  • by Result Change ie. Fixes, Regressions, New and No
    Changes
    ; the change is determined between the current result and
    the previous result from previous run
  • by Age where age is a number of previous runs with the same result
    as in the current run; so e.g. age 0 means that in the previous run test
    case had a different result than it has now (so these are regressions
    and fixes); age 1 means that the current run and the previous run has the
    same result but the result before the previous one is different
  • New is a shortcut to set Age between 0 and 0, ie. show changes
    that happened in the last run (fixes + regressions)
  • by Instability where instability is a count of result status
    changes in the last 10 runs; if this is 0 then in the last 10 runs test
    case has the same result (it was pretty stable); if this is 10 then
    it means that in every run of the last 10 runs the status was
    changing compared to the previous one (it was very unstable)
  • by Test Case name
  • by Job name of ID
  • New is a shortcut to set Age between 0 and 0, ie. show changes
    that happend in last run (fixes + regressions)
  • by Instability where instability is a count of result status
    changes in last 10 runs; if this is 0 then in the last 10 runs test
    case has the same result (it was pretty stable); if this is 10 then
    it means that in every run of the last 10 runs the status was
    changing comparing to previous one (it was very unstable)
  • by Test Case name
  • by Job name of ID

Sorting Test Case Results

  • name
  • result
  • change
  • age
  • instability
  • relevancy
  • 0 is initial
  • +1 for not passed
  • +1 for not root caused problem
  • +1 for failure
  • +1 for instability <= 3
  • +1 for age < 5
  • +1 for regression

Comments

  • new
  • investigating
  • bug in product
  • bug in test

Test Case History

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Michal Nowikowski

Michal Nowikowski

Kraken CI Founder. I’m software engineer focused on full-stack programming and improving software processes.