What is Wrong with Your Testing?

  • a project was not that big, so it was not that hard to achieve 100%
  • lots of failing or problematic tests were marked as skip, so they are not executed
  • there was not many tests or no tests at all, i.e. tests coverage was non-existing


ISTQB defines several goals for testing. Two of them that are pretty important are:

  1. To find defects and failures thus reduce the level of risk of inadequate software quality.
  2. To provide sufficient information to stakeholders to allow them to
    make informed decisions, especially regarding the level of quality of
    the test object.

Result Changes in the Last Flow

The basic approach to testing is observing results. The results
include passes, failures, errors and sometimes some other types of
status. Looking at failures and errors allows for finding some
problems, these lead to some debugging, root-casing, defect
submission and fixing eventually.

New button

In the case of Kraken CI, the table with test case results allows
immediately for filtering and showing only these results that are
regressions or fixes. For that purpose, a `New` button shows only the
test cases that changed the status in the current build compared to
the previous one, i.e. shows “new”, interesting results that require
some attention.

Filter by Result Change

It is possible to do more fine-grain filtering and select a particular
change, e.g., regression or fix. For this purpose, there is a dropdown Result Change.


Even if only the regressions are presented, some tests are still not
interesting or maybe interesting but for different reasons. These not
interesting results are sporadic or unstable tests. Sometimes a test
case passes, sometimes it fails. It changes its results status quite
often. It can be concluded that this is all the time the same problem:
either there is a bug in the product or in the test. If this is a
known problem, it is better to filter out such results. This is
possible by filtering results by their instability. There is a column
called Ins. (for instability) that shows a count of status changes
in the last 10 builds. If it is equal to 9 it means that it was
flipping from passed to failed, back and forth. So using a slider in
the filtering pane, the instability can be narrowed to, e.g. 0–3


Another way for getting important results is sorting them by
relevancy. This is a metric built on properties of particular
results. The higher the `relevancy` is, the more important the result

  • +1 if the result is not passed
  • +1 if there is no comment attached to the result ie. it is not root-caused yet
  • +1 if the result is failure
  • +1 if the result was pretty stable (so it is the same over several test runs), ie. instability is <= 3
  • +1 if the result is rather recent (the age is < 5) ie. ignore old failures
  • +1 if this is a regression


To remember the conclusions of investigation of given failure it is
good to store them. For that purpose, there are test case comments.

Finding Changes in the Past

It is hard to spot regressions if the results are not analyzed
immediately after tests execution. The failures are getting stable, so
it is not visible as regressions anymore. To find such results,
results that were regressions during, e.g. last week, we need to
filter by age. Age allows filtering results that are not recent but
happened in the near past. If, e.g. we set age to 1–5 range, then the
test case results table will show results that persist (are stable in
the last 1–5 runs) the change happened in the previous 1–5
executions. This allows for filtering out recent failures (because
e.g. we already have analyzed them) and very old, stable results
(because e.g. no one cares about them anymore).



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store