After many months of pain and stress, it was decided to perform so-called bring-up test (or smoke test etc). This was supposed to be executed after every build, with some rules to be applied:
- If bring-up of local build fails then it is not allowed to submit labels to common build
- If bring-up of common build fails - whole development team is informed and baseline is marked as unstable thus not usable. In this case, fix build was scheduled with only few labels solving the issue.
Episode 2: How bring-up test execution stabilized negative feedback loop of bad quality.
As every change also this one came with certain resistance. In these times the typical discussion was moving around resources. Who is supposed to perform tests? Do we get more people, or shall we cut some of our tasks? We won't be able to finish our previous tasks on time if we have to execute these tests...etc
The process was finally established with the help of power, time and positive feedback loop.
Whole effect can be now described in few sentences:
- As tests were executed by particular teams itself - it got the power and thus also the responsibility to judge if quality is sufficient or not
- Increased awareness of possible negative side-effects has stressed testing before change is submitted.
- The "fix velocity" was increased by fast feedback - in fact the test result was known just few hours after label submission (however still only twice a week)
Picture above shows the entire process. The small picture below shows then main consequence - bring-up tests stabilized, before negative, reinforcement loop.