Showing posts with label configuration management. Show all posts
Showing posts with label configuration management. Show all posts

Wednesday, February 16, 2011

Configuration management story - part 2

This story is very short. Nevertheless the improvement helped the project to survive and converge on the final delivery.

After many months of pain and stress, it was decided to perform so-called bring-up test (or smoke test etc). This was supposed to be executed after every build, with some rules to be applied:
  • If bring-up of local build fails then it is not allowed to submit labels to common build  
  • If bring-up of common build fails - whole development team is informed and baseline is marked as unstable thus not usable. In this case, fix build was scheduled with only few labels solving the issue. 
The idea itself is not new, it is not worth of a patent - nevertheless I was surprised of its positive effect.

Episode 2: How bring-up test execution stabilized negative feedback loop of bad quality.
As every change also this one came with certain resistance. In these times the typical discussion was moving around resources. Who is supposed to perform tests? Do we get more people, or shall we cut some of our tasks? We won't be able to finish our previous tasks on time if we have to execute these tests...etc
The process was finally established with the help of power, time and positive feedback loop.
Whole effect can be now described in few sentences:
  • As tests were executed by particular teams itself - it got the power and thus also the responsibility to judge if quality is sufficient or not
  • Increased awareness of possible negative side-effects has stressed testing before change is submitted.
  • The "fix velocity" was increased by fast feedback - in fact the test result was known just few hours after label submission (however still only twice a week) 



Picture above shows the entire process. The small picture below shows then main consequence - bring-up tests stabilized, before negative, reinforcement loop.





    Friday, January 7, 2011

    Configuration management story - part 1

    In next days I'd like to write down our CM story. How a CM process has changed over the last years in our organization and project.

    Once upon a time - before has the project really started - few people wrote a configuration management plan. All the experience was written down and reviewed until all participants were tired of reviewing. That was a trigger to consider the plan as final version and document was successfully released under complex process.

    CM process itself was pretty simple. Three groups of developers were defined: two of them distributed and local one. Every supplier was supposed to submit their changes to the local build once a week. Together with changes from local team all changes were submitted to the common build. This was then next day used as a new baseline for further development.

    Episode 1: How CM process contributes to the dependency between feature velocity and baseline quality.

    Picture below shows the causal loop diagram [1], [2]. The term “build quality” should be interpreted as stability expressed e.g. in number of exceptions per hour…
    1. As every developer submits his changes to the local build without the integration and testing of unwanted side effects the quality of local build goes down.
    2. Local build result is submitted to the common baseline which goes consequently also down.
    3. Decreased baseline quality has bad influence on the development velocity on both constructions: featured development and also the bug fixing.
    4. The lower is feature velocity the higher project pressure is created by management.
    5. This makes the developers morale low but also decreases the bug fixing rate, while developer will rather decide to implement new feature as to fix reported bug.
    6. Low bug fixing rate decreases again the baseline quality etc…
    Consequence of this is a reinforcing loop depicted in the red box.

    With some delay a process change came introducing so called bring-up tests. And the story continues with the episode 2.


    Reference: