SME Server:Documentation:QA:Verification

Not all bugs require code fixes but a substantial part of them do. An important part of the QA process is verifying that the issue addressed in a bug is fixed when any of the code is changed. We try our best not to overlook things but it can happen, proper verification is important to reduce the risk of such things slipping through. Therefore we try and stick to a certain work flow when doing verification. Since the development team and resources are rather limited it is difficult enough to try and fix all bugs raised, let alone also verify them. Since you do not need to be a developer to verify the fixes that have been produced this guide is written to help all members of our community to help us in the process of verification, it is a vital part of the ongoing maintenance of SME Server.

General work flow

As already mentioned we try and stick to a certain work flow when verifying bugs. In general the process can be split in the following steps:

1. . status of test box
2. . description of the bug
3. . list new package fixing the problem
4. . check which package is currently installed
5. . replicate problem in detail
6. . install new package
7. . check that new package has been installed
8. . repeat testing above (5th line),
9. . fixed/not fixed based on new testing, conclusion: VERIFIED or REOPEN
10. . detail any documentation impact, i.e. new DB or whatever
11. . short summary of what the new package has achieve, eg. fixed this or that.

Template

The above process is documented in the comment field of a bug using the following template, more information on the steps and the template sections can be found below.

VERIFICATION TEMPLATE

= ENVIRONMENT: 

= ORIGINAL PROBLEM:

= RESOLUTION:

= CURRENT VERSION INSTALLED:

= TESTING:

= UPDATED VERSION INSTALLED:

= PROBLEM FIXED:

= VERIFIED OR REOPEN:

= DOCUMENTATION IMPACT:

= SUGGESTED RELEASE NOTES:

AS AN EXAMPLE:

VERIFICATION

= ENVIRONMENT:
[description of test system (version, installation methods, upgrade history. etc).
If you have some non-core package installed, run /sbin/e-smith/audittools/newrpms and provide output]

= ORIGINAL PROBLEM:
[Summarise bug]

= RESOLUTION:
[Insert changelog for new package]
In example:
Fixed in e-smith-base
* Mon Apr 21 2013 chris burnat <devlist@burnat.com> 5.4.0-27.sme
- Fix the way '.' works in bash [SME: 7532]

= CURRENT VERSION INSTALLED:
[rpm -qa <package name>]

=TESTING:
[Reproduce bug if you can, showing steps taken]

= UPDATED VERSION INSTALLED:
[Update to new package, show steps taken]
and:
[rpm -qa <package name>]

= PROBLEM FIXED?:
[Repeat steps carried out under TESTING above.

= VERIFIED OR REOPEN:
[Problem fixed, then VERIFIED - not fixed, then REOPEN]
Note: if you may not have rights to toggle resolution, someone will do it for you

= DOCUMENTATION IMPACT:
[Does something need documentation, eg a db or new procedure? if affirmative, please provide details, someone will later punt to docteam for action]

= SUGGESTED RELEASE NOTES:
[Summary of what was fixed]

Environment

Since some bugs might appear in multiple versions of our products (e. g. SME Server 7.x as well as SME Server 8.x and now SME Server 9.x) we need to make sure you are performing the process of verification using the proper environment. Most of the time it would be something like the following:

SME Server 8.0 fully updated, no contribs installed

Original problem

This section is used to show that you can reproduce the original problem on your test machine. It is important that you do this as accurately and methodologically as possible as you will have to reproduce these steps after you have updated to the newer pacakage(s) to show that the problem is actually fixed. Make sure to conduct your testing as extensively as required but on the other hand try and keep things as clear and concise as possible. In the case of New Feature Requests (NFR) or if a previous version of the package with the reported bug is not available this step might be skipped.

Resolution

This section is used to briefly describe which steps are needed to be taken in order to upgrade to the fixed version, usually it will be something like this:

Install package1 and package2 from smeupdates-testing and/or smetest or patch xyz was applied to etc

After that it is time to actually upgrade to the new package(s).

Current version installed

When the bug is reported and diagnosed normally the package that should be fixed, as well as the version of the package might be documented, but certainly the package name and the version on which the issue was fixed is meant to be reported by the developer, it will be something like below:

Fixed in package-name

* Thu Nov 25 2010 John Doe <jdoe@fqdn.org> x.y.z-r.sme
- Short description of the fix [SME: bugnumber]

The version intended here is the version of the package with the bug present, usually the output of the rpm -q command will do, so most of the time this will look something like this:

[root@smetest ~]# rpm -q e-smith-base
e-smith-base-5.2.0-28.el5.sme
[root@smetest ~]#

Testing

Show in detail existing problem being fixed by new package

Updated version installed

This section is analogue to the Current version installed and is meant to show that the updated package(s) are in fact installed before verifying the bug has been fixed, usually a rpm -q for the updated package(s) would do.

Problem fixed

This is the heart of the verification process as here is where we show the problem is actually fixed. You can use the same steps for this as used in the Original problem section. Be sure to check that not only the problem is fixed, but also make sure no error messages are found in the logfiles or on the console when entering the comments. If errors are present please report them to the bug report if it affects the fix, or open a new bug when a new issue is discovered. Also make sure to test normal functionality related to the changes are still working properly.

Verified or Reopen

This is the section where we conclude on the verification process. If the fixed package works and there are no ill side effects we can conclude the package is VERIFIED, otherwise it might be REOPENED. If you have found other bugs in the process you can state them here (briefly) as well, but please keep in mind that new issues should be reported in a new bug report. When REOPENING the bug, please justify this briefly.

	Note:
	Please, do not forget to also set the status field, at the bottom of the comment field, to the proper value matching your conclusion.

Documentation Impact

Eg: is a DB variable created/removed?

Suggested release notes

If you feel capable please suggest the information we can put in the release notes in one or two lines, if you cannot please leave this empty, or for instance specify it is to be determined by specifying T. B. D..