Difference between revisions of "SME Server:Documentation:QA:Verification"

From SME Server
Jump to navigationJump to search
 
(33 intermediate revisions by 4 users not shown)
Line 1: Line 1:
−
Not all bugs require code fixes but a substantial part of them do. An important part of the QA process is verifying the issue addressed in a bug is fixed when code is changed.
+
{{Languages}}
−
We try our best not to overlook things but we might do, proper verification is important to reduce the risk of such things slipping through. Therefore we try and stick to a certain work flow when doing verification.
+
Not all bugs require code fixes but a substantial part of them do. An important part of the QA process is verifying that the issue addressed in a bug is fixed when any of the code is changed.
−
Since the development team and resources are rather limited it is already hard to try and fix all bugs raised, let alone also verify them. Since you do not need to be a developer to verify the fixes we make this guide is written to help people from our community to help us in the process of verification.
+
We try our best not to overlook things but it can happen, proper verification is important to reduce the risk of such things slipping through. Therefore we try and stick to a certain work flow when doing verification.
 +
Since the development team and resources are rather limited it is difficult enough to try and fix all bugs raised, let alone also verify them. Since you do not need to be a developer to verify the fixes that have been produced this guide is written to help all members of our community to help us in the process of verification, it is a vital part of the ongoing maintenance of SME Server.
 +
 
 +
{{Warning box|KOOZALI SME 10. To install a test machine, see [[Testing_Environments]].}}
 +
1. Install and configure a server with iso.
 +
 
 +
2. smeserver-yum, e-smith-base and smeserver-support in smeupdates-testing needs to be updated first with their dependancies :
 +
yum update smeserver-yum smeserver-support e-smith-base --enablerepo=smeupdates-testing
 +
3. after reboot apply full :
 +
yum update
 +
 
 +
An update to tinydns & dnscache did break dns.
 +
You have to manually start them after a reboot and then update.
 +
 
 +
Verification should be done with yum update —enablerepo=smeupdates-testing packagename.
 +
 
 +
Before that you must use the same commande but with smeserver-yum e-smith-base smeserver-support to ensure you have the last running scripts.
 +
 
  
 
===General work flow===
 
===General work flow===
 
As already mentioned we try and stick to a certain work flow when verifying bugs. In general the process can be split in the following steps:
 
As already mentioned we try and stick to a certain work flow when verifying bugs. In general the process can be split in the following steps:
  
−
#. Describe the environment briefly
+
#. status of test box
−
#. Show the installed versions of affected packages before upgrading
+
#. description of the bug
−
#. Show you can reproduce the problem as reported in the bug
+
#. list new package fixing the problem
−
#. Short synopsis of the resolution of the problem
+
#. check which package is currently installed
−
#. Update the affected packages
+
#. replicate problem in detail
−
#. Show the problem is fixed (and no ill side affects have appeared)
+
#. install new package
−
#. Suggestion of the release note information
+
#. check that new package has been installed
−
#. Final conclusion: VERIFIED or REOPEN
+
#. repeat testing above (5th line),
 +
#. fixed/not fixed based on new testing, conclusion: VERIFIED or REOPEN
 +
#. detail any documentation impact, i.e. new DB or whatever
 +
#. short summary of what the new package has achieve, eg. fixed this or that.
 +
 
 +
===How upgrade the new packages===
 +
{{Note box|Of course we are doing good code but we can brake your system, so you are strongly advised for doing verification on a virtual SME Server.}}
 +
 
 +
The bug report will detail the exact package and version needed. These are normally in smeupdates-testing and can be installed by :
 +
yum --enablerepo=smeupdates-testing update <package>
 +
 
 +
for example
 +
yum --enablerepo=smeupdates-testing update e-smith-ldap
 +
 
 +
As it is a manual step to move packages into smeupdates-testing the most recent packages will be in smetest and can be installed from there if necessary.
 +
for example
 +
  yum --enablerepo=smetest update e-smith-ldap
 +
or if you need several repositories :
 +
  yum --enablerepo=smetest,smeupdates-testing,smedev update e-smith-ldap
  
 
===Template===
 
===Template===
 
The above process is documented in the comment field of a bug using the following template, more information on the steps and the template sections can be found below.
 
The above process is documented in the comment field of a bug using the following template, more information on the steps and the template sections can be found below.
  
−
  = Environment:
+
'''VERIFICATION TEMPLATE'''
 +
 
 +
  = ENVIRONMENT:  
 
   
 
   
−
  = Current version installed:
+
  = ORIGINAL PROBLEM:
 
   
 
   
−
  = Original problem:
+
  = RESOLUTION:
 
   
 
   
−
  = Resolution:
+
  = CURRENT VERSION INSTALLED:
 
   
 
   
−
  = Updated version installed:
+
  = TESTING:
 
   
 
   
−
  = Problem fixed:
+
  = UPDATED VERSION INSTALLED:
 
   
 
   
−
  = Release note information suggestion:
+
  = PROBLEM FIXED:
 
   
 
   
−
  = Verified or Reopen:
+
  = VERIFIED OR REOPEN:
 +
 +
= DOCUMENTATION IMPACT:
 +
 +
= SUGGESTED RELEASE NOTES:
  
 +
AS AN EXAMPLE:
 +
 +
VERIFICATION
 +
 +
= ENVIRONMENT:
 +
[description of test system (version, installation methods, upgrade history. etc).
 +
If you have some non-core package installed, run /sbin/e-smith/audittools/newrpms and provide output]
 +
 +
= ORIGINAL PROBLEM:
 +
[Summarise bug]
 +
 +
= RESOLUTION:
 +
[Insert changelog for new package]
 +
In example:
 +
Fixed in e-smith-base
 +
* Mon Apr 21 2013 chris burnat <devlist@burnat.com> 5.4.0-27.sme
 +
- Fix the way '.' works in bash [SME: 7532]
 +
 +
= CURRENT VERSION INSTALLED:
 +
[rpm -qa <package name>]
 +
 +
=TESTING:
 +
[Reproduce bug if you can, showing steps taken]
 +
 +
= UPDATED VERSION INSTALLED:
 +
[Update to new package, show steps taken]
 +
and:
 +
[rpm -qa <package name>]
 +
 +
= PROBLEM FIXED?:
 +
[Repeat steps carried out under TESTING above]
 +
 +
= VERIFIED OR REOPEN:
 +
[Problem fixed, then VERIFIED - not fixed, then REOPEN]
 +
Note: if you may not have rights to toggle resolution, someone will do it for you
 +
 +
= DOCUMENTATION IMPACT:
 +
[Does something need documentation, eg a db or new procedure? if affirmative, please provide details, someone will later punt to docteam for action]
 +
 +
= SUGGESTED RELEASE NOTES:
 +
[Summary of what was fixed]
 
====Environment====
 
====Environment====
−
Since some bugs might appear in multiple versions of our products (e. g. SME Server 7.x as well as SME Server 8.x) we need to make sure you are doing verification using the proper environment. Most of the times it would be something like the following:
+
Since some bugs might appear in multiple versions of our products (e. g. SME Server 7.x as well as SME Server 8.x and now SME Server 9.x) we need to make sure you are performing the process of verification using the proper environment. Most of the time it would be something like the following:
  
−
  SME Server 7.5 fully updated, no contribs installed
+
  SME Server 8.0 fully updated, no contribs installed
 +
 
 +
====Original problem====
 +
This section is used to show that you can reproduce the original problem on your test machine. It is important that you do this as accurately and methodologically as possible as you will have to reproduce these steps after you have updated to the newer pacakage(s) to show that the problem is actually fixed.
 +
Make sure to conduct your testing as extensively as required but on the other hand try and keep things as clear and concise as possible.
 +
{{{Note box|In the case of New Feature Requests (NFR) or if a previous version of the package with the reported bug is not available this step might be skipped.}}}
 +
 
 +
====Resolution====
 +
This section is used to briefly describe which steps are needed to be taken in order to upgrade to the fixed version, usually it will be something like this:
 +
 
 +
Install package1 and package2 from smeupdates-testing and/or smetest or patch xyz was applied to etc
 +
 
 +
After that it is time to actually upgrade to the new package(s).
  
 
====Current version installed====
 
====Current version installed====
−
When the bug is reported and diagnosed normally the package that should be fixed, as well as the version of the package might be documented, but certainly the package name and the version on which the issue was fixed is reported most of the time, it will be something like below:
+
When the bug is reported and diagnosed normally the package that should be fixed, as well as the version of the package might be documented, but certainly the package name and the version on which the issue was fixed is meant to be reported by the developer, it will be something like below:
  
−
Fixed in package-name
+
Fixed in package-name
 
  * Thu Nov 25 2010 John Doe <jdoe@fqdn.org> x.y.z-r.sme
 
  * Thu Nov 25 2010 John Doe <jdoe@fqdn.org> x.y.z-r.sme
 
  - Short description of the fix [SME: bugnumber]
 
  - Short description of the fix [SME: bugnumber]
−
 
+
−
The version intended here is the version of the package with the bug present, usualy the output of the ''rpm -q'' command will do, so most of the time this will look something like this:
+
The version intended here is the version of the package with the bug present, usually the output of the ''rpm -q'' command will do, so most of the time this will look something like this:
  
 
  [root@smetest ~]# rpm -q e-smith-base
 
  [root@smetest ~]# rpm -q e-smith-base
Line 52: Line 146:
 
  [root@smetest ~]#
 
  [root@smetest ~]#
  
−
====Original problem====
+
====Testing====
−
This section is used to show that you can reproduce the original problem on your test machine. It is important that you do this as accurate and methodological as possible as you will have to reproduce these steps after you have updated to the newer pacakage(s) to show that the problem is actually fixed.
+
Show in detail existing problem being fixed by new package
−
Make sure to make your test as extensive as required but on the other hand try and keep things as clear and dense as possible.
 
−
{{{Note box|In the case of New Feature Requests (NFR) or if a previous version of the package with the reported bug is not available this step might be skipped.}}}
 
−
 
 
−
====Resolution====
 
−
This section is used to shortly describe which steps need to be taken in order to upgrade to the fixed version, usually it will be something like this:
 
−
 
 
−
Install package1 and package2 from smeupdates-testing and/or smetest
 
−
 
 
−
After that it is time to actually upgrade to the new package(s).
 
  
 
====Updated version installed====
 
====Updated version installed====
Line 71: Line 156:
 
Be sure to check that not only the problem is fixed, but also make sure no error messages are found in the logfiles or on the console when entering the comments. If errors are present please report them to the bug report if it affects the fix, or open a new bug when a new issue is discovered.
 
Be sure to check that not only the problem is fixed, but also make sure no error messages are found in the logfiles or on the console when entering the comments. If errors are present please report them to the bug report if it affects the fix, or open a new bug when a new issue is discovered.
 
Also make sure to test normal functionality related to the changes are still working properly.
 
Also make sure to test normal functionality related to the changes are still working properly.
−
 
−
====Release note information suggestion====
 
−
If you feel capable please suggest the information we can put in the release notes in one or two lines, if you cannot please leave this empty, or for instance specify it is to be determined by specifying ''T. B. D.''.
 
  
 
====Verified or Reopen====
 
====Verified or Reopen====
 
This is the section where we conclude on the verification process. If the fixed package works and there are no ill side effects we can conclude the package is VERIFIED, otherwise it might be REOPENED. If you have found other bugs in the process you can state them here (briefly) as well, but please keep in mind that new issues should be reported in a new bug report.
 
This is the section where we conclude on the verification process. If the fixed package works and there are no ill side effects we can conclude the package is VERIFIED, otherwise it might be REOPENED. If you have found other bugs in the process you can state them here (briefly) as well, but please keep in mind that new issues should be reported in a new bug report.
−
When REOPENING the bug, please motivate this briefly.
+
When REOPENING the bug, please justify this briefly.
 +
 
 
{{Note box|Please, do not forget to also set the status field, at the bottom of the comment field, to the proper value matching your conclusion.}}
 
{{Note box|Please, do not forget to also set the status field, at the bottom of the comment field, to the proper value matching your conclusion.}}
 +
 +
====Documentation Impact====
 +
Eg: is a DB variable created/removed?
 +
 +
====Suggested release notes====
 +
If you feel capable please suggest the information we can put in the release notes in one or two lines, if you cannot please leave this empty, or for instance specify it is to be determined by specifying ''T. B. D.''.
 +
 +
===Abbreviated Verification===
 +
Only to be used when circumstances dictate urgency in completing multiple verification in a short time.
 +
 +
= CURRENT VERSION INSTALLED:
 +
[rpm -qa <package name>]
 +
If its an actual Bug show the issue if you can, showing steps taken, otherwise just show it is working, location, any templates,
 +
config settings, db settings etc that will or may be changed
 +
 +
= UPDATED VERSION INSTALLED:
 +
Update to new package, show steps taken
 +
# yum updat/install e-smith-base --enablerepo=smeupdates=testing or smetest (check for latest
 +
version, it evolves rapidly just now with changes)
 +
# reboot reconfig if required, good idea just in case
 +
and:
 +
[rpm -qa <package name>]
 +
Show its working if possible and/or no errors evident in loges etc
 +
 +
= VERIFIED OR REOPEN:
 +
[Problem fixed, then VERIFIED - not fixed, then REOPEN]
 +
Note: if you do not have rights to toggle resolution, someone will do it for you
 +
 +
= DOCUMENTATION IMPACT:
 +
[Does something need documentation, eg a db or new procedure? if affirmative, please provide details, someone will later punt to
 +
docteam for action]
 +
 +
[[Category:SME Server]][[Category:Howto]][[Category:Help]][[Category:Developer]]

Latest revision as of 14:54, 19 June 2023

Geographylogo.png Languages: 

English  • français


Not all bugs require code fixes but a substantial part of them do. An important part of the QA process is verifying that the issue addressed in a bug is fixed when any of the code is changed. We try our best not to overlook things but it can happen, proper verification is important to reduce the risk of such things slipping through. Therefore we try and stick to a certain work flow when doing verification. Since the development team and resources are rather limited it is difficult enough to try and fix all bugs raised, let alone also verify them. Since you do not need to be a developer to verify the fixes that have been produced this guide is written to help all members of our community to help us in the process of verification, it is a vital part of the ongoing maintenance of SME Server.


Warning.png Warning:
KOOZALI SME 10. To install a test machine, see Testing_Environments.


1. Install and configure a server with iso.

2. smeserver-yum, e-smith-base and smeserver-support in smeupdates-testing needs to be updated first with their dependancies : 

yum update smeserver-yum smeserver-support e-smith-base --enablerepo=smeupdates-testing

3. after reboot apply full : 

yum update

An update to tinydns & dnscache did break dns. You have to manually start them after a reboot and then update.

Verification should be done with yum update —enablerepo=smeupdates-testing packagename.

Before that you must use the same commande but with smeserver-yum e-smith-base smeserver-support to ensure you have the last running scripts.


General work flow

As already mentioned we try and stick to a certain work flow when verifying bugs. In general the process can be split in the following steps:

  1. . status of test box
  2. . description of the bug
  3. . list new package fixing the problem
  4. . check which package is currently installed
  5. . replicate problem in detail
  6. . install new package
  7. . check that new package has been installed
  8. . repeat testing above (5th line),
  9. . fixed/not fixed based on new testing, conclusion: VERIFIED or REOPEN
  10. . detail any documentation impact, i.e. new DB or whatever
  11. . short summary of what the new package has achieve, eg. fixed this or that.

How upgrade the new packages

Important.png Note:
Of course we are doing good code but we can brake your system, so you are strongly advised for doing verification on a virtual SME Server.


The bug report will detail the exact package and version needed. These are normally in smeupdates-testing and can be installed by : 

yum --enablerepo=smeupdates-testing update <package>

for example 

yum --enablerepo=smeupdates-testing update e-smith-ldap

As it is a manual step to move packages into smeupdates-testing the most recent packages will be in smetest and can be installed from there if necessary. for example 

 yum --enablerepo=smetest update e-smith-ldap

or if you need several repositories : 

 yum --enablerepo=smetest,smeupdates-testing,smedev update e-smith-ldap

Template

The above process is documented in the comment field of a bug using the following template, more information on the steps and the template sections can be found below.

VERIFICATION TEMPLATE 

= ENVIRONMENT: 

= ORIGINAL PROBLEM:

= RESOLUTION:

= CURRENT VERSION INSTALLED:

= TESTING:

= UPDATED VERSION INSTALLED:

= PROBLEM FIXED:

= VERIFIED OR REOPEN:

= DOCUMENTATION IMPACT:

= SUGGESTED RELEASE NOTES:

AS AN EXAMPLE: 

VERIFICATION

= ENVIRONMENT:
[description of test system (version, installation methods, upgrade history. etc).
If you have some non-core package installed, run /sbin/e-smith/audittools/newrpms and provide output]

= ORIGINAL PROBLEM:
[Summarise bug]

= RESOLUTION:
[Insert changelog for new package]
In example:
Fixed in e-smith-base
* Mon Apr 21 2013 chris burnat <devlist@burnat.com> 5.4.0-27.sme
- Fix the way '.' works in bash [SME: 7532]

= CURRENT VERSION INSTALLED:
[rpm -qa <package name>]

=TESTING:
[Reproduce bug if you can, showing steps taken]

= UPDATED VERSION INSTALLED:
[Update to new package, show steps taken]
and:
[rpm -qa <package name>]

= PROBLEM FIXED?:
[Repeat steps carried out under TESTING above]

= VERIFIED OR REOPEN:
[Problem fixed, then VERIFIED - not fixed, then REOPEN]
Note: if you may not have rights to toggle resolution, someone will do it for you

= DOCUMENTATION IMPACT:
[Does something need documentation, eg a db or new procedure? if affirmative, please provide details, someone will later punt to docteam for action]

= SUGGESTED RELEASE NOTES:
[Summary of what was fixed]

Environment

Since some bugs might appear in multiple versions of our products (e. g. SME Server 7.x as well as SME Server 8.x and now SME Server 9.x) we need to make sure you are performing the process of verification using the proper environment. Most of the time it would be something like the following: 

SME Server 8.0 fully updated, no contribs installed

Original problem

This section is used to show that you can reproduce the original problem on your test machine. It is important that you do this as accurately and methodologically as possible as you will have to reproduce these steps after you have updated to the newer pacakage(s) to show that the problem is actually fixed. Make sure to conduct your testing as extensively as required but on the other hand try and keep things as clear and concise as possible. In the case of New Feature Requests (NFR) or if a previous version of the package with the reported bug is not available this step might be skipped.

Resolution

This section is used to briefly describe which steps are needed to be taken in order to upgrade to the fixed version, usually it will be something like this: 

Install package1 and package2 from smeupdates-testing and/or smetest or patch xyz was applied to etc

After that it is time to actually upgrade to the new package(s).

Current version installed

When the bug is reported and diagnosed normally the package that should be fixed, as well as the version of the package might be documented, but certainly the package name and the version on which the issue was fixed is meant to be reported by the developer, it will be something like below:

Fixed in package-name 

* Thu Nov 25 2010 John Doe <jdoe@fqdn.org> x.y.z-r.sme
- Short description of the fix [SME: bugnumber]

The version intended here is the version of the package with the bug present, usually the output of the rpm -q command will do, so most of the time this will look something like this: 

[root@smetest ~]# rpm -q e-smith-base
e-smith-base-5.2.0-28.el5.sme
[root@smetest ~]#

Testing

Show in detail existing problem being fixed by new package

Updated version installed

This section is analogue to the Current version installed and is meant to show that the updated package(s) are in fact installed before verifying the bug has been fixed, usually a rpm -q for the updated package(s) would do.

Problem fixed

This is the heart of the verification process as here is where we show the problem is actually fixed. You can use the same steps for this as used in the Original problem section. Be sure to check that not only the problem is fixed, but also make sure no error messages are found in the logfiles or on the console when entering the comments. If errors are present please report them to the bug report if it affects the fix, or open a new bug when a new issue is discovered. Also make sure to test normal functionality related to the changes are still working properly.

Verified or Reopen

This is the section where we conclude on the verification process. If the fixed package works and there are no ill side effects we can conclude the package is VERIFIED, otherwise it might be REOPENED. If you have found other bugs in the process you can state them here (briefly) as well, but please keep in mind that new issues should be reported in a new bug report. When REOPENING the bug, please justify this briefly.


Important.png Note:
Please, do not forget to also set the status field, at the bottom of the comment field, to the proper value matching your conclusion.


Documentation Impact

Eg: is a DB variable created/removed?

Suggested release notes

If you feel capable please suggest the information we can put in the release notes in one or two lines, if you cannot please leave this empty, or for instance specify it is to be determined by specifying T. B. D..

Abbreviated Verification

Only to be used when circumstances dictate urgency in completing multiple verification in a short time. 

= CURRENT VERSION INSTALLED:
[rpm -qa <package name>]
If its an actual Bug show the issue if you can, showing steps taken, otherwise just show it is working, location, any templates, 
config settings, db settings etc that will or may be changed

= UPDATED VERSION INSTALLED:
Update to new package, show steps taken
# yum updat/install e-smith-base --enablerepo=smeupdates=testing or smetest (check for latest
version, it evolves rapidly just now with changes)
# reboot reconfig if required, good idea just in case
and:
[rpm -qa <package name>]
Show its working if possible and/or no errors evident in loges etc

= VERIFIED OR REOPEN:
[Problem fixed, then VERIFIED - not fixed, then REOPEN]
Note: if you do not have rights to toggle resolution, someone will do it for you

= DOCUMENTATION IMPACT:
[Does something need documentation, eg a db or new procedure? if affirmative, please provide details, someone will later punt to 
docteam for action]
Retrieved from "https://wiki.koozali.org/index.php?title=SME_Server:Documentation:QA:Verification&oldid=42114"