SME Server:Documentation:QA
This should become the starting point for SME Server QA.
Introduction
Since SME Server is a collaborative effort to produce a stable and secure platform we would like the SME Server community to help out in helping us create that. Since resources are always limited and with the small amount of developers doing all the task of diagnosing bugs, fixing them and verifying them it would be nice if others could step up. This page is meant to be a starting point for the SME Server community on how we do QA and how we think you can help us with that.
Available information
At the moment we do have some limited information available in a few places which we would like to consolidate in a well structured document.
For now just to provide a general overview please collect the links of information here that you are aware of:
Bugtracker
The bug tracker is the spider in the web of our development as we use it to document new feature requests and issues with the current code base of SME Server. Problems are investigated, solutions discussed and testing of fixes is documented in it. Although it might seem daunting to use it at first, once you get used to it, is not much harder than for instance the forums.
The SME Server bug tracker can be found at http://bugs.contribs.org.
Help on our bugtracker
Although this page is intended to explain the SME Server bug tracker this page is not intended to be a general help concerning our bugtracker. We use Bugzilla to track our bugs and general help on this product can be found using the Help link, which is more or less context sensitive/page aware or you can access it's main help page.
Registration
To be able to post anything to the bug tracker we require you to register. At the moment it is not possible to do this with the account you might be using for the wiki and/or the forums. Registration is required as the process of tracking bugs and fixing them depends on feedback, therefore please use a valid e-mail address as notifications and comments added to bugs will be e-mailed to you. Registration is done using your e-mail address and can be done at http://bugs.contribs.org/createaccount.cgi.
Layout
Bugzilla is organized to keep a clear few on all issues and feature requests, we have tried to define our products as clear as possible in order to help you raise bugs in the proper product and components.
Products
All active products can be found here. As you can see we have products for our current and future release, as well as for translations, documentation, the contribs and the different websites SME Server is providing. When raising a new bug please take care in selecting the proper product.
Components
Each product holds one or more components, which help us categorize bugs and requests within a product. After selection a product the next thing that needs to be chosen is the component. For some products that might be easy, such as in SME Server Contribs, but for instance for SME Server 7.x or SME Server 8.x this might be a little more complex. When in doubt please choose on you think is as close as possible or us the component labeled Unknown if available.
Versions
Each product might also have a version box, where possible try and fill this in as good as you can. If you are running SME Server 7.5 please select that version, if you are running on of our beta releases of SME Server 8, please specify the proper beta version.
Reporting a bug
In the previous section a few hints were given already on actions to do when reporting a bug or feature request. Users reporting bugs to us are valuable, but the quality of the bug reports is also important to us, in order to address the raised issue as good as we can. We therefore kindly ask you to read the contents of "How to report bugs effectively" as it will give you a very good idea of how we like you to report our bugs so we can diagnose and where required, fix them as adequately as possible.
In general the principles are:
- Be precise
- Be clear - explain it so others can reproduce the bug
- One bug per report
- No bug is too trivial to report - small bugs may hide big bugs
- Clearly separate fact from speculation
Please fill in the form as precise as you can. Specify a accurate summary that describes your issue as descriptive as possible, yet shortly. In the comment field please specify the details on your setup as well as on the problem. Make sure you specify error messages exactly as you see them and do not paraphrase them. Don't elaborate too much, but make sure you are not too brief either. Generally a food bug report would answer the following questions:
What is the history of your server? What are you expecting to happen? What is actually happening? What did you do that triggered the issue or might have lead to it appearing?
Verification
Not all bugs require code fixes but a substantial part of them do. An important part of the QA process is verifying that the issue addressed in a bug is fixed when any of the code is changed. We try our best not to overlook things but it can happen, proper verification is important to reduce the risk of such things slipping through. Therefore we try and stick to a certain work flow when doing verification. Since the development team and resources are rather limited it is difficult enough to try and fix all bugs raised, let alone also verify them. Since you do not need to be a developer to verify the fixes that have been produced this guide is written to help all members of our community to help us in the process of verification, it is a vital part of the ongoing maintenance of SME Server.
1. Install and configure a server with iso.
2. smeserver-yum, e-smith-base and smeserver-support in smeupdates-testing needs to be updated first with their dependancies :
yum update smeserver-yum smeserver-support e-smith-base --enablerepo=smeupdates-testing
3. after reboot apply full :
yum update
An update to tinydns & dnscache did break dns. You have to manually start them after a reboot and then update.
Verification should be done with yum update —enablerepo=smeupdates-testing packagename.
Before that you must use the same commande but with smeserver-yum e-smith-base smeserver-support to ensure you have the last running scripts.
General work flow
As already mentioned we try and stick to a certain work flow when verifying bugs. In general the process can be split in the following steps:
- . status of test box
- . description of the bug
- . list new package fixing the problem
- . check which package is currently installed
- . replicate problem in detail
- . install new package
- . check that new package has been installed
- . repeat testing above (5th line),
- . fixed/not fixed based on new testing, conclusion: VERIFIED or REOPEN
- . detail any documentation impact, i.e. new DB or whatever
- . short summary of what the new package has achieve, eg. fixed this or that.
How upgrade the new packages
The bug report will detail the exact package and version needed. These are normally in smeupdates-testing and can be installed by :
yum --enablerepo=smeupdates-testing update <package>
for example
yum --enablerepo=smeupdates-testing update e-smith-ldap
As it is a manual step to move packages into smeupdates-testing the most recent packages will be in smetest and can be installed from there if necessary. for example
yum --enablerepo=smetest update e-smith-ldap
or if you need several repositories :
yum --enablerepo=smetest,smeupdates-testing,smedev update e-smith-ldap
Template
The above process is documented in the comment field of a bug using the following template, more information on the steps and the template sections can be found below.
VERIFICATION TEMPLATE
= ENVIRONMENT: = ORIGINAL PROBLEM: = RESOLUTION: = CURRENT VERSION INSTALLED: = TESTING: = UPDATED VERSION INSTALLED: = PROBLEM FIXED: = VERIFIED OR REOPEN: = DOCUMENTATION IMPACT: = SUGGESTED RELEASE NOTES:
AS AN EXAMPLE:
VERIFICATION = ENVIRONMENT: [description of test system (version, installation methods, upgrade history. etc). If you have some non-core package installed, run /sbin/e-smith/audittools/newrpms and provide output] = ORIGINAL PROBLEM: [Summarise bug] = RESOLUTION: [Insert changelog for new package] In example: Fixed in e-smith-base * Mon Apr 21 2013 chris burnat <devlist@burnat.com> 5.4.0-27.sme - Fix the way '.' works in bash [SME: 7532] = CURRENT VERSION INSTALLED: [rpm -qa <package name>] =TESTING: [Reproduce bug if you can, showing steps taken] = UPDATED VERSION INSTALLED: [Update to new package, show steps taken] and: [rpm -qa <package name>] = PROBLEM FIXED?: [Repeat steps carried out under TESTING above] = VERIFIED OR REOPEN: [Problem fixed, then VERIFIED - not fixed, then REOPEN] Note: if you may not have rights to toggle resolution, someone will do it for you = DOCUMENTATION IMPACT: [Does something need documentation, eg a db or new procedure? if affirmative, please provide details, someone will later punt to docteam for action] = SUGGESTED RELEASE NOTES: [Summary of what was fixed]
Environment
Since some bugs might appear in multiple versions of our products (e. g. SME Server 7.x as well as SME Server 8.x and now SME Server 9.x) we need to make sure you are performing the process of verification using the proper environment. Most of the time it would be something like the following:
SME Server 8.0 fully updated, no contribs installed
Original problem
This section is used to show that you can reproduce the original problem on your test machine. It is important that you do this as accurately and methodologically as possible as you will have to reproduce these steps after you have updated to the newer pacakage(s) to show that the problem is actually fixed. Make sure to conduct your testing as extensively as required but on the other hand try and keep things as clear and concise as possible. In the case of New Feature Requests (NFR) or if a previous version of the package with the reported bug is not available this step might be skipped.
Resolution
This section is used to briefly describe which steps are needed to be taken in order to upgrade to the fixed version, usually it will be something like this:
Install package1 and package2 from smeupdates-testing and/or smetest or patch xyz was applied to etc
After that it is time to actually upgrade to the new package(s).
Current version installed
When the bug is reported and diagnosed normally the package that should be fixed, as well as the version of the package might be documented, but certainly the package name and the version on which the issue was fixed is meant to be reported by the developer, it will be something like below:
Fixed in package-name
* Thu Nov 25 2010 John Doe <jdoe@fqdn.org> x.y.z-r.sme - Short description of the fix [SME: bugnumber]
The version intended here is the version of the package with the bug present, usually the output of the rpm -q command will do, so most of the time this will look something like this:
[root@smetest ~]# rpm -q e-smith-base e-smith-base-5.2.0-28.el5.sme [root@smetest ~]#
Testing
Show in detail existing problem being fixed by new package
Updated version installed
This section is analogue to the Current version installed and is meant to show that the updated package(s) are in fact installed before verifying the bug has been fixed, usually a rpm -q for the updated package(s) would do.
Problem fixed
This is the heart of the verification process as here is where we show the problem is actually fixed. You can use the same steps for this as used in the Original problem section. Be sure to check that not only the problem is fixed, but also make sure no error messages are found in the logfiles or on the console when entering the comments. If errors are present please report them to the bug report if it affects the fix, or open a new bug when a new issue is discovered. Also make sure to test normal functionality related to the changes are still working properly.
Verified or Reopen
This is the section where we conclude on the verification process. If the fixed package works and there are no ill side effects we can conclude the package is VERIFIED, otherwise it might be REOPENED. If you have found other bugs in the process you can state them here (briefly) as well, but please keep in mind that new issues should be reported in a new bug report. When REOPENING the bug, please justify this briefly.
Documentation Impact
Eg: is a DB variable created/removed?
Suggested release notes
If you feel capable please suggest the information we can put in the release notes in one or two lines, if you cannot please leave this empty, or for instance specify it is to be determined by specifying T. B. D..
Abbreviated Verification
Only to be used when circumstances dictate urgency in completing multiple verification in a short time.
= CURRENT VERSION INSTALLED: [rpm -qa <package name>] If its an actual Bug show the issue if you can, showing steps taken, otherwise just show it is working, location, any templates, config settings, db settings etc that will or may be changed
= UPDATED VERSION INSTALLED: Update to new package, show steps taken # yum updat/install e-smith-base --enablerepo=smeupdates=testing or smetest (check for latest version, it evolves rapidly just now with changes) # reboot reconfig if required, good idea just in case and: [rpm -qa <package name>] Show its working if possible and/or no errors evident in loges etc
= VERIFIED OR REOPEN: [Problem fixed, then VERIFIED - not fixed, then REOPEN] Note: if you do not have rights to toggle resolution, someone will do it for you = DOCUMENTATION IMPACT: [Does something need documentation, eg a db or new procedure? if affirmative, please provide details, someone will later punt to docteam for action]
