TULIP Framework
TULIP stands for
Testing Upgrades of Linux Images Program
Its purpose is to drive upgrade tests of various linux distributions to ensure both fine grained QA at the package level and testing the reliability of the upgrade result.
Presentation
Based on fresh linux installations, twiked to be an initial desired state, and day after day upgrades, the framework will detect packets installation failures, interactions leading to binary incompatibility and launch administrator unit tests. Various upgrade periodicity can also be tested such as daily, weekly and monthly deltas.
The results is gathered in a database for every test case and reports showing state and evolution of the packages upgrade quality can be built.
TULIP is based on qemu images (other virtual environments may also be plugged into the system).
On defining a new test campain, the base images are generated as well as the rotation policiy. Then, on a daily basis, the upgrade process is exercised. Once upgraded, the images are tested regarding binary dependencies using dependencies testing tools like ldd. Explicit QA scripts can also be run in this phase.
TULIP has been first presented at RMLL EDOS Workshop
nuxeo-tulip-rmll.pdf
Synopsis
When a session is launched, Tulip searches for available Delta archives for this project. As an example, we will consider a daily-periodic project.
In order to cover some different state of the images and experience sightly different conditions, we define an historic of images and keep the five (in case of daily) last images (Day-1, Day-2, ...).
Each delta image is a compressed incremental image based on a read-only initial qemu image. This allow qemu images to preserve space by only writing the modification regarding the base.
So for each delta image (form 1 to 5), Tulip uncompresses the image and launches qemu with a TCP redir parameter allowing to connect though ssh once the image is up.
qemu -m 256 -redir tcp:2022::22 QemuIncrementalDeltaImage.img
Once the image is up, tulip connects via ssh as root into the image
For each image, the purpose is to upgrade the system with the defaults responses.
First, a clean of any previous logfiles is performed
Then, an update of the sources is launched.
Once done, the upgrade, assuming default responses is first done in simulation mode and then done really
The simulation list will give the list of packages that where planned and the installer logfile will store what has been really done
Once the upgrade achieved, the files are uploaded to the tulip server using scp command line
- note: qemu defines an internal network with dhcp and the server launching any qemu image has the 10.0.2.2 address - a qemu image has 10.0.2.15 ip address
Then all the files needed by the system tests are downloaded to the image in a test directory. These tests have to check that no packet have been broken. As a first implementation a binary dependency test
lddcheck is performed on every executable file found in the PATH environment variable. If a dependency is not found, the package containing the binary is retrieved and issued as
failing.
Other tests, dedicated to specific packages, or more general, may be used as qatr developed inside EDOS or CETOO dedicated to OpenOffice.org GUI testing for example.
Once all the tests finished and their results uploaded to the tulip machine, the qemu image is stopped
all the log files are then parsed :
- upgrades : differences between planned and installed are detected (planned but not installed, installed but not planned) and packaged reported as failed are also identified. The results are stored in the database
- tests : tests files returns faulty packages list. This list is processed and database filled
Then if we are processing delta 1, we compress and archive this image
A new loop is the initiated, performing the work in the next delta
Once all deltas are performed, all the archives are rotated to increase their delta value. Older images are deleted as not needed anymore
If possible an email is sent,containing the console output (stdout and stderr) of the whole process.
Programs, helpers and requirements
Tulip is mainly written in python using SQLAlchemy for database abstraction. It drives qemu images through ssh connections using python pexpect module
The image has to allow ssh root connection, have a working upgrade mechanism that has been defined at tulip level (commands and parser)
A python version has to be available in the image for some tests (see lddcheck) but more generally all tools needed to run the test
Some helpers have created
- to allow defining a new project and create initial base image (that has to be setup manually to be in the desired initial state with teh set of packages representing a use-case)
- to associate a test to a list of projects
- to associate a list of tests to a project
The reporting is done through BIRT reports designed under eclipse
Birt is an eclipse tool allowing generating reports connected to a database though JDBC. It allows PDF and HTML exports and has an integrated viewer.
This is this viewer that is used inside a Tomcat server. The viewer allows to use built-in parameters selections features.
Tests
While designing tulip only few tests have been introduced
defining a test
Test can be defined at Tulip level and then attached to any project. An helper defining it is available.
All the needed files and test engine will be imported in running image inside a
Tests directory in home. the commandline to be launched is from the home.
The results of teh test in the tulip framework is a list of faulty packages. So the test has to generate this information on teh base of what he is testing (eg. and OpenOffice.org GUI testing may return openoffice.org package as failing if the test is not fullfilled)
A python based helper has been designed to give the packages containing a filename that is passed as argument. This also work on a list. So the test writer has not to care about the installer type but only use the utility
Tests can also generate multiple arbitrary outputs that can be retreived and stored in the Tulip framework for later anaysis or display
lddcheck test
One basic test that has been written is the binary dependancy analyzer lddcheck
It analyzes any path passed as argument or if nothing, the ones that are in the PATH environement variable.
For each path, lddcheck searces for binaries and issue an ldd command to lst dependancies. If a dependancy is reported as missing, then the package containing this binary file is reported as broken
At the end, all the broken packaged are returned as a list in a file.
CETOOo : Continuous Effort Testing OpenOffice.org
CETOOo is an OpenOffice.org project dedicated to continuous testing of OOo GUI through funtional tests. More on
OOo Wiki
Context
OpenOffice.org is a multiplateform popular office suite that is often the main tool of an end-user desktop installation. Then it needs to be checked that upgrades of it will not break any functionality.
OpenOffice.org project has a dedicated funtionnal testing framework that can be used to verify that the it is still functional though tests written by the community and available on the OOo CVS. Using the TestTools program, we will integrate these functional testing in the TULIP framework.
It will benefit the Linux distribution so that they have a tool to heavily check OpenOffice.org is working but also the OpenOffice.org project itself by the definition of the
CETOOo program.
OpneOffice.org has a dedicated project for QA :
http://qa.openoffice.org. Each Native-lang project (French, German, Japanese … ) as its own subset of QA structure and involves regular contributors dedicated to QA testing.
Development time
Each time a feature or a patch is integrated into OOo, the developper needs a QA representative.
The QA representative is usually a skilled developper and is in charge to verify that the development fits the specification documents and coding guidelines (and that it works). He generally checks on 2 main operating system (Linux and MS Windows). If all is ok, the child workspace CWS is officially interated into the HEAD branch.
A new milestone is built on a regularly basis (one week) and only building process is tested here. A new milestone integrates all the released CWS that has been accpted by their QA representative.
Release time
Relase time occurs every 3-4 months and a this time a deep participation of the community worldwide volunteers occur to :
- manually test internationalization and translations
- smoke tests and Test Case Management
- use automated functional testing on various plateforms
The process is quite efficient as many problems are reported each time. And then it ends with a lot of Release Candidate versions that imply to loop over the whole QA process in a short period of time.
This overloaded looping in a short time release process is not efficient and may discourage a lot of volunteers
The testools engine is in charge of sending commands to an OpenOffice.org instance it has started. It uses the accessibility features to pilot OOo GUI and achieve the functional testing written in a basic like language.
Some tests are for example clicking all menus, submenus and verify there is no crash nor wrong window opened. An other is to test the presence of each listed help file. More specific tests dedicated to each element of the suite (writer, calc, base ...) are also available.
There's to be noticed that the whole test sets requires 60-80 hours to process on a middle-strength PC. A shell script is availble that launches automatically all of them
More detail can be found in the guides :
http://qa.openoffice.org/qatesttool/index.html
CETOOo : Continuous Effort Testing OpenOffice.org
The QA process is mainly done at release time and then functional regressions are detected very late. This inevitabely leads to slow the release process and increase risk of instability of the release.
Obviously, the solution is to do functional testing as soon as possible in the development process. That is the aim of the CETOOo project.
Every time a milestone is built, a set of volunteers use the functional testing framework. The TesTools binary is shipped within OOo installation and tests sets are retreived from CVS according to the milestone tag.
The generated report is then reported and automatically parsed for errors. The concerned developpers are then informed that their last modification shows functional problems or side effects.
Linux Distribution OpenOffice.org Packages
Linux distributions propose their own OpenOffice.org version through their pacakging system. The main distribution do also integrate the testtools (Debian, Mandriva, SuSe, Fedora, ...)
As the files layout and the build environnement are changed compared to the legacy OOo build, some functionality may not work properly. Other packages may also disturb an OpenOffice.org installation
For example, previous version of the debian version were displaying a menu but the fonctionality was never called as the maintainer disabled the feature without removing the menu entry. This is quite disturbing for the user and can be seen as poor quality software.
TULIP integration
The purpose of TULIP is to schedule regular packages upgrade on Linux distributions and see if the system is still working. It is typically a framework that could suite the CETOOo project.
First, a tulip project is built based on Linux distribution. Then on a regular basis, weekly, or on demand, the upgade of packages, including OpenOffice.org ones, is requested.
Once installed OOo version and milestone on wich it built is retreived and The tests sets according to the milestone are retreived from OOo anonymous CVS.
Then the testools are launched ....
Once finished many hours later, the result files are retreived and parsed to detect failed tests and report them.
Collected data
Actually, database engine is sqlite. Using SQLAlchemy is not a problem changing the engine.
main tables
TULIP is project based. The
PROJECT table defines basic settings such as periodicity, file locations and installer engines (apt-get, urpmi, apt-get-rpm)
| Field Name | Type | Comments |
|---|
| project_id | INTEGER NOT NULL PRIMARY KEY | Each project, based on a distribution image and state has an unique ID |
| name | TEXT | Name of the project, used to build working filenames |
| installer | TEXT | Code of supported upgrade technologies : urpmi, apt-get, apt-get-rpm. Every new code needs adding upgrade and update commands, log files locations, parser of log files ... |
| periodicity | TEXT | Defines how often the project has to launched - Daily, Weekly, Monthly |
| base_filename | TEXT | Qemu base image used for incremental definition |
| work_directory | TEXT | Directory where delta images will be stored |
| logfile_location | TEXT | Directory where upgrade an tests logfiles will be stored (defaults to work_directory) |
| work_filename | TEXT | Base filename for the image naming |
| number_history | INTEGER | (unused) Number of delta images to be handled at each turn |
| is_ready | TEXT | (unused) state of the project (waiting, running, closed) |
| creation_date | TIMESTAMP | Date of the project creation |
| ssh_port | INTEGER | Which ssh port will be used to connect |
| image_root_account | TEXT | Root account name of the qemu image |
| image_root_password | TEXT | Root password |
Each session run is parsed in the
LOG table that summarize the process, launched commands, counting upgrades packages and failures. Every delta (day-1, day-2, day-3 ...) of the tulip images stack is stored
| Field Name | Type | Comments |
|---|
| log_id | INTEGER NOT NULL PRIMARY KEY | Each qemu session as an unique ID |
| project_id | INTEGER REFERENCES project(project_id) | The project that this qemu session belongs to |
| status | TEXT | (unused) |
| log_date | TIMESTAMP | Date of this tulip run |
| archive_name | TEXT | Name of the compressed archive |
| command_lines | TEXT | list of command lines that have been launched |
| duration | NUMERIC(10,2) | total time to run update, upgrade and all tests that have been submited |
| total_modules | INTEGER | Number of installed/upgraded packages |
| failures | INTEGER | Total number of errors |
| delta_rank | INTEGER | Delta position of the image (range depends of periodicity) |
| previous_id | INTEGER | log_id of the previous delta the running qemu image was the origin |
| module_list | TEXT | List of involved packages |
| master_log_id | INTEGER | log_id of delta 1 of this tulip session |
The
DETAIL table stores the package (module) upgrade history. The success field indicates that the process ran smoothly or if some tests failed. (Values to be defined !!!)
| Field Name | Type | Comments |
|---|
| detail_id | INTEGER NOT NULL PRIMARY KEY | each package detail has an unique identifier |
| log_id | INTEGER REFERENCES log(log_id) | the qemu session (log_id) this detail is part of |
| module | TEXT | Package name |
| version | TEXT | Upgraded version |
| previous_version | NUMERIC(10,2) | Previous version (if availablz in log file) |
| success | INTEGER | Positive number is a success. Negative is a failure (-1 to -9 reserved for geenral failures; other for test failures - has to be reworked) |
| delta_rank | TEXT | What is the delta of the image where this detail occured |
| previous_id | INTEGER | previous ID for this image (allow to track history of a failure) |
Test Suite Management
The
TEST_SUITE table defines a test to be launched. The source directory, relatives to the tulipe/tests directory has to be defined. resfile_list are the retrieved files of the tests and faulty_file contains the list of the packages that failed on this tests. A code helper has been created giving the package for a given binary for each supported installers.
| Field Name | Type | Comments |
|---|
| test_id | INTEGER NOT NULL PRIMARY KEY | Every test has a unique identifier |
| test_name | TEXT | Name of the test |
| src_dir | TEXT | Where are the test sources to upload in the image |
| command_line | TEXT | Command line to be launched (todo : change this) |
| faulty_file | TEXT | File containing the list of faulty packages for which the test failed |
| resfile_list | TEXT | List of extra files to be retrieved |
Once a test defined, a relational table
PROJECT_TEST_REL associates this
TEST_SUITE to the
PROJECT.
| Field Name | Type | Comments |
|---|
| project_test_rel_id | INTEGER NOT NULL PRIMARY KEY | unique Id of this test-project relation |
| project_id | INTEGER REFERENCES project(project_id) | project_id for the relation |
| test_id | INTEGER REFERENCES test_suite(test_id) | test_id defining the test |
raw datas
All these information are parsed from the logfiles. Each logfile and output is kept archived identified with the log ID in a separate zip file
The deliberate choice was made not to overload the database. Raw data are then kept compressed with a filename convention
< projectID >.< logfineName >.< logSessionID >.tgz
No problem including this in the database though but tulip sees database fields more as workable datas. A separate table could be used not to load the
LOG table to much.
Reports
Reports are designed for many things such as monitoring daily activity or trace an information on a longer term scope.
For Tulip daily automatic reporting, an e-mail is sent to the administrator containing the console and various step informations outputs
For more elaborated reports, the BIRT reporting tool is used to display informations collected from the database. Until now, 3 types of report have been designed
- Session Report
- Session History Report
- Package history Report
Based on templates and having internal selection tools, Birt reports allow on-line browsing and PDF export.
Session report
A session is the run of a project at its period time.
The filter parameters asked is the log_id. A dropdown list gives all the sessions for all the projects.
We then have new upgrade and N delta upgrades data performed at one day
The report gives an overview of this daily activity.
First it presents some basic information on the project that is run such as ID, name, periodicity …
Then an overview with the total number of packages involved and the number of encountered errors. The list of all packages name is given for the current (delta 1) image
For analysing the delta, a cumulative bargraph involving the N images results is provided with for each upgrade showing in green successfully upgraded/installed packages and in red the proportion of errors
Then the name of the faulty packages is reported with the delta in which they occurred
Finally, a complete detailed list with delta, package name, version and status is displayed. A Failed package is highlighted by a light red background colour
Session History Report
Every delta 1 images gives the periodic (eg. daily) status of the upgrade. The Session History report gives, for a given project the history of installed/upgraded packages and failures
the filter asks for the project that is to be reported. A dropdown list gives all the project names that are in the database and let teh user make its choice.
Then, on report, the project basic informations are displayed. Then the number of delta 1 sessions and the time range it covers is displayed.
Then, for each session a proportion bargraph is displayed, giving in green the success and light red the failures for every non empty session. A second Y axis gives directly the number of errors
A second graph, gives a time-scale view with the number of successful and failing packages.
Package History
The filter asks for the package history that is to be reported. A dropdown list gives all the packages names that are in the database (with the first project they were found in) and let the user make its choice.
The package history report gives the history of its upgrade ordered by
project / date / delta
it gives the status (ok or failed), the session id (log_id) and the detail reference (detail_id) and the involved versions
If an other project has the same package name, a new list is displayed
First results
The provided PDF files as attachments show the various reports generated for the projects that has been used to design tulip. Mainly Mandriva and Debian distributions both in stable and unstable versions.
As the database is the one used for design, these results are not fully reliable before early September 2006. Caixa Magica distribution has been setup later and the actual results are non significant do not reported
The results are very encouraging as we can see that unstable delta versions show broken packages in the
daily reports. Approximativelly 50% of packages are reported with problems. Reading the
history reorts we can confirm that the upgrade for unstable ditrbutions remains risky.
Looking at
packages reporting, one can see that once an upgrade error occurs, its reparation can be followed until the last delta error vanishes. please note again that early results are not significnat and that they should be deleted.
perspectives
The following points will be worked as evolving features
- more reports and tests
- enhance the flow between reports and obtain detailed information and see raw outputs
- enhance communication to EDOS framework
- improve modularity on defining new installer
Conclusion
Tulip has been proven as a versatile framework enable to give promising results regarding packages upgrade process. it can also be used to test very special sets or even, only one package (like openoffice.org).Tulip engine now running allow to enhance all its tools and possibilities
Obviously works remain to do, allowing a smooth navigation in the reporting and some more helpers adding tests and projects. All the points in perspectives will be addressed and the integration in the test farm is also now possible and will be the next major focus.
Acknowledgements
Developed by Nuxeo team, Tulip has been inspired by prior works from Nexedi (
Umigumi-Umitester) and Caixa Magica (
eqatool)
Comments: 0