Debian Data Retrieval and Database Filling Procedure

Debian Data Retrieval and Database Filling Procedure

The current versions of history and anla use the MySQL backend, which is quite slow. However, while we wait for the development of the new versions, we will put it into production. This requires automatizing the database filling procedure.

Location of source files

The sources for anla, debsqlfill and the various libraries are under edos/software/dependencies. The scripts are under edos/software/anla-scripts.

Organization

The current system comprises the following parts :

• A MySQL database, currently running on cadet.inria.fr. The database is "edos-debian".
• A directory where the raw metadata files (Contents.gz for various distributions) downloaded from snapshot.debian.net are stored. This is currently the /edos/data/snapshot directory on cadet.
• A crontab entry that calls, everyday at 23:45 Paris Time, a script, download.sh, which downloads the "metadata du jour" from snapshot.debian.net. The script calls debsqlfill with the -download option ; debsqlfill then uses the Netclient module to download the data.
• The anla server, which is actually a single process that runs on cadet, port 9998, under the user "anla", in /home/anla.
• The /edos/data/download.sh script
• The /edos/data/fill.sh script
• The /home/anla directory contains :

Filling the database

This part has been automated. The procedure for adding new data to the database and getting anla to display lifetables for the extended part follows.

• First, we need to fill the SQL database.
  • Enter the SQL database, for example by launching edosdeb.sh
  • Type

select max(day) from lifetimes;

• • Enter the directory /edos/data
  • Type

./debsqlfill.opt -start START_DATE -fill

• • This will start filling the SQL database. It takes about 4 to 5 minutes per day.

Updating the cache and solving the installability

We now need to launch a copy of the anla process, which we call the recomputer,

1. reload the SQL data,
2. recompute the installability,
3. save the SQL data cache into history-cache,
4. save the installability information into installability.msh.

The procedure is therefore as follows:

• Become anla
• Let DATE be today's date string (like 2006-05-17).
• Do

mkdir -f /home/anla/backups/DATE/
cp installability.msh /home/anla/backups/DATE/
cp history-cache /home/anla/backups/DATE/

• Launch ./recompute.sh
• This script should recreate fresh installability.msh and history-cache files.
• Relaunch anla with ./run.sh