Ubiquitous storage

This section presents some ideas and solution requirements, which can be extended and developed, to help solve the problems presented in the problems summary and throughout the process descriptions and overviews. This is a first draft and the ideas are meant as a basis for discussion and brain-storming.

Ubiquitous storage

Ubiquitous storage infracstructure such as OceanStore can be a possible direction for keeping packages - either new or updated.

Some OceanStore properties:

• By design, files stored in OceanStore are "read only" and changes result in new versions. This can support both source packages and binary packages. New versions will be available but so will be the old ones.
• The more a file is read in OceanStore, the more available it becomes (replicated), while unread files can lose replicas. This way, the more popular a file is - the easier it will be to locate a copy.
• OceanStore is decentralized and uses Tapestry as the P2P overlay network. Other possibilities for the overlay network exist, such as Chord.
• OceanStore utilizes localization through Tapestry, bringing files closer to where many clients requested them.
• Byzantine agreement methods are employed and guarantee that all non-faulty replicas agree (as long as no more than about one third of the replicas are faulty).

A direction for research could possibly be "downgrading" OceanStore to fit the open source distribution model efficiently - for source code distribution as well as binary packages. Other ubiquitous storage models can be investigated.

Package search

An interesting approach for easying the retrieval of information about new software is a search engine for packages. Today the de-facto standard for getting information about new software is Freshmeat. Information about new software is posted on Freshmeat after the software maintainer herself has sent that information to the Freshmeat moderators. One can coceive an analogy between Freshmeat and Yahoo.com, as a portal. What's missing is a "Google" kind of portal. It may be possible to build a crawler that will analyze web pages for available software. This search engine will have to be proven more efficient than waiting for Freshmeat updates. Also, the search engine will have to be resilient to trojan horses posing as legitimate packages and of course spam.

Dependency graph viewer

A useful tool for a distribution editor can be a dependency graph viewer and manager. Packages will be connected to each other according to their dependencies. The links will be categorized according to the dependency type: replaces, recommends, depends etc. Dependencies between specific versions will also have to be supported since not all versions integrate well. This tool could help the editor to define and maintain distribution baselines and analyze consequences of upgrading versions of certain packages. Since tens of thousands of packages and versions exist, an important challenge will be the presentation of the information to the editor so it will be useful and not too much all at once. The viewer could support different views of the information, such as essential or core packages, multimedia-related packages, QA-approved packages and so on.

Putting it all together

Certain solutions can be constructed from existing tools. For example, the process could be automated as such: a Luau client will get updates straight from the developers; upon getting the update - an automatic script will configure, compile and build a new package from the source code; some automatic regression testing could be employed; the newly-made package will enter the unstable branch (according to the vendor's policies), perhaps with caveats to users; this will enable users who want to stay on the "bleeding edge" to get updates as soon as possible - with the vendor's package format; other users could wait until the new package moves to a more stable branch and gets a "stamp of approval".

Contemplations on Distribution

• Current Status
• Hierarchy of mirror sites
• Bottlenecks (e.g. no support for more than 5 RSync clients)
• Incomplete "Image" due to disconnections/bandwidth
• Some mirrors don't store all versions
• Sporadic downloading of packages, sources from various sites
• From specific developers (e.g. KDE, gcc and so)
• "Solution" Package
• P2P leads the way
• How to make the leap from a main site to the P2P networks (mirror sites as seeds that support only P2P downloading?)
• Selecting the P2P implementation which is best suitable
• Variations on the P2P network
• RSync on sources (packages too?!) between peers
• Compare versions between sources/package/ISOs
• Whenever possible - Download "diff" files and patch local sources
• Reliability of downloaded software
• PGP/GPG metadata, as published on "the main site"
• Trust, built over time (e.g. E-Bay sellers)
• When to download?
• Update on demand
• Periodic (Security patches only?), by version info
• Auto-Compile sources? (downloading a specific binary package from cooker will probably not work with older build, need to compile from source)

• • Main.AssafSagi - 31 Jan 2005

{metadata}

Topics Wp2 Wp4

{metadata}