perfsonar development work

[Gijs Molenaar <>]

Frederic LOUI wrote:
> Hello there,
> 
> My comments in line this mail.
> 
> Gijs Molenaar a écrit :
>> Hello,
>>   
> Fred> Hi Gijs, hope you're doing well :-)
>> I was asked to help on the release team for the perfsonar bundle 3.1.
>> below are some points written down that I want to do, but maybe some of
>> you have some suggestions about a thing or two. Nothing in here is
>> definitive yet, this is the first draft of the thoughts in my mind about
>> perfSONAR packages.
>>   
> Fred> Welcome on board !
>> I want to streamline the package creation of the microreleases, and
>> build a package distribution architecture based upon repositories. I
>> want to create deb/rpm build scripts for every service and put it in the
>> source, so packages can be created from source directly with one or 2
>> commands.
>>
>> Advantage is:
>>  - more flexible package creation
>>  - One time work, if there are no big changes in the micro release no
>>    adjustments have to be done to the package files
>>  - package creation time improvement
>>  - If everything automates well, the whole perfSONAR project can be
>>    compiled and packaged with one command (but maybe that is to
>>    ambitious).
>>   
> Fred>
> I agree with all the previous points. But apart the package management
> issue, maybe I miss the point
> but is there somewhere a description of a PerfSONAR web service common
> behaviour ? That is to say:
> 
> 1) Common logging rules. (Syslog or not and where ?) This has an impact
> on package management if one service
>    is writing is log in
> /opt/perfsonar/<web-service-name>-<version>/webapps/logs and the other
> one writing its log
>   in /var/log/perfsonar/<web-service-name>-<version>. (if we want to be
> LSB compliant ?)

I don't know, but our service is configured to write to
<tomcat-log-dir>/<servicename>.log. This works fine and is LSB compatible.

> 2) Same goes for web-service configuration file. Should we use /etc or
> an other folder such as
> /opt/perfsonar/<web-service-name>-<version>/webapps/config ?

My proposal is to put the configuration files (like service.properties)
in /etc/<servicename> and symlink them to the WEB-INF folder. Most
packaging is done this way.

> 
> The two previous points could impact the rpm/deb build script elaboration.
>> These packages will be put in a repository from where they can be
>> installed automatically or manually, what the user wants.
>>   
> Fred> I'm 100% with you. At last installing PerfSONAR using APT / YUM,
> that would be a great major change
>> I want to create two repositories for RPM and deb based distro's:
>>  - One for testing. This way testers can install new packages with one
>>    command and give feedback faster.
>>  - One for every stable release. advantages:
>>    - now when there is a 3.0 bundle release, there is no much space for
>>      error. With the stable repository small fixes can be done after the
>>      release
>>    - now there is no mechanism for security updates (not that perfSONAR
>>      worries about that now)
>>   
> Fred>
> Maybe use the same process "à la DEBIAN or UBUNTU" ? (STABLE / TESTING /
> UNSTABLE ?)
> STABLE would include all recommended package as per the release
> management team : All software in "production" state.
> TESTING would encompass the package agreed upon the next PerfSONAR macro
> release.
> UNSTABLE woud include day to day CVS version of the web services.
>> using repositories makes life so much easier. you only need to add the
>> repository one time to your apt/yum configuration, and installing and
>> updating is very easy.
>>   
> => At last ! ;-)
> Just keep in mind that managing a package repositories involves MANPOWER
> (Security/organisation) and network resources (BANDWIDTH)

yes, we need to think about that, but using a repository saves so much
release management time that maybe this is compensated.

bandwidth isn't really a concern I think, maybe there will be a bit more
queriing, but I don't believe there will be a lot more downloads of the
packages as there is now.

>> Ideally we only need to create 1 RPM for all RPM based distro's, but
>> this needs to be investigated (for example different tomcat versions
>> with different paths are possible).
>>
>>
>> For every service I need:
>> - Short name (for file names)
>> - Long Name / Short description
>> - Long description  (package description)
>> - A building STABLE branch source with building instructions, I prefer
>>   maven.
>> - Special (build) dependencies (again, I prefer maven)
>> - Contact information (for questions and testing)
>> - Where everything should be installed, why, and read-write permissions
>>   on file system. default will be _no_ write permissions on filesystem.
>> - A quick way of testing if the service is working properly
>>
>>
>> I need:
>> - SSH/FTP access to a perfSONAR ftp/http server for package publication
>> - People helping me to test packages
>>
>>
>> My intention is:
>> - Investigate popular GNU/Linux distributions:
>>   - Red Hat Enterprise
>>   - CentOS
>>   - Fedora
>>   - OpenSUSE (?)
>>   - Ubuntu
>>   - Debian
>>   For latest release and latest release - 1. I don't want to make this
>>   list to big, because all distro's have small differences which makes
>>   the packages too complicated. It would be fantastic to support
>>   FreeBSD, Mac OS X, Solaris, BeOS and UNIX System V, but I really think
>>   we need to minimise the packaging support for now to get this
>>   streamlined. People can always use the source if they really don't
>>   want to use a supported distro.
>>   
> 
> Fred>   Maybe an extensive manual/document describing how to elaborate
> PerfSONAR repositories
>   including web-service intrinsic (All the point you requested above)
> could be a start for someone
>   that wants to build its own repository, but for an other platform.
> (Such as Solaris 2008-05 for instance)

For now, I don't have too much time to write big manuals and documents
:) But it is something that would be nice.

> 
>> - Design structure to use the tomcat of the distribution
>> - For tomcat services create RPM / DEB build files
>>   architecture all/any, i386 and x86_64?
>> - Standardisation of installation of non java package
>> - goal is minimal installation effort, so no adjusting LD_LIBRARY_PATH
>>   or JAVA_HOME
>> - Create yum/apt repositories:
>>   - For every bundle release
>>   - unstable for package testing
>> - Making the tomcat services as secure as possible (tomcat security
>>   manager)
>> - Write basic package build instructions
>> - optional: Write basic install instructions
>>
>>
>> So work flow for 3.1 will be:
>> - Micro release of service is done (in a branch)
>> - I'll create deb/rpm build scripts
>> - optional: create security profile for tomcat security manager.
>> - I'll build packages for service and do small test
>> - Package is uploaded to testing repositories
>> - Developers of service will test the package
>> - feedback goes back to me
>> - New package to testing repositories
>> - Cycle repeats until ready
>> - If all micro release packages are ready a stable repository is created
>>
>> Work flow of future will be:
>> - Micro release of service is done (in a branch)
>> - Developer will build package with deb/rpm build scripts, or this can
>>   be done centrally by releae team.
>> - Test the package
>> - package is given to release team, checks if everything is okay.
>> - package is uploaded to repository
>>
>>
>> Implications for the end users are only positive, they will have the
>> option to add a yum/apt repository and install all services with simple
>> apt-get/yum commands. Direct installation of the packages is also
>> possible (download rpm/deb manually).
>>
>> phew, now I'm going to drink a coffee.
>>   
> Fred>  Good luck !

Thanks! :)

> 
> Just my 2 cents.
> Cheers/Fred
> 


-- 
Gijs Molenaar

fingerprint C660 BABA 4B91 4B5C EB60 7739 4385 8ABA 72EE 99CA

Attachment: signature.asc
Description: OpenPGP digital signature