perfsonar development work

[Stijn Melis <>]

Hi,

My comments are in line as well.

Frederic LOUI wrote:
> Hello there,
>
> My comments in line this mail.
>
> Gijs Molenaar a écrit :
> > Hello,
> >   
> Fred> Hi Gijs, hope you're doing well :-)
> > I was asked to help on the release team for the perfsonar bundle 3.1.
> > below are some points written down that I want to do, but maybe some of
> > you have some suggestions about a thing or two. Nothing in here is
> > definitive yet, this is the first draft of the thoughts in my mind about
> > perfSONAR packages.
> >   
> Fred> Welcome on board !
> > I want to streamline the package creation of the microreleases, and
> > build a package distribution architecture based upon repositories. I
> > want to create deb/rpm build scripts for every service and put it in the
> > source, so packages can be created from source directly with one or 2
> > commands.

This would be great, because it is a rather dull job at the moment 
creating these packages. I have to first create a release tar.gz from my 
compiled source code, then install and deploy everything on Fedora and 
on Debian, and then create the rpm and deb packages. This can take up 
quiet some time to do.


> > Advantage is:
> >  - more flexible package creation
> >  - One time work, if there are no big changes in the micro release no
> >    adjustments have to be done to the package files
> >  - package creation time improvement
> >  - If everything automates well, the whole perfSONAR project can be
> >    compiled and packaged with one command (but maybe that is to
> >    ambitious).
> >   
> Fred>
> I agree with all the previous points. But apart the package management 
> issue, maybe I miss the point
> but is there somewhere a description of a PerfSONAR web service common 
> behaviour ? That is to say:
>
> 1) Common logging rules. (Syslog or not and where ?) This has an impact 
> on package management if one service
>    is writing is log in 
> /opt/perfsonar/<web-service-name>-<version>/webapps/logs and the other 
> one writing its log
>   in /var/log/perfsonar/<web-service-name>-<version>. (if we want to be 
> LSB compliant ?)
> 2) Same goes for web-service configuration file. Should we use /etc or 
> an other folder such as 
> /opt/perfsonar/<web-service-name>-<version>/webapps/config ?

I think that at the moment there are no rules for this. For example, the 
SSHTELNET MP uses 
/opt/perfsonar/services/geant2-java-sshtelnet-mp/WEB-INF/classes/perfsonar/logs 
and 
/opt/perfsonar/services/geant2-java-sshtelnet-mp/WEB-INF/classes/perfsonar/conf 
for these files. But I am not sure, if there is a standard way to do 
this among all services.

> The two previous points could impact the rpm/deb build script elaboration.
> > These packages will be put in a repository from where they can be
> > installed automatically or manually, what the user wants.

> Fred> I'm 100% with you. At last installing PerfSONAR using APT / YUM, 
> that would be a great major change
> > I want to create two repositories for RPM and deb based distro's:
> >  - One for testing. This way testers can install new packages with one
> >    command and give feedback faster.
> >  - One for every stable release. advantages:
> >    - now when there is a 3.0 bundle release, there is no much space for
> >      error. With the stable repository small fixes can be done after the
> >      release
> >    - now there is no mechanism for security updates (not that perfSONAR
> >      worries about that now)
> >   
> Fred>
> Maybe use the same process "à la DEBIAN or UBUNTU" ? (STABLE / TESTING / 
> UNSTABLE ?)
> STABLE would include all recommended package as per the release 
> management team : All software in "production" state.
> TESTING would encompass the package agreed upon the next PerfSONAR macro 
> release.
> UNSTABLE woud include day to day CVS version of the web services.

Seems like a good idea to me as well.

> > using repositories makes life so much easier. you only need to add the
> > repository one time to your apt/yum configuration, and installing and
> > updating is very easy.
> >   
> => At last ! ;-)
> Just keep in mind that managing a package repositories involves MANPOWER 
> (Security/organisation) and network resources (BANDWIDTH)
> > Ideally we only need to create 1 RPM for all RPM based distro's, but
> > this needs to be investigated (for example different tomcat versions
> > with different paths are possible).
> >
> >
> > For every service I need:

I don't know if I am supposed to give this information now, but if you 
want it for the SSHTELNET MP, I can give most of it. Just ask :)

> > - Short name (for file names)
> > - Long Name / Short description
> > - Long description  (package description)
> > - A building STABLE branch source with building instructions, I prefer
> >   maven.
> > - Special (build) dependencies (again, I prefer maven)

Are these two previous points like the ant targets at the moment, i.e. 
scripts for creating the packages?

> > - Contact information (for questions and testing)
> > - Where everything should be installed, why, and read-write permissions
> >   on file system. default will be _no_ write permissions on filesystem.
> > - A quick way of testing if the service is working properly
> >
> >
> > I need:
> > - SSH/FTP access to a perfSONAR ftp/http server for package publication
> > - People helping me to test packages
> >
> >
> > My intention is:
> > - Investigate popular GNU/Linux distributions:
> >   - Red Hat Enterprise
> >   - CentOS
> >   - Fedora
> >   - OpenSUSE (?)
> >   - Ubuntu
> >   - Debian
> >   For latest release and latest release - 1. I don't want to make this
> >   list to big, because all distro's have small differences which makes
> >   the packages too complicated. It would be fantastic to support
> >   FreeBSD, Mac OS X, Solaris, BeOS and UNIX System V, but I really think
> >   we need to minimise the packaging support for now to get this
> >   streamlined. People can always use the source if they really don't
> >   want to use a supported distro.
> >   
>
> Fred>   Maybe an extensive manual/document describing how to elaborate 
> PerfSONAR repositories
>   including web-service intrinsic (All the point you requested above) 
> could be a start for someone
>   that wants to build its own repository, but for an other platform. 
> (Such as Solaris 2008-05 for instance)
>
> > - Design structure to use the tomcat of the distribution
> > - For tomcat services create RPM / DEB build files
> >   architecture all/any, i386 and x86_64?
> > - Standardisation of installation of non java package
> > - goal is minimal installation effort, so no adjusting LD_LIBRARY_PATH
> >   or JAVA_HOME
> > - Create yum/apt repositories:
> >   - For every bundle release
> >   - unstable for package testing
> > - Making the tomcat services as secure as possible (tomcat security
> >   manager)
> > - Write basic package build instructions
> > - optional: Write basic install instructions
> >
> >
> > So work flow for 3.1 will be:
> > - Micro release of service is done (in a branch)
> > - I'll create deb/rpm build scripts
> > - optional: create security profile for tomcat security manager.
> > - I'll build packages for service and do small test
> > - Package is uploaded to testing repositories
> > - Developers of service will test the package
> > - feedback goes back to me
> > - New package to testing repositories
> > - Cycle repeats until ready
> > - If all micro release packages are ready a stable repository is created

Will you still be in the middle of creating the scripts during 3.1? 
Because it seems a bit redundant to me that you would build the 
packages. I think this would probably cost a lot more time compared to 
we (i.e. the developers) doing it ourselves. Sometimes there are just 
small issues, like typos or something, when I build a new package for a 
release, and then I fix it right away. It seems to me that we'd lose a 
rather big amount of time if you would have to build all packages for 
all services.

> > Work flow of future will be:
> > - Micro release of service is done (in a branch)
> > - Developer will build package with deb/rpm build scripts, or this can
> >   be done centrally by releae team.
> > - Test the package
> > - package is given to release team, checks if everything is okay.
> > - package is uploaded to repository

This on the other hand looks very good to me. Again, I would prefer to 
build the package myself.

> > Implications for the end users are only positive, they will have the
> > option to add a yum/apt repository and install all services with simple
> > apt-get/yum commands. Direct installation of the packages is also
> > possible (download rpm/deb manually).
> >
> > phew, now I'm going to drink a coffee.
> >   
> Fred>  Good luck !

Indeed, the best of luck with this. I think it is a very good initiative!

Cheers,

Stijn

> Just my 2 cents.
> Cheers/Fred