perfsonar-dev - Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal
Subject: perfsonar development work
List archive
- From: Gijs Molenaar <>
- To: Stijn Melis <>
- Cc: , "" <>, Loukik Kudarimoti <>
- Subject: Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal
- Date: Fri, 16 May 2008 15:12:25 +0200
- Openpgp: id=72EE99CA
more inline comments.
By the way, I've already started packaging 'the new way'. Next week I'll
create a repository where people can start testing the new packaging. if
they want.
- gijs
Stijn Melis wrote:
> Hi,
>
> My comments are in line as well.
>
> Frederic LOUI wrote:
>> Hello there,
>>
>> My comments in line this mail.
>>
>> Gijs Molenaar a écrit :
>>> Hello,
>>>
>> Fred> Hi Gijs, hope you're doing well :-)
>>> I was asked to help on the release team for the perfsonar bundle 3.1.
>>> below are some points written down that I want to do, but maybe some of
>>> you have some suggestions about a thing or two. Nothing in here is
>>> definitive yet, this is the first draft of the thoughts in my mind about
>>> perfSONAR packages.
>>>
>> Fred> Welcome on board !
>>> I want to streamline the package creation of the microreleases, and
>>> build a package distribution architecture based upon repositories. I
>>> want to create deb/rpm build scripts for every service and put it in the
>>> source, so packages can be created from source directly with one or 2
>>> commands.
>
> This would be great, because it is a rather dull job at the moment
> creating these packages. I have to first create a release tar.gz from my
> compiled source code, then install and deploy everything on Fedora and
> on Debian, and then create the rpm and deb packages. This can take up
> quiet some time to do.
>
>
>>> Advantage is:
>>> - more flexible package creation
>>> - One time work, if there are no big changes in the micro release no
>>> adjustments have to be done to the package files
>>> - package creation time improvement
>>> - If everything automates well, the whole perfSONAR project can be
>>> compiled and packaged with one command (but maybe that is to
>>> ambitious).
>>>
>> Fred>
>> I agree with all the previous points. But apart the package management
>> issue, maybe I miss the point
>> but is there somewhere a description of a PerfSONAR web service common
>> behaviour ? That is to say:
>>
>> 1) Common logging rules. (Syslog or not and where ?) This has an
>> impact on package management if one service
>> is writing is log in
>> /opt/perfsonar/<web-service-name>-<version>/webapps/logs and the other
>> one writing its log
>> in /var/log/perfsonar/<web-service-name>-<version>. (if we want to
>> be LSB compliant ?)
>> 2) Same goes for web-service configuration file. Should we use /etc or
>> an other folder such as
>> /opt/perfsonar/<web-service-name>-<version>/webapps/config ?
>
> I think that at the moment there are no rules for this. For example, the
> SSHTELNET MP uses
> /opt/perfsonar/services/geant2-java-sshtelnet-mp/WEB-INF/classes/perfsonar/logs
> and
> /opt/perfsonar/services/geant2-java-sshtelnet-mp/WEB-INF/classes/perfsonar/conf
> for these files. But I am not sure, if there is a standard way to do
> this among all services.
>
>> The two previous points could impact the rpm/deb build script
>> elaboration.
>>> These packages will be put in a repository from where they can be
>>> installed automatically or manually, what the user wants.
>
>> Fred> I'm 100% with you. At last installing PerfSONAR using APT / YUM,
>> that would be a great major change
>>> I want to create two repositories for RPM and deb based distro's:
>>> - One for testing. This way testers can install new packages with one
>>> command and give feedback faster.
>>> - One for every stable release. advantages:
>>> - now when there is a 3.0 bundle release, there is no much space for
>>> error. With the stable repository small fixes can be done after the
>>> release
>>> - now there is no mechanism for security updates (not that perfSONAR
>>> worries about that now)
>>>
>> Fred>
>> Maybe use the same process "à la DEBIAN or UBUNTU" ? (STABLE / TESTING
>> / UNSTABLE ?)
>> STABLE would include all recommended package as per the release
>> management team : All software in "production" state.
>> TESTING would encompass the package agreed upon the next PerfSONAR
>> macro release.
>> UNSTABLE woud include day to day CVS version of the web services.
the daily CVS builds would require a lot of work in automating the
builds... I don't think I can put this up alone or in this time span.
>
> Seems like a good idea to me as well.
>
>>> using repositories makes life so much easier. you only need to add the
>>> repository one time to your apt/yum configuration, and installing and
>>> updating is very easy.
>>>
>> => At last ! ;-)
>> Just keep in mind that managing a package repositories involves
>> MANPOWER (Security/organisation) and network resources (BANDWIDTH)
>>> Ideally we only need to create 1 RPM for all RPM based distro's, but
>>> this needs to be investigated (for example different tomcat versions
>>> with different paths are possible).
>>>
>>>
>>> For every service I need:
>
> I don't know if I am supposed to give this information now, but if you
> want it for the SSHTELNET MP, I can give most of it. Just ask :)
>
>>> - Short name (for file names)
>>> - Long Name / Short description
>>> - Long description (package description)
>>> - A building STABLE branch source with building instructions, I prefer
>>> maven.
>>> - Special (build) dependencies (again, I prefer maven)
>
> Are these two previous points like the ant targets at the moment, i.e.
> scripts for creating the packages?
No, maven is a advanced build and dependecy management tool for your
code. There are even plugins for it so it can create DEB and RPM files,
but I didn't look at that yet. It just makes ant completely obsolete.
The build files for perfsonar are one big spagetti mess, I've never
understood them (but I'm not a java developer, so that's the problem
maybe).
>
>>> - Contact information (for questions and testing)
>>> - Where everything should be installed, why, and read-write permissions
>>> on file system. default will be _no_ write permissions on filesystem.
>>> - A quick way of testing if the service is working properly
>>>
>>>
>>> I need:
>>> - SSH/FTP access to a perfSONAR ftp/http server for package publication
>>> - People helping me to test packages
>>>
>>>
>>> My intention is:
>>> - Investigate popular GNU/Linux distributions:
>>> - Red Hat Enterprise
>>> - CentOS
>>> - Fedora
>>> - OpenSUSE (?)
>>> - Ubuntu
>>> - Debian
>>> For latest release and latest release - 1. I don't want to make this
>>> list to big, because all distro's have small differences which makes
>>> the packages too complicated. It would be fantastic to support
>>> FreeBSD, Mac OS X, Solaris, BeOS and UNIX System V, but I really think
>>> we need to minimise the packaging support for now to get this
>>> streamlined. People can always use the source if they really don't
>>> want to use a supported distro.
>>>
>>
>> Fred> Maybe an extensive manual/document describing how to elaborate
>> PerfSONAR repositories
>> including web-service intrinsic (All the point you requested above)
>> could be a start for someone
>> that wants to build its own repository, but for an other platform.
>> (Such as Solaris 2008-05 for instance)
>>
>>> - Design structure to use the tomcat of the distribution
>>> - For tomcat services create RPM / DEB build files
>>> architecture all/any, i386 and x86_64?
>>> - Standardisation of installation of non java package
>>> - goal is minimal installation effort, so no adjusting LD_LIBRARY_PATH
>>> or JAVA_HOME
>>> - Create yum/apt repositories:
>>> - For every bundle release
>>> - unstable for package testing
>>> - Making the tomcat services as secure as possible (tomcat security
>>> manager)
>>> - Write basic package build instructions
>>> - optional: Write basic install instructions
>>>
>>>
>>> So work flow for 3.1 will be:
>>> - Micro release of service is done (in a branch)
>>> - I'll create deb/rpm build scripts
>>> - optional: create security profile for tomcat security manager.
>>> - I'll build packages for service and do small test
>>> - Package is uploaded to testing repositories
>>> - Developers of service will test the package
>>> - feedback goes back to me
>>> - New package to testing repositories
>>> - Cycle repeats until ready
>>> - If all micro release packages are ready a stable repository is created
>
> Will you still be in the middle of creating the scripts during 3.1?
> Because it seems a bit redundant to me that you would build the
> packages. I think this would probably cost a lot more time compared to
> we (i.e. the developers) doing it ourselves. Sometimes there are just
> small issues, like typos or something, when I build a new package for a
> release, and then I fix it right away. It seems to me that we'd lose a
> rather big amount of time if you would have to build all packages for
> all services.
>
>>> Work flow of future will be:
>>> - Micro release of service is done (in a branch)
>>> - Developer will build package with deb/rpm build scripts, or this can
>>> be done centrally by releae team.
>>> - Test the package
>>> - package is given to release team, checks if everything is okay.
>>> - package is uploaded to repository
>>>
>
> This on the other hand looks very good to me. Again, I would prefer to
> build the package myself.
Well, the idea was to centralise the packaging, so it is standardized
and everything has the same structure. The packaging script was created
because some developers where having trouble creating packages. My
intention was to ease this process for these developers and save them time.
The script is nice, but my opinion is that it can't manage the exeptions
for all packages correctly (special permissions, etc).
From this point of view I really don't mind of you want to do it
yourself, this will save me time also. We just need to discuss your
packaging style (paths, permissions) to confirm to a certain perfsonar
standard (not yet defined, but not very complex).
>
>>> Implications for the end users are only positive, they will have the
>>> option to add a yum/apt repository and install all services with simple
>>> apt-get/yum commands. Direct installation of the packages is also
>>> possible (download rpm/deb manually).
>>>
>>> phew, now I'm going to drink a coffee.
>>>
>> Fred> Good luck !
>
> Indeed, the best of luck with this. I think it is a very good initiative!
>
> Cheers,
>
> Stijn
>
>> Just my 2 cents.
>> Cheers/Fred
>>
--
Gijs Molenaar
fingerprint C660 BABA 4B91 4B5C EB60 7739 4385 8ABA 72EE 99CA
Attachment:
signature.asc
Description: OpenPGP digital signature
- perfSONAR 3.1 packaging adjustment proposal, Gijs Molenaar, 05/09/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Frederic LOUI, 05/16/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Stijn Melis, 05/16/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Gijs Molenaar, 05/16/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Gijs Molenaar, 05/16/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Stijn Melis, 05/16/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Jochen Reinwand, 05/19/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Jeff W. Boote, 05/19/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Gijs Molenaar, 05/20/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Jochen Reinwand, 05/21/2008
- Re: [pS-dev] perfSONAR 3.1 packaging adjustment proposal, Frederic LOUI, 05/16/2008
Archive powered by MHonArc 2.6.16.