Shibboleth Developers

[Russell Beall <>]

Chad,

Allow me to allay some concerns on these issues.

Terracotta has proven to be robust and resilient to failures.  There  
has been a learning curve and a growing period of working through the  
initial issues.  However, now that I have it working in a production  
setting and knocked out the kinks, it is working very efficiently.

The main garbage collection issue has been handled.  By default,  
Terracotta runs a garbage collector over its disk store every hour.   
This is relatively useless when session data is not expiring that  
fast.  It consumes much processor on the active terracotta server, and  
some on the passive server when it deletes content as selected by the  
active server.

Factually, this process does not need to run more than once a day.  I  
now have it set to run at 2:30am.  Last night it took one hour to  
flush 20mil expired objects.  The applications are not paused while  
this is occurring and there is far more than enough power to handle  
the limited late night login activity.

In a setting with more login activity, it could need to run more  
often, but in that case it would have a machine with more processors  
which could handle the load, or even separate hardware for the  
terracotta instances.

The latest garbage collection issue is relatively trivial.  I have  
only been attempting to tweak settings so that the occasional slight  
pause for Old Generation collection (every two hours or so) does not  
make users wait a bit.  This is almost a non-issue.

I would expect it to take far more effort to switch off onto some  
other HA service than it would take to deal with the occasional  
upgrade woes.

Most of my experiences with Terracotta have caused me to be impressed  
and happy with the product.  The tomcat servers efficiently select a  
new active terracotta server when I take one down (practically without  
any application pause time the last time -- it is a pause only,  
without error), and other knocking around I have given it, which it  
has stood up to, cause me to believe that we are now running in a  
safe, highly available environment with protection from hardware  
failure.

Just my 2¢,
Russ.

On Dec 2, 2008, at 11:44 PM, Chad La Joie wrote:

> Sorry, I said I would move this to the dev list a while ago and it  
> just
> fell off my list.
>
> Taylor, if I can summarize, you said you saw the following benefits to
> creating a TIM:
> - Removal of most, if not all, of the information in the <application>
> section of the TC config
> - Expression of support versions of Shib and TC
> - Distribution of the file via tim-get
>
> I actually don't see these benefits.  Users can safely ignore the
> information in the TC config file, our instructions are pretty clear
> they just need to touch the top part.  In the event that they're using
> an extension that would need them to instrument classes I believe  
> having
> the other classes there as a guide is also helpful.  In terms of
> expressing support for particular versions of Shibboleth we only  
> support
> one version at a time.  So when that version comes out I update the  
> docs
> (or at least try to) and thats the end.  Distribution of the config  
> file
> is also a non-issue.  It comes with the Shib distribution or people  
> can
> pull it off the site from the page that provides directions on setting
> up clustering.
>
> Now, one thing I do find worrying about our use of TC to date is that
> when 2.7.1, in theory just a bug release, came out we had to change  
> our
> directions for installing TC because a TIM that had worked no longer
> did.  I'm willing to believe that this is because I mistakenly used a
> SNAPSHOT version and not an official release of the vector TIM.  If
> however that wasn't the case and such changes will be common we'll
> probably need to move to something other than Terracotta because it
> won't be supportable given our resources.
>
> Additionally I'm having growing concerns as I watch the interaction
> between Russ and you.  I am very grateful to Russ for putting this  
> stuff
> through it's paces in a way that I can not but I see him bring up a
> number of a issues (garbage collection currently) and you are  
> providing
> answers (which is good, and I appreciate that too).  However, the
> information you provide isn't something I can find in the TC
> documentation.  I don't mind telling people to go read the docs, I do
> that all the time, but if there are no docs for them to read than,
> again, it's not a supportable product for us.
>
> -------- Original Message --------
> Subject: Re: [Shib-Users] Roll call - Terracotta usage?
> Date: Thu, 6 Nov 2008 10:11:25 -0800 (PST)
> From: Taylor Gautier 
> <>
> Reply-To: 
>
> To: 
>
>
> It's not as much as I would like, but should give you a pretty good  
> idea
> how to get going:
>
>
> http://www.terracotta.org/web/display/orgsite/Terracotta+Integration+Modules+Manual
>
>
>
> Basically, TIMs are OSGi modules with a bit more meta-data in them.
> There's also a maven-archetype to get you going quickly that is listed
> in the doc.  If you don't use Maven to build Shibboleth, then you will
> find it a bit harder to manage and create TIMs because Maven manages a
> lot of the dependencies and other tasks such as updating the meta-data
> in the JAR manifest.
>
>
> Compass would provide a good example of how to build a TIM using Ant
> since it uses Ant, not Maven.
>
>
>
> ----- Original Message -----
> From: "Chad La Joie" 
> <>
> To: 
>
> Sent: Thursday, November 6, 2008 10:06:37 AM GMT -08:00 US/Canada  
> Pacific
> Subject: Re: [Shib-Users] Roll call - Terracotta usage?
>
> Is there any documentation on creating these?
>
> Taylor Gautier wrote:
> > Thanks for the response Chad.
> >
> >
> > I think the largest benefit to using the TIM method is hiding or  
> > abstracting all the uninteresting bits of config - from the end- 
> > users perspective - into the TIM.  Only the shib devs need to  
> > manage that stuff, so I think it's cleaner.  It also provides for a  
> > mechanism to track dependencies, both as a dependency on versions  
> > of shibboleth, and on Terracotta.  This may not be so important  
> > now, but may be more important over time.  Finally, you can track  
> > versions of the implementation - I find that in real deployments,  
> > rather than diffing configs, it is a lot easier to simply state  
> > that "tim-shibboleth version 1.0.1 is out of date, you should use  
> > 1.0.2".
> >
> >
> > As for the distribution using tim-get (and the resulting extra  
> > step) this is simply one way to manage the distribution - if you  
> > would like to update the shibboleth build process to produce a TIM,  
> > that is absolutely fine.  The Compass project is doing this  
> > currently:  http://www.compass-project.org/  The Compass project is  
> > simply a Java library, and thus produces a single JAR file.  To  
> > modify this output JAR to "be" a TIM was a simple matter of  
> > updating only slightly the Compass build process.  TIMs are JARs  
> > with extra meta-data in them (and the config file) so this is  
> > usually pretty easy.  There are two options here - one would be if  
> > Shibboleth has a natural "jar" file - and the other would be to  
> > produce a new build artifact that is the TIM.
> >
> >
> >
> > ----- Original Message -----
> > From: "Chad La Joie" 
> > <>
> > To: 
> >
> > Sent: Thursday, November 6, 2008 9:48:18 AM GMT -08:00 US/Canada  
> > Pacific
> > Subject: Re: [Shib-Users] Roll call - Terracotta usage?
> >
> >
> >
> > Taylor Gautier wrote:
> > > Hello fellow Shibboleth users. Â
> > >
> > >
> > > I've noticed a slight increase in traffic pertaining to Terracotta  
> > > on this list, thought I would join and see how I can help. Â So  
> > > first question, how many people have tried Terracotta with  
> > > Shibboleth - and how many are running the setup in production? Â I  
> > > don't mind if that number is zero, just want to know! :)
> >
> > My guess would be zero currently.
> >
> > > For anyone that has tried the Terracotta integration, there is a  
> > > simpler way to package up the Shibboleth configuration for  
> > > Terracotta. Â To do so, Terracotta supports the notion of plugin  
> > > modules called "TIMs" - Terracotta Integration Modules. Â I am  
> > > thinking it would be a lot easier to package up the Shibboleth  
> > > configuration provided in the source tree as a Terracotta module  
> > > and would like your feedback on this idea.
> >
> > Do you have documentation on creating the TIMs, I couldn't find any  
> > when
> > I looked.
> >
> > > In doing so, your configuration would be greatly simplified. Â So  
> > > would installation. Â After downloading and installing Shibboleth  
> > > and then Terracotta, to install the Terracotta module you only  
> > > have to run the integrated TIM update tool to get the shibboleth  
> > > configuration, like so:
> >
> > This is one more command than what people currently would have to do.
> >
> > > From there, it would be a simple matter of putting together a tc- 
> > > config.xml file that would be very small. Â It would only include  
> > > the server and client configuration and the link to the shibboleth  
> > > TIM, like so:
> >
> > While the current config file needs a couple updates, once it's set
> > people aren't going to have to change anything other than the server
> > information.  So using a TIM doesn't offer any additional benefit.
> >
> > > The final benefit is that having a tim-shibboleth would allow  
> > > updates such as the recent additions listed by Greg Haverkamp to  
> > > be updated into the TIM, and released back to the community in the  
> > > TIM. Â The tim-get tool could then be used to get the updated TIM  
> > > simply by using the update commnand:
> >
> > Or people could just grab an update file from the web.  Again, I  
> > don't
> > see the actual benefit of the TIM method.  But then, as I also  
> > said, I
> > couldn't find any real documentation on the TIM modules.
>
> --
> SWITCH
> Serving Swiss Universities
> --------------------------
> Chad La Joie, Software Engineer, Net Services
> Werdstrasse 2, P.O. Box, 8021 Zürich, Switzerland
> phone +41 44 268 15 75, fax +41 44 268 15 68
> ,
>  http://www.switch.ch
>
>
>
> --
> SWITCH
> Serving Swiss Universities
> --------------------------
> Chad La Joie, Software Engineer, Net Services
> Werdstrasse 2, P.O. Box, 8021 Zürich, Switzerland
> phone +41 44 268 15 75, fax +41 44 268 15 68
> ,
>  http://www.switch.ch