Shibboleth Developers

[André Cruz <>]

At least part of the leak I had was with the memcache client API I  
was using...

I'm adapting my code to a new API to see what difference it makes.
Sorry for the possible false alarm.

André

On 2007/10/30, at 02:11, Scott Cantor wrote:

> After a little research, I'm still convinced the "leak" people are  
> seeing is
> Linux glibc doing a lousy job of heap management. My testing shows  
> that
> Linux almost never lowers the process' VM footprint once it's been  
> bumped to
> a certain level, but after purging sessions, it mostly reuses the  
> VM it does
> allocate. That's what my original testing of 1.3 showed, which is  
> why I
> closed the book on leak checking.
>
> Windows, by contrast, lowers the VM footprint quite a bit once  
> sessions are
> purged, which is why it stablizes at a much lower level when the  
> load falls
> off. The conventional wisdom is that Windows doesn't do this, but  
> this is
> what I'm seeing.
>
> I can't however explain why the Linux process size would reach 2G so
> quickly. That isn't my experience while load testing at all. It  
> does not
> appear to keep growing boundlessly unless the sessions stay in  
> memory, which
> is why I'm still asking whether the log shows them being purged.
>
> Related to this, Xerces 2.8 is not a great fit for 1.3 after all,
> unfortunately. I lost the argument to favor my kind of usage  
> pattern, but I
> thought the compromise numbers it uses would work better. It's not  
> doing
> well, from what I can tell, taking up roughly 48k per typical  
> document,
> which works out to a lousy 150k or so per session, about double or  
> more what
> my patch was doing.
>
> I'm going to do a custom build of it again for inclusion with 1.3.1 on
> Windows, where I can control the build easily. I *may* do a custom  
> package
> for other platforms, but only as a binary. My main solution to this  
> problem
> is to get 2.0 done, which should have a different memory footprint and
> doesn't cache XML in sessions.
>
> I don't know what to tell people in the meantime about any leaks,  
> other than
> to say that glibc doesn't appear to be very good. We may find that  
> the best
> solution is to explore alternative heap libraries that can load  
> ahead of
> glibc, such as Hoard perhaps. I haven't tried it, but it might be  
> worth a
> look. If somebody has a tool they want to try that does a better  
> job than
> valgrind at finding leaks, that's fine too.
>
> -- Scott