Shibboleth Developers

["Scott Cantor" <>]

After a little research, I'm still convinced the "leak" people are seeing is
Linux glibc doing a lousy job of heap management. My testing shows that
Linux almost never lowers the process' VM footprint once it's been bumped to
a certain level, but after purging sessions, it mostly reuses the VM it does
allocate. That's what my original testing of 1.3 showed, which is why I
closed the book on leak checking.

Windows, by contrast, lowers the VM footprint quite a bit once sessions are
purged, which is why it stablizes at a much lower level when the load falls
off. The conventional wisdom is that Windows doesn't do this, but this is
what I'm seeing.
 
I can't however explain why the Linux process size would reach 2G so
quickly. That isn't my experience while load testing at all. It does not
appear to keep growing boundlessly unless the sessions stay in memory, which
is why I'm still asking whether the log shows them being purged.

Related to this, Xerces 2.8 is not a great fit for 1.3 after all,
unfortunately. I lost the argument to favor my kind of usage pattern, but I
thought the compromise numbers it uses would work better. It's not doing
well, from what I can tell, taking up roughly 48k per typical document,
which works out to a lousy 150k or so per session, about double or more what
my patch was doing.

I'm going to do a custom build of it again for inclusion with 1.3.1 on
Windows, where I can control the build easily. I *may* do a custom package
for other platforms, but only as a binary. My main solution to this problem
is to get 2.0 done, which should have a different memory footprint and
doesn't cache XML in sessions.

I don't know what to tell people in the meantime about any leaks, other than
to say that glibc doesn't appear to be very good. We may find that the best
solution is to explore alternative heap libraries that can load ahead of
glibc, such as Hoard perhaps. I haven't tried it, but it might be worth a
look. If somebody has a tool they want to try that does a better job than
valgrind at finding leaks, that's fine too.

-- Scott