Shibboleth Developers

["Scott Cantor" <>]

> I know shiny new toys (2.0) are more motivating than old stuff (1.3)
> but, as far as I know, 1.3 is the ONLY stable version of the
> shibboleth SP released so please bear with me...

Trust me, that has *nothing* to do with it. You're saying you think the RPC
code is responsible, and that code is entirely gone in 2.0. So what else can
I do with it? It's gone...that seems like a pretty comprehensive fix,
doesn't it?

I think maybe you think there's a simple fix for this, but I really doubt
that. I think Linux is broken when it comes to long running processes. 2.0
might be better and it might not (my guess is not), but 1.3 is really not
going to get any better than it is now.

So far, I get the feeling people don't believe me, but everything I'm seeing
points to that as the problem. That isn't "good", but it would mean it's a
better use of time to reproduce this on 2.0 and then see what kind of
redesign would be needed there to mitigate it. For example, even without a
new cache plugin, it could be modified to support "dumping" its state and
then reloading it so that graceful restarts might be possible. That is very
doable using the 2.0 code as a starting point.

The point being that if it takes a complete redesign to fix it (and I think
it would have to if Linux is the problem), that isn't going to happen on the
old branch. That wouldn't make sense.

So this is *not* about what's "new" or what I would prefer to work on, it's
just practicality.
 
> I can see that svc_create() is called when a thread is started. In
> this function a bunch of functions that allocate memory are called
> (svcfd_create() for example). If a problem happens when svc_register()
> is called some cleanup code is called: svc_destroy() or the inline
> version. But I don't see this kind of code being called when the
> thread finishes...

I believe the ONC library handles it when the socket closes in some fashion.
I'm pretty sure I've been over this ground, but I'd have to spend a day or
so to find that out. I traced the entire code path at one time and I think I
would have noted this.
 
> When the while cycle finishes on run(), shouldn't
> there be the same cleanup code? I must be missing something here. If
> this code is really missing then that would explain the memory growth.

If you were correct, the same problem would happen on Windows, and more
importantly, it wouldn't exhibit the behavior I see, which is growth but
then *reuse* of the same memory if you let it purge and then load it down
again.

But I really don't think that's where the RAM usage comes from. I can give
you a simple test...run the test-client applet that's in the code in a loop
or hack on it to fire up a ton of connected threads. That makes an RPC, same
as the real code does, and runs all that code you're looking at. If the
memory usage is actually just due to RPC handling and not due to XML
processing like I think it is, you'd see it there.

-- Scott