Skip to Content.
Sympa Menu

grouper-dev - Re: [grouper-dev] Reducing PSPNG's memory footprint

Subject: Grouper Developers Forum

List archive

Re: [grouper-dev] Reducing PSPNG's memory footprint


Chronological Thread 
  • From: "Gettes, Michael" <>
  • To: Bert Lindgren <>
  • Cc: "" <>
  • Subject: Re: [grouper-dev] Reducing PSPNG's memory footprint
  • Date: Tue, 4 Jun 2019 18:01:47 +0000

hi Bert - I replied to your private Slack query about all this as well. If
my estimates on the size of a cached entry are correct (I supposed 600bytes)
then 500K (which is what I am wanting to use) would be only 300MB of cache.
If my math is right - then I don’t believe putting this on disk is worth it -
just keep it all in memory. If you do this kind of caching for each LDAP
server defined for Grouper - and 10 LDAP servers - then it would be 3GB of
caching. I would still say to keep this in memory.

math and opinions are my own.

/mrg

> On Jun 4, 2019, at 1:22 PM, Bee-Lindgren, Bert
> <> wrote:
>
> PSPNG takes a Collections-intensive approach to processing identities. In
> other words, there are methods to fetch hundreds of thousands of objects
> and return them as a single Collection... obviously sucking lots of memory.
>
> I'm testing more of a Streaming approach (processing results as they come
> in rather than collecting them in a collection) as well as holding the
> remaining, necessary, large structures in btrees so they can be on disk.
>
> A couple questions:
> 1) Does the Loader or other parts of grouper do anything with 3-500k'ish
> lists of identities with less memory impact? Any tricks?
>
> 2) Does GrouperCache support hybrid (some memory, mostly disk) caching? I
> can see that ehCache seems to have max-memory and max-disk options, but
> didn't see if GrouperCache supported them.
>
> 3) If I find btrees particularly useful for either short- or long-term
> tasks, most btree libraries (like btree4j) seem to not be in maven central.
> One (MapDB ) is in maven central... How would I add a non-maven-central or
> a maven-central jar to PSPNG's dependencies?
>
>
>
> Thanks,
> Bert




Archive powered by MHonArc 2.6.19.

Top of Page