Grouper Developers Forum

["Bee-Lindgren, Bert" <>]

Thank you. The small amount of ldap data you swag just doesn’t seem to mesh 
with actual memory usage. I’m inclined to believe your estimates are right 
and there is huge data-structure bloat. 

I’ll try to do some logging and analysis for how much memory is being used 
when compared to the ldif text (what I’ll call the true size). 



> On Jun 4, 2019, at 2:01 PM, Gettes, Michael <> wrote:
> 
> hi Bert - I replied to your private Slack query about all this as well.  If 
> my estimates on the size of a cached entry are correct (I supposed 
> 600bytes) then 500K (which is what I am wanting to use) would be only 300MB 
> of cache.  If my math is right - then I don’t believe putting this on disk 
> is worth it - just keep it all in memory.  If you do this kind of caching 
> for each LDAP server defined for Grouper - and 10 LDAP servers - then it 
> would be 3GB of caching.  I would still say to keep this in memory.
> 
> math and opinions are my own.
> 
> /mrg
> 
>> On Jun 4, 2019, at 1:22 PM, Bee-Lindgren, Bert 
>> <> wrote:
>> 
>> PSPNG takes a Collections-intensive approach to processing identities. In 
>> other words, there are methods to fetch hundreds of thousands of objects 
>> and return them as a single Collection... obviously sucking lots of memory.
>> 
>> I'm testing more of a Streaming approach (processing results as they come 
>> in rather than collecting them in a collection) as well as holding the 
>> remaining, necessary, large structures in btrees so they can be on disk.
>> 
>> A couple questions:
>> 1) Does the Loader or other parts of grouper do anything with 3-500k'ish 
>> lists of identities with less memory impact? Any tricks?
>> 
>> 2) Does GrouperCache support hybrid (some memory, mostly disk) caching? I 
>> can see that ehCache seems to have max-memory and max-disk options, but 
>> didn't see if GrouperCache supported them.
>> 
>> 3) If I find btrees particularly useful for either short- or long-term 
>> tasks, most btree libraries (like btree4j) seem to not be in maven 
>> central. One (MapDB ) is in maven central... How would I add a 
>> non-maven-central or a maven-central jar to PSPNG's dependencies?
>> 
>> 
>> 
>> Thanks,
>>  Bert
>