Grouper Users - Open Discussion List

["Omaraie, Brad" <>]

Hi Chris,

Unfortunately we are under a very tight schedule right now and can 't wait for the release of 2.1.5. We ended up dividing our data set to even smaller chunks so we can handle the data in memory. I agree with you regarding no need for reading all the subject's data into memory at first. My suggestion is may be instead of bring in all the subjects for each group into memory before starting to load that group into grouper registry,  we can do it through a loop with certain number of subjects for each iteration. Let's say if we can set a value of 10000 for number of subjects read each time in properties file, then for each iteration we get only 10000 subjects from source and commit them to grouper DB and go back and get another 10000 till all the members of that group are committed. This way we can handle the big loads with much smaller heap size.

I'll keep you posted on how the load goes.

Thanks,

Brad

From: "" <>
Date: Friday, August 16, 2013 9:51 AM
To: ucla <>
Cc: "" <>
Subject: RE: GC overhead limit exceeded error

Brad, I put this in the 2.1.5 candidate:

 

https://bugs.internet2.edu/jira/browse/GRP-929

 

It should reduce the memory required for the loader.

 

If you can apply this new code, or wait a few days for 2.1.5 to be released and upgrade your loader, I would be interested in hearing feedback about the improvement.

 

Thanks.

Chris

 

Ps. note, there is an option to revert to the old algorithm in the grouper-loader.properties…

 

 

 

 

From: Chris Hyzer
Sent: Friday, August 16, 2013 1:06 AM
To: Omaraie, Brad
Cc:
Subject: RE: GC overhead limit exceeded error

 

Can you switch to a 64 bit linux machine?  :)

 

http://stackoverflow.com/questions/2457514/understanding-max-jvm-heap-size-32bit-vs-64bit

 

I noticed that we are loading the entire member objects (which are getting large), instead of just the subject_id and source_id (which is all that is generally used).  We could have an option where it retrieves less data if you like...

 

(from GrouperLoaderType.java)

 

        Set<Member> members = group[0].getImmediateMembers();

        for (Member member : GrouperUtil.nonNull(members)) {

          currentMembers.add(new LoaderMemberWrapper(member));

        }

      }

      

      //now lets remove data from each since the member is there and is supposed to be there

      Iterator<LoaderMemberWrapper> iterator = currentMembers.iterator();

      

      int count = 0;

      

      while (iterator.hasNext()) {

        

        LoaderMemberWrapper member = iterator.next();

        //see if it is in the current list

        Row row = grouperLoaderResultset.find(member.getSubjectId(), member.getSourceId());

 

 

---

From: Omaraie, Brad []
Sent: Thursday, August 15, 2013 7:07 PM
To: Chris Hyzer
Subject: GC overhead limit exceeded error

Hi Chris,

I was using a windows 7 64bit machine with 4G ram for testing the load to our grouper QA database before and load was successful. I was setting the maximum heap size to 3072m. I noticed when loading some of our bigger groups, the heap size would go up to 2.8g. Now I want to load the production data to our grouper prod database and we're using a 32 bit Linux machine with 3g of ram. The GSH session did not start on this machine when I was using 3072m as maximum size of heap. When I get a top from that machine, I see only 800k used memory and 2.2g free. But when I tried different maximum heap sizes,  GSH session finally started with 1024m max heap size although there's more free memory available! 

 

We started the load and now we are getting to some of those bigger groups. Looks like grouper tries to load all the member subject's for each group into memory first and then start the load to the DB. That's when it hits the 1024m size and crashes. I was wondering if you know of any setting to force grouper bring in smaller chunks of each group members into memory (not the entire group which in this case can go up to 450k member) before entering them to DB? Or maybe any suggestion on how we can increase the max heap size on Linux systems so the GSH session starts with higher max heap size?

 

Thanks

Brad