Grouper Users - Open Discussion List

[Chris Hyzer <>]

 

> 1) At the time of loading the data for the first time, if I break the dataset to

> three chunks and load the first chunk, when I start to load the second chunk,

> grouper starts to delete the first chuck (members of those groups) before loading

> the second one, even though I set loader.sqlTable.likeString.removeGroupIfNotUsed = false 

> in grouper-loader.properties file and also set

> grouperLoaderGroupsLike to root:nameOfStem1:nameOfStem2:%_systemOfRecord in loader group

> attributes. What setting should I use to force grouper not to delete other groups while

> updating with a new sql query? (I guess that's what you call orphant groups)

> 

 

I think if you don’t set the likeString, then it shouldn’t affect other groups.  Though how would orphans be deleted then?  Are you sure you cants set a like string for the batch of groups or just keep it as 1 large batch?

 

 

> 2) When I have a group loaded to grouper, if I set loader group's

> grouperLoaderQuartzCron attribute to one minute for testing, any

> time the loader runs it deletes most of the group members and add

> them again although no changes was done to the group. I said most

> of them because I noticed for a group with 55 members, it deleted

> 19 of them and then added the same 19 again back. And it keeps

> doing this every minute although no changes was done to the group

> through grouper UI or to the source view that the data is being

> read from! Shouldn't the correct behavior be like if the group is

> loaded and there's no changes to the members of it on the source

> view side, no changes should happen on grouper side either? Why

> does it keep deleting the group members? This can be very problematic

> in our case with some of our groups having two or three hundred

> thousands of members and if they get deleted to be re-populated on

> nightly syncs, the sync will never finish!

 

Yes, it should definitely not do that!  J  Hmmm… are you using 2.1.4?  Is the query for the subjects by subjectId or subjectIdentifier?  subjectId is better, though it should work with either.  Do you have a simple example that I could reproduce?  Are you a Java person and you can debug this and see why?  J Can you set logging to debug and let me know the output?

 

log4j.logger.edu.internet2.middleware.grouper.app.loader = DEBUG

 

Thanks,

Chris

 

From: Omaraie, Brad [mailto:]
Sent: Thursday, June 20, 2013 8:54 PM
To: Chris Hyzer
Subject: Grouper loader sync problem

 

Hi Chris,

We are getting to the point that we want to go to production with our Grouper installation. For that to happen we need to load three set of large data to grouper and we want to have nightly sql loader sync with our data sources. We are using change log consumers for provisioning to ldap and that seems to be working fine. But when I try to load and and sync our data, I have the following problems:

 

1) At the time of loading the data for the first time, if I break the dataset to three chunks and load the first chunk, when I start to load the second chunk, grouper starts to delete the first chuck (members of those groups) before loading the second one, even though I set loader.sqlTable.likeString.removeGroupIfNotUsed = false  in grouper-loader.properties file and also set grouperLoaderGroupsLike to root:nameOfStem1:nameOfStem2:%_systemOfRecord in loader group attributes. What setting should I use to force grouper not to delete other groups while updating with a new sql query? (I guess that's what you call orphant groups)

 

2) When I have a group loaded to grouper, if I set loader group's grouperLoaderQuartzCron attribute to one minute for testing, any time the loader runs it deletes most of the group members and add them again although no changes was done to the group. I said most of them because I noticed for a group with 55 members, it deleted 19 of them and then added the same 19 again back. And it keeps doing this every minute although no changes was done to the group through grouper UI or to the source view that the data is being read from! Shouldn't the correct behavior be like if the group is loaded and there's no changes to the members of it on the source view side, no changes should happen on grouper side either? Why does it keep deleting the group members? This can be very problematic in our case with some of our groups having two or three hundred thousands of members and if they get deleted to be re-populated on nightly syncs, the sync will never finish!

 

I know these might be repeated questions, but I looked everywhere in documentation and couldn't find a good solution. I really appreciate your help.

 

Thanks,

Brad