Grouper Users - Open Discussion List

[Jeffrey Williams <>]

Those look like PSPNG giving up on trying to sync a group(most likely since it's locked by another process.  Maybe the manual sync?).  There's a couple of things you can do:

• Update the record in GROUPER_CHANGE_LOG_CONSUMER corresponding to the provisioner in question to have the LAST_SEQUENCE_PROCESSED to be the max value in the table.  This will skip the remaining changes and make the provisioner ready for any future changes to come in.  No restart is necessary.  You will miss any incrementals between where the provisioner was and the present, but those should get effectively done during the next full-sync.  Pros: Get the provisioner back to handle new incrementals.  Cons: some incrementals get missed and will get trued up by the full sync.
• You can wait and let it process the jobs in order and get caught up.  You can optionally configure a lower timeout value using  coordinationTimeout_secs and restart the daemons if you want to hurry them along.  Pros: You'll catch any unrelated incrementals on the way and not disrupt the service.  Cons, it may take a while..

On Tue, Mar 10, 2020 at 12:25 PM Weston, Todd <> wrote:

OK - created a setenv.sh file in the $GROUPER_HOME/bin directory and set the -Xms and -Xmx values. Restarted the grouper service and ran a full manual sync in GSH. All groups populated in AD! Thank you all for your assistance. No more Java GC errors...but...

Now I'm getting a couple of groups (originally populated fine) in the error log that won't process. Error messages:

2020-03-10 09:11:16,438: [DefaultQuartzScheduler_Worker-4] ERROR ProvisioningWorkItem.setStatus(137) -  - Work item not handled; ProvisioningWorkItem[done=true,successful=false,msg=FullSync timed out after 300 seconds,clog=clog #9475230 / ChangeLog type: group: updateGroup,group=wsu.edu:bundle:AD:role.groups:students:Students.Admitted.Undergraduate]

2020-03-10 09:14:16,317: [DefaultQuartzScheduler_Worker-6] ERROR ProvisioningWorkItem.setStatus(137) -  - Work item not handled; ProvisioningWorkItem[done=true,successful=false,msg=FullSync timed out after 300 seconds,clog=clog #9475185 / ChangeLog type: group: updateGroup,group=wsu.edu:bundle:AD:role.groups:employees:Employees.Active.Hourly]

2020-03-10 09:16:16,504: [DefaultQuartzScheduler_Worker-4] ERROR ProvisioningWorkItem.setStatus(137) -  - Work item not handled; ProvisioningWorkItem[done=true,successful=false,msg=FullSync timed out after 300 seconds,clog=clog #9475231 / ChangeLog type: group: updateGroup,group=wsu.edu:bundle:AD:role.groups:students:Students.Admitted.Undergraduate]

2020-03-10 09:19:16,395: [DefaultQuartzScheduler_Worker-6] ERROR ProvisioningWorkItem.setStatus(137) -  - Work item not handled; ProvisioningWorkItem[done=true,successful=false,msg=FullSync timed out after 300 seconds,clog=clog #9475186 / ChangeLog type: group: updateGroup,group=wsu.edu:bundle:AD:role.groups:employees:Employees.Active.Hourly]

I've restarted the service and rebooted the server - these keep coming up like they are leftovers from the last full sync. Is there a way to kill them or ?

Todd Weston

Washington State University

---

From: <> on behalf of Weston, Todd <>
Sent: Monday, March 9, 2020 9:03 AM
To: <>
Subject: Re: [grouper-users] PSPNG mostly working

 

I'm certain that the server host has that memory - as for the JVM: is this a catalina.sh setting? The only related entry in that file is: JAVA_OPTS="-server -Xmx512M -XX:MaxPermSize=256M"

I'm somewhat familiar with the -Xms and -Xmx tags, but haven't see the -XX:MaxPermSize before. It seems to be in conflict with the -Xmx setting. I'm not an experienced Linux/Apache/Tomcat admin, so advice on settings would be appreciated.

Todd Weston

WSU

---

From: Hyzer, Chris <>
Sent: Monday, March 9, 2020 8:38 AM
To: Weston, Todd <>; <>
Subject: RE: PSPNG mostly working

 

37k is not a lot of users.  That means you need more memory, but 16gb is way more than enough.  Are you sure the JVM has that much memory available?  And the host has that much memory?

 

From: On Behalf Of Weston, Todd
Sent: Monday, March 09, 2020 11:35 AM
To:
Subject: Re: [grouper-users] PSPNG mostly working

 

Thanks for the direction - that got all but one group provisioning. The one in question has the largest population (~37K user subjects) and is coughing up this error:

 

2020-03-09 08:26:21,596: [FullSyncer(pspng_ad_role_students)-Thread] ERROR FullSyncProvisioner.fullSyncGroup(674) -  - FullSyncer(pspng_ad_role_students): Problem doing full sync. Requeuing group wsu.edu:bundle:AD:role.groups:students:Students.Enrolled.Undergraduate
java.lang.OutOfMemoryError: GC overhead limit exceeded

 

I've looked for the shell commands that launch java instances, but found too many to feel confident in just adjusting any of them. Any clues as to which one should be targeted? Or am I barking up the wrong tree? The server originally had 8GB RAM and was crushing the ceiling. I increased the allocation to 16GB and now it is running at about 10GB consumed.

 

I appreciate any additional direction -

 

Todd Weston 

Washington State University

 

---

From: <> on behalf of Black, Carey M. <>
Sent: Tuesday, March 3, 2020 10:14 AM
To: Weston, Todd <>; <>
Subject: RE: [grouper-users] PSPNG mostly working

 

Todd,

 

I would suggest you try to manually trigger a full sync and watch what happens with respect to those groups. ( Maybe an AD access restriction? Just a guess. )

https://spaces.at.internet2.edu/display/Grouper/Grouper+Provisioning%3A+PSPNG#GrouperProvisioning:PSPNG-FullSYNCProvisioning

 

 

NOTE: You might also want to turn up the log level too. See the section a bit farther down the page “Adjusting log level.” And you can set it in GSH before you manually start the full sync.

 

--

Carey Matthew

 

From: <> On Behalf Of Weston, Todd
Sent: Tuesday, March 3, 2020 11:34 AM
To:
Subject: [grouper-users] PSPNG mostly working

 

So – I have figured out the majority of provisioning workflow into AD – but have 5 groups that will not provision users into AD. My pspng attribute assignments have been made at the folder level and we have three different pspng configs to push groups into different OUs:

The PSPNG attribute assignments are applied at the affiliates, employees and students folders. All but two of the groups in employees is provisioning users into the groups:

These two groups show membership in the Grouper database of 2063 and 1736, respectively. But their AD counterparts are empty. A similar issue presents in our student groups:

These groups show memberships in the database of 37,543, 1341 and 16,055, respectively, yet AD groups are empty.

All of the remaining groups in both folders have populated their AD groups with no issues and maintain their dynamic membership on a daily basis.

 

I’m pretty certain the PSPNG config in the grouper-loader.properties is configured correctly as these other groups are provisioning properly and update regularly. And they are in the same folders as the ones that are not working. There are roll-up groups in the above output that we will populate with the child groups eventually, so most of the zero-membership groups in the list aren’t concerning – just the highlighted ones that should be getting memberships.

 

I have not gone as far as deleting the AD groups and allowing Grouper to recreate them as this is not a desirable method once we go into production. We which to retain the SIDs of these existing groups to make the transition from our current group populator to grouper as seamless as possible. We did empty all of these groups prior to turning on provisioning settings in Grouper – and it’s mostly worked…

 

One other note: we were running into memory issues originally (Java heap full in the logs) – went from 8GB to 16GB on the app server. The level memory consumption on the server is now close to 10GB, so I don’t believe we’re memory constrained any longer  and I don’t see the Java memory errors.

 

Any Ideas? I can post filtered logs fi I know what to look for (The grouper_error.log is incredibly noisy).

 

-

Todd Weston Information Security Analyst, Identity | Information Technology Services | Washington State University Phone 509.335.4479 PO Box 641222 Pullman, WA  99164-1222

 

 

--

Jeffrey Williams 

Identity Engineer
Identity & Access Services
https://its.uncg.edu