Grouper Developers Forum

[Chris Hyzer <>]

> It could well do, although I haven't yet digested it all. Perhaps you
> could add short summary at the beginning of the document in which you
> describe the general approach. For example I hadn't been aware that
> there was a loader pre-run hook.

I added the pre-hook for the loader work I did for generic org management 
last month.  I sent an email and wiki about it, but not a Jira, I just added 
one:

https://bugs.internet2.edu/jira/browse/GRP-279
https://wiki.internet2.edu/confluence/display/GrouperWG/Organization+hierarchies

I added a summary of the Penn org implementation.  Let me know if you need 
more info:

General approach

    * We have a view of orgs and parent id's, and a view of assignments of 
org to person

    * Since we want the orgs in a tree structure in Grouper, and the loader 
expects the group_name matched with person id, we use the built-in grouper 
hook on pre-run of loader job, to take the org table and calculate the 
hierarchical group names.  You could do this part a variety of ways, 
including a db function (e.g. PLSQL for oracle)

    * There are a few loader jobs:

          o One that manages all the leaf node orgs groups (assigns people to 
org groups)

          o One that manages all the non-leaf rollup nodes (assigns orgs to 
parent org groups)

          o One that manages Penn "centers" which are our high level 
organizations (assigns direct center children orgs to center groups)

          o One that manages contractor groups.  We dont have contractors 
assigned in our org structure, so we manage them through attributes in our 
person database.  Then we can manually make other groups next to the org 
rollup which includes the org rollup and the contractor group

    * We will run all these jobs once daily after our payroll jobs run

    * Grouper include/exclude automatically creates include and exclude lists 
which are tacked on to the system of record group.  This is useful in loader 
jobs since the loader system of record list is resynced with the DB query 
with each run, so manual changes will be undone.  Penn actually only needs 
"include" lists, not exclude at this point (use case is VP's are in a 
different org than the org they manage, so we want to add them in).  So we 
will have include/exclude groups on the rollup groups, the centers groups, 
and the contractor groups.  We dont think we will need them on the leaf 
nodes, though if we have a need later, we can add it in.  The reason not to 
do this is performance, it creates a bunch more groups and memberships.

Here are the steps to creating a loader SQL_GROUP_LIST job:

    * First of all, you should prefer using views, and simply put simple 
select from the view in the loader config.  It is also easy to tell what the 
loader is going to do without hunting through the loader configs, and easy to 
make changes at runtime (though I believe the loader allows runtime query 
changes as well)

    * Note: if you are using include/exclude, then the group names should 
have the system of record suffix which is configured in the grouper.properties

    * Make a view of groups where each row represents a group.

          o There is a col for the group name, display name (optional), 
description (optional), and security (e.g. readers, viewers, etc: optional)

          o This view will set these attributes of group and auto-create 
groups which have no members (might be useful for orgs since apps can refer 
to groups which have no members)

          o Note: if you can give groups a unique suffix in the stem 
structure, then the job can use the setting "grouperLoaderGroupsLike" which 
will delete groups which are no longer in the group list

    * Make the query which returns the subjectId (and sourceId if not the 
default loader source), and group name

          o For leaf nodes, this is generally a simple query that assigns 
people to org groups

          o For rollup nodes, this can be a little complex.

                + First of all, you might union the direct rollup children 
with the direct rollup leaf nodes

                + The subjectId for groups is the group_id.  for Grouper 1.4, 
you can join to the grouper_attributes table.  For 1.5 you can simply join to 
the grouper_groups table.  In both, you can join to grouper_groups_v if you 
like.  What I did for 1.4 is: grouper_attributes ga, grouper_fields gf where 
gf.NAME = 'name' AND gf.ID = ga.field_id and ga.VALUE = 
ocrv.MEMBER_GROUP_NAME.  I would definitely keep these query in a view.

                + You should also specify the source id for groups: 'g:gsa' 
as SUBJECT_SOURCE_ID

    * Configure the config group for each loader job.  I put this next to the 
top level loaded stem.  I generally do this in GSH, though you could also do 
this in the UI

    * Kick off the loader job manually in GSH so you can verify the results 
without waiting for the cron to run

    * Restart your loader so it picks up the new job


> -----Original Message-----
> From: Loris Bennett 
> [mailto:]
> Sent: Friday, May 22, 2009 4:31 AM
> To: Chris Hyzer
> Cc: Grouper Users Mailing List; 
> 
> Subject: Re: [grouper-users] Penn's organizational hierarchy
> 
> On Thu, 2009-05-21 at 15:37 -0400, Chris Hyzer wrote:
> > Hey,
> >
> > I implemented the Grouper org hook at Penn.  I have permission to
> share
> >  my experience outside of Penn so I made a document here:
> >
> >
> https://wiki.internet2.edu/confluence/display/GrouperWG/Penn+organizati
> onal+hierarchy
> >
> > This is the size of our org implementation:
> >
> >  27,000 people in orgs at Penn
> >   2,200 orgs
> >   3,000 org groups (more due to include/exclude lists)
> > 500,000 org memberships (there are a lot due to the rollups, and
> include/exclude lists)
> >
> > Note that the bulk of the work was in figuring out how our org data
> is
> >  structured, and how I can expose it in views to something that the
> >  loader can process.  If you want to get this working at your
> >  institution this document should be helpful.
> >
> > Btw, Loris, does this help with your org issues?
> 
> It could well do, although I haven't yet digested it all. Perhaps you
> could add short summary at the beginning of the document in which you
> describe the general approach. For example I hadn't been aware that
> there was a loader pre-run hook.
> 
> Cheers
> 
> Loris
> 
> 
> 
> 
> 
> > Thanks,
> > Chris
> --
> Dr. Loris Bennett (Mr.)
> Freie Universität Berlin
> ZEDAT - Zentraleinrichtung für Datenverarbeitung / Computer Center
> Compute & Media Service
> Fabeckstr. 32, Room 221
> D-14195 Berlin
> Tel ++49 30 838 51024
> Fax ++49 30 838 56721
> Email 
> 
> Web www.zedat.fu-berlin.de