Grouper Users - Open Discussion List

[Chris Hyzer <>]

This is done.  And this is my last coding required for 1.4.  :)  After 1.4 
RC1, I will use a lot of these features at Penn, and make a wiki document 
about it, so you can see an example of how it works.

https://bugs.internet2.edu/jira/browse/GRP-149
https://wiki.internet2.edu/confluence/display/GrouperWG/Grouper+-+Loader

1. You dont have to add the type and attributes in GSH anymore, it is 
automatic if you set the grouper-loader.properties setting:

loader.autoadd.typesAttributes = true

Make sure if you are running the loader now, if you dont set that to true, 
add the two new attributes:

subj=SubjectFinder.findById("GrouperSystem")
sess=GrouperSession.start(subj)
type=GroupTypeFinder.find("grouperLoader")
type.addAttribute(sess, "grouperLoaderGroupTypes", read, admin, false)
type.addAttribute(sess, "grouperLoaderGroupsLike", read, admin, false)

2. grouperLoaderGroupTypes attribute: whatever you put in the value should be 
comma separated GroupTypes which will be applied to the loaded groups.  The 
reason this enhancement exists is so we can do a SQL_GROUP_LIST query and 
attach addIncludeExclude to the groups.  Note, if you do this (or use some 
requireGroups), the group name in the loader query should end in the system 
of record suffix, which by default is _systemOfRecord.

3. grouperLoaderGroupsLike attribute: this should be a sql like string (e.g. 
school:orgs:%org%_systemOfRecord), and the loader should be able to query 
group names to see which names are managed by this loader job.  So if a group 
falls off the loader resultset (or is moved), this will help the loader 
remove the members from this group.  Note, if the group is used anywhere as a 
member or composite member, it wont be removed.  All 
include/exclude/requireGroups will be removed.  Though the two groups, 
include and exclude, will not be removed if they have members.  There is a 
grouper-loader.properties setting to note remove loader groups if empty and 
not used:


#if using a sql table, and specifying the name like string, then should the 
group (in addition to memberships)
# be removed if not used anywhere else?
loader.sqlTable.likeString.removeGroupIfNotUsed = true

4. added sensible defaults to loader settings.
   - If there is a query, and it has "group_name" before "from", then 
GrouperLoadertype
   defaults to SQL_GROUP_LIST else defaults to SQL_SIMPLE
   - grouperLoaderQuery defaults to "grouper" for sql type loaders
   - grouperLoaderScheduleType defaults to CRON if there is a
   grouperLoaderCronString, else defaults to START_TO_START_INTERVAL
   - grouperLoaderIntervalSeconds defaults to 86400 (1 day)
   if grouperLoaderScheduleType is not CRON

The bottom line is, its still probably a good idea to be explicit, but if you 
just fill in a query, it will run daily...

5. added gsh command to run one loader job.  This is useful to run once at 
the beginning, and not have to wait for the schedule.  Or to troubleshoot e.g.

        loaderGroup = 
GroupFinder.findByName(GrouperSession.startRootSession(), 
"school:orgs:orgGroup");
        loaderRunOneJob(loaderGroup);
        loaderRunOneJob("MAINTENANCE_cleanLogs");


6. added debug log messages, and info log messages (less frequent).  to see 
these, set log level in log4j.properties

## Log debug info on loader to see progress etc
log4j.logger.edu.internet2.middleware.grouper.app.loader = DEBUG
-or-
log4j.logger.edu.internet2.middleware.grouper.app.loader = INFO

7. added periodic DB updates on the grouper_loader_log record which shows the 
job running and making progress

8. added loader transaction switch in properties file 
(loader.use.transactions), since
loading large groups uses a lot of memory.  This defaults to not use 
transactions (the more progress we can make, the better...)

9. added type security which defaults to only wheel members being able to add 
grouperLoader type or edit grouperLoader attributes


> -----Original Message-----
> From: Loris Bennett 
> [mailto:]
> Sent: Tuesday, November 11, 2008 9:39 AM
> To: Chris Hyzer
> Cc: Grouper Users Mailing List
> Subject: RE: [grouper-users] Grouper-WS: Strategies for moving
> groups/stems?
>
> Would it be possible to make the removal configurable? It has occurred
> to me that there could be a situation in which a group is only
> temporarily empty and is still needed. Implementing a mechanism for
> defining which groups should be deleted and which shouldn't might be a
> bit complicated, but it would be handy if the deletion could at least
> be
> turned on and off globally.
>
> Thanks
>
> Loris
>
> On Mon, 2008-11-10 at 09:43 -0500, Chris Hyzer wrote:
> > I was planning to do this (probably today) for 1.4...
> >
> > > -----Original Message-----
> > > From: Loris Bennett 
> > > [mailto:]
> > > Sent: Monday, November 10, 2008 4:38 AM
> > > To: Chris Hyzer; Grouper Users Mailing List
> > > Subject: RE: [grouper-users] Grouper-WS: Strategies for moving
> > > groups/stems?
> > >
> > > Could that be included in 1.5?
> > >
> > > Until it is available I shall implement a solution using getGroups.
> > >
> > > Thanks
> > >
> > > Loris
> > >
> > > On Fri, 2008-11-07 at 09:46 -0500, Chris Hyzer wrote:
> > > > I could do that in the loader... if a group is not used anywhere,
> > > remove it.
> > > > Also, there is a web service to list the groups an entity is the
> > > member of: getGroups:
> > > >
> > > >
> https://wiki.internet2.edu/confluence/display/GrouperWG/Get+Groups
> > > >
> > > > Chris
> > > >
> > > > > -----Original Message-----
> > > > > From: Dr. Loris Bennett 
> > > > > [mailto:]
> > > > > Sent: Friday, November 07, 2008 3:25 AM
> > > > > To: Chris Hyzer
> > > > > Subject: RE: [grouper-users] Grouper-WS: Strategies for moving
> > > > > groups/stems?
> > > > >
> > > > > Hi Chris,
> > > > >
> > > > > I have been talking to some colleagues about our needs for
> removing
> > > > > members from groups which no longer appear in the query.
> > > > >
> > > > > Am I correct in thinking that a missing group would have all
> its
> > > > > members
> > > > > removed but the group itself would not be deleted?
> > > > >
> > > > > In this case we would then need to implement some mechanism to
> > > check
> > > > > whether the empty group is used in any other groups before we
> could
> > > > > actually remove it.
> > > > >
> > > > > Looking briefly at the web service I couldn't see a method
> which
> > > would
> > > > > give me the groups an entity is a member of, so I assume I
> would
> > > have
> > > > > to
> > > > > go through all the groups and check if the empty group is used
> > > > > anywhere.
> > > > > Or am I missing something?
> > > > >
> > > > > Regards
> > > > >
> > > > > Loris
> > > > >
> > > > > PS I noticed the release notes for Grouper 1.4 were updated
> > > recently.
> > > > > Is
> > > > > the actual release date imminent?
> > > > >
> > > > >
> > > > >
> > > > > On Fri, 2008-10-17 at 05:07 -0400, Chris Hyzer wrote:
> > > > > > Ok, so the two requested changes to the loader are:
> > > > > >
> > > > > > 1. Dry run mode where it just print out what it will do, but
> > > doesn't
> > > > > actually do it
> > > > > > 2. Attribute which has the SQL "like" pattern of the group
> names
> > > that
> > > > > loader job is responsible for, so it can remove memberships
> from
> > > groups
> > > > > not listed in the query
> > > > > >
> > > > > > Are you also interested in the idea of auto-creating the
> > > composite
> > > > > groups, perhaps with a built-in group type?  And the loader
> having
> > > a
> > > > > way to associate a created group with that type, which would
> auto-
> > > > > create those composites?
> > > > > >
> > > > > > As far as timeline, these changes should be there in the next
> > > release
> > > > > which is the end of the month.  And if not, then shortly
> > > thereafter.
> > > > > >
> > > > > > Also we discussed the group detail being created by web
> service,
> > > > > which you said was not as important.
> > > > > >
> > > > > > Regards,
> > > > > > Chris
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Loris Bennett 
> > > > > > > [mailto:-
> berlin.de]
> > > > > > > Sent: Friday, October 17, 2008 4:40 AM
> > > > > > > To: Chris Hyzer
> > > > > > > Subject: RE: [grouper-users] Grouper-WS: Strategies for
> moving
> > > > > > > groups/stems?
> > > > > > >
> > > > > > > On Thu, 2008-10-16 at 11:59 -0400, Chris Hyzer wrote:
> > > > > > > > >
> > > > > > > > > Deleting empty groups my hand would be OK for us. We
> are
> > > > > already a
> > > > > > > > > little chicken about the idea of updating our basic org
> > > groups
> > > > > 100%
> > > > > > > > > automatically and hoping that nothing breaks in the
> > > composite
> > > > > > > groups.
> > > > > > > > > Ideally we would like to generate some sort of summary
> of
> > > the
> > > > > > > changes
> > > > > > > > > before these are actually propagated. This would give
> us an
> > > > > > > overview of
> > > > > > > > > the changes occurring and enable us to anticipate the
> > > effects
> > > > > on
> > > > > > > > > derived
> > > > > > > > > groups.
> > > > > > > >
> > > > > > > > Are you saying that all grouper loader changes should
> > > generate a
> > > > > > > summary, or only when you delete groups?
> > > > > > >
> > > > > > > At least initially, we would like a summary of all group
> > > additions
> > > > > and
> > > > > > > deletions in our org data. However, I see this as
> essentially
> > > > > > > independent of grouper-loader, although I suppose it could
> be
> > > built
> > > > > > > into
> > > > > > > grouper-loader as a dry run mode. If deletion were by hand,
> the
> > > > > summary
> > > > > > > would also serve to keep tabs on which groups were in fact
> > > empty
> > > > > and
> > > > > > > needed to be removed. But as I said, I don't think a
> summary
> > > > > feature is
> > > > > > > necessarily something grouper-loader should have.
> > > > > >
> > > > > --
> > > > > Dr. Loris Bennett (Mr.)
> > > > > Freie Universität Berlin
> > > > > Zentraleinrichtung für Datenverarbeitung / Computer Center
> > > > > ZEDAT
> > > > > Compute & Media Service
> > > > > Fabeckstr. 32, Room 221
> > > > > D-14195 Berlin
> > > > > Tel ++49 30 838 51024
> > > > > Fax ++49 30 838 56721
> > > > > Email 
> > > > > 
> > > > > Web www.zedat.fu-berlin.de
> > > > >
> > >
> >