Grouper Users - Open Discussion List

["Hyzer, Chris" <>]

I implemented an export to gsh.

 

https://spaces.internet2.edu/display/Grouper/Grouper+export+to+a+GSH+script

 

This is in the latest patch for v2.2.2.  Note, a lot of classes were changed though with this new feature core Grouper code did not get touched so it is still low risk.  I was hoping this enhancement would be smaller and affect fewer files which is why I started working in the v2.2 branch, but it got more and more complex.

 

Grouper can export a folder or the entire registry as a GSH script.   This is currently experimental, take a backup or verify in a test env first.  In order to run the script you need a Grouper v2.2.2+ patched env.  Note, you should capture the output of running the GSH script so you can review it for errors.

Performance on demo server (local mysql database)

Exporting 6000 records to GSH takes 26 seconds

The GSH script ran 6000 records 1min 6sec with no changes

If there were 6000 changes (new registry) it took 6 minutes.

The script is idempotent (run it twice, its ok).  It will not delete what is there in general though it will delete attribute assignments on exported assignments.  It is assumed that the general configuration of the source environment is equivalent to the destination environment.  e.g. if name is not a required external subject attribute in the source env, then it shouldnt be in the destination either.

 

It will not export

audits

point in time data

daemon logs

change log entries

if you are importing and the objects already exist, it will not sync up the object ID uuid's.  In some cases will be use different uuid even if the object doesnt exist

membership fields are not imported (note, privileges are, but if you are using a membership list that is not "members" (not common), that will be skipped

if you export a folder, and some necessary objects do not exist in the target (e.g. attribute definitions), they will be reported as an error and skipped

note: international characters have been tested on the demo server and works there though your mileage may vary, be careful here (test it)

unresolvable subjects will be exported but will throw an error on import (maybe they shouldnt be exported)

 

Start an export

 

grouperSession = GrouperSession.startRootSession();

scriptName = "c:/temp/script.gsh";

new File(scriptName).delete();

XmlExportGshScript().assignStemName(":").assignFileNameToWriteTo(scriptName).exportGsh();

The end of the script will print a summary of changes that occurred:

Script complete: total objects: 6222, expected approx total: 6327, changes: 133, known errors (view output for full list): 36

 

 

Thanks,

Chris

 

From: Sean Mason [mailto:]
Sent: Wednesday, December 16, 2015 4:16 PM
To: Hyzer, Chris <>; Julio Polo <>
Cc:
Subject: RE: [grouper-users] Grouper Import-Export and managing objects

 

Bah, right at the top too!  Sorry about that.

This looks like a fine method to manage and migrate stem structure and loader definitions from one instance to another, unless someone else chimes in with a more accepted practice.

 

Thanks again,

Sean.

 

From: Chris Hyzer []
Sent: Wednesday, December 16, 2015 3:59 PM
To: Sean Mason <>; Julio Polo <>
Cc:
Subject: RE: [grouper-users] Grouper Import-Export and managing objects

 

Yes:

 

 

gsh filename

 

https://spaces.internet2.edu/pages/viewpage.action?pageId=14517859

 

 

Read gsh commands from a script file:

$GROUPER_HOME/bin/gsh.sh /path/to/your/script.gsh

 

 

Thanks,

Chris

 

 

From: Sean Mason []
Sent: Wednesday, December 16, 2015 3:56 PM
To: Chris Hyzer; Julio Polo
Cc:
Subject: RE: [grouper-users] Grouper Import-Export and managing objects

 

Is there a way to feed gsh a list of commands similar to something like the Postgres client?

Something like gsh < commands.txt

Or

gsh% executeCommands(“/some/dir/gsh_commands.txt”);

 

I could use this as a way to build group definitions and assign attributes that would become the loader definitions, and then just run those command collections on the various instances.  I don’t see any examples or documentation like this, but maybe I’ve missed something.  I have seen similar examples in the grouperClient documentation about testing that suggests perhaps that may be the way to go?

 

Thanks,
Sean.

 

 

From: Chris Hyzer []
Sent: Wednesday, December 16, 2015 3:41 PM
To: Julio Polo <>; Sean Mason <>
Cc:
Subject: RE: [grouper-users] Grouper Import-Export and managing objects

 

He wants something to copy the loader settings which are attributes of groups…

 

Thanks,

Chris

 

 

From: Julio Polo []
Sent: Wednesday, December 16, 2015 3:25 PM
To: Sean Mason
Cc: Chris Hyzer;
Subject: Re: [grouper-users] Grouper Import-Export and managing objects

 

I am not familiar with loader groups, so maybe someone else can chime in.  Maybe the Grouper loader feature can take care of creating the stems and groups you want. Instead of writing gsh scripts, you might just need to promote new loader settings to the other Grouper environments.  Again, I'm not familiar with the loader, so this is just speculation on my part.

-julio

 

On Wed, Dec 16, 2015 at 4:25 AM, Sean Mason <> wrote:

I’ve replied to ‘all’, as some posts to the list haven’t gotten through over the past couple of days.

 

Thank you for your feedback.  It was not my intention to move memberships or subject references between instances, only stems and loader groups.  It makes sense that one would create processes that create and maintain those structures instead to prepare target environments.  What is the most popular and practical way to develop those processes?  GSH scripts?  (Can GSH be passed a script?) Use of the Grouper Client?  Some other means?

 

Thank you,

Sean.

 

 

From: Julio Polo [mailto:]
Sent: Tuesday, December 15, 2015 6:50 PM
To: Chris Hyzer <>
Cc: Sean Mason <>;

Subject: Re: [grouper-users] Grouper Import-Export and managing objects

 

Thanks Chris, but it was indeed the permissions and attributes that were difficult to export/import.  The other stuff was relatively easy to code.

I think it would definitely be helpful to provide an option to import/export only the objects that fall under a parent folder, and to use group ID path and subject ID/Identifier instead of UUIDs (maybe export them as wsSubjectLookup and wsGroupLookup?).   Another useful option would be to assume that we do not want to create the subjects or groups that are members of the groups being imported/exported; assume they exist if they are not under that parent folder.

Going back to Sean's original question on how to perform this type of object management, I can only say that we do not move data across environments.  We develop subsystems for each type of group (e.g. registration groups, curriculum groups, faculty groups, etc.) and we promote the subsytem code rather than the data through dev, test and production.  The groups/objects are then created by the code in each environment.    If we were to move objects (such as memberships), we'd need to make sure that data across dev, test and prod be in sync, and that's not usually true (for dev, at least)

-julio

 

 

On Tue, Dec 15, 2015 at 11:33 AM, Chris Hyzer <> wrote:

Thanks Julio.  Yes, the XML export inport isn’t really for partial transfers.  What do you want to transfer?  Folders, groups, memberships, privileges?  Based on a parent folder?  Is it ok if create dates, last modify dates, member uuids, etc don’t match?   I could try to make something that generates some GSH for you…  if you need attributes, permissions, rules, etc, that is more complicated…

 

Thanks,

Chris

 

From: [mailto:] On Behalf Of Julio Polo
Sent: Tuesday, December 15, 2015 4:08 PM
To: Sean Mason
Cc:
Subject: Re: [grouper-users] Grouper Import-Export and managing objects

 

I suspect that you will spend a lot of time trying to get that import/export tool to do what you want.  I don't think it was designed for partial transfers.

I recently spent a lot of time trying to preserve one stem when migrating from one Grouper instance to another, and I ended up writing code to migrate *most* of what I needed, manually recreating certain things (fortunately my stem only had a handful of these things), and living with certain values being changed (original creator, timestamps, losing all audit logs).

In the end, the Grouper import/export tool was mostly used to verify that I didn't miss anything during the transfer.  I had to write a utility to de-normalize the data in the XML and change uuid values into group names and subject IDs. That made it easier for me to spot check that I didn't miss transferring anything important.

Some things I learned about the Grouper import/export tool as I used it:

gsh  -xmlexport  -excludeAudits  -stems some:small:stem  someOutputFile.xml

-excludeAudits was necessary; otherwise I would wait forever (days?) for the export to complete, even when exporting a stem with little data.  Not sure why you got audit data if you specified -excludeAudits

 

It doesn't look like folder privileges (stem, create) are exported (unless that's buried somewhere in <memberships>)

All subjects are exported, regardless of whether the subject is a member of the stem I wanted.  For us, that meant our entire population in LDAP was exported.  Not sure if that's the case when you specify -objectNames.

I tried modifying the exported XML to skip the subjects/memberships, and import only the folders, groups, attributes, but ran into some error during the gsh -xmlimport of this modified XML.  Maybe I broke the integrity of the exported XML when I did that.

Hopefully a future Grouper upgrade can better support partial exports/imports.   Best of luck.

 

-julio

Julio Polo
Enterprise Middleware, Identity and Access Management
Information Technology Services
University of Hawaii

 

 

 

On Mon, Dec 14, 2015 at 4:39 AM, Sean Mason <> wrote:

Hi There,

I am looking at introducing Grouper to provide some access management services for the campus, and am currently stuck trying to figure out how to move stem structures and group definition objects between instances to manage the development lifecycle of the product.  This will be a new implementation, using the latest released version of Grouper (2.2.2).  I had sent this question to the group early last week, but it seems to have been lost in the shuffle or perhaps I'm just too impatient.

I am attempting to build a process that would allow migration of new stem structures and preconfigured groups (like loader jobs) from development, to test, to production.  It would also be nice to be able to maintain an external repository of these types of objects so that new environments may be built in a more automated way.  It seemed to me the Import-Export functionality would be a good candidate for this, but I am either doing something wrong, or it does not operate as I current understand.

An example: gsh -xmlexport -excludeAudits -objectNames etc:LoaderJobs:SomeLoaderJob ~/SomeLoaderJob.xml The object in question is a group definition with attribute values to configure a (successful) SQL_GROUP_LIST loader job with no direct members.

My understanding of the documentation on https://spaces.internet2.edu/display/Grouper/Import-Export made me believe that the result would be an XML representation of the supplied group.  Instead the result is the export of a very large number (reportedly 600,000+, though it "may be less") of objects that includes thousands of memberships, audit events, and perhaps more since I haven't let a process complete through the extraction of audit events.

Have I misunderstood what the xmlexport utility is for and how it works, or perhaps I am doing something wrong?  The inclusion of audit events alone makes me believe I've missed something, since I thought I was explicitly avoiding the extraction of that type of object with the -excludeAudits option.

If this is not the utility to perform such tasks, does the community have suggestions on how this type of object management might be performed?

Thank you,
Sean