Skip to Content.
Sympa Menu

grouper-users - Re: Re: [grouper-users] Loading Module Information into Grouper usingTalend

Subject: Grouper Users - Open Discussion List

List archive

Re: Re: [grouper-users] Loading Module Information into Grouper usingTalend


Chronological Thread 
  • From:
  • To:
  • Subject: Re: Re: [grouper-users] Loading Module Information into Grouper usingTalend
  • Date: Mon, 19 Oct 2009 11:57:48 -0400 (EDT)

Thanks for your questions.

The normal run of the job on a nightly basis, according to Talend's stat
catching, is around 73 seconds. When we first ran the script to load all
modules and enrolments it did take up to an hour to run, this was creating
around 4500 modules, with around 90,000 enrolments. We do try and have
updates to grouper running sequentially and try to avoid having them overlap
if possible.

For the "true" responses, I think this would be quite useful for
auditing/logging purposes in certain circumstances. I think this would be
even more beneficial when a command fails, for example if we load in a file
which is adding 1000 new memberships, we direct the error output to an error
log, however the error output isn't always usable. In some circumstances, for
example when an attempt to add a member to a group that doesn't exists, the
error is output to the error file, displaying why it fails but you cannot
determine the user it was trying to add,
e.g. // Error: Cannot find group with name:
'CorporateData:Modules:EEE:EEE2009'
It would be useful in these cases for the command to be linked to the error.

In this job we are using SSH to fire off the GSH, so if there was a failure
within one of the scripts, this would be dealt with on the server side, by us
directing errors to our error logs as discussed above. We could then
configure Talend to monitor these logs and orchestrate any necessary actions
if errors were occurring. However if there was a problem with starting the
GSH environment up, or the Talend job was unable to connect to the Grouper
server for any reason, then Talend can catch these errors and handle these
errors with the required response.

Regarding web services, we have successfully connected through to Grouper's
web services, and we are able to insert/update/retrieve data in grouper. We
did attempt to use them for the export we require for the room booking
service, but found that it didn’t provide the most efficient way of getting
the data out, and we therefore are accessing the database directly. I think
this is more the case to do with the amount of data that we need to process
for the export rather than the web services themselves, I will continue to
have a look at this and report back if we get it working more effectively.

Many Thanks

Richard


  • Re: Re: [grouper-users] Loading Module Information into Grouper usingTalend, richard . james, 10/19/2009

Archive powered by MHonArc 2.6.16.

Top of Page