Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] Loading Module Information into Grouper using Talend

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] Loading Module Information into Grouper using Talend

Chronological Thread 
  • From: Tom Barton <>
  • To:
  • Cc:
  • Subject: Re: [grouper-users] Loading Module Information into Grouper using Talend
  • Date: Mon, 19 Oct 2009 09:13:58 -0500

Very cool!

A couple of questions, if you don't mind. Do you know how long the whole process takes for your production environment? Do you perform other updates to grouper over the same time period, or do you try to sequence one update process at a time?

Seeing those "True" responses from gsh made me wonder if that's satisfactory, or if you'd rather see a reflection of the operation that was successful.

If you plant a gsh operation that will fail into the process, how does the system respond, ie, does gsh + talend work well in that scenario?

Can talend use web services directly to reflect changes?


Hi Everyone,

Inspired by Chris’ email regarding the management of class lists
using the grouper loader I thought I would share what we have been
doing at Newcastle University to provision students module enrolments
into Grouper.

Module information, including course details, and student enrolments
for Newcastle University are held in a database which the Faculty of
Medical Sciences (FMSC) holds the responsibility for. We would
normally take advantage of the grouper loader, however due to factors
including database permissions and some required restructuring of the
data we have not been able to this in this scenario. We find the
loader to be a very useful feature, and as Grouper has access to the
corporate data for the University’s departmental structure, we are
able to use the Grouper Loader to create our Organisational
structure, which we find very beneficial to provide an up to date
representation of the organisation. This would also be our desired
route for the provisioning of module data into grouper, and as the
IDMAPs project continues to develop, we hope to be able implement
this in the future.

As a consequence of not being able to use the loader, we needed a
method of extracting the appropriate information from the database
and then feed this into Grouper, the method to do this has been
created using the Open Source data integration suite, Talend. As part
of a JISC funded project, IDMAPS, we have recently started to using
Talend to try and address the problems posed by data flows which are
inconsistent, badly documented and hard to maintain.

With Talend, we have been able to produce a job which allows us to
extract the data from the FMSC’s database, manipulate the data into
the required format, and feed this data into Grouper using the
Grouper Shell, also allowing us to keep an audit trail of any changes
made to modules and module enrolments. To demonstrate how we have
achieved this we have created some videos which show the process of
extracting the data from the source location and feeding this into
Grouper, the videos can be found at This is the first time
we have created videos for this purpose, and we have encountered some
problems with the videos being temperamental at times, we have tried
to overcome this by splitting the video into a number of parts.

Talend alone is a very useful tool for the creation and
maintainability of the numerous data feeds throughout an
organisation, in the short time of using it we have already seen how
it can improve the current flows of data throughout the University.
It is provides a quick and efficient way of producing data feeds to
meet a systems required data rules, and allows you to easily augment
data into one overall data flow. This was recently required to
provision role based access control from Grouper into the Scientia
Syllabus Plus Room Booking service, for this we needed to bring
together data from Grouper and various SAP data sources. We are
currently in the process of producing a video to demonstrate how we
do this, and will share this in the near future.

I hope the videos are of some interest,


Richard James ISS Middleware Team Newcastle University

Archive powered by MHonArc 2.6.16.

Top of Page