Grouper Users - Open Discussion List

["Gettes, Michael" <>]

Okay, I yield.  I honed in on Carey’s note where he mentioned the ETL and I got stuck on how to reliably kick off the loader job but only at the right time.  So yes, we need some way to kick off a loader job.  Barf to firing up GSH just to kick the loader.
  I could see otherJob being used to look for some state some where (a scratch in a table or a file timestamp being updated) to indicate it’s okay to run a loaderJob corresponding to the state test.  A single otherJob with a map some place of stateTest to loaderJob
 might do it.  Something like this could tie into grouper diagnostics nicely for monitoring status of these loaderJob kickers.  But, once again, I will yield to your good judgment Mr. Hyzer.

Carey, I am looking at Talend.  It’s a beast.  I’ll start small with it.  Fortunately, I have some groovy clue now due to the grouper graphing exercise (no chris, not raw java :0) so I believe I can make use of this.  Jim, yes, the loaderJob is appealing because of the scale of what it can do.  Yes, there’s web services but I don’t believe it will scale like a loaderJob.  If I was just updating a few groups/objects, WS for sure.  But when differencing hundreds of thousands or millions of memberships across 10s of thousands of groups, the loader really is the better way to go.  On my laptop I run a loaderJob for 850K memberships across 2500 groups in about 8-10 minutes (the new subject caching adjustments in 2.4 are key and will be better when Mr. Hyzer addresses the filesystem storage issue :-) ).

Thank you all for the conversation and the hints.  Especially, Carey - THANK YOU!  (Note: I respectfully don’t agree with your assessment on the complexity of rest and the integration issues you cited but I do appreciate the ETL approach and will go that route).

Good nite all.

/mrg

On Feb 13, 2019, at 11:52 PM, Hyzer, Chris <> wrote:

Yes Michael I will give you a free java lesson…

 

Seriously though, I would prefer not to start another process / executable.  Web service?  Perhaps…

 

You could send a message that replaces the members of a group with the members in the message 😊  WS does that too

 

The ETL way with sql is what we do at penn for this… or cron a process that updates a table with data, loader takes it from there.  Maybe we should support a message to kick off a full loader job?

 

Thanks

Chris

 

 

 

From:  <> On Behalf Of Greg Haverkamp
Sent: Wednesday, February 13, 2019 11:19 PM
To: Gettes, Michael <>
Cc: 
Subject: Re: [grouper-users] Grouper Loader - of another type

 

Got it.

 

Yes.  Shortly after I replied, it dawned on me that one would have to recreate all of the comparison and lookup logic that the loader already takes care of.  And while it looks like it _might_ be possible to wedge yourself into the middle of all of that, then you'd be coding to what I imagine are moderately "private" API's.

 

One way to do this would be to have the external program be callable via JDBC or LDAP.  To be honest, I don't know what the former would entail, because I've never written a server receiving JDBC calls.  I've written a number of LDAP proxies, and it wouldn't be too bad, and there are lots of options for LDAP server implementations, but the LDAP loader options seem a little more limited.  (Even OpenLDAP with its sock or shell backend  would give some options.)

 

But I do understand that if there's a way to make something like this generally available, that's probably better.

 

Greg

 

On Wed, Feb 13, 2019 at 7:33 PM Gettes, Michael <> wrote:

I thought this was an interesting idea, thanks Greg.  From my short investigation into otherJob it appears it is a way to hook the scheduling capabilities of grouper (quartz) with running grouper code/functions which in itself is kinda cool.  But, it’s not evident to me how this would achieve what I am asking.  I want to run a loader job.  I just don’t want to be limited by SQL/LDAP.  I guess what I am seeking is the essential components of the SQL loader job modified into 3 components:

 

1) Run an external program to produce the dataset associated with the grouperLoaderQuery phase of processing in a SQL loader job. Most likely the value of this attribute is the full filesystem pathname to the program to execute.  The output (maybe stored in a filename passed to the program as a parameter on execution) would be the same output as defined by the grouperLoaderQuery SQL job attribute.  The first line of output would be tab-delimited “column names” and the remaining lines would be tab-delimited “columns” in the same order to be consumed by the grouper loader processor.

 

2) Run an external program to produce the dataset associated with the the grouperLoaderGroupQuery phase of processing in a SQL loader job. Most likely the value of this attribute is the full filesystem pathname to the program to execute.  The output (maybe stored in a filename passed to the program as a parameter on execution) would be the same output as defined by the grouperLoaderGroupQuery SQL job attribute.  The first line of output would be tab-delimited “column names” and the remaining lines would be tab-delimited “columns” in the same order to be consumed by the grouper loader processor.

 

3) with all the other possible parameters of a loader SQL type job, process the resulting datasets from 1 and 2 as usual.

 

I am not seeing how to influence/alter a loader job to behave in this fashion without our fearless grouper devs providing an extended capability of the loaderJob itself.  Doing this would seem to provide infinite extensibility to the essential grouper loader function.

 

I hope this helps.

 

/mrg

On Feb 13, 2019, at 9:02 PM, Greg Haverkamp <> wrote:

 

Currently, grouper supports loader jobs of LDAP and SQL and an additional capability to inject messages to process changes related to an individual - a way of sparking loader jobs for one person instead of in bulk - at least this is my interpretation.

 

Perhaps I don't understand the nuances of what you're looking for, but could you use the otherJob construct for this?  Primarily what you want is something running as Grouper in the JVM with the daemon, right?

 

Or it's possible that I'm overestimating what's possible with an otherJob, and there's a limitation I don't know of.

 

Greg

 

On Wed, Feb 13, 2019 at 2:38 PM Gettes, Michael <> wrote:

Hi all,

Currently, grouper supports loader jobs of LDAP and SQL and an additional capability to inject messages to process changes related to an individual - a way of sparking loader jobs for one person instead of in bulk - at least this is my interpretation.

I have a need for loader jobs to be of an arbitrary nature - call a program, written in any language, which might do REST calls or whatever and return, in bulk, something similar to what the loader job now receives from SQL/LDAP.  This way I can go against alternative sources without the need of staging the data into LDAP/SQL but get all the benefits and scale of a grouper loader job.

Does anyone else see a need for this?  Grouper dev dudes… (and dudettes)… have you considered this?  I can only assume you have since you guys have thought of a great many things for grouper.

Many thanks for your time and consideration especially if you choose to respond.

/mrg