Grouper Users - Open Discussion List

[Kathryn Huxtable <>]

I should add that it looks as if I'm able to add or remove one subject per
50 seconds. That's averaged over about 100,000 of them.

-K


On 8/13/06 9:06 AM, "Kathryn Huxtable" 
<>
 wrote:

> Sure. First of all, I miscounted. There were 50,000 subjects. There were
> 240,000 memberships, all JNDI subjects added as members to 638 groups.
> 
> I generated the XML import file using a Perl script that ran against my
> Oracle database in about 100 seconds.
> 
> I'm afraid I don't really know the hardware specs for my test server, but
> it's the same as my production server. It's a Dell Poweredge running RedHat
> Linux. I'm using Java1.5.
> 
> The subject connector is JNDI running against an extremely oversized Solaris
> box running SunJava 5.2. Would JDBC against Oracle be faster? I can do that,
> but I'll have to worry about FERPA suppressing the name and description, so
> I'll have to make my own JDBC connector to add some code. I get that for
> free with my ACIs in LDAP.
> 
> The grouper database is Oracle 9i running on some sort of SunFire box.
> 
> I made the mistake of running this in a shell inside XEmacs, so I had to
> leave my laptop on for all that time.
> 
> Anyway, does adding 240,000 memberships take 47.5 hours normally?
> 
> I've rewritten _processMembershipLists in XmlImporter so that instead of
> removing the old memberships and adding the new ones, it gets the old
> membership IDs into a HashSet, does the same with the new membership IDs and
> then only removes or adds those that aren't present in both. This does
> updates faster, since from day to day most of these groups' memberships
> won't change that much (except for three times per year at semester breaks).
> 
> But that was after running this.
> 
> -K
> 
> 
> On 8/12/06 8:56 AM, "Tom Barton" 
> <>
>  wrote:
> 
>> Can you supply further details of the operation, the execution
>> environment, and exactly how it was conducted? That does sound absurdly
>> slow, and far off from my own experience.
>> 
>> Tom
>> 
>> Kathryn Huxtable wrote:
>>> I just added external provisioning for all the base groups we might want 
>>> to
>>> use in composite operations. It took almost 48 hours. This is ridiculous. 
>>> My
>>> home-grown group management can populate from empty in about four hours. 
>>> My
>>> nightly update only does adds/removes and so runs very quickly.
>>> 
>>> I can obviously write similar update code, but really, what all is going 
>>> on
>>> that adding 20,000 subjects takes so long? Is a table not indexed on some
>>> attribute, or are you creating new subject records from the JNDI data in 
>>> my
>>> subject configs?
>>> 
>>> Inquiring minds want to know (fnord).
>>> 
>>> -K
>>> 
>>> 
>>> On 8/4/06 3:43 PM, "Tom Barton" 
>>> <>
>>>  wrote:
>>> 
>>>> You should be able to use the XML import/export tool for this purpose.
>>>> See 
>>>> 
> 
<https://wiki.internet2.edu/confluence/display/GrouperWG/Import-Export+v1.0>>>>
>
> .
>>>> 
>>>> <hint>
>>>> It'd be good to include an example of how to "prune & graft", like
>>>> you're wanting to do, in that wiki page.
>>>> </hint>
>>>> 
>>>> Tom
>>>> 
>>>> Will Norris wrote:
>>>>> can groups or stems be moved to somewhere else in the group hierarchy?
>>>>> Looking at the database I can't think of any technical reason why this
>>>>> wouldn't be possible, unless I'm overlooking something.
>>>>> 
>>>>> -will
>>> 
>