Grouper Users - Open Discussion List

[Kathryn Huxtable <>]

Thanks for the suggestion. I don't think this is memory related, but rather
is related to inefficient JDBC queries from Hibernate. I've gotten an
enormous speed improvement by writing an HQL join query between membership
and member and only selecting the subject elements that would be present in
the input.

I'll post this code in the next several days. I've generally lobotomized the
XmlImporter in my own class and need to move my relatively small number of
changes back into the original source.

BTW, I use Hibernate3 normally. Are you folks planning to move to it at some
point?

-K


On 8/14/06 11:24 AM, "Monica Crawford" 
<>
 wrote:

> Hi Kathryn,
> 
> Was the initial/max heap explicitly stated for the JVM when the script
> was kicked off?  How many MB is the xml file being processed?
> 
> Setting the heap explicitly may help a bit, it looks like a dom parser
> is used and the time spent processing larger files can significantly
> increase with the size of the file.  Doing that may shave down the time
> down, however there still could be additional bottlenecks on the db side.
> 
> Monica
> Kathryn Huxtable wrote:
>> Sure. First of all, I miscounted. There were 50,000 subjects. There were
>> 240,000 memberships, all JNDI subjects added as members to 638 groups.
>> 
>> I generated the XML import file using a Perl script that ran against my
>> Oracle database in about 100 seconds.
>> 
>> I'm afraid I don't really know the hardware specs for my test server, but
>> it's the same as my production server. It's a Dell Poweredge running RedHat
>> Linux. I'm using Java1.5.
>> 
>> The subject connector is JNDI running against an extremely oversized 
>> Solaris
>> box running SunJava 5.2. Would JDBC against Oracle be faster? I can do 
>> that,
>> but I'll have to worry about FERPA suppressing the name and description, so
>> I'll have to make my own JDBC connector to add some code. I get that for
>> free with my ACIs in LDAP.
>> 
>> The grouper database is Oracle 9i running on some sort of SunFire box.
>> 
>> I made the mistake of running this in a shell inside XEmacs, so I had to
>> leave my laptop on for all that time.
>> 
>> Anyway, does adding 240,000 memberships take 47.5 hours normally?
>> 
>> I've rewritten _processMembershipLists in XmlImporter so that instead of
>> removing the old memberships and adding the new ones, it gets the old
>> membership IDs into a HashSet, does the same with the new membership IDs 
>> and
>> then only removes or adds those that aren't present in both. This does
>> updates faster, since from day to day most of these groups' memberships
>> won't change that much (except for three times per year at semester 
>> breaks).
>> 
>> But that was after running this.
>> 
>> -K
>> 
>> 
>> On 8/12/06 8:56 AM, "Tom Barton" 
>> <>
>>  wrote:
>> 
>>> Can you supply further details of the operation, the execution
>>> environment, and exactly how it was conducted? That does sound absurdly
>>> slow, and far off from my own experience.
>>> 
>>> Tom
>>> 
>>> Kathryn Huxtable wrote:
>>>> I just added external provisioning for all the base groups we might want 
>>>> to
>>>> use in composite operations. It took almost 48 hours. This is ridiculous.
>>>> My
>>>> home-grown group management can populate from empty in about four hours. 
>>>> My
>>>> nightly update only does adds/removes and so runs very quickly.
>>>> 
>>>> I can obviously write similar update code, but really, what all is going 
>>>> on
>>>> that adding 20,000 subjects takes so long? Is a table not indexed on some
>>>> attribute, or are you creating new subject records from the JNDI data in 
>>>> my
>>>> subject configs?
>>>> 
>>>> Inquiring minds want to know (fnord).
>>>> 
>>>> -K
>>>> 
>>>> 
>>>> On 8/4/06 3:43 PM, "Tom Barton" 
>>>> <>
>>>>  wrote:
>>>> 
>>>>> You should be able to use the XML import/export tool for this purpose.
>>>>> See 
>>>>> 
>> <https://wiki.internet2.edu/confluence/display/GrouperWG/Import-Export+v1.0>>
>> >>
>> .
>>>>> <hint>
>>>>> It'd be good to include an example of how to "prune & graft", like
>>>>> you're wanting to do, in that wiki page.
>>>>> </hint>
>>>>> 
>>>>> Tom
>>>>> 
>>>>> Will Norris wrote:
>>>>>> can groups or stems be moved to somewhere else in the group hierarchy?
>>>>>> Looking at the database I can't think of any technical reason why this
>>>>>> wouldn't be possible, unless I'm overlooking something.
>>>>>> 
>>>>>> -will
>>