Grouper Developers Forum

[Tom Zeller <>]

I think 9.0.4, the latest from MacPorts, as of last week.

On Oct 1, 2011, at 1:11 PM, Shilen Patel 
<>
 wrote:

> Actually, never mind on that.. :)  I wasn't running my test properly.
> 
> -- Shilen
> 
> On 10/1/11 2:08 PM, "Shilen Patel" 
> <>
>  wrote:
> 
>> Tom,
>> 
>> What version of postgres are you running?  I noticed the problem with
>> 8.4.7.  I just upgraded to the latest 8.4 release (8.4.9) and the test now
>> runs in about 4 seconds for me.  Can you try 8.4.9 or 9.0.5 or 9.1.1,
>> which were all released this week?
>> 
>> Thanks!
>> 
>> -- Shilen
>> 
>> 
>> On 10/1/11 9:57 AM, "Shilen Patel" 
>> <>
>>  wrote:
>> 
>>> I can reproduce this problem in both 2.0 and trunk.  For me, using a
>>> local
>>> mysql database takes about 4 seconds, but using a local postgres database
>>> takes over 70 seconds.  Most of the time is spent processing the temp
>>> change log.  After doing a bit of profiling, it seems the queries to
>>> determine flattened permission changes are not performing well in
>>> postgres.  Those queries will be changing for 2.1 or 2.2 anyways since
>>> they don't currently handle deny permissions.  You can avoid the issue
>>> right now by setting changeLog.includeFlattenedPermissions to false in
>>> grouper-loader.properties and ignore the tests for flattened permissions.
>>> I'll let you know if I find a better workaround for the near term.
>>> 
>>> Thanks!
>>> 
>>> -- Shilen
>>> 
>>> 
>>> On 9/30/11 3:59 PM, "Chris Hyzer" 
>>> <>
>>>  wrote:
>>> 
>>>> Can you put a handful of tests in there so we can see the breakdown
>>>> between startup time, and test time.
>>>> 
>>>> Thanks,
>>>> Chris
>>>> 
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Tom
>>>> Zeller
>>>> Sent: Friday, September 30, 2011 3:57 PM
>>>> To: Shilen Patel
>>>> Cc: Grouper Dev
>>>> Subject: Re: [grouper-dev] help with writing changeLog tests ?
>>>> 
>>>> Right.
>>>> 
>>>> The following is a test I just cobbled together. The changelog entries
>>>> are from grouper initialization.
>>>> 
>>>> postgres : Time: 70.81
>>>> hsql: Time: 4.415
>>>> 
>>>> So postgres needed over a minute and hsql a few seconds.
>>>> 
>>>> 
>>>> package edu.internet2.middleware.grouper.pit;
>>>> 
>>>> import junit.textui.TestRunner;
>>>> 
>>>> import org.apache.commons.lang.time.StopWatch;
>>>> import org.slf4j.Logger;
>>>> import org.slf4j.LoggerFactory;
>>>> 
>>>> import edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity;
>>>> import edu.internet2.middleware.grouper.helper.GrouperTest;
>>>> 
>>>> public class TomTest extends GrouperTest {
>>>> 
>>>> protected static final Logger LOG =
>>>> LoggerFactory.getLogger(TomTest.class);
>>>> 
>>>> public void testOne() {
>>>>   LOG.info("starting");
>>>>   StopWatch sw = new StopWatch();
>>>>   sw.start();
>>>>   ChangeLogTempToEntity.convertRecords();
>>>>   sw.stop();
>>>>   LOG.info(sw.toString());
>>>> }
>>>> }
>>>> 
>>>> On Fri, Sep 30, 2011 at 1:52 PM, Shilen Patel 
>>>> <>
>>>>  wrote:
>>>>> The time it takes to run that depends on how many changes you have in
>>>>> your
>>>>> temp change log among other things.  Is there a particular junit test
>>>>> that
>>>>> is having this slow response time when using postgres?  If so, I can
>>>>> try
>>>>> to reproduce it and take a look.
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> -- Shilen
>>>>> 
>>>>> On 9/30/11 2:22 PM, "Tom Zeller" 
>>>>> <>
>>>>>  wrote:
>>>>> 
>>>>>> Thanks.
>>>>>> 
>>>>>> I found that in GrouperLoaderType, in the CHANGE_LOG enum, the
>>>>>> following
>>>>>> 
>>>>>> 826 ChangeLogTempToEntity.convertRecords(hib3GrouploaderLog);
>>>>>> 
>>>>>> seems slow (10s per test) using postgres, it is faster with hsql.
>>>>>> 
>>>>>> I guess it is the PIT saveOrUpdate() stuff.
>>>>>> 
>>>>>> This may be of concern, perhaps the postgres ddl needs tweaking, I
>>>>>> have no idea how :-)
>>>>>> 
>>>>>> On Fri, Sep 30, 2011 at 11:06 AM, Chris Hyzer 
>>>>>> <>
>>>>>> wrote:
>>>>>>> I think you can make it think it is actually running to be a more
>>>>>>> realistic test... i.e. configure it for tests in the grouper-loader
>>>>>>> testConfig in a startUp() [and remove in tearDown()]:
>>>>>>> 
>>>>>>> 
>>>>>>> GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.class
>>>>>>> "
>>>>>>> ,
>>>>>>> LdappcngConsumer.class.getName());
>>>>>>> 
>>>>>>> GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.quart
>>>>>>> z
>>>>>>> C
>>>>>>> ro
>>>>>>> n", "0 0 8 * * ?");
>>>>>>> 
>>>>>>> Then run it to register the consumer and clear out the change log:
>>>>>>> 
>>>>>>>   GrouperLoader.runOnceByJobName(grouperSession,
>>>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>>>>   GrouperLoader.runOnceByJobName(grouperSession,
>>>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>>> 
>>>>>>> Do stuff:
>>>>>>> 
>>>>>>> Call those again and it will process whatever you did
>>>>>>> 
>>>>>>>   GrouperLoader.runOnceByJobName(grouperSession,
>>>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>>>>   GrouperLoader.runOnceByJobName(grouperSession,
>>>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>>> 
>>>>>>> Ok?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Chris
>>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: 
>>>>>>> 
>>>>>>>  
>>>>>>> [mailto:]
>>>>>>>  On Behalf Of
>>>>>>> Tom
>>>>>>> Zeller
>>>>>>> Sent: Friday, September 30, 2011 10:31 AM
>>>>>>> To: Grouper Dev
>>>>>>> Subject: [grouper-dev] help with writing changeLog tests ?
>>>>>>> 
>>>>>>> I am starting to write tests for the ldappcng changelog consumer. Is
>>>>>>> this the correct way to write changelog tests ?
>>>>>>> 
>>>>>>> public void testSomething() {
>>>>>>> 
>>>>>>> ldappcngConsumer.runChangeLog();
>>>>>>> 
>>>>>>> ChangeLogTempToEntity.convertRecords();
>>>>>>> 
>>>>>>> // clear changelog
>>>>>>> HibernateSession.byHqlStatic().createQuery("delete from
>>>>>>> ChangeLogEntryTemp").executeUpdate();
>>>>>>> HibernateSession.byHqlStatic().createQuery("delete from
>>>>>>> ChangeLogEntryEntity").executeUpdate();
>>>>>>> 
>>>>>>> // do something e.g. group.addMember();
>>>>>>> 
>>>>>>> ChangeLogTempToEntity.convertRecords();
>>>>>>> 
>>>>>>> ldappcngConsumer.runChangeLog();
>>>>>>> 
>>>>>>> // verify test
>>>>>>> }
>>>>>>> 
>>>>>>> And here is the runChangeLog() method :
>>>>>>> 
>>>>>>> public void runChangeLog() {
>>>>>>>   LOG.info("runChangeLog");
>>>>>>>   Hib3GrouperLoaderLog hib3GrouploaderLog = new
>>>>>>> Hib3GrouperLoaderLog();
>>>>>>>   hib3GrouploaderLog.setHost(GrouperUtil.hostname());
>>>>>>>   hib3GrouploaderLog.setJobName(CHANGELOG_JOB_NAME);
>>>>>>>   hib3GrouploaderLog.setStatus(GrouperLoaderStatus.RUNNING.name());
>>>>>>>   hib3GrouploaderLog.store();
>>>>>>> 
>>>>>>>   try {
>>>>>>>     ChangeLogHelper.processRecords(CHANGELOG_JOB_NAME,
>>>>>>> hib3GrouploaderLog, this);
>>>>>>> 
>>>>>>> hib3GrouploaderLog.setStatus(GrouperLoaderStatus.SUCCESS.name());
>>>>>>>   } catch (Exception e) {
>>>>>>>     LOG.error("Error processing records", e);
>>>>>>>     hib3GrouploaderLog.setStatus(GrouperLoaderStatus.ERROR.name());
>>>>>>>   }
>>>>>>>   hib3GrouploaderLog.store();
>>>>>>> }
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 
>