Grouper Developers Forum

[Shilen Patel <>]

Tom,

What version of postgres are you running?  I noticed the problem with
8.4.7.  I just upgraded to the latest 8.4 release (8.4.9) and the test now
runs in about 4 seconds for me.  Can you try 8.4.9 or 9.0.5 or 9.1.1,
which were all released this week?

Thanks!

-- Shilen


On 10/1/11 9:57 AM, "Shilen Patel" 
<>
 wrote:

>I can reproduce this problem in both 2.0 and trunk.  For me, using a local
>mysql database takes about 4 seconds, but using a local postgres database
>takes over 70 seconds.  Most of the time is spent processing the temp
>change log.  After doing a bit of profiling, it seems the queries to
>determine flattened permission changes are not performing well in
>postgres.  Those queries will be changing for 2.1 or 2.2 anyways since
>they don't currently handle deny permissions.  You can avoid the issue
>right now by setting changeLog.includeFlattenedPermissions to false in
>grouper-loader.properties and ignore the tests for flattened permissions.
>I'll let you know if I find a better workaround for the near term.
>
>Thanks!
>
>-- Shilen
>
>
>On 9/30/11 3:59 PM, "Chris Hyzer" 
><>
> wrote:
>
>>Can you put a handful of tests in there so we can see the breakdown
>>between startup time, and test time.
>>
>>Thanks,
>>Chris
>>
>>-----Original Message-----
>>From: 
>>
>> 
>>[mailto:]
>> On Behalf Of Tom
>>Zeller
>>Sent: Friday, September 30, 2011 3:57 PM
>>To: Shilen Patel
>>Cc: Grouper Dev
>>Subject: Re: [grouper-dev] help with writing changeLog tests ?
>>
>>Right.
>>
>>The following is a test I just cobbled together. The changelog entries
>>are from grouper initialization.
>>
>> postgres : Time: 70.81
>> hsql: Time: 4.415
>>
>>So postgres needed over a minute and hsql a few seconds.
>>
>>
>>package edu.internet2.middleware.grouper.pit;
>>
>>import junit.textui.TestRunner;
>>
>>import org.apache.commons.lang.time.StopWatch;
>>import org.slf4j.Logger;
>>import org.slf4j.LoggerFactory;
>>
>>import edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity;
>>import edu.internet2.middleware.grouper.helper.GrouperTest;
>>
>>public class TomTest extends GrouperTest {
>>
>>  protected static final Logger LOG =
>>LoggerFactory.getLogger(TomTest.class);
>>
>>  public void testOne() {
>>    LOG.info("starting");
>>    StopWatch sw = new StopWatch();
>>    sw.start();
>>    ChangeLogTempToEntity.convertRecords();
>>    sw.stop();
>>    LOG.info(sw.toString());
>>  }
>>}
>>
>>On Fri, Sep 30, 2011 at 1:52 PM, Shilen Patel 
>><>
>> wrote:
>>> The time it takes to run that depends on how many changes you have in
>>>your
>>> temp change log among other things.  Is there a particular junit test
>>>that
>>> is having this slow response time when using postgres?  If so, I can
>>>try
>>> to reproduce it and take a look.
>>>
>>> Thanks!
>>>
>>> -- Shilen
>>>
>>> On 9/30/11 2:22 PM, "Tom Zeller" 
>>> <>
>>>  wrote:
>>>
>>>>Thanks.
>>>>
>>>>I found that in GrouperLoaderType, in the CHANGE_LOG enum, the
>>>>following
>>>>
>>>> 826 ChangeLogTempToEntity.convertRecords(hib3GrouploaderLog);
>>>>
>>>>seems slow (10s per test) using postgres, it is faster with hsql.
>>>>
>>>>I guess it is the PIT saveOrUpdate() stuff.
>>>>
>>>>This may be of concern, perhaps the postgres ddl needs tweaking, I
>>>>have no idea how :-)
>>>>
>>>>On Fri, Sep 30, 2011 at 11:06 AM, Chris Hyzer 
>>>><>
>>>>wrote:
>>>>> I think you can make it think it is actually running to be a more
>>>>>realistic test... i.e. configure it for tests in the grouper-loader
>>>>>testConfig in a startUp() [and remove in tearDown()]:
>>>>>
>>>>>
>>>>>GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.class"
>>>>>,
>>>>>LdappcngConsumer.class.getName());
>>>>>
>>>>>GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.quartz
>>>>>C
>>>>>ro
>>>>>n", "0 0 8 * * ?");
>>>>>
>>>>> Then run it to register the consumer and clear out the change log:
>>>>>
>>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>
>>>>> Do stuff:
>>>>>
>>>>> Call those again and it will process whatever you did
>>>>>
>>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>
>>>>> Ok?
>>>>>
>>>>> Thanks,
>>>>> Chris
>>>>>
>>>>> -----Original Message-----
>>>>> From: 
>>>>> 
>>>>>  
>>>>> [mailto:]
>>>>>  On Behalf Of Tom
>>>>>Zeller
>>>>> Sent: Friday, September 30, 2011 10:31 AM
>>>>> To: Grouper Dev
>>>>> Subject: [grouper-dev] help with writing changeLog tests ?
>>>>>
>>>>> I am starting to write tests for the ldappcng changelog consumer. Is
>>>>> this the correct way to write changelog tests ?
>>>>>
>>>>> public void testSomething() {
>>>>>
>>>>>  ldappcngConsumer.runChangeLog();
>>>>>
>>>>>  ChangeLogTempToEntity.convertRecords();
>>>>>
>>>>>  // clear changelog
>>>>>  HibernateSession.byHqlStatic().createQuery("delete from
>>>>> ChangeLogEntryTemp").executeUpdate();
>>>>>  HibernateSession.byHqlStatic().createQuery("delete from
>>>>> ChangeLogEntryEntity").executeUpdate();
>>>>>
>>>>>  // do something e.g. group.addMember();
>>>>>
>>>>>  ChangeLogTempToEntity.convertRecords();
>>>>>
>>>>>  ldappcngConsumer.runChangeLog();
>>>>>
>>>>>  // verify test
>>>>> }
>>>>>
>>>>> And here is the runChangeLog() method :
>>>>>
>>>>> public void runChangeLog() {
>>>>>    LOG.info("runChangeLog");
>>>>>    Hib3GrouperLoaderLog hib3GrouploaderLog = new
>>>>>Hib3GrouperLoaderLog();
>>>>>    hib3GrouploaderLog.setHost(GrouperUtil.hostname());
>>>>>    hib3GrouploaderLog.setJobName(CHANGELOG_JOB_NAME);
>>>>>    hib3GrouploaderLog.setStatus(GrouperLoaderStatus.RUNNING.name());
>>>>>    hib3GrouploaderLog.store();
>>>>>
>>>>>    try {
>>>>>      ChangeLogHelper.processRecords(CHANGELOG_JOB_NAME,
>>>>> hib3GrouploaderLog, this);
>>>>>      
>>>>>hib3GrouploaderLog.setStatus(GrouperLoaderStatus.SUCCESS.name());
>>>>>    } catch (Exception e) {
>>>>>      LOG.error("Error processing records", e);
>>>>>      hib3GrouploaderLog.setStatus(GrouperLoaderStatus.ERROR.name());
>>>>>    }
>>>>>    hib3GrouploaderLog.store();
>>>>>  }
>>>>>
>>>
>>>
>