Grouper Developers Forum

[Shilen Patel <>]

I can reproduce this problem in both 2.0 and trunk.  For me, using a local
mysql database takes about 4 seconds, but using a local postgres database
takes over 70 seconds.  Most of the time is spent processing the temp
change log.  After doing a bit of profiling, it seems the queries to
determine flattened permission changes are not performing well in
postgres.  Those queries will be changing for 2.1 or 2.2 anyways since
they don't currently handle deny permissions.  You can avoid the issue
right now by setting changeLog.includeFlattenedPermissions to false in
grouper-loader.properties and ignore the tests for flattened permissions.
I'll let you know if I find a better workaround for the near term.

Thanks!

-- Shilen


On 9/30/11 3:59 PM, "Chris Hyzer" 
<>
 wrote:

>Can you put a handful of tests in there so we can see the breakdown
>between startup time, and test time.
>
>Thanks,
>Chris
>
>-----Original Message-----
>From: 
>
> 
>[mailto:]
> On Behalf Of Tom
>Zeller
>Sent: Friday, September 30, 2011 3:57 PM
>To: Shilen Patel
>Cc: Grouper Dev
>Subject: Re: [grouper-dev] help with writing changeLog tests ?
>
>Right.
>
>The following is a test I just cobbled together. The changelog entries
>are from grouper initialization.
>
> postgres : Time: 70.81
> hsql: Time: 4.415
>
>So postgres needed over a minute and hsql a few seconds.
>
>
>package edu.internet2.middleware.grouper.pit;
>
>import junit.textui.TestRunner;
>
>import org.apache.commons.lang.time.StopWatch;
>import org.slf4j.Logger;
>import org.slf4j.LoggerFactory;
>
>import edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity;
>import edu.internet2.middleware.grouper.helper.GrouperTest;
>
>public class TomTest extends GrouperTest {
>
>  protected static final Logger LOG =
>LoggerFactory.getLogger(TomTest.class);
>
>  public void testOne() {
>    LOG.info("starting");
>    StopWatch sw = new StopWatch();
>    sw.start();
>    ChangeLogTempToEntity.convertRecords();
>    sw.stop();
>    LOG.info(sw.toString());
>  }
>}
>
>On Fri, Sep 30, 2011 at 1:52 PM, Shilen Patel 
><>
> wrote:
>> The time it takes to run that depends on how many changes you have in
>>your
>> temp change log among other things.  Is there a particular junit test
>>that
>> is having this slow response time when using postgres?  If so, I can try
>> to reproduce it and take a look.
>>
>> Thanks!
>>
>> -- Shilen
>>
>> On 9/30/11 2:22 PM, "Tom Zeller" 
>> <>
>>  wrote:
>>
>>>Thanks.
>>>
>>>I found that in GrouperLoaderType, in the CHANGE_LOG enum, the following
>>>
>>> 826 ChangeLogTempToEntity.convertRecords(hib3GrouploaderLog);
>>>
>>>seems slow (10s per test) using postgres, it is faster with hsql.
>>>
>>>I guess it is the PIT saveOrUpdate() stuff.
>>>
>>>This may be of concern, perhaps the postgres ddl needs tweaking, I
>>>have no idea how :-)
>>>
>>>On Fri, Sep 30, 2011 at 11:06 AM, Chris Hyzer 
>>><>
>>>wrote:
>>>> I think you can make it think it is actually running to be a more
>>>>realistic test... i.e. configure it for tests in the grouper-loader
>>>>testConfig in a startUp() [and remove in tearDown()]:
>>>>
>>>>
>>>>GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.class",
>>>>LdappcngConsumer.class.getName());
>>>>
>>>>GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.quartzC
>>>>ro
>>>>n", "0 0 8 * * ?");
>>>>
>>>> Then run it to register the consumer and clear out the change log:
>>>>
>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>
>>>> Do stuff:
>>>>
>>>> Call those again and it will process whatever you did
>>>>
>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>
>>>> Ok?
>>>>
>>>> Thanks,
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: 
>>>> 
>>>>  
>>>> [mailto:]
>>>>  On Behalf Of Tom
>>>>Zeller
>>>> Sent: Friday, September 30, 2011 10:31 AM
>>>> To: Grouper Dev
>>>> Subject: [grouper-dev] help with writing changeLog tests ?
>>>>
>>>> I am starting to write tests for the ldappcng changelog consumer. Is
>>>> this the correct way to write changelog tests ?
>>>>
>>>> public void testSomething() {
>>>>
>>>>  ldappcngConsumer.runChangeLog();
>>>>
>>>>  ChangeLogTempToEntity.convertRecords();
>>>>
>>>>  // clear changelog
>>>>  HibernateSession.byHqlStatic().createQuery("delete from
>>>> ChangeLogEntryTemp").executeUpdate();
>>>>  HibernateSession.byHqlStatic().createQuery("delete from
>>>> ChangeLogEntryEntity").executeUpdate();
>>>>
>>>>  // do something e.g. group.addMember();
>>>>
>>>>  ChangeLogTempToEntity.convertRecords();
>>>>
>>>>  ldappcngConsumer.runChangeLog();
>>>>
>>>>  // verify test
>>>> }
>>>>
>>>> And here is the runChangeLog() method :
>>>>
>>>> public void runChangeLog() {
>>>>    LOG.info("runChangeLog");
>>>>    Hib3GrouperLoaderLog hib3GrouploaderLog = new
>>>>Hib3GrouperLoaderLog();
>>>>    hib3GrouploaderLog.setHost(GrouperUtil.hostname());
>>>>    hib3GrouploaderLog.setJobName(CHANGELOG_JOB_NAME);
>>>>    hib3GrouploaderLog.setStatus(GrouperLoaderStatus.RUNNING.name());
>>>>    hib3GrouploaderLog.store();
>>>>
>>>>    try {
>>>>      ChangeLogHelper.processRecords(CHANGELOG_JOB_NAME,
>>>> hib3GrouploaderLog, this);
>>>>      hib3GrouploaderLog.setStatus(GrouperLoaderStatus.SUCCESS.name());
>>>>    } catch (Exception e) {
>>>>      LOG.error("Error processing records", e);
>>>>      hib3GrouploaderLog.setStatus(GrouperLoaderStatus.ERROR.name());
>>>>    }
>>>>    hib3GrouploaderLog.store();
>>>>  }
>>>>
>>
>>