Grouper Developers Forum

[Tom Zeller <>]

Thanks. I thought postgres seemed sluggish.

On Oct 1, 2011, at 8:57 AM, Shilen Patel 
<>
 wrote:

> I can reproduce this problem in both 2.0 and trunk.  For me, using a local
> mysql database takes about 4 seconds, but using a local postgres database
> takes over 70 seconds.  Most of the time is spent processing the temp
> change log.  After doing a bit of profiling, it seems the queries to
> determine flattened permission changes are not performing well in
> postgres.  Those queries will be changing for 2.1 or 2.2 anyways since
> they don't currently handle deny permissions.  You can avoid the issue
> right now by setting changeLog.includeFlattenedPermissions to false in
> grouper-loader.properties and ignore the tests for flattened permissions.
> I'll let you know if I find a better workaround for the near term.
> 
> Thanks!
> 
> -- Shilen
> 
> 
> On 9/30/11 3:59 PM, "Chris Hyzer" 
> <>
>  wrote:
> 
>> Can you put a handful of tests in there so we can see the breakdown
>> between startup time, and test time.
>> 
>> Thanks,
>> Chris
>> 
>> -----Original Message-----
>> From: 
>> 
>>  
>> [mailto:]
>>  On Behalf Of Tom
>> Zeller
>> Sent: Friday, September 30, 2011 3:57 PM
>> To: Shilen Patel
>> Cc: Grouper Dev
>> Subject: Re: [grouper-dev] help with writing changeLog tests ?
>> 
>> Right.
>> 
>> The following is a test I just cobbled together. The changelog entries
>> are from grouper initialization.
>> 
>> postgres : Time: 70.81
>> hsql: Time: 4.415
>> 
>> So postgres needed over a minute and hsql a few seconds.
>> 
>> 
>> package edu.internet2.middleware.grouper.pit;
>> 
>> import junit.textui.TestRunner;
>> 
>> import org.apache.commons.lang.time.StopWatch;
>> import org.slf4j.Logger;
>> import org.slf4j.LoggerFactory;
>> 
>> import edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity;
>> import edu.internet2.middleware.grouper.helper.GrouperTest;
>> 
>> public class TomTest extends GrouperTest {
>> 
>> protected static final Logger LOG =
>> LoggerFactory.getLogger(TomTest.class);
>> 
>> public void testOne() {
>>   LOG.info("starting");
>>   StopWatch sw = new StopWatch();
>>   sw.start();
>>   ChangeLogTempToEntity.convertRecords();
>>   sw.stop();
>>   LOG.info(sw.toString());
>> }
>> }
>> 
>> On Fri, Sep 30, 2011 at 1:52 PM, Shilen Patel 
>> <>
>>  wrote:
>>> The time it takes to run that depends on how many changes you have in
>>> your
>>> temp change log among other things.  Is there a particular junit test
>>> that
>>> is having this slow response time when using postgres?  If so, I can try
>>> to reproduce it and take a look.
>>> 
>>> Thanks!
>>> 
>>> -- Shilen
>>> 
>>> On 9/30/11 2:22 PM, "Tom Zeller" 
>>> <>
>>>  wrote:
>>> 
>>>> Thanks.
>>>> 
>>>> I found that in GrouperLoaderType, in the CHANGE_LOG enum, the following
>>>> 
>>>> 826 ChangeLogTempToEntity.convertRecords(hib3GrouploaderLog);
>>>> 
>>>> seems slow (10s per test) using postgres, it is faster with hsql.
>>>> 
>>>> I guess it is the PIT saveOrUpdate() stuff.
>>>> 
>>>> This may be of concern, perhaps the postgres ddl needs tweaking, I
>>>> have no idea how :-)
>>>> 
>>>> On Fri, Sep 30, 2011 at 11:06 AM, Chris Hyzer 
>>>> <>
>>>> wrote:
>>>>> I think you can make it think it is actually running to be a more
>>>>> realistic test... i.e. configure it for tests in the grouper-loader
>>>>> testConfig in a startUp() [and remove in tearDown()]:
>>>>> 
>>>>> 
>>>>> GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.class",
>>>>> LdappcngConsumer.class.getName());
>>>>> 
>>>>> GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.quartzC
>>>>> ro
>>>>> n", "0 0 8 * * ?");
>>>>> 
>>>>> Then run it to register the consumer and clear out the change log:
>>>>> 
>>>>>   GrouperLoader.runOnceByJobName(grouperSession,
>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>>   GrouperLoader.runOnceByJobName(grouperSession,
>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>> 
>>>>> Do stuff:
>>>>> 
>>>>> Call those again and it will process whatever you did
>>>>> 
>>>>>   GrouperLoader.runOnceByJobName(grouperSession,
>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>>   GrouperLoader.runOnceByJobName(grouperSession,
>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>> 
>>>>> Ok?
>>>>> 
>>>>> Thanks,
>>>>> Chris
>>>>> 
>>>>> -----Original Message-----
>>>>> From: 
>>>>> 
>>>>>  
>>>>> [mailto:]
>>>>>  On Behalf Of Tom
>>>>> Zeller
>>>>> Sent: Friday, September 30, 2011 10:31 AM
>>>>> To: Grouper Dev
>>>>> Subject: [grouper-dev] help with writing changeLog tests ?
>>>>> 
>>>>> I am starting to write tests for the ldappcng changelog consumer. Is
>>>>> this the correct way to write changelog tests ?
>>>>> 
>>>>> public void testSomething() {
>>>>> 
>>>>> ldappcngConsumer.runChangeLog();
>>>>> 
>>>>> ChangeLogTempToEntity.convertRecords();
>>>>> 
>>>>> // clear changelog
>>>>> HibernateSession.byHqlStatic().createQuery("delete from
>>>>> ChangeLogEntryTemp").executeUpdate();
>>>>> HibernateSession.byHqlStatic().createQuery("delete from
>>>>> ChangeLogEntryEntity").executeUpdate();
>>>>> 
>>>>> // do something e.g. group.addMember();
>>>>> 
>>>>> ChangeLogTempToEntity.convertRecords();
>>>>> 
>>>>> ldappcngConsumer.runChangeLog();
>>>>> 
>>>>> // verify test
>>>>> }
>>>>> 
>>>>> And here is the runChangeLog() method :
>>>>> 
>>>>> public void runChangeLog() {
>>>>>   LOG.info("runChangeLog");
>>>>>   Hib3GrouperLoaderLog hib3GrouploaderLog = new
>>>>> Hib3GrouperLoaderLog();
>>>>>   hib3GrouploaderLog.setHost(GrouperUtil.hostname());
>>>>>   hib3GrouploaderLog.setJobName(CHANGELOG_JOB_NAME);
>>>>>   hib3GrouploaderLog.setStatus(GrouperLoaderStatus.RUNNING.name());
>>>>>   hib3GrouploaderLog.store();
>>>>> 
>>>>>   try {
>>>>>     ChangeLogHelper.processRecords(CHANGELOG_JOB_NAME,
>>>>> hib3GrouploaderLog, this);
>>>>>     hib3GrouploaderLog.setStatus(GrouperLoaderStatus.SUCCESS.name());
>>>>>   } catch (Exception e) {
>>>>>     LOG.error("Error processing records", e);
>>>>>     hib3GrouploaderLog.setStatus(GrouperLoaderStatus.ERROR.name());
>>>>>   }
>>>>>   hib3GrouploaderLog.store();
>>>>> }
>>>>> 
>>> 
>>> 
>