Grouper Developers Forum

[Shilen Patel <>]

Actually, never mind on that.. :)  I wasn't running my test properly.

-- Shilen

On 10/1/11 2:08 PM, "Shilen Patel" 
<>
 wrote:

>Tom,
>
>What version of postgres are you running?  I noticed the problem with
>8.4.7.  I just upgraded to the latest 8.4 release (8.4.9) and the test now
>runs in about 4 seconds for me.  Can you try 8.4.9 or 9.0.5 or 9.1.1,
>which were all released this week?
>
>Thanks!
>
>-- Shilen
>
>
>On 10/1/11 9:57 AM, "Shilen Patel" 
><>
> wrote:
>
>>I can reproduce this problem in both 2.0 and trunk.  For me, using a
>>local
>>mysql database takes about 4 seconds, but using a local postgres database
>>takes over 70 seconds.  Most of the time is spent processing the temp
>>change log.  After doing a bit of profiling, it seems the queries to
>>determine flattened permission changes are not performing well in
>>postgres.  Those queries will be changing for 2.1 or 2.2 anyways since
>>they don't currently handle deny permissions.  You can avoid the issue
>>right now by setting changeLog.includeFlattenedPermissions to false in
>>grouper-loader.properties and ignore the tests for flattened permissions.
>>I'll let you know if I find a better workaround for the near term.
>>
>>Thanks!
>>
>>-- Shilen
>>
>>
>>On 9/30/11 3:59 PM, "Chris Hyzer" 
>><>
>> wrote:
>>
>>>Can you put a handful of tests in there so we can see the breakdown
>>>between startup time, and test time.
>>>
>>>Thanks,
>>>Chris
>>>
>>>-----Original Message-----
>>>From: 
>>>
>>> 
>>>[mailto:]
>>> On Behalf Of Tom
>>>Zeller
>>>Sent: Friday, September 30, 2011 3:57 PM
>>>To: Shilen Patel
>>>Cc: Grouper Dev
>>>Subject: Re: [grouper-dev] help with writing changeLog tests ?
>>>
>>>Right.
>>>
>>>The following is a test I just cobbled together. The changelog entries
>>>are from grouper initialization.
>>>
>>> postgres : Time: 70.81
>>> hsql: Time: 4.415
>>>
>>>So postgres needed over a minute and hsql a few seconds.
>>>
>>>
>>>package edu.internet2.middleware.grouper.pit;
>>>
>>>import junit.textui.TestRunner;
>>>
>>>import org.apache.commons.lang.time.StopWatch;
>>>import org.slf4j.Logger;
>>>import org.slf4j.LoggerFactory;
>>>
>>>import edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity;
>>>import edu.internet2.middleware.grouper.helper.GrouperTest;
>>>
>>>public class TomTest extends GrouperTest {
>>>
>>>  protected static final Logger LOG =
>>>LoggerFactory.getLogger(TomTest.class);
>>>
>>>  public void testOne() {
>>>    LOG.info("starting");
>>>    StopWatch sw = new StopWatch();
>>>    sw.start();
>>>    ChangeLogTempToEntity.convertRecords();
>>>    sw.stop();
>>>    LOG.info(sw.toString());
>>>  }
>>>}
>>>
>>>On Fri, Sep 30, 2011 at 1:52 PM, Shilen Patel 
>>><>
>>> wrote:
>>>> The time it takes to run that depends on how many changes you have in
>>>>your
>>>> temp change log among other things.  Is there a particular junit test
>>>>that
>>>> is having this slow response time when using postgres?  If so, I can
>>>>try
>>>> to reproduce it and take a look.
>>>>
>>>> Thanks!
>>>>
>>>> -- Shilen
>>>>
>>>> On 9/30/11 2:22 PM, "Tom Zeller" 
>>>> <>
>>>>  wrote:
>>>>
>>>>>Thanks.
>>>>>
>>>>>I found that in GrouperLoaderType, in the CHANGE_LOG enum, the
>>>>>following
>>>>>
>>>>> 826 ChangeLogTempToEntity.convertRecords(hib3GrouploaderLog);
>>>>>
>>>>>seems slow (10s per test) using postgres, it is faster with hsql.
>>>>>
>>>>>I guess it is the PIT saveOrUpdate() stuff.
>>>>>
>>>>>This may be of concern, perhaps the postgres ddl needs tweaking, I
>>>>>have no idea how :-)
>>>>>
>>>>>On Fri, Sep 30, 2011 at 11:06 AM, Chris Hyzer 
>>>>><>
>>>>>wrote:
>>>>>> I think you can make it think it is actually running to be a more
>>>>>>realistic test... i.e. configure it for tests in the grouper-loader
>>>>>>testConfig in a startUp() [and remove in tearDown()]:
>>>>>>
>>>>>>
>>>>>>GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.class
>>>>>>"
>>>>>>,
>>>>>>LdappcngConsumer.class.getName());
>>>>>>
>>>>>>GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.quart
>>>>>>z
>>>>>>C
>>>>>>ro
>>>>>>n", "0 0 8 * * ?");
>>>>>>
>>>>>> Then run it to register the consumer and clear out the change log:
>>>>>>
>>>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>>
>>>>>> Do stuff:
>>>>>>
>>>>>> Call those again and it will process whatever you did
>>>>>>
>>>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>>>    GrouperLoader.runOnceByJobName(grouperSession,
>>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>>
>>>>>> Ok?
>>>>>>
>>>>>> Thanks,
>>>>>> Chris
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: 
>>>>>> 
>>>>>>  
>>>>>> [mailto:]
>>>>>>  On Behalf Of
>>>>>>Tom
>>>>>>Zeller
>>>>>> Sent: Friday, September 30, 2011 10:31 AM
>>>>>> To: Grouper Dev
>>>>>> Subject: [grouper-dev] help with writing changeLog tests ?
>>>>>>
>>>>>> I am starting to write tests for the ldappcng changelog consumer. Is
>>>>>> this the correct way to write changelog tests ?
>>>>>>
>>>>>> public void testSomething() {
>>>>>>
>>>>>>  ldappcngConsumer.runChangeLog();
>>>>>>
>>>>>>  ChangeLogTempToEntity.convertRecords();
>>>>>>
>>>>>>  // clear changelog
>>>>>>  HibernateSession.byHqlStatic().createQuery("delete from
>>>>>> ChangeLogEntryTemp").executeUpdate();
>>>>>>  HibernateSession.byHqlStatic().createQuery("delete from
>>>>>> ChangeLogEntryEntity").executeUpdate();
>>>>>>
>>>>>>  // do something e.g. group.addMember();
>>>>>>
>>>>>>  ChangeLogTempToEntity.convertRecords();
>>>>>>
>>>>>>  ldappcngConsumer.runChangeLog();
>>>>>>
>>>>>>  // verify test
>>>>>> }
>>>>>>
>>>>>> And here is the runChangeLog() method :
>>>>>>
>>>>>> public void runChangeLog() {
>>>>>>    LOG.info("runChangeLog");
>>>>>>    Hib3GrouperLoaderLog hib3GrouploaderLog = new
>>>>>>Hib3GrouperLoaderLog();
>>>>>>    hib3GrouploaderLog.setHost(GrouperUtil.hostname());
>>>>>>    hib3GrouploaderLog.setJobName(CHANGELOG_JOB_NAME);
>>>>>>    hib3GrouploaderLog.setStatus(GrouperLoaderStatus.RUNNING.name());
>>>>>>    hib3GrouploaderLog.store();
>>>>>>
>>>>>>    try {
>>>>>>      ChangeLogHelper.processRecords(CHANGELOG_JOB_NAME,
>>>>>> hib3GrouploaderLog, this);
>>>>>>      
>>>>>>hib3GrouploaderLog.setStatus(GrouperLoaderStatus.SUCCESS.name());
>>>>>>    } catch (Exception e) {
>>>>>>      LOG.error("Error processing records", e);
>>>>>>      hib3GrouploaderLog.setStatus(GrouperLoaderStatus.ERROR.name());
>>>>>>    }
>>>>>>    hib3GrouploaderLog.store();
>>>>>>  }
>>>>>>
>>>>
>>>>
>>
>