Skip to Content.
Sympa Menu

grouper-dev - Re: [grouper-dev] help with writing changeLog tests ?

Subject: Grouper Developers Forum

List archive

Re: [grouper-dev] help with writing changeLog tests ?


Chronological Thread 
  • From: Shilen Patel <>
  • To: Tom Zeller <>
  • Cc: Grouper Dev <>, Chris Hyzer <>
  • Subject: Re: [grouper-dev] help with writing changeLog tests ?
  • Date: Sat, 1 Oct 2011 18:08:04 +0000
  • Accept-language: en-US

Tom,

What version of postgres are you running? I noticed the problem with
8.4.7. I just upgraded to the latest 8.4 release (8.4.9) and the test now
runs in about 4 seconds for me. Can you try 8.4.9 or 9.0.5 or 9.1.1,
which were all released this week?

Thanks!

-- Shilen


On 10/1/11 9:57 AM, "Shilen Patel"
<>
wrote:

>I can reproduce this problem in both 2.0 and trunk. For me, using a local
>mysql database takes about 4 seconds, but using a local postgres database
>takes over 70 seconds. Most of the time is spent processing the temp
>change log. After doing a bit of profiling, it seems the queries to
>determine flattened permission changes are not performing well in
>postgres. Those queries will be changing for 2.1 or 2.2 anyways since
>they don't currently handle deny permissions. You can avoid the issue
>right now by setting changeLog.includeFlattenedPermissions to false in
>grouper-loader.properties and ignore the tests for flattened permissions.
>I'll let you know if I find a better workaround for the near term.
>
>Thanks!
>
>-- Shilen
>
>
>On 9/30/11 3:59 PM, "Chris Hyzer"
><>
> wrote:
>
>>Can you put a handful of tests in there so we can see the breakdown
>>between startup time, and test time.
>>
>>Thanks,
>>Chris
>>
>>-----Original Message-----
>>From:
>>
>>
>>[mailto:]
>> On Behalf Of Tom
>>Zeller
>>Sent: Friday, September 30, 2011 3:57 PM
>>To: Shilen Patel
>>Cc: Grouper Dev
>>Subject: Re: [grouper-dev] help with writing changeLog tests ?
>>
>>Right.
>>
>>The following is a test I just cobbled together. The changelog entries
>>are from grouper initialization.
>>
>> postgres : Time: 70.81
>> hsql: Time: 4.415
>>
>>So postgres needed over a minute and hsql a few seconds.
>>
>>
>>package edu.internet2.middleware.grouper.pit;
>>
>>import junit.textui.TestRunner;
>>
>>import org.apache.commons.lang.time.StopWatch;
>>import org.slf4j.Logger;
>>import org.slf4j.LoggerFactory;
>>
>>import edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity;
>>import edu.internet2.middleware.grouper.helper.GrouperTest;
>>
>>public class TomTest extends GrouperTest {
>>
>> protected static final Logger LOG =
>>LoggerFactory.getLogger(TomTest.class);
>>
>> public void testOne() {
>> LOG.info("starting");
>> StopWatch sw = new StopWatch();
>> sw.start();
>> ChangeLogTempToEntity.convertRecords();
>> sw.stop();
>> LOG.info(sw.toString());
>> }
>>}
>>
>>On Fri, Sep 30, 2011 at 1:52 PM, Shilen Patel
>><>
>> wrote:
>>> The time it takes to run that depends on how many changes you have in
>>>your
>>> temp change log among other things. Is there a particular junit test
>>>that
>>> is having this slow response time when using postgres? If so, I can
>>>try
>>> to reproduce it and take a look.
>>>
>>> Thanks!
>>>
>>> -- Shilen
>>>
>>> On 9/30/11 2:22 PM, "Tom Zeller"
>>> <>
>>> wrote:
>>>
>>>>Thanks.
>>>>
>>>>I found that in GrouperLoaderType, in the CHANGE_LOG enum, the
>>>>following
>>>>
>>>> 826 ChangeLogTempToEntity.convertRecords(hib3GrouploaderLog);
>>>>
>>>>seems slow (10s per test) using postgres, it is faster with hsql.
>>>>
>>>>I guess it is the PIT saveOrUpdate() stuff.
>>>>
>>>>This may be of concern, perhaps the postgres ddl needs tweaking, I
>>>>have no idea how :-)
>>>>
>>>>On Fri, Sep 30, 2011 at 11:06 AM, Chris Hyzer
>>>><>
>>>>wrote:
>>>>> I think you can make it think it is actually running to be a more
>>>>>realistic test... i.e. configure it for tests in the grouper-loader
>>>>>testConfig in a startUp() [and remove in tearDown()]:
>>>>>
>>>>>
>>>>>GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.class"
>>>>>,
>>>>>LdappcngConsumer.class.getName());
>>>>>
>>>>>GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.quartz
>>>>>C
>>>>>ro
>>>>>n", "0 0 8 * * ?");
>>>>>
>>>>> Then run it to register the consumer and clear out the change log:
>>>>>
>>>>> GrouperLoader.runOnceByJobName(grouperSession,
>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>> GrouperLoader.runOnceByJobName(grouperSession,
>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>
>>>>> Do stuff:
>>>>>
>>>>> Call those again and it will process whatever you did
>>>>>
>>>>> GrouperLoader.runOnceByJobName(grouperSession,
>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>> GrouperLoader.runOnceByJobName(grouperSession,
>>>>>GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>
>>>>> Ok?
>>>>>
>>>>> Thanks,
>>>>> Chris
>>>>>
>>>>> -----Original Message-----
>>>>> From:
>>>>>
>>>>>
>>>>> [mailto:]
>>>>> On Behalf Of Tom
>>>>>Zeller
>>>>> Sent: Friday, September 30, 2011 10:31 AM
>>>>> To: Grouper Dev
>>>>> Subject: [grouper-dev] help with writing changeLog tests ?
>>>>>
>>>>> I am starting to write tests for the ldappcng changelog consumer. Is
>>>>> this the correct way to write changelog tests ?
>>>>>
>>>>> public void testSomething() {
>>>>>
>>>>> ldappcngConsumer.runChangeLog();
>>>>>
>>>>> ChangeLogTempToEntity.convertRecords();
>>>>>
>>>>> // clear changelog
>>>>> HibernateSession.byHqlStatic().createQuery("delete from
>>>>> ChangeLogEntryTemp").executeUpdate();
>>>>> HibernateSession.byHqlStatic().createQuery("delete from
>>>>> ChangeLogEntryEntity").executeUpdate();
>>>>>
>>>>> // do something e.g. group.addMember();
>>>>>
>>>>> ChangeLogTempToEntity.convertRecords();
>>>>>
>>>>> ldappcngConsumer.runChangeLog();
>>>>>
>>>>> // verify test
>>>>> }
>>>>>
>>>>> And here is the runChangeLog() method :
>>>>>
>>>>> public void runChangeLog() {
>>>>> LOG.info("runChangeLog");
>>>>> Hib3GrouperLoaderLog hib3GrouploaderLog = new
>>>>>Hib3GrouperLoaderLog();
>>>>> hib3GrouploaderLog.setHost(GrouperUtil.hostname());
>>>>> hib3GrouploaderLog.setJobName(CHANGELOG_JOB_NAME);
>>>>> hib3GrouploaderLog.setStatus(GrouperLoaderStatus.RUNNING.name());
>>>>> hib3GrouploaderLog.store();
>>>>>
>>>>> try {
>>>>> ChangeLogHelper.processRecords(CHANGELOG_JOB_NAME,
>>>>> hib3GrouploaderLog, this);
>>>>>
>>>>>hib3GrouploaderLog.setStatus(GrouperLoaderStatus.SUCCESS.name());
>>>>> } catch (Exception e) {
>>>>> LOG.error("Error processing records", e);
>>>>> hib3GrouploaderLog.setStatus(GrouperLoaderStatus.ERROR.name());
>>>>> }
>>>>> hib3GrouploaderLog.store();
>>>>> }
>>>>>
>>>
>>>
>




Archive powered by MHonArc 2.6.16.

Top of Page