Skip to Content.
Sympa Menu

grouper-dev - Re: [grouper-dev] help with writing changeLog tests ?

Subject: Grouper Developers Forum

List archive

Re: [grouper-dev] help with writing changeLog tests ?


Chronological Thread 
  • From: Tom Zeller <>
  • To: Shilen Patel <>
  • Cc: Grouper Dev <>
  • Subject: Re: [grouper-dev] help with writing changeLog tests ?
  • Date: Sat, 1 Oct 2011 11:01:27 -0700 (MST)

Thanks. I thought postgres seemed sluggish.

On Oct 1, 2011, at 8:57 AM, Shilen Patel
<>
wrote:

> I can reproduce this problem in both 2.0 and trunk. For me, using a local
> mysql database takes about 4 seconds, but using a local postgres database
> takes over 70 seconds. Most of the time is spent processing the temp
> change log. After doing a bit of profiling, it seems the queries to
> determine flattened permission changes are not performing well in
> postgres. Those queries will be changing for 2.1 or 2.2 anyways since
> they don't currently handle deny permissions. You can avoid the issue
> right now by setting changeLog.includeFlattenedPermissions to false in
> grouper-loader.properties and ignore the tests for flattened permissions.
> I'll let you know if I find a better workaround for the near term.
>
> Thanks!
>
> -- Shilen
>
>
> On 9/30/11 3:59 PM, "Chris Hyzer"
> <>
> wrote:
>
>> Can you put a handful of tests in there so we can see the breakdown
>> between startup time, and test time.
>>
>> Thanks,
>> Chris
>>
>> -----Original Message-----
>> From:
>>
>>
>> [mailto:]
>> On Behalf Of Tom
>> Zeller
>> Sent: Friday, September 30, 2011 3:57 PM
>> To: Shilen Patel
>> Cc: Grouper Dev
>> Subject: Re: [grouper-dev] help with writing changeLog tests ?
>>
>> Right.
>>
>> The following is a test I just cobbled together. The changelog entries
>> are from grouper initialization.
>>
>> postgres : Time: 70.81
>> hsql: Time: 4.415
>>
>> So postgres needed over a minute and hsql a few seconds.
>>
>>
>> package edu.internet2.middleware.grouper.pit;
>>
>> import junit.textui.TestRunner;
>>
>> import org.apache.commons.lang.time.StopWatch;
>> import org.slf4j.Logger;
>> import org.slf4j.LoggerFactory;
>>
>> import edu.internet2.middleware.grouper.changeLog.ChangeLogTempToEntity;
>> import edu.internet2.middleware.grouper.helper.GrouperTest;
>>
>> public class TomTest extends GrouperTest {
>>
>> protected static final Logger LOG =
>> LoggerFactory.getLogger(TomTest.class);
>>
>> public void testOne() {
>> LOG.info("starting");
>> StopWatch sw = new StopWatch();
>> sw.start();
>> ChangeLogTempToEntity.convertRecords();
>> sw.stop();
>> LOG.info(sw.toString());
>> }
>> }
>>
>> On Fri, Sep 30, 2011 at 1:52 PM, Shilen Patel
>> <>
>> wrote:
>>> The time it takes to run that depends on how many changes you have in
>>> your
>>> temp change log among other things. Is there a particular junit test
>>> that
>>> is having this slow response time when using postgres? If so, I can try
>>> to reproduce it and take a look.
>>>
>>> Thanks!
>>>
>>> -- Shilen
>>>
>>> On 9/30/11 2:22 PM, "Tom Zeller"
>>> <>
>>> wrote:
>>>
>>>> Thanks.
>>>>
>>>> I found that in GrouperLoaderType, in the CHANGE_LOG enum, the following
>>>>
>>>> 826 ChangeLogTempToEntity.convertRecords(hib3GrouploaderLog);
>>>>
>>>> seems slow (10s per test) using postgres, it is faster with hsql.
>>>>
>>>> I guess it is the PIT saveOrUpdate() stuff.
>>>>
>>>> This may be of concern, perhaps the postgres ddl needs tweaking, I
>>>> have no idea how :-)
>>>>
>>>> On Fri, Sep 30, 2011 at 11:06 AM, Chris Hyzer
>>>> <>
>>>> wrote:
>>>>> I think you can make it think it is actually running to be a more
>>>>> realistic test... i.e. configure it for tests in the grouper-loader
>>>>> testConfig in a startUp() [and remove in tearDown()]:
>>>>>
>>>>>
>>>>> GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.class",
>>>>> LdappcngConsumer.class.getName());
>>>>>
>>>>> GrouperLoaderConfig.testConfig.put("changeLog.consumer.ldappcng.quartzC
>>>>> ro
>>>>> n", "0 0 8 * * ?");
>>>>>
>>>>> Then run it to register the consumer and clear out the change log:
>>>>>
>>>>> GrouperLoader.runOnceByJobName(grouperSession,
>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>> GrouperLoader.runOnceByJobName(grouperSession,
>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>
>>>>> Do stuff:
>>>>>
>>>>> Call those again and it will process whatever you did
>>>>>
>>>>> GrouperLoader.runOnceByJobName(grouperSession,
>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_TEMP_TO_CHANGE_LOG);
>>>>> GrouperLoader.runOnceByJobName(grouperSession,
>>>>> GrouperLoaderType.GROUPER_CHANGE_LOG_CONSUMER_PREFIX + "ldappcng");
>>>>>
>>>>> Ok?
>>>>>
>>>>> Thanks,
>>>>> Chris
>>>>>
>>>>> -----Original Message-----
>>>>> From:
>>>>>
>>>>>
>>>>> [mailto:]
>>>>> On Behalf Of Tom
>>>>> Zeller
>>>>> Sent: Friday, September 30, 2011 10:31 AM
>>>>> To: Grouper Dev
>>>>> Subject: [grouper-dev] help with writing changeLog tests ?
>>>>>
>>>>> I am starting to write tests for the ldappcng changelog consumer. Is
>>>>> this the correct way to write changelog tests ?
>>>>>
>>>>> public void testSomething() {
>>>>>
>>>>> ldappcngConsumer.runChangeLog();
>>>>>
>>>>> ChangeLogTempToEntity.convertRecords();
>>>>>
>>>>> // clear changelog
>>>>> HibernateSession.byHqlStatic().createQuery("delete from
>>>>> ChangeLogEntryTemp").executeUpdate();
>>>>> HibernateSession.byHqlStatic().createQuery("delete from
>>>>> ChangeLogEntryEntity").executeUpdate();
>>>>>
>>>>> // do something e.g. group.addMember();
>>>>>
>>>>> ChangeLogTempToEntity.convertRecords();
>>>>>
>>>>> ldappcngConsumer.runChangeLog();
>>>>>
>>>>> // verify test
>>>>> }
>>>>>
>>>>> And here is the runChangeLog() method :
>>>>>
>>>>> public void runChangeLog() {
>>>>> LOG.info("runChangeLog");
>>>>> Hib3GrouperLoaderLog hib3GrouploaderLog = new
>>>>> Hib3GrouperLoaderLog();
>>>>> hib3GrouploaderLog.setHost(GrouperUtil.hostname());
>>>>> hib3GrouploaderLog.setJobName(CHANGELOG_JOB_NAME);
>>>>> hib3GrouploaderLog.setStatus(GrouperLoaderStatus.RUNNING.name());
>>>>> hib3GrouploaderLog.store();
>>>>>
>>>>> try {
>>>>> ChangeLogHelper.processRecords(CHANGELOG_JOB_NAME,
>>>>> hib3GrouploaderLog, this);
>>>>> hib3GrouploaderLog.setStatus(GrouperLoaderStatus.SUCCESS.name());
>>>>> } catch (Exception e) {
>>>>> LOG.error("Error processing records", e);
>>>>> hib3GrouploaderLog.setStatus(GrouperLoaderStatus.ERROR.name());
>>>>> }
>>>>> hib3GrouploaderLog.store();
>>>>> }
>>>>>
>>>
>>>
>



Archive powered by MHonArc 2.6.16.

Top of Page