Skip to Content.
Sympa Menu

grouper-users - Re: [grouper-users] activemq changelog consumer bug?

Subject: Grouper Users - Open Discussion List

List archive

Re: [grouper-users] activemq changelog consumer bug?


Chronological Thread 
  • From: Daniel Hutchison <>
  • To: "Hyzer, Chris" <>, "" <>
  • Subject: Re: [grouper-users] activemq changelog consumer bug?
  • Date: Fri, 06 Apr 2018 14:59:08 -0600
  • Ironport-phdr: 9a23:konGEhGhiMrMTc4FxDlgu51GYnF86YWxBRYc798ds5kLTJ76p8+zbnLW6fgltlLVR4KTs6sC17KN9fi4EUU7or+5+EgYd5JNUxJXwe43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6arXK99yMdFQviPgRpOOv1BpTSj8Oq3Oyu5pHfeQpFiCazbL9oMBm6sRjau9ULj4dlNqs/0AbCrGFSe+RRy2NoJFaTkAj568yt4pNt8Dletuw4+cJYXqr0Y6o3TbpDDDQ7KG81/9HktQPCTQSU+HQRVHgdnwdSDAjE6BH6WYrxsjf/u+Fg1iSWIdH6QLYpUjmk8qxlSgLniD0fOjA67W/ZitJ+gqxZrxKvuxNxxIHbbZqPO/ZiZK7QZ8kXSXZDU8tXSidPApm8b4wKD+cZM+hXtY/9p1oQohu6GAKiBv7vyiNUiX/23K06yOMhEQfb1wEnAt0BqnLUrNTrO6cWS+y1zLTIzSnfYPNW2Df984/IchY6rPGRQ7JwbNDdxlcyGAPYl1idr5HuMT2S1uQIqWeb7uxgWPq1hGE7twFxoyavydkpionQnYIUykvE9Tlnz4YwP9G4SEl7YcW4EJdIrS6VK5d2QsI8Q252vSY606cKuZihcygP1psr3xHfa+aAc4iV+hLsSvyRLS1ii3J9YLKznxSy8U+7xe3mS8m01k1KripbndnIsnABzQbT6tWBSvdl40iuxziP2BrO6uFAPUA4j6zbJIQkwrEujpYcq1jMHjP3mEXokqCWc1wk+vOo6uT8frXpuJqcO5Vyig7kN6QhhMqyCvk7PAgWR2WX5OWx2KD58UHnQ7hGlOM6nrXDvJzHOcgWpqi0CBJP3Ik58RawFTKm3cwYnXYZKFJFfwqKj43zNFHPJPD4F/C/g0mwkDh13fDKJKPuApDXIXjClLftZ6t961BByAYp099Q+o9UBqkAIPL1REDxqMTVAgI4PgCp2errFtZw2pkAVW6TGKOVLb/evF+Q6uIqOeaMZYsVuDjnK/gi4v7jlWI2lkUbfamow5QXbGq0HvR7I0WFfXXsgtYBEXwNvgo4V+HqlUOCUSNJa3msQq08+yk3CJi6AofbWoCtnLuB0T+0Hp1MYWBGF0iMHmnyd4mdRvcMczydItV6kjEfUbihSpQh1Q21tAPk0bZnL+zU+jEGupL50th6+fHTmQ8o+TNqEsudzjLFc2Yh1EMZVTIsmOhUoVZ80R3Lha1zg+1KGMZ75uhCFBoiOJjaieF2Foa2EkjOZNCUUFu8B8i9DCsqZtM339IUZUthQZOvgg2Jl36lGbgIj7GRQYEv/7jH93n3O8tnzXvaju8sg0RwEeVVMmjzqqd59gXXT67Uj06Zlr2tc+xI0CfL9WqHi2qToEpYURV3V43YQ21Za0fL+4eqrnjeRqOjXOx0ejBKztSPf/NH

Yeah, in my $GROUPER_HOME/conf/grouper-loader.properties, the setting is set to true:

/apps/grouper/grouper-api/conf# grep autocreate.objects *
grouper.client.properties:loader.messaging.settings.autocreate.objects = true
grouper-loader.base.properties:loader.messaging.settings.autocreate.objects = true
grouper-loader.properties:loader.messaging.settings.autocreate.objects = true

Where do I go to submit bug reports?

Thanks,

-daniel

On Fri, 2018-04-06 at 20:33 +0000, Hyzer, Chris wrote:

Are you sure that property is in the grouper-loader.properties for the loader process that is running?  i.e. you need to make sure your config files are in sync…  otherwise, yes, file a bug and we can fix it.

 

Thanks,

Chris

 

From: [mailto:] On Behalf Of Daniel Hutchison
Sent: Friday, April 06, 2018 3:37 PM
To:
Subject: [grouper-users] activemq changelog consumer bug?

 

Hey All,

 

I'm a long time lurker, but first time posting here.   I recently upgraded to 2.3 to setup the new box group provisioner.  As part of this upgrade, I'm trying to switch to using the native aqmp activemq connector (I was using the http post method before for my default membership consumer).

 

However, I am getting the following exception when the change log tries to send a new event:

 

2018-04-06 12:34:52,802: [DefaultQuartzScheduler_Worker-2] ERROR EsbConsumer.processChangeLogEntries(551) -  - problem
java.lang.IllegalArgumentException: For ActiveMQ, autoCreateObjects has to be set to true.
        at edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem.validate(GrouperMessagingActiveMQSystem.java:219)
        at edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem.send(GrouperMessagingActiveMQSystem.java:61)
        at edu.internet2.middleware.grouperClient.messaging.GrouperMessagingEngine.send(GrouperMessagingEngine.java:60)
        at edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher.dispatchEvent(EsbMessagingPublisher.java:58)
        at edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer.processChangeLogEntries(EsbConsumer.java:525)
        at edu.internet2.middleware.grouper.changeLog.ChangeLogHelper.processRecords(ChangeLogHelper.java:245)
        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$5.runJob(GrouperLoaderType.java:640)
        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:465)
        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:345)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
 

Here is what my grouper-loader.properties looks like:

=======================
loader.messaging.settings.autocreate.objects = true
 
changeLog.consumer.esb.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer
changeLog.consumer.esb.publisher.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher
changeLog.consumer.esb.publisher.messagingSystemName = activemq
changeLog.consumer.esb.publisher.messageQueueType = topic
changeLog.consumer.esb.publisher.queueOrTopicName = IAM.account.membership
#run every 30 secs
changeLog.consumer.esb.quartzCron = 0,30 * * * * ?
changeLog.consumer.esb.elfilter = event.eventType eq 'GROUP_DELETE' || event.eventType eq 'GROUP_ADD' || event.eventType eq 'MEMBERSHIP_DELETE' || event.eventType eq 'MEMBERSHIP_ADD'
 
 
changeLog.consumer.boxEsb.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer
changeLog.consumer.boxEsb.publisher.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher
changeLog.consumer.boxEsb.quartzCron = 0,30 * * * * ?
# carefully adjust this filter e.g. for sourceId and groupName
changeLog.consumer.boxEsb.elfilter = (event.sourceId == null || event.sourceId eq 'jdbc') && (event.groupName =~ '^svc:box:groups:.*$' || event.groupName eq 'svc:box:boxUser' || event.name =~ '^svc:box:groups:.*$' || event.name eq 'svc:box:boxUser') && (event.eventType eq 'GROUP_DELETE' || event.eventType eq 'GROUP_ADD' || event.eventType eq 'GROUP_UPDATE' || event.eventType eq 'MEMBERSHIP_DELETE' || event.eventType eq 'MEMBERSHIP_ADD' || event.eventType eq 'MEMBERSHIP_UPDATE')
changeLog.consumer.boxEsb.publisher.messagingSystemName = activemq
changeLog.consumer.boxEsb.publisher.messageQueueType = topic
changeLog.consumer.boxEsb.publisher.queueOrTopicName = IAM.box.membership
# this is optional if not using "id" for subjectId, need to be a subject attribute in the sources.xml
changeLog.consumer.boxEsb.publisher.addSubjectAttributes = email
=======================
 

And the grouper.client.properties:

=======================
loader.messaging.settings.autocreate.objects = true
 
# name of messaging system which is the default
grouper.messaging.default.name.of.messaging.system = activemq
 
# name of a messaging system, required
grouper.messaging.system.activemq.name = activemq
grouper.messaging.system.activemq.defaultSystemName = activemqSystem
grouper.messaging.system.activemq.username = USERNAME
grouper.messaging.system.activemq.password = PASSWORD
 
# Default settings for activemq (can use as base to change userids)
grouper.messaging.system.activemqSystem.name = activemqSystem
grouper.messaging.system.activemqSystem.class = edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem
grouper.messaging.system.activemqSystem.host = HOSTNAME
grouper.messaging.system.activemqSystem.port = 5672
grouper.messaging.system.activemqSystem.defaultPageSize = 10
grouper.messaging.system.activemqSystem.maxPageSize = 50
 
=======================
 

Looking at the code in github, it appears that assignAutocreateObjects method is never called on the GrouperMessageSendParam object in the code path that my configuration is using.   In EsbMessagingPublisher:

 

 46   public boolean dispatchEvent(String eventJsonString, String consumerName) {
 47     String messagingSystemName = GrouperLoaderConfig.retrieveConfig()
 48         .propertyValueString("changeLog.consumer."
 49             + consumerName + ".publisher.messagingSystemName", "grouperBuiltinMessaging");
 50     String queueOrTopicName = GrouperLoaderConfig.retrieveConfig().propertyValueString(
 51         "changeLog.consumer."
 52             + consumerName + ".publisher.queueOrTopicName");
 53 
 54     String messageQueueType = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."
 55         + consumerName + ".publisher.messageQueueType");
 56     GrouperMessageQueueType grouperMessageQueueType = GrouperMessageQueueType.valueOfIgnoreCase(messageQueueType, true);
 57 
 58     GrouperMessagingEngine.send(new GrouperMessageSendParam()
 59         .assignGrouperMessageSystemName(messagingSystemName)
 60         .assignQueueType(grouperMessageQueueType)
 61         .addMessageBody(eventJsonString).assignQueueOrTopicName(queueOrTopicName));
 62     return true;
 63   }
 

However, in the ChangeLogConsumerToMessage class file, it pulls the loader.messaging.settings.autocreate.objects property and adds it to Params being generated in the send() method:

 

 33   public long processChangeLogEntries(List<ChangeLogEntry> changeLogEntryList,
 34       ChangeLogProcessorMetadata changeLogProcessorMetadata) {
 35 
 36     long lastProcessed = -1;
 37 
 38     String messagingSystemName = GrouperLoaderConfig.retrieveConfig().propertyValueString("changeLog.consumer."
 39         + changeLogProcessorMetadata.getConsumerName() + ".messagingSystemName");
 40     String queueOrTopicName = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."
 41         + changeLogProcessorMetadata.getConsumerName() + ".queueOrTopicName");
 42     String messageQueueType = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."
 43         + changeLogProcessorMetadata.getConsumerName() + ".messageQueueType");
 44     GrouperMessageQueueType grouperMessageQueueType = GrouperMessageQueueType.valueOfIgnoreCase(messageQueueType, true);
 45 
 46     boolean autocreateObjects = GrouperLoaderConfig.retrieveConfig().propertyValueBoolean("loader.messaging.settings.autocreate.objects", true);
 47 
 48     for (ChangeLogEntry changeLogEntry : GrouperUtil.nonNull(changeLogEntryList)) {
 49 
 50       try {
 51         String json = changeLogEntry.toJson(true);
 52         GrouperMessagingEngine.send(new GrouperMessageSendParam().assignGrouperMessageSystemName(messagingSystemName)
 53             .assignAutocreateObjects(autocreateObjects)
 54             .assignQueueType(grouperMessageQueueType).assignQueueOrTopicName(queueOrTopicName).addMessageBody(json));
 55         lastProcessed = changeLogEntry.getSequenceNumber();
 56       } catch (Exception e) {
 57         LOG.error("Error processing event: " + changeLogEntry.getId(), e);
 58         changeLogProcessorMetadata.registerProblem(e, "Error processing event: " + changeLogEntry.getId() + ", " + e.getMessage(), changeLogEntry.getSequenceNumber());
 59         break;
 60       }
 61     }
 62     return lastProcessed;
 63 
 64   }
 
 

So, I'm wondering if perhaps I didn't set something up right, and I'm using a code path that is not compatible with activemq...   Or, maybe this is a bug (and if so, where would be the proper place to submit the bug)?

 

Thanks,

 

-daniel




Archive powered by MHonArc 2.6.19.

Top of Page