Grouper Users - Open Discussion List

["Hyzer, Chris" <>]

Are you sure that property is in the grouper-loader.properties for the loader process that is running?  i.e. you need to make sure your config files are in sync…  otherwise, yes, file a bug and we can fix it.

 

Thanks,

Chris

 

From: [mailto:] On Behalf Of Daniel Hutchison
Sent: Friday, April 06, 2018 3:37 PM
To:
Subject: [grouper-users] activemq changelog consumer bug?

 

Hey All,

 

I'm a long time lurker, but first time posting here.   I recently upgraded to 2.3 to setup the new box group provisioner.  As part of this upgrade, I'm trying to switch to using the native aqmp activemq connector (I was using the http post method before for my default membership consumer).

 

However, I am getting the following exception when the change log tries to send a new event:

 

2018-04-06 12:34:52,802: [DefaultQuartzScheduler_Worker-2] ERROR EsbConsumer.processChangeLogEntries(551) -  - problem

java.lang.IllegalArgumentException: For ActiveMQ, autoCreateObjects has to be set to true.

        at edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem.validate(GrouperMessagingActiveMQSystem.java:219)

        at edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem.send(GrouperMessagingActiveMQSystem.java:61)

        at edu.internet2.middleware.grouperClient.messaging.GrouperMessagingEngine.send(GrouperMessagingEngine.java:60)

        at edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher.dispatchEvent(EsbMessagingPublisher.java:58)

        at edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer.processChangeLogEntries(EsbConsumer.java:525)

        at edu.internet2.middleware.grouper.changeLog.ChangeLogHelper.processRecords(ChangeLogHelper.java:245)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$5.runJob(GrouperLoaderType.java:640)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:465)

        at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:345)

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Here is what my grouper-loader.properties looks like:

=======================

loader.messaging.settings.autocreate.objects = true

changeLog.consumer.esb.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer

changeLog.consumer.esb.publisher.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher

changeLog.consumer.esb.publisher.messagingSystemName = activemq

changeLog.consumer.esb.publisher.messageQueueType = topic

changeLog.consumer.esb.publisher.queueOrTopicName = IAM.account.membership

#run every 30 secs

changeLog.consumer.esb.quartzCron = 0,30 * * * * ?

changeLog.consumer.esb.elfilter = event.eventType eq 'GROUP_DELETE' || event.eventType eq 'GROUP_ADD' || event.eventType eq 'MEMBERSHIP_DELETE' || event.eventType eq 'MEMBERSHIP_ADD'

changeLog.consumer.boxEsb.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer

changeLog.consumer.boxEsb.publisher.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher

changeLog.consumer.boxEsb.quartzCron = 0,30 * * * * ?

# carefully adjust this filter e.g. for sourceId and groupName

changeLog.consumer.boxEsb.elfilter = (event.sourceId == null || event.sourceId eq 'jdbc') && (event.groupName =~ '^svc:box:groups:.*$' || event.groupName eq 'svc:box:boxUser' || event.name =~ '^svc:box:groups:.*$' || event.name eq 'svc:box:boxUser') && (event.eventType eq 'GROUP_DELETE' || event.eventType eq 'GROUP_ADD' || event.eventType eq 'GROUP_UPDATE' || event.eventType eq 'MEMBERSHIP_DELETE' || event.eventType eq 'MEMBERSHIP_ADD' || event.eventType eq 'MEMBERSHIP_UPDATE')

changeLog.consumer.boxEsb.publisher.messagingSystemName = activemq

changeLog.consumer.boxEsb.publisher.messageQueueType = topic

changeLog.consumer.boxEsb.publisher.queueOrTopicName = IAM.box.membership

# this is optional if not using "id" for subjectId, need to be a subject attribute in the sources.xml

changeLog.consumer.boxEsb.publisher.addSubjectAttributes = email

=======================

And the grouper.client.properties:

=======================

loader.messaging.settings.autocreate.objects = true

# name of messaging system which is the default

grouper.messaging.default.name.of.messaging.system = activemq

# name of a messaging system, required

grouper.messaging.system.activemq.name = activemq

grouper.messaging.system.activemq.defaultSystemName = activemqSystem

grouper.messaging.system.activemq.username = USERNAME

grouper.messaging.system.activemq.password = PASSWORD

# Default settings for activemq (can use as base to change userids)

grouper.messaging.system.activemqSystem.name = activemqSystem

grouper.messaging.system.activemqSystem.class = edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem

grouper.messaging.system.activemqSystem.host = HOSTNAME

grouper.messaging.system.activemqSystem.port = 5672

grouper.messaging.system.activemqSystem.defaultPageSize = 10

grouper.messaging.system.activemqSystem.maxPageSize = 50

=======================

Looking at the code in github, it appears that assignAutocreateObjects method is never called on the GrouperMessageSendParam object in the code path that my configuration is using.   In EsbMessagingPublisher:

 

 46   public boolean dispatchEvent(String eventJsonString, String consumerName) {

 47     String messagingSystemName = GrouperLoaderConfig.retrieveConfig()

 48         .propertyValueString("changeLog.consumer."

 49             + consumerName + ".publisher.messagingSystemName", "grouperBuiltinMessaging");

 50     String queueOrTopicName = GrouperLoaderConfig.retrieveConfig().propertyValueString(

 51         "changeLog.consumer."

 52             + consumerName + ".publisher.queueOrTopicName");

 53 

 54     String messageQueueType = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."

 55         + consumerName + ".publisher.messageQueueType");

 56     GrouperMessageQueueType grouperMessageQueueType = GrouperMessageQueueType.valueOfIgnoreCase(messageQueueType, true);

 57 

 58     GrouperMessagingEngine.send(new GrouperMessageSendParam()

 59         .assignGrouperMessageSystemName(messagingSystemName)

 60         .assignQueueType(grouperMessageQueueType)

 61         .addMessageBody(eventJsonString).assignQueueOrTopicName(queueOrTopicName));

 62     return true;

 63   }

However, in the ChangeLogConsumerToMessage class file, it pulls the loader.messaging.settings.autocreate.objects property and adds it to Params being generated in the send() method:

 

 33   public long processChangeLogEntries(List<ChangeLogEntry> changeLogEntryList,

 34       ChangeLogProcessorMetadata changeLogProcessorMetadata) {

 35 

 36     long lastProcessed = -1;

 37 

 38     String messagingSystemName = GrouperLoaderConfig.retrieveConfig().propertyValueString("changeLog.consumer."

 39         + changeLogProcessorMetadata.getConsumerName() + ".messagingSystemName");

 40     String queueOrTopicName = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."

 41         + changeLogProcessorMetadata.getConsumerName() + ".queueOrTopicName");

 42     String messageQueueType = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."

 43         + changeLogProcessorMetadata.getConsumerName() + ".messageQueueType");

 44     GrouperMessageQueueType grouperMessageQueueType = GrouperMessageQueueType.valueOfIgnoreCase(messageQueueType, true);

 45 

 46     boolean autocreateObjects = GrouperLoaderConfig.retrieveConfig().propertyValueBoolean("loader.messaging.settings.autocreate.objects", true);

 47 

 48     for (ChangeLogEntry changeLogEntry : GrouperUtil.nonNull(changeLogEntryList)) {

 49 

 50       try {

 51         String json = changeLogEntry.toJson(true);

 52         GrouperMessagingEngine.send(new GrouperMessageSendParam().assignGrouperMessageSystemName(messagingSystemName)

 53             .assignAutocreateObjects(autocreateObjects)

 54             .assignQueueType(grouperMessageQueueType).assignQueueOrTopicName(queueOrTopicName).addMessageBody(json));

 55         lastProcessed = changeLogEntry.getSequenceNumber();

 56       } catch (Exception e) {

 57         LOG.error("Error processing event: " + changeLogEntry.getId(), e);

 58         changeLogProcessorMetadata.registerProblem(e, "Error processing event: " + changeLogEntry.getId() + ", " + e.getMessage(), changeLogEntry.getSequenceNumber());

 59         break;

 60       }

 61     }

 62     return lastProcessed;

 63 

 64   }

So, I'm wondering if perhaps I didn't set something up right, and I'm using a code path that is not compatible with activemq...   Or, maybe this is a bug (and if so, where would be the proper place to submit the bug)?

 

Thanks,

 

-daniel