Grouper Users - Open Discussion List

[Daniel Hutchison <>]

Hey All,

I'm a long time lurker, but first time posting here.   I recently upgraded to 2.3 to setup the new box group provisioner.  As part of this upgrade, I'm trying to switch to using the native aqmp activemq connector (I was using the http post method before for my default membership consumer).

However, I am getting the following exception when the change log tries to send a new event:

2018-04-06 12:34:52,802: [DefaultQuartzScheduler_Worker-2] ERROR EsbConsumer.processChangeLogEntries(551) -  - problem

java.lang.IllegalArgumentException: For ActiveMQ, autoCreateObjects has to be set to true.

	at edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem.validate(GrouperMessagingActiveMQSystem.java:219)

	at edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem.send(GrouperMessagingActiveMQSystem.java:61)

	at edu.internet2.middleware.grouperClient.messaging.GrouperMessagingEngine.send(GrouperMessagingEngine.java:60)

	at edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher.dispatchEvent(EsbMessagingPublisher.java:58)

	at edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer.processChangeLogEntries(EsbConsumer.java:525)

	at edu.internet2.middleware.grouper.changeLog.ChangeLogHelper.processRecords(ChangeLogHelper.java:245)

	at edu.internet2.middleware.grouper.app.loader.GrouperLoaderType$5.runJob(GrouperLoaderType.java:640)

	at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.runJob(GrouperLoaderJob.java:465)

	at edu.internet2.middleware.grouper.app.loader.GrouperLoaderJob.execute(GrouperLoaderJob.java:345)

	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Here is what my grouper-loader.properties looks like:

=======================

loader.messaging.settings.autocreate.objects = true


changeLog.consumer.esb.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer

changeLog.consumer.esb.publisher.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher

changeLog.consumer.esb.publisher.messagingSystemName = activemq

changeLog.consumer.esb.publisher.messageQueueType = topic

changeLog.consumer.esb.publisher.queueOrTopicName = IAM.account.membership

#run every 30 secs

changeLog.consumer.esb.quartzCron = 0,30 * * * * ?

changeLog.consumer.esb.elfilter = event.eventType eq 'GROUP_DELETE' || event.eventType eq 'GROUP_ADD' || event.eventType eq 'MEMBERSHIP_DELETE' || event.eventType eq 'MEMBERSHIP_ADD'



changeLog.consumer.boxEsb.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer

changeLog.consumer.boxEsb.publisher.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher

changeLog.consumer.boxEsb.quartzCron = 0,30 * * * * ?

# carefully adjust this filter e.g. for sourceId and groupName

changeLog.consumer.boxEsb.elfilter = (event.sourceId == null || event.sourceId eq 'jdbc') && (event.groupName =~ '^svc:box:groups:.*$' || event.groupName eq 'svc:box:boxUser' || event.name =~ '^svc:box:groups:.*$' || event.name eq 'svc:box:boxUser') && (event.eventType eq 'GROUP_DELETE' || event.eventType eq 'GROUP_ADD' || event.eventType eq 'GROUP_UPDATE' || event.eventType eq 'MEMBERSHIP_DELETE' || event.eventType eq 'MEMBERSHIP_ADD' || event.eventType eq 'MEMBERSHIP_UPDATE')

changeLog.consumer.boxEsb.publisher.messagingSystemName = activemq

changeLog.consumer.boxEsb.publisher.messageQueueType = topic

changeLog.consumer.boxEsb.publisher.queueOrTopicName = IAM.box.membership

# this is optional if not using "id" for subjectId, need to be a subject attribute in the sources.xml

changeLog.consumer.boxEsb.publisher.addSubjectAttributes = email

=======================

And the grouper.client.properties:

=======================

loader.messaging.settings.autocreate.objects = true


# name of messaging system which is the default

grouper.messaging.default.name.of.messaging.system = activemq


# name of a messaging system, required

grouper.messaging.system.activemq.name = activemq

grouper.messaging.system.activemq.defaultSystemName = activemqSystem

grouper.messaging.system.activemq.username = USERNAME

grouper.messaging.system.activemq.password = PASSWORD


# Default settings for activemq (can use as base to change userids)

grouper.messaging.system.activemqSystem.name = activemqSystem

grouper.messaging.system.activemqSystem.class = edu.internet2.middleware.grouperMessagingActiveMQ.GrouperMessagingActiveMQSystem

grouper.messaging.system.activemqSystem.host = HOSTNAME

grouper.messaging.system.activemqSystem.port = 5672

grouper.messaging.system.activemqSystem.defaultPageSize = 10

grouper.messaging.system.activemqSystem.maxPageSize = 50


=======================

Looking at the code in github, it appears that assignAutocreateObjects method is never called on the GrouperMessageSendParam object in the code path that my configuration is using.   In EsbMessagingPublisher:

 46   public boolean dispatchEvent(String eventJsonString, String consumerName) {

 47     String messagingSystemName = GrouperLoaderConfig.retrieveConfig()

 48         .propertyValueString("changeLog.consumer."

 49             + consumerName + ".publisher.messagingSystemName", "grouperBuiltinMessaging");

 50     String queueOrTopicName = GrouperLoaderConfig.retrieveConfig().propertyValueString(

 51         "changeLog.consumer."

 52             + consumerName + ".publisher.queueOrTopicName");

 53 

 54     String messageQueueType = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."

 55         + consumerName + ".publisher.messageQueueType");

 56     GrouperMessageQueueType grouperMessageQueueType = GrouperMessageQueueType.valueOfIgnoreCase(messageQueueType, true);

 57 

 58     GrouperMessagingEngine.send(new GrouperMessageSendParam()

 59         .assignGrouperMessageSystemName(messagingSystemName)

 60         .assignQueueType(grouperMessageQueueType)

 61         .addMessageBody(eventJsonString).assignQueueOrTopicName(queueOrTopicName));

 62     return true;

 63   }

However, in the ChangeLogConsumerToMessage class file, it pulls the loader.messaging.settings.autocreate.objects property and adds it to Params being generated in the send() method:

 33   public long processChangeLogEntries(List<ChangeLogEntry> changeLogEntryList,

 34       ChangeLogProcessorMetadata changeLogProcessorMetadata) {

 35 

 36     long lastProcessed = -1;

 37 

 38     String messagingSystemName = GrouperLoaderConfig.retrieveConfig().propertyValueString("changeLog.consumer."

 39         + changeLogProcessorMetadata.getConsumerName() + ".messagingSystemName");

 40     String queueOrTopicName = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."

 41         + changeLogProcessorMetadata.getConsumerName() + ".queueOrTopicName");

 42     String messageQueueType = GrouperLoaderConfig.retrieveConfig().propertyValueStringRequired("changeLog.consumer."

 43         + changeLogProcessorMetadata.getConsumerName() + ".messageQueueType");

 44     GrouperMessageQueueType grouperMessageQueueType = GrouperMessageQueueType.valueOfIgnoreCase(messageQueueType, true);

 45 

 46     boolean autocreateObjects = GrouperLoaderConfig.retrieveConfig().propertyValueBoolean("loader.messaging.settings.autocreate.objects", true);

 47 

 48     for (ChangeLogEntry changeLogEntry : GrouperUtil.nonNull(changeLogEntryList)) {

 49 

 50       try {

 51         String json = changeLogEntry.toJson(true);

 52         GrouperMessagingEngine.send(new GrouperMessageSendParam().assignGrouperMessageSystemName(messagingSystemName)

 53             .assignAutocreateObjects(autocreateObjects)

 54             .assignQueueType(grouperMessageQueueType).assignQueueOrTopicName(queueOrTopicName).addMessageBody(json));

 55         lastProcessed = changeLogEntry.getSequenceNumber();

 56       } catch (Exception e) {

 57         LOG.error("Error processing event: " + changeLogEntry.getId(), e);

 58         changeLogProcessorMetadata.registerProblem(e, "Error processing event: " + changeLogEntry.getId() + ", " + e.getMessage(), changeLogEntry.getSequenceNumber());

 59         break;

 60       }

 61     }

 62     return lastProcessed;

 63 

 64   }

So, I'm wondering if perhaps I didn't set something up right, and I'm using a code path that is not compatible with activemq...   Or, maybe this is a bug (and if so, where would be the proper place to submit the bug)?

Thanks,

-daniel