Grouper Users - Open Discussion List

[Beth Halsema <>]

Oops.. I just had to be patient.  Microsoft's audit log and group update appear to be a little slow.  

That did it!  Now pondering the best way to manage the workaround for the course groups.  :)

Thank you!   This has been a good week!

Beth

-------------------------------------------------------------------------------------------------------------
Beth A. Halsema - M.S. in Computer Science, GSEC
Sr. Software Engineer, Identity & Access Management
OVPIT - IT Security and Policy

www.itap.purdue.edu

On Fri, Jul 24, 2020 at 5:35 PM Beth Halsema <> wrote:

You are correct.  There is no such attribute associated with the group that was created.  

The attempt to backfill the group's object ID didn't work, but that is not to say that it won't.  Still experimenting...

Thank you!

Beth

-------------------------------------------------------------------------------------------------------------
Beth A. Halsema - M.S. in Computer Science, GSEC
Sr. Software Engineer, Identity & Access Management
OVPIT - IT Security and Policy

www.itap.purdue.edu

On Fri, Jul 24, 2020 at 5:06 PM Black, Carey M. <> wrote:

Beth,

 

I did not look down that road… but a quick scan of the CLC….

 

After a group is added in o365, then an attribute should be set on the group with an identifier from o365.

My guess is that the “boom” happens before that get’s a chance to be attempted.

https://github.com/Internet2/grouper/blob/6f349441e4272b8c17ece2e2fec4c15b9ff85677/grouper-misc/grouper-azure/src/main/java/edu/internet2/middleware/grouper/changeLog/consumer/Office365ChangeLogConsumer.java#L247

 

So… if you look at the group in Grouper ( after the CLC creates the group in O365…) and look at the “More actions” à “Attribute Assignments” and see if the attribute “etc:attribute:office365:o365Id" is on the group. ( I am guessing “no”.)

 

And … when I look at “addMembership” … It uses the attribute value to add members. So.. No attribute value then no memberships  can be added.

 

 

 

On the other hand…

   If you backfill that attribute assignment and value “by hand” then you might get it to provision memberships. ( for a very limited POC  till the bug get’s fixed.)

 

--

Carey Matthew

 

From: Beth Halsema <>
Sent: Friday, July 24, 2020 4:52 PM
To: Black, Carey M. <>
Cc: Grouper Users <>
Subject: Re: [grouper-users] New Office 365 Consumer Log and Unified Groups

 

Carey,

 

Your findings match mine.  Since the groups are being created in Office 365, I think that the initial operation, the addGroup(),  succeeds.  The error is being generated as the result of the response from Microsoft Graph API.

 

Would that failure then result in the subsequent operations  for those groups (group members additions/deletions and the group removal) not even be attempted for those groups? 

 

I am starting to wander through the code to get a better feel for the design and expected behaviors.

 

Thanks again and have a great weekend!

 

Beth

-------------------------------------------------------------------------------------------------------------
Beth A. Halsema - M.S. in Computer Science, GSEC
Sr. Software Engineer, Identity & Access Management
OVPIT - IT Security and Policy

www.itap.purdue.edu

 

 

On Fri, Jul 24, 2020 at 4:30 PM Black, Carey M. <> wrote:

Beth,

 

Not much of an answer.. but I find it unfortunately "normal" to see stuff like this in "API"s....  ( Sloppy docs, and/or sloppy API's...)

  https://docs.microsoft.com/en-us/search/?scope=graph&view=graph-rest-1.0&terms=HiddenMembership

                There are 3 separate forms of "HiddenMembership" in those M$ docs.

               

                "HiddenMembership" : https://docs.microsoft.com/en-us/graph/api/group-post-groups

                "Hiddenmembership" : https://docs.microsoft.com/en-us/graph/api/resources/group

                "hiddenMembership" : https://docs.microsoft.com/en-us/graph/changelog

 

 

The error you are reporting looks like a JSON parser error to my eye.

                But I don't know (100% certain) if that is before the client sent a string, or after the client got a message back from M$.

 

Based on a quick read of the Grouper code it looks like it only is using the "Hiddenmembership" form ( in edu.internet2.middleware.grouper.changeLog.consumer.o365.model.Group ).

 

So that string likely is coming back from o365 and not matching the "Hiddenmembership".

But my read of the official o365 docs seems to suggest that "Hiddenmembership" is the correct string. So maybe they are sending back an invalid value?

 

 

And to be fair and complete...

                https://spaces.at.internet2.edu/display/Grouper/Grouper+Azure+provisioner

                                has: "public private, and hiddenmembership", "[Public* | Private | Hiddenmembership]" and "Private, Public, HiddenMembership,".

 

 

So yea... The defintion is "clear as mud" for the correct value for that string. 😊

 

--

Carey Matthew

 

-----Original Message-----
From: <> On Behalf Of Beth Halsema
Sent: Friday, July 24, 2020 3:47 PM
To:
Subject: [grouper-users] New Office 365 Consumer Log and Unified Groups

 

 

Our team is attempting to use the new Office 365 consumer log to maintain

course groups in Office 365.

 

We had successfully done so using the security groups; however, we wanted

to use the "Hiddenmembership" Unified groups in order to comply with

FERPA regulations and support the collaboration team's need to create

Microsoft Teams from these groups.

 

We modified the values in the grouper-loader.properties file:

 

changeLog.consumer.o365.class = edu.internet2.middleware.grouper.changeLog.consumer.Office365ChangeLogConsumer

# fire every 5 seconds

changeLog.consumer.o365.quartzCron = 0,5,10,15,20,25,30,35,40,45,50,55 * * * * ?

changeLog.consumer.o365.syncAttributeName = etc:attribute:office365:o365Sync

changeLog.consumer.o365.retryOnError = true

changeLog.consumer.o365.tenantId = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

changeLog.consumer.o365.clientId = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

changeLog.consumer.o365.clientSecret = :)

changeLog.consumer.o365.domain = domain.purdue.edu

changeLog.consumer.o365.idAttribute = uid

#changeLog.consumer.o365.upnAttribute =

changeLog.consumer.o365.groupJexl = group.name.replaceAll("^app:office365:groups:courses:service:policy:","GROUPER-courses-")

#changeLog.consumer.o365.mailNicknameJexl =

#changeLog.consumer.o365.descriptionJexl =

#changeLog.consumer.o365.subjectJexl =

#changeLog.consumer.o365.groupType = [Security* | Unified]

changeLog.consumer.o365.groupType = Unified

#changeLog.consumer.o365.visibility = [Public* | Private | Hiddenmembership]  * Only works with Unified groups

changeLog.consumer.o365.visibility = Hiddenmembership

#changeLog.consumer.o365.proxyType = [http | socks]

#changeLog.consumer.o365.proxyHost =

#changeLog.consumer.o365.proxyPort =

 

I modified the file

 

                /opt/grouper/grouperWebapp/WEB-INF/classes/log4j.properties

 

in our Docker containers in order to increase the logging.

 

                log4j.logger.edu.internet2.middleware.grouper.changeLog.consumer.Office365ChangeLogConsumer = DEBUG

 

The outcome was:

 

1. The groups are created in Office 365.  According to the Office 365

   admin portal, we created Microsoft 365 HiddenMembership groups.

 

   No owners and no members.

 

   In the logs/grouper_daemon.log file, the following error is logged:

 

Did not get all the way through the batch! 1848027 != 1848101java.lang.RuntimeException: Error in loader job: null, check logs: Error: o365 threw an exception processing change log entry sequence number 1848027., sequenceNumber: 1848027, com.squareup.moshi.JsonDataException: Expected one of [Public, Private, Hiddenmembership] but was HiddenMembership at path $.visibility

                at com.squareup.moshi.StandardJsonAdapters$EnumJsonAdapter.fromJson(StandardJsonAdapters.java:258)

                at com.squareup.moshi.StandardJsonAdapters$EnumJsonAdapter.fromJson(StandardJsonAdapters.java:227)

                at com.squareup.moshi.JsonAdapter$2.fromJson(JsonAdapter.java:137)

                at com.squareup.moshi.ClassJsonAdapter$FieldBinding.read(ClassJsonAdapter.java:194)

                at com.squareup.moshi.ClassJsonAdapter.fromJson(ClassJsonAdapter.java:156)

                at com.squareup.moshi.JsonAdapter$2.fromJson(JsonAdapter.java:137)

                at retrofit2.converter.moshi.MoshiResponseBodyConverter.convert(MoshiResponseBodyConverter.java:45)

                at retrofit2.converter.moshi.MoshiResponseBodyConverter.convert(MoshiResponseBodyConverter.java:27)

                at retrofit2.OkHttpCall.parseResponse(OkHttpCall.java:225)

                at retrofit2.OkHttpCall.execute(OkHttpCall.java:188)

                at edu.internet2.middleware.grouper.changeLog.consumer.o365.GraphApiClient.invoke(GraphApiClient.java:164)

                at edu.internet2.middleware.grouper.changeLog.consumer.o365.GraphApiClient.addGroup(GraphApiClient.java:201)

                at edu.internet2.middleware.grouper.changeLog.consumer.Office365ChangeLogConsumer.addGroup(Office, threadId: 104, elapsed: 1285 ms

 

 

2. No members were ever added to the groups via changeLog.consumer.o365.

3. We were unable to delete the groups from Office 365 through their deletion

   in Grouper.

 

The groups are created but no subsequent operations on them seem to be

performed.

 

Is there something that we have overlooked?  Or is the development of support

for unified groups still a work-in-progress?

 

Thank you!

 

Beth

-------------------------------------------------------------------------

Beth A. Halsema - M.S. in Computer Science, GSEC

Sr. Sofware Engineer, Identity & Access Management

OVPIT - IT Security and Policy

https://urldefense.com/v3/__http://www.itap.purdue.edu__;!!KGKeukY!kIgRCaDOkKFEUnBHzPZEJu8pk_3ipJwriqvV0H15-kRKFWkrNDld2bNcdf2dMLDVM3E$