Grouper Developers Forum

[Chris Hyzer <>]

Im done with the discovery client in grouper 2.1

 

https://spaces.internet2.edu/display/Grouper/Grouper+discovery+client

 

this is for the always-available web service client.

 

Let me know any comments…

 

The Grouper discovery client is a way to discovery information about services using a failsafe cache.  This is available in Grouper v2.1+ in grouperClient.jar.  It simply downloads a file from a web server.  The file can be a properties file that holds information about the URLs and other metadata about a service.  There should be multiple discovery URLs, and it uses the GrouperClient failover client so that communication is always available if there are no shared components of the multiple discovery web servers.  However, if there is only one server, and it has been downloaded once in the past, then it will be in a failsafe cache so that it will use that information instead of failing.

This client will be used for the GrouperClient WS and GrouperClient LDAP.  It can used as a command line utility or Java library for other non-Grouper services.

Configuring

These settings in the grouper.client.properties configure the discovery client:

# path of a writable directory where files can be created or stored

# for example, cache of discovery configuration, or failover state

# dot is the current directory...  note, this directory must exist

# or it will be created (attempted)

# if this is blank, none of these features will be used, and

# no files will be saved

grouperClient.cacheDirectory = .

########################################

## Service discovery settings

########################################

# number of minutes to remember that a connection had errors

grouperClient.minutesToKeepErrors = 2

# if you are using a discovery service, but a discovery properties

# at a URL (preferably SSL with valid certificate)

# you should have multiple discovery URLs hosted at independent locations

# to add more, increment the integer

grouperClient.urlOfDiscovery.0 = https://firstserver.school.edu/path/

grouperClient.urlOfDiscovery.1 = https://secondserver.school.edu/path/

# if your app has a slow startup time, and the initial connections are timing out

# esp if you arent just using the command line client (e.g. if using it as a jar),

# then add more time here

grouperClient.secondsForClassesToLoad = 20

# this will save the failover state to a file so if the JVM is stopped, it

# will be there when it starts again.  

# Set to 0 to store on every use (recommended if used command line)

# or set to -1 to not store or read ever

# grouperClient.cacheDirectory must be set

grouperClient.saveFailoverStateEverySeconds = 60

# this will cache the discovery properties in memory or on disk

# if you want to cache the discovery properties locally, put a directory here:

# this is recommended especially if you are using the grouper client as a command

# line application and the process is constantly restarting

# note, this will be used for a failsafe cache if all discovery servers are unavailable

# grouperClient.cacheDirectory must be set

# set to 0 or -1 to not cache

grouperClient.cacheDiscoveryPropertiesForSeconds = 120

Using as a Java library

Just include the grouperClient.jar and grouper.client.properties in the classpath, and call like this:

boolean throwExceptionIfNotFound = true;

File configFile = edu.internet2.middleware.grouperClient.discovery.DiscoveryClient.retrieveFile("myConfigFile.properties", throwExceptionIfNotFound);

Note, if the URLs in the grouper.client.properties are:

https://firstserver.school.edu/path/https://secondserver.school.edu/path/

then the path of the file downloaded from the web server would be

https://firstserver.school.edu/path/myConfigFile.properties

-or-

https://secondserver.school.edu/path/myConfigFile.properties

depending on which server is available and responds successfully.  See the FailoverClient documentation for more information about how this works

Using as a command line utility

C:\mchyzer\grouper\trunk\grouperClient\dist>java -cp grouperClient.jar edu.internet2.middleware.grouperClient.discovery.DiscoveryClient --operation=getFile --fileName=myConfigFile.properties

C:\mchyzer\grouper\trunk\grouperClient\dist\myConfigFile_20120208_141052_088_Q8O0R25L.properties

C:\mchyzer\grouper\trunk\grouperClient\dist>

See the Java library section above for more information, it just calls that method and will fail if not there.

How it works

Not that the file returned does not have the same name as the file requested, this is so that multiple processes can access the discovery client at the same time and never have contention.

1. First see if the file is in the memory expirable cache.  If so, use it
2. Next, if file not found yet, check the file system cache directory, for all files that match this name.  Note, there can be directories on the name, though all cache files are in the same parent directory.  If there is a file in the directory, then get the newest one, and if it is new enough to be expired by the setting grouperClient.cacheDiscoveryPropertiesForSeconds, then cache it and use it
3. Next, if file not found yet, try the URLs in the config file, one then the other.  If there are no URLs throw an exception.  Save the file to the filesystem with a temporary name, e.g. myConfigFile_20120208_141052_088_Q8O0R25L.properties.discoverytmp, then when done downloading, rename to myConfigFile_20120208_141052_088_Q8O0R25L.properties.  This file has the date, time, millis, random unique string in it.  Add the file to cache and use it.
4. Next, if file not found yet, use the latest file on the file system, add to cache, and use it (failsafe part)
5. If nothing found throw exception

Note: you cannot rename or edit the file returned from the DiscoveryClient since it will be reused by subsequent calls until it is expired.

Example

Get a couple of web servers, and add a text file.  For example:

http://www.internet2.edu/grouper/temp/temp.txt

https://grouperdemo.internet2.edu/temp/temp.txt

Then set the base URLS in the grouper.client.properties:

grouperClient.urlOfDiscovery.0 = https://grouperdemo.internet2.edu/temp/

grouperClient.urlOfDiscovery.1 = http://www.internet2.edu/grouper/temp/

Then request the file, e.g. from command line:

C:\mchyzer\grouper\trunk\grouperClient\dist>java -cp grouperClient.jar edu.internet2.middleware.grouperClient.discovery.DiscoveryClient --operation=getFile --fileName=temp.txt

C:\mchyzer\grouper\trunk\grouperClient\dist\temp_20120208_145738_982_Q8O2F8SB.txt

C:\mchyzer\grouper\trunk\grouperClient\dist>

Open that file on the local machine and make sure the contents are correct:

C:\mchyzer\grouper\trunk\grouperClient\dist>more C:\mchyzer\grouper\trunk\grouperClient\dist\temp_20120208_145738_982_Q8O2F8SB.txt

Hey, this is my file, internet2.edu

That came from internet2.  Change the grouper.client.properties to make the internet2 URL wrong, and rename the result to be too old, and try again.  Note, there is a stderr stack, but it is successful and the stdout still has the file contents ok

C:\mchyzer\grouper\trunk\grouperClient\dist>ren temp_20120208_145738_982_Q8O2F8SB.txt temp_20120208_145238_982_Q8O2F8SB.txt

grouperClient.urlOfDiscovery.0 = https://grouperdemo.internet2.edu/temp/

grouperClient.urlOfDiscovery.1 = http://www.internet2.edu/grouper/temp2/

C:\mchyzer\grouper\trunk\grouperClient\dist>java -cp grouperClient.jar edu.internet2.middleware.grouperClient.discovery.DiscoveryClient --operation=getFile --fileName=temp.txt

Feb 8, 2012 3:03:23 PM edu.internet2.middleware.grouperClient.failover.FailoverClient internal_failoverLogic

WARNING: NON-FATAL error in failover connection: http://www.internet2.edu/grouper/temp2/, will try others in pool (1 tot

al errors, 0 total timeouts, 0 total fatal errors): [THIS STACK]: java.lang.RuntimeException: this stack

        at edu.internet2.middleware.grouperClient.failover.FailoverClient.internal_failoverLogic(FailoverClient.java:721)

        at edu.internet2.middleware.grouperClient.failover.FailoverClient.failoverLogic(FailoverClient.java:521)

        at edu.internet2.middleware.grouperClient.failover.FailoverClient.failoverLogic(FailoverClient.java:503)

        at edu.internet2.middleware.grouperClient.discovery.DiscoveryClient.retrieveFileFromDiscoveryServer(DiscoveryClient.java:614)

        at edu.internet2.middleware.grouperClient.discovery.DiscoveryClient.retrieveFile(DiscoveryClient.java:308)

        at edu.internet2.middleware.grouperClient.discovery.DiscoveryClient.main(DiscoveryClient.java:130)

java.lang.RuntimeException: Problem with url: http://www.internet2.edu/grouper/temp2/temp.txt, and local file: C:\mchyzer\grouper\trunk\grouperClient\dist\temp_20120208_150615_695_Q8O2RBNU.txt.discoverytmp

        at edu.internet2.middleware.grouperClient.discovery.DiscoveryClient$1.logic(DiscoveryClient.java:755)

        at edu.internet2.middleware.grouperClient.discovery.DiscoveryClient$1.logic(DiscoveryClient.java:614)

        at edu.internet2.middleware.grouperClient.failover.FailoverClient$1.call(FailoverClient.java:631)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:619)

Caused by: java.lang.RuntimeException: Expected 200, but received response code: 404

        at edu.internet2.middleware.grouperClient.discovery.DiscoveryClient$1.logic(DiscoveryClient.java:723)

        ... 7 more

C:\mchyzer\grouper\trunk\grouperClient\dist\temp_20120208_150323_221_Q8O2NMKU.txt

C:\mchyzer\grouper\trunk\grouperClient\dist>more temp_20120208_150323_221_Q8O2NMKU.txt

This is my file, from grouper build server

If you rename one to be 20 minutes old the old files will not be automatically deleted yet, however, rename two to be more than 20 minutes old, and it will automatically delete the ones more than 20 minutes old except the newest one over 20 minutes old.

C:\mchyzer\grouper\trunk\grouperClient\dist>dir temp*

 Volume in drive C is OS

 Volume Serial Number is 4651-8CE1

 Directory of C:\mchyzer\grouper\trunk\grouperClient\dist

02/08/2012  02:10 PM                35 temp_20120208_141052_088_Q8O0R25L.txt

02/08/2012  02:57 PM                35 temp_20120208_145238_982_Q8O2F8SB.txt

02/08/2012  03:03 PM                42 temp_20120208_150323_221_Q8O2NMKU.txt

               3 File(s)            112 bytes

               0 Dir(s)  270,957,670,400 bytes free

C:\mchyzer\grouper\trunk\grouperClient\dist>ren temp_20120208_145238_982_Q8O2F8SB.txt temp_20120208_143238_982_Q8O2F8SB.

txt

C:\mchyzer\grouper\trunk\grouperClient\dist>dir temp*

 Volume in drive C is OS

 Volume Serial Number is 4651-8CE1

 Directory of C:\mchyzer\grouper\trunk\grouperClient\dist

02/08/2012  02:10 PM                35 temp_20120208_141052_088_Q8O0R25L.txt

02/08/2012  02:57 PM                35 temp_20120208_143238_982_Q8O2F8SB.txt

02/08/2012  03:03 PM                42 temp_20120208_150323_221_Q8O2NMKU.txt

               3 File(s)            112 bytes

               0 Dir(s)  270,956,908,544 bytes free

C:\mchyzer\grouper\trunk\grouperClient\dist>java -cp grouperClient.jar edu.internet2.middleware.grouperClient.discovery.DiscoveryClient --operation=getFile --fileName=temp.txt

C:\mchyzer\grouper\trunk\grouperClient\dist\temp_20120208_150615_884_Q8O2RBNV.txt

C:\mchyzer\grouper\trunk\grouperClient\dist>dir temp*

 Volume in drive C is OS

 Volume Serial Number is 4651-8CE1

 Directory of C:\mchyzer\grouper\trunk\grouperClient\dist

02/08/2012  02:57 PM                35 temp_20120208_143238_982_Q8O2F8SB.txt

02/08/2012  03:03 PM                42 temp_20120208_150323_221_Q8O2NMKU.txt

02/08/2012  03:06 PM                42 temp_20120208_150615_884_Q8O2RBNV.txt

               3 File(s)            119 bytes

               0 Dir(s)  270,956,711,936 bytes free

C:\mchyzer\grouper\trunk\grouperClient\dist>

Note this file was deleted: temp_20120208_141052_088_Q8O0R25L.txt, since it was 20 minutes old, and the one from 14:32.

To do

• Add basic authn for discovery
• Add in an end date for discovery information so things can be transitioned in the future?