Grouper Users - Open Discussion List

[Keith Hazelton <>]

Non-rhetorical question: Is there a reason NOT to go with UTF-8 in this 
situation?  In general it gives more consistent and broader coverage of 
non-English character sets
          --Keith
______________

On Mar 29, 2011, at 08:10, Chris Hyzer wrote:

> Fyi when I query my DB command line or in sqlyog, I don't see what I 
> expect, but Im not sure why, Grouper shows the right thing on the screen...
> 
> mysql> select name from grouper_groups where name like 'test:%';
> +----------------+
> | name           |
> +----------------+
> | test:tást     | 
> | test:testGroup | 
> +----------------+
> 2 rows in set (0.00 sec)
> 
> -----Original Message-----
> From: Richard James 
> [mailto:]
>  
> Sent: Tuesday, March 29, 2011 6:53 AM
> To: Chris Hyzer; 
> 
> Subject: RE: Foreign characters not displaying correctly in user interfaces
> 
> Hi Chris,
> 
> We are running this on a Unix machine. Yep that is the main problem we have 
> had, trying to identify at what stage the characters aren't being 
> interpreted correctly.
> 
> Currently our character variables show up as follows;
> 
> mysql> SHOW VARIABLES LIKE 'char%';
> +--------------------------+----------------------------+
> | Variable_name            | Value                      |
> +--------------------------+----------------------------+
> | character_set_client     | latin1                     |
> | character_set_connection | latin1                     |
> | character_set_database   | latin1                     |
> | character_set_filesystem | binary                     |
> | character_set_results    | latin1                     |
> | character_set_server     | latin1                     |
> | character_set_system     | utf8                       |
> | character_sets_dir       | /usr/share/mysql/charsets/ |
> +--------------------------+----------------------------+
> 8 rows in set (0.00 sec)
> 
> It may be the case that these need to be utf8, although when querying the 
> database via the command line or PhpMyAdmin the accented characters are 
> displayed correctly. I will try to change these on our test box to see if 
> it makes any difference. (The local variables shown on PhpMyAdmin are set 
> to UTF-8). 
> 
> It's the subject where we are coming across the problem, we load subject 
> details into a subject table in the grouper database, when loading this 
> data we have tried loading it using ISO-8859-1 and UTF-8 but this doesn't 
> seem to change anything. The table has been setup so that it is using the 
> utf8_unicode_ci collation. I was wondering if it might be when the JDBC 
> connector for the subject sources connects to the database that it isn't 
> using the correct char encoding? Just a guess not sure about the internals.
> 
> This might well be a separate issue from Grouper, I will keep on having a 
> look into it and feedback any possible solutions I come across.
> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
> 
> 
> Regards
> 
> Richard James
> Infrastructure Systems Administrator
> ISS Systems Architecture
> Newcastle University
> 
> 
>> -----Original Message-----
>> From: Chris Hyzer 
>> [mailto:]
>> Sent: 28 March 2011 17:33
>> To: Richard James; 
>> 
>> Subject: RE: Foreign characters not displaying correctly in user
>> interfaces
>> 
>> Are you running the server on unix or windows?  Ive seen difficulties in
>> windows if it is setup a certain way to support these chars.
>> Unfortunately there are a lot of factors here: database OS, app server
>> OS, database charset, app server OS, browser, etc
>> 
>> 
>> Does this show utf-8 on all lines that it should?  (mine doesn't, does
>> yours?)
>> 
>> mysql> SHOW VARIABLES LIKE 'char%';
>> +--------------------------+----------------------------+
>> | Variable_name            | Value                      |
>> +--------------------------+----------------------------+
>> | character_set_client     | utf8                       |
>> | character_set_connection | utf8                       |
>> | character_set_database   | latin1                     |
>> | character_set_filesystem | binary                     |
>> | character_set_results    | utf8                       |
>> | character_set_server     | latin1                     |
>> | character_set_system     | utf8                       |
>> | character_sets_dir       | /usr/share/mysql/charsets/ |
>> +--------------------------+----------------------------+
>> 8 rows in set (0.00 sec)
>> 
>> 
>> SHOW VARIABLES LIKE 'char%';
>> 
>> 
>> 
>> The grouperdemo server is on unix, and I didn't do anything special do
>> it, and it seems fine.  I created a group with a special char, and
>> attached are the screenshots for the UI and UI-lite with a windows
>> browser (unix server/db mysql)...
>> 
>> Thanks,
>> Chris
>> 
>> 
>> 
>> 
>> -----Original Message-----
>> From: 
>> 
>>  [
>> ]
>>  On Behalf Of Richard James
>> Sent: Monday, March 28, 2011 12:04 PM
>> To: 
>> 
>> Subject: [grouper-users] Foreign characters not displaying correctly in
>> user interfaces
>> 
>> Hi,
>> 
>> We've recently come across a problem with foreign characters (most
>> notably accented characters) not displaying correctly in the main
>> Grouper UI and the Lite UI. It replaces these characters with a question
>> mark as shown in the attached screenshot. I saw a post in the mailing
>> list from February last year
>> (https://lists.internet2.edu/sympa/arc/grouper-users/2010-
>> 02/msg00012.html) which discussed possible solutions to rectify the
>> problem, unfortunately we have not been able to come up with a solution.
>> I have tried amending the following files to use UTF-8, unfortunately
>> although the changes are reflected in the source code and in the browser
>> meta data, the character is still displayed incorrectly.
>> 
>> edu.internet2.middleware.grouper.ui.util.HttpContentType
>> grouper.html
>> head.jsp
>> template.jsp
>> 
>> The characters are displayed correctly in the database (mysql), the
>> collation for the column is set to utf8_unicode_ci. I've tried
>> explicitly setting a LANG environment variable to en_GB.UTF-8 on the
>> server, and also in the tomcat start-up script in case there was a
>> conflict at the end, but again to no avail.
>> 
>> A few little tests I tried to test behaviour was to, via the main UI,
>> change a name of group to include an accented character, i.e. tást. This
>> displays correctly in the UI, but if you look in the database it is
>> represented by tást.
>> 
>> I was wondering if anyone else has come across this issue? And if so if
>> can you suggest how we can overcome this problem.
>> 
>> Unfortunately within the subject search in the lite UI, the accented
>> character does also cause an xml load error (screenshot attached). This
>> happens when searching for a user where the name contains an foreign
>> character.
>> 
>> Thanks
>> 
>> Richie
>> 
>> 
>> Richard James
>> Infrastructure Systems Administrator
>> ISS Systems Architecture
>> Newcastle University
>> 
>