Grouper Users - Open Discussion List

[Chris Hyzer <>]

This is fixed.

https://bugs.internet2.edu/jira/browse/GRP-1032

Here are two commits in order 

https://github.com/Internet2/grouper/commit/a08b5cc4d48105bd1c3e19be0af6d6257c924144 

https://github.com/Internet2/grouper/commit/3b8697e6f7bb7a83d0e37faa6fa426bd4482234b

This is from the installer (new):

Do you want to set URIEncoding to UTF-8 in tomcat server.xml <Connector> elements (t|f)? [t]: 

 - adding tomcat URIEncoding attribute for element <Connector AJP with value: 'UTF-8'

 - adding tomcat URIEncoding attribute for element <Connector HTTP with value: 'UTF-8'

Thanks,

Chris

---

From: Chris Hyzer
Sent: Thursday, August 21, 2014 6:28 PM
To: Yoann Delattre;
Subject: RE: [grouper-users] Accented characters and ComboBox in New UI

Ok, thanks a lot for debugging and submitting the solution and testing this.  What I will do is:

 

1.       Open a jira

2.       Change the default on that setting

3.       Change the installer to edit the tomcat setting

4.       Add a note to the install/upgrade instruction

5.       Make a wiki about international chars, explain the solution, how to test, and make sure we test on new releases

6.       Change the demo server, make sure it works with AJP connectors

 

Will get back to you

 

For everyone installing 2.2 in the meantime, you should set this in the server.xml in tomcat:

 

<Connector connectionTimeout="20000" port="8090" protocol="HTTP/1.1" redirectPort="8443"

    URIEncoding="UTF-8"

    />

 

Add this to the grouper-ui.properties

 

convertInputToUtf8 = false

 

 

 

Thanks,

Chris

 

From: [mailto:] On Behalf Of Yoann Delattre
Sent: Thursday, August 21, 2014 4:49 AM
To: Chris Hyzer;
Subject: Re: [grouper-users] Accented characters and ComboBox in New UI

 

Yes, it works.
Because i think in server.xml tomcat file, you don't have URIEncoding="UTF-8".
But with this option it's the breadcrumb now who doesn't work...

If you click on "accentó"



You have this error :




I'l try to resume :-D

You set URIEncoding="UTF-8" in server.xml tomcat file. The breadcumbs works but not auto-completion with ComboBox.
In this case, with the getParameter method, you have a double encoding.

You don't set URIEncoding="UTF-8". Auto-completion with ComboBox works but not the breadcumb.
Without URIEncoding parameter tomcat default encoding is ISO-8859-1. So it should works, but the Content Type of the http request (when you click on accentó) is : application/x-www-form-urlencoded; charset=UTF-8. It's UTF-8 but tomcat will treat that like ISO-8859-1. Conclusion : it doesn't work.

I don't know if it's really what happened. But it should be something like that.

The best way (at least, for us), i think, it's to assume that everything is in UTF-8 and configure convertInputToUtf8 to false with URIEncoding="UTF-8" in server.xml.
Don't know if it's the best solution but, for us, it seems to be a logical solution.

Yoann Delattre	✆	03 20 95 69 61	✉
Équipe SIAD (Systèmes d'Information et Aide à la Décision) DSI de l'académie de Lille (Direction des Systèmes d'Information) 110 avenue Gaston Berger - 59000 Lille

Le 21/08/2014 00:32, Chris Hyzer a écrit :

Im trying to reproduce this on the demo server and I cannot.  Can you reproduce on the demo server?  Please let me know the exact steps.

 

Folder with accent:

 

https://grouperdemo.internet2.edu/grouper_v2_2/grouperUi/app/UiV2Main.index?operation=UiV2Stem.viewStem&stemId=ecc7d19e9b874f73b10748e73e0cff3e

 

Here is a search for an accented char:

 

 

 

 

Here is an accented folder found in the combobox:

 

 

 

 

From: Yoann Delattre []
Sent: Wednesday, August 20, 2014 11:49 AM
To: Chris Hyzer;
Subject: Re: [grouper-users] Accented characters and ComboBox in New UI

 

Hi,

i agree, i use a PostgreSQL database, there is no problem doing a basic search with accented character.
It's only when i was using ComboBox. I had no result at all. Because GET variable "name" was badly encoded using getParameters method.



In DojoComboLogic.java, with this debug code :

        if (StringUtils.isBlank(query)) {
                  LOG.debug(request.getCharacterEncoding());
                        LOG.debug(request.getParameter("name"));
          isLookup = false;

        query = StringUtils.trimToEmpty(request.getParameter("name"));
        }

i had something like this in my log :

2014-08-20 09:24:42,671: [http-8080-4] DEBUG DojoComboLogic.logic(109) -  - UTF-8
2014-08-20 09:24:42,671: [http-8080-4] DEBUG DojoComboLogic.logic(112) -  - Recherche avec :éété*

With the normal search, like this :



There was no problem :). It was only when i was using ComboBox and the auto-completion...

I don't know if it's a default option but in server.xml from tomcat6, there is this parameter who already encoded in UTF-8 all GET/POST variables :
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               redirectPort="8443" />

so in addition with getParameters method, i think the result is a double encoding. And for me, that's why the GET variable "name" was badly encoded.

I actually play with Grouper 2.2 and with convertInputToUtf8= false, i have no problem... I check frequently our database if the data is still in UTF-8 and no problem...

I don't know if my explication is easy to understand :-\

Yoann Delattre	✆	03 20 95 69 61	✉
Équipe SIAD (Systèmes d'Information et Aide à la Décision) DSI de l'académie de Lille (Direction des Systèmes d'Information) 110 avenue Gaston Berger - 59000 Lille

Le 20/08/2014 16:41, Chris Hyzer a écrit :

When I use 2.2 without any changes, I created a folder with an accented character, I am using mysql, I did a search, and it sent the accented char to the DB (from p6spy).  Mysql by default will treat that accented char as an “e” I guess…  so you get more results than you think you should, but it seems to be working, right?  If you don’t convert input to utf8, will creating objects as utf8 still work?  I think there was something that wasn’t working that it helped with J

 

 

2014/08/20 10:35:46:840, 7ms, statement: ByHqlStatic.java.listSet() line 421, Hib3StemDAO.java.getAllStemsSecureHelper() line 1533, Hib3StemDAO.java.getAllStemsSecure() line 2132, StemFinder.java.findStems() line 628, UiV2Stem.java.search() line 1484, DojoComboLogic.java.logic() line 133, UiV2Stem.java.createGroupParentFolderFilter() line 1462, GrouperUtil.java.invokeMethod() line 3951, GrouperUtil.java.callMethod() line 3902, GrouperUiRestServlet.java.doGet() line 277, GrouperUiFilter.java.doFilter() line 985

   select distinct stem0_.id as id32_, stem0_.hibernate_version_number as hibernate2_32_, stem0_.parent_stem as parent3_32_, stem0_.name as name32_, stem0_.display_name as display5_32_, stem0_.creator_id as creator6_32_, stem0_.create_time as create7_32_, stem0_.modifier_id as modifier8_32_, stem0_.modify_time as modify9_32_, stem0_.display_extension as display10_32_, stem0_.extension as extension32_, stem0_.description as descrip12_32_, stem0_.last_membership_change as last13_32_, stem0_.alternate_name as alternate14_32_, stem0_.context_id as context15_32_, stem0_.id_index as id16_32_ from grouper_stems stem0_ where lower(stem0_.name) like '%ést%' or lower(stem0_.display_name) like '%ést%' or lower(stem0_.description) like '%ést%' or stem0_.id='ést' order by stem0_.display_name asc limit 200

2014/08/20 10:35:46:870, resultset: ALIASES: 0: id32_, 1: hibernate2_32_, 2: parent3_32_, 3: name32_, 4: display5_32_, 5: creator6_32_, 6: create7_32_, 7: modifier8_32_, 8: modify9_32_, 9: display10_32_, 10: extension32_, 11: descrip12_32_, 12: last13_32_, 13: alternate14_32_, 14: context15_32_, 15: id16_32_

2014/08/20 10:35:46:871, resultset: 0: 183821004210420fade310db3a72923f, 1: 0, 2: 267b5c51292a46c1bb1a2022a7b2fc68, 3: someStem, 4: someStem, 5: dc7cb1680c3c48bab2499cd3169f6418, 6: 1408003384426, 7: null, 8: 0, 9: someStem, 10: someStem, 11: null, 12: null, 13: null, 14: d1afb6813bde439a8a1ce05ce22d8e57, 15: 10160

2014/08/20 10:35:46:876, resultset: 0: f19aaeb3580f4aa499d67567b9da5947, 1: 0, 2: 267b5c51292a46c1bb1a2022a7b2fc68, 3: someStem3, 4: someStem3, 5: dc7cb1680c3c48bab2499cd3169f6418, 6: 1408004060711, 7: null, 8: 0, 9: someStem3, 10: someStem3, 11: null, 12: null, 13: null, 14: 92ece379ee5a484cbf1d234425c35a18, 15: 10171

2014/08/20 10:35:46:879, resultset: 0: 7e2672121fe0496ea27b0cf6d679b7c0, 1: 0, 2: 267b5c51292a46c1bb1a2022a7b2fc68, 3: test, 4: test, 5: dc7cb1680c3c48bab2499cd3169f6418, 6: 1400681869188, 7: null, 8: 0, 9: test, 10: test, 11: null, 12: null, 13: null, 14: 10f9212216b94b2abbaba6565e92caed, 15: 10031

 

From: [] On Behalf Of Yoann Delattre
Sent: Wednesday, August 20, 2014 9:26 AM
To:
Subject: [grouper-users] Accented characters and ComboBox in New UI

 

Hi everyone !

Actually, i'm working to get our grouper 2.2 ready for production (thanks again for your work on this version !).
Don't know if it's usefull but i had a problem with ComboBox in New UI. When i was trying to search a group/folder who contain an accented character, i had no result...

It was an encoding problem despiste the fact that UTF-8 is use everywhere...

The problem was the getParameter method (use in DojoComboLogic.java) in GrouperRequestWrapper.java, especially those lines (408 - 415) :

      if (param != null && StringUtils.equals("GET", this.getMethod()) && GrouperUiConfig.retrieveConfig().propertyValueBoolean("convertInputToUtf8", true)) {
        try {
          byte[] bytes = param.getBytes("ISO-8859-1");
          param = new String(bytes, "UTF-8");
        } catch (Exception e) {
          throw new RuntimeException(e);
        }
      }

So i change convertInputToUtf8 to false in grouper-ui.properties and everything works now ! :-)

But if everything is in UTF-8, i don't know if this part of code is still useful ???

Maybe some people will have the same problem...That's why i share the solution :)

I hope this will help.

Yoann.

--

Yoann Delattre	✆	03 20 95 69 61	✉
Équipe SIAD (Systèmes d'Information et Aide à la Décision) DSI de l'académie de Lille (Direction des Systèmes d'Information) 110 avenue Gaston Berger - 59000 Lille