ZnClient and percent characters

The plus "+", dollar "$", and comma "," characters have been added to
   those in the "reserved" set, since they are treated as reserved
   within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip


Thanks.

Jimmie

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin [hidden email]:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

On 10 Jun 2015, at 17:24, David [hidden email] wrote:

El Wed, 10 Jun 2015 10:14:37 -0500
Jimmie Houchin [hidden email]
escribió:

Hello,

I am attempting to use ZnClient to request data. The request requires
a %2C (comma) delimited string as part of the query. Below is a
snippet.

znClient
        addPath: '/v1/instruments';
        queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip';
        get ;
        contents)

The string  'displayName%2Cinstrument%2Cpip'
is being converted to  'displayName%252Cinstrument%252Cpip'
which causes the request to fail.

The query needs to be
fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly.
Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to
make itself encode that? Like
znClient
         addPath: '/v1/instruments';
         queryAt: 'fields' putAll: 'displayName,instrument,pip';
         get ;
         contents)

I suspect it is using encoding internally, that is why % is also
encoded if you try to put it.

I hope that works

Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version.
Xubuntu 15.04



That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new
       https;
       host: 'google.com';
       addPath: '/commaTest';
       queryAt: 'fields' put: 'displayName,instrument,pip';
       yourself

View the  request / requestLine / uri.  The commas are still present in the URI.
So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

There is a simpler way than Norbert’s suggestion. Find the class ZnResourceMetaUtils (in the package Zinc-Resource-Meta-Core). Locate the class side method #queryKeyValueSafeSet. Remove the comma from the string. With this change your ‘Google’ example generates the query line with the ‘%2C’ encoding of the commas.

 

Of course, with this change a comma in the value field of a query will always be encoded. It is not clear to me whether this is correcting an error in Zinc, or just a hack to solve your problem – earlier posts here endorse both views. No doubt Sven is the person to sort this out. Meanwhile, this will enable you to get moving.

 

Hope this helps

 

Peter Kenny

 

From: Pharo-users [[hidden email]] On Behalf Of Norbert Hartl
Sent: 10 June 2015 23:56
To: Pharo users users
Subject: Re: [Pharo-users] ZnClient and percent characters

 

Just to clarify:

 

"Characters in the "reserved" set are not reserved in all contexts.

   The set of characters actually reserved within any given URI

   component is defined by that component. In general, a character is

   reserved if the semantics of the URI changes if the character is

   replaced with its escaped US-ASCII encoding."

If I were you I'd subclass ZnUrl and implement 

#encodeQuery:on:

on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url:

Cannot think of anything better for a quick resolve of your problem.

Norbert

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <[hidden email]>:

 

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt

The plus "+", dollar "$", and comma "," characters have been added to

   those in the "reserved" set, since they are treated as reserved

   within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip

Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin [hidden email]:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

On 10 Jun 2015, at 17:24, David [hidden email] wrote:

El Wed, 10 Jun 2015 10:14:37 -0500

Jimmie Houchin [hidden email]

escribió:

Hello,

I am attempting to use ZnClient to request data. The request requires

a %2C (comma) delimited string as part of the query. Below is a

snippet.

znClient

        addPath: '/v1/instruments';

        queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip';

        get ;

        contents)

The string  'displayName%2Cinstrument%2Cpip'

is being converted to  'displayName%252Cinstrument%252Cpip'

which causes the request to fail.

The query needs to be

fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly.

Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to

make itself encode that? Like

znClient

         addPath: '/v1/instruments';

         queryAt: 'fields' putAll: 'displayName,instrument,pip';

         get ;

         contents)

I suspect it is using encoding internally, that is why % is also

encoded if you try to put it.

I hope that works

Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version.

Xubuntu 15.04

That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new

       https;

       host: 'google.com';

       addPath: '/commaTest';

       queryAt: 'fields' put: 'displayName,instrument,pip';

       yourself

View the  request / requestLine / uri.  The commas are still present in the URI.

So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie

 

 

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded

Just to clarify:

"Characters in the "reserved" set are not reserved in all contexts.

   The set of characters actually reserved within any given URI
   component is defined by that component. In general, a character is
   reserved if the semantics of the URI changes if the character is
   replaced with its escaped US-ASCII encoding."

If I were you I'd subclass ZnUrl and implement 

#encodeQuery:on:

on that class. You could have an extension method in ZnResourceMetaUtils that returns the character set you need to have encoded. In ZnClient you just set your ZnUrl derived class object as #url:

Cannot think of anything better for a quick resolve of your problem.

Norbert

Am 11.06.2015 um 00:26 schrieb Jimmie Houchin <[hidden email]>:

I am not an expert on URIs or encoding. However, this is a requirement of the API I am using and I am required to submit an encoded URI with %2C and no commas.

As far as commas needing to be escaped, it seems from other sources that they should be.

From https://www.ietf.org/rfc/rfc2396.txt

The plus "+", dollar "$", and comma "," characters have been added to
   those in the "reserved" set, since they are treated as reserved
   within the query component.

States that commas are reserved within the query component.

http://stackoverflow.com/questions/8828702/why-is-the-comma-url-encoded

Regardless of what is or is not required, I do need the ability to have a query string with commas encoded as %2C in order to satisfy and use the API which states.

fields: Optional An URL encoded (%2C) comma separated list of instrument fields that are to be returned in the response. The instrument field will be returned regardless of the input to this query parameter. Please see the Response Parameters section below for a list of valid values.

Which will look like this or something similar.

fields=displayName%2Cinstrument%2Cpip


Thanks.

Jimmie

On 06/10/2015 03:27 PM, Norbert Hartl wrote:

That's because the comma does not need to be escaped in the query part of the uri.

Norbert

Am 10.06.2015 um 22:00 schrieb Jimmie Houchin [hidden email]:

On 06/10/2015 10:32 AM, Sven Van Caekenberghe wrote:

On 10 Jun 2015, at 17:24, David [hidden email] wrote:

El Wed, 10 Jun 2015 10:14:37 -0500
Jimmie Houchin [hidden email]
escribió:

Hello,

I am attempting to use ZnClient to request data. The request requires
a %2C (comma) delimited string as part of the query. Below is a
snippet.

znClient
        addPath: '/v1/instruments';
        queryAt: 'fields' putAll: 'displayName%2Cinstrument%2Cpip';
        get ;
        contents)

The string  'displayName%2Cinstrument%2Cpip'
is being converted to  'displayName%252Cinstrument%252Cpip'
which causes the request to fail.

The query needs to be
fields=displayName%2Cinstrument%2Cpip

I have not found how to do this correctly.
Any help greatly appreciated.

Thanks.

Jimmie

Maybe a silly thing, but since %2C = , ... Did you tried already to
make itself encode that? Like
znClient
         addPath: '/v1/instruments';
         queryAt: 'fields' putAll: 'displayName,instrument,pip';
         get ;
         contents)

I suspect it is using encoding internally, that is why % is also
encoded if you try to put it.

I hope that works

Not silly and no need to suspect, but absolutely correct !

Sven

My apologies for not having full disclosure.

Pharo 4, new image, freshly installed Zinc stable version.
Xubuntu 15.04



That is what I thought would happen and what I tried first. But it is not being encoded from what I can find.

Inspect this in a workspace/playground.

ZnClient new
       https;
       host: 'google.com';
       addPath: '/commaTest';
       queryAt: 'fields' put: 'displayName,instrument,pip';
       yourself

View the  request / requestLine / uri.  The commas are still present in the URI.
So I tried encoding myself and get the other error.

Of course Google won't understand this and in this snippet won't receive it.

And please let me know if I am doing something wrong.

Any help greatly appreciated.

Jimmie