Looking for advice on SstByteMessage and UTF-8

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Looking for advice on SstByteMessage and UTF-8

jtuchel
Dear VAST community,

I am dusting off some old RESTful web services code I once wrote because our Web Application needs a public API.

Most things work out of the box now, but I am having a little problem with UTF-8 (i hear some readers sigh).

I'd like to subclass SstByteMessage in a way so that I can access the contents of an incoming request automatically converted from UTF-8 to the local code page (#convertFromCodePage:) and of course I'D like the setting of content of a response on the server side to automatically convert that to UTF-8 when I set it.

So the simplest thing that could possibly work is to sumply subclass SstByteMessage with my own and have it do the conversion in the #contents and #contents: methods.

My naive implementation lloks like this:

contents
  ^super contents asString convertFromCodePage: 'utf-8'

contents: str
  super contents: (str convertToCodePage: 'utf-8').


In theory, this should work. But it doesn't. Interestingly, the #contents: setter seems to work nicely. I use Postman as a development client and there I get correct utf-8 Umlauts in responses.
The #contents getter, however, must be remoived to make things work. As long as the overridden #contents method is in the image, a GET or POST request is never answered. I also get no Exception (http forwarding is disabled). I wonder why that is?

Am I on the right track anyways? One of the reasons why chose this approach is that this way the calculation of the content's size for the response is correctly taking the extra bytes for Umlauts into account....

If I am on the right track, I am facing another problem:

For now I implemented #messageClass in my SstHttpServer subclass to return my own SstByteMessage subclass. IIUC, this will often not work, because there are some places where the actual messageClass to instantiate is not retrieved from the server, but from the Transport configuration... I am not sure how to register  an individual Transport Configuration, however. The only references in the image are loaded or post-load intializers and seem to be global. I'd like to only set this for one Applicatipon context....

So I would be grateful for pointers and ideas.


Joachim







--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Looking for advice on SstByteMessage and UTF-8

Louis LaBrunda
Hi Joachim,

I think the problem lies with the fact that SstByteMessage>contents is returning an #SstOSPtr and not a string as your overwritten code does.

contents
^SstOSPtr on: contents size: self contentsSize offset: self headerSize

Maybe you can return an #SstOSPtr that points to converted data?

Here is a completely untested wild ass guess:

 
contents
| utf8 |

utf8 := super contents asString convertFromCodePage: 'utf-8'.

^SstOSPtr on: utf8 size: utf8 size.


Lou


On Friday, September 28, 2018 at 3:42:49 AM UTC-4, Joachim Tuchel wrote:
Dear VAST community,

I am dusting off some old RESTful web services code I once wrote because our Web Application needs a public API.

Most things work out of the box now, but I am having a little problem with UTF-8 (i hear some readers sigh).

I'd like to subclass SstByteMessage in a way so that I can access the contents of an incoming request automatically converted from UTF-8 to the local code page (#convertFromCodePage:) and of course I'D like the setting of content of a response on the server side to automatically convert that to UTF-8 when I set it.

So the simplest thing that could possibly work is to sumply subclass SstByteMessage with my own and have it do the conversion in the #contents and #contents: methods.

My naive implementation lloks like this:

contents
  ^super contents asString convertFromCodePage: 'utf-8'

contents: str
  super contents: (str convertToCodePage: 'utf-8').


In theory, this should work. But it doesn't. Interestingly, the #contents: setter seems to work nicely. I use Postman as a development client and there I get correct utf-8 Umlauts in responses.
The #contents getter, however, must be remoived to make things work. As long as the overridden #contents method is in the image, a GET or POST request is never answered. I also get no Exception (http forwarding is disabled). I wonder why that is?

Am I on the right track anyways? One of the reasons why chose this approach is that this way the calculation of the content's size for the response is correctly taking the extra bytes for Umlauts into account....

If I am on the right track, I am facing another problem:

For now I implemented #messageClass in my SstHttpServer subclass to return my own SstByteMessage subclass. IIUC, this will often not work, because there are some places where the actual messageClass to instantiate is not retrieved from the server, but from the Transport configuration... I am not sure how to register  an individual Transport Configuration, however. The only references in the image are loaded or post-load intializers and seem to be global. I'd like to only set this for one Applicatipon context....

So I would be grateful for pointers and ideas.


Joachim







--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Looking for advice on SstByteMessage and UTF-8

jtuchel
Lou,

I'll try over the weekend. Thanks for the tip. Sure, SstOSPtr is a completely different beast than a String... How could I overlook that?

Any tips for the registration part of teh SstByteMessage subclass?


Joachim



Am Freitag, 28. September 2018 15:46:32 UTC+2 schrieb Louis LaBrunda:
Hi Joachim,

I think the problem lies with the fact that SstByteMessage>contents is returning an #SstOSPtr and not a string as your overwritten code does.

contents
^SstOSPtr on: contents size: self contentsSize offset: self headerSize

Maybe you can return an #SstOSPtr that points to converted data?

Here is a completely untested wild ass guess:

 
contents
| utf8 |

utf8 := super contents asString convertFromCodePage: 'utf-8'.

^SstOSPtr on: utf8 size: utf8 size.


Lou


On Friday, September 28, 2018 at 3:42:49 AM UTC-4, Joachim Tuchel wrote:
Dear VAST community,

I am dusting off some old RESTful web services code I once wrote because our Web Application needs a public API.

Most things work out of the box now, but I am having a little problem with UTF-8 (i hear some readers sigh).

I'd like to subclass SstByteMessage in a way so that I can access the contents of an incoming request automatically converted from UTF-8 to the local code page (#convertFromCodePage:) and of course I'D like the setting of content of a response on the server side to automatically convert that to UTF-8 when I set it.

So the simplest thing that could possibly work is to sumply subclass SstByteMessage with my own and have it do the conversion in the #contents and #contents: methods.

My naive implementation lloks like this:

contents
  ^super contents asString convertFromCodePage: 'utf-8'

contents: str
  super contents: (str convertToCodePage: 'utf-8').


In theory, this should work. But it doesn't. Interestingly, the #contents: setter seems to work nicely. I use Postman as a development client and there I get correct utf-8 Umlauts in responses.
The #contents getter, however, must be remoived to make things work. As long as the overridden #contents method is in the image, a GET or POST request is never answered. I also get no Exception (http forwarding is disabled). I wonder why that is?

Am I on the right track anyways? One of the reasons why chose this approach is that this way the calculation of the content's size for the response is correctly taking the extra bytes for Umlauts into account....

If I am on the right track, I am facing another problem:

For now I implemented #messageClass in my SstHttpServer subclass to return my own SstByteMessage subclass. IIUC, this will often not work, because there are some places where the actual messageClass to instantiate is not retrieved from the server, but from the Transport configuration... I am not sure how to register  an individual Transport Configuration, however. The only references in the image are loaded or post-load intializers and seem to be global. I'd like to only set this for one Applicatipon context....

So I would be grateful for pointers and ideas.


Joachim







--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Looking for advice on SstByteMessage and UTF-8

Louis LaBrunda
HI Joachim,

SstTransportConfiguration accepts the #messageClass: message, so, if you can find the right spot to set it to your subclass of SstByteMessage, you won't need a subclass of SstTransportConfiguration.  Unfortunately, I don't any idea where that spot might be.

Lou


On Saturday, September 29, 2018 at 1:49:47 AM UTC-4, Joachim Tuchel wrote:
Lou,

I'll try over the weekend. Thanks for the tip. Sure, SstOSPtr is a completely different beast than a String... How could I overlook that?

Any tips for the registration part of teh SstByteMessage subclass?


Joachim



Am Freitag, 28. September 2018 15:46:32 UTC+2 schrieb Louis LaBrunda:
Hi Joachim,

I think the problem lies with the fact that SstByteMessage>contents is returning an #SstOSPtr and not a string as your overwritten code does.

contents
^SstOSPtr on: contents size: self contentsSize offset: self headerSize

Maybe you can return an #SstOSPtr that points to converted data?

Here is a completely untested wild ass guess:

 
contents
| utf8 |

utf8 := super contents asString convertFromCodePage: 'utf-8'.

^SstOSPtr on: utf8 size: utf8 size.


Lou


On Friday, September 28, 2018 at 3:42:49 AM UTC-4, Joachim Tuchel wrote:
Dear VAST community,

I am dusting off some old RESTful web services code I once wrote because our Web Application needs a public API.

Most things work out of the box now, but I am having a little problem with UTF-8 (i hear some readers sigh).

I'd like to subclass SstByteMessage in a way so that I can access the contents of an incoming request automatically converted from UTF-8 to the local code page (#convertFromCodePage:) and of course I'D like the setting of content of a response on the server side to automatically convert that to UTF-8 when I set it.

So the simplest thing that could possibly work is to sumply subclass SstByteMessage with my own and have it do the conversion in the #contents and #contents: methods.

My naive implementation lloks like this:

contents
  ^super contents asString convertFromCodePage: 'utf-8'

contents: str
  super contents: (str convertToCodePage: 'utf-8').


In theory, this should work. But it doesn't. Interestingly, the #contents: setter seems to work nicely. I use Postman as a development client and there I get correct utf-8 Umlauts in responses.
The #contents getter, however, must be remoived to make things work. As long as the overridden #contents method is in the image, a GET or POST request is never answered. I also get no Exception (http forwarding is disabled). I wonder why that is?

Am I on the right track anyways? One of the reasons why chose this approach is that this way the calculation of the content's size for the response is correctly taking the extra bytes for Umlauts into account....

If I am on the right track, I am facing another problem:

For now I implemented #messageClass in my SstHttpServer subclass to return my own SstByteMessage subclass. IIUC, this will often not work, because there are some places where the actual messageClass to instantiate is not retrieved from the server, but from the Transport configuration... I am not sure how to register  an individual Transport Configuration, however. The only references in the image are loaded or post-load intializers and seem to be global. I'd like to only set this for one Applicatipon context....

So I would be grateful for pointers and ideas.


Joachim







--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Looking for advice on SstByteMessage and UTF-8

jtuchel
That's abou t how far I got.

Sometimes I am fascinated what's in the box with Sst. Sometimes I think the creators of it "overdid a little" in its complexity...


Joachim

Am Samstag, 29. September 2018 16:16:13 UTC+2 schrieb Louis LaBrunda:
HI Joachim,

SstTransportConfiguration accepts the #messageClass: message, so, if you can find the right spot to set it to your subclass of SstByteMessage, you won't need a subclass of SstTransportConfiguration.  Unfortunately, I don't any idea where that spot might be.

Lou


On Saturday, September 29, 2018 at 1:49:47 AM UTC-4, Joachim Tuchel wrote:
Lou,

I'll try over the weekend. Thanks for the tip. Sure, SstOSPtr is a completely different beast than a String... How could I overlook that?

Any tips for the registration part of teh SstByteMessage subclass?


Joachim



Am Freitag, 28. September 2018 15:46:32 UTC+2 schrieb Louis LaBrunda:
Hi Joachim,

I think the problem lies with the fact that SstByteMessage>contents is returning an #SstOSPtr and not a string as your overwritten code does.

contents
^SstOSPtr on: contents size: self contentsSize offset: self headerSize

Maybe you can return an #SstOSPtr that points to converted data?

Here is a completely untested wild ass guess:

 
contents
| utf8 |

utf8 := super contents asString convertFromCodePage: 'utf-8'.

^SstOSPtr on: utf8 size: utf8 size.


Lou


On Friday, September 28, 2018 at 3:42:49 AM UTC-4, Joachim Tuchel wrote:
Dear VAST community,

I am dusting off some old RESTful web services code I once wrote because our Web Application needs a public API.

Most things work out of the box now, but I am having a little problem with UTF-8 (i hear some readers sigh).

I'd like to subclass SstByteMessage in a way so that I can access the contents of an incoming request automatically converted from UTF-8 to the local code page (#convertFromCodePage:) and of course I'D like the setting of content of a response on the server side to automatically convert that to UTF-8 when I set it.

So the simplest thing that could possibly work is to sumply subclass SstByteMessage with my own and have it do the conversion in the #contents and #contents: methods.

My naive implementation lloks like this:

contents
  ^super contents asString convertFromCodePage: 'utf-8'

contents: str
  super contents: (str convertToCodePage: 'utf-8').


In theory, this should work. But it doesn't. Interestingly, the #contents: setter seems to work nicely. I use Postman as a development client and there I get correct utf-8 Umlauts in responses.
The #contents getter, however, must be remoived to make things work. As long as the overridden #contents method is in the image, a GET or POST request is never answered. I also get no Exception (http forwarding is disabled). I wonder why that is?

Am I on the right track anyways? One of the reasons why chose this approach is that this way the calculation of the content's size for the response is correctly taking the extra bytes for Umlauts into account....

If I am on the right track, I am facing another problem:

For now I implemented #messageClass in my SstHttpServer subclass to return my own SstByteMessage subclass. IIUC, this will often not work, because there are some places where the actual messageClass to instantiate is not retrieved from the server, but from the Transport configuration... I am not sure how to register  an individual Transport Configuration, however. The only references in the image are loaded or post-load intializers and seem to be global. I'd like to only set this for one Applicatipon context....

So I would be grateful for pointers and ideas.


Joachim







--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Looking for advice on SstByteMessage and UTF-8

jtuchel
In reply to this post by Louis LaBrunda
Lou,

I tried your suggestion. Unfortunately, it "somewhat works" - and I am really puzzled.

Funnily, using a curl -X POST seems to work , while Postman shows an error message. I am sitting at an airport terminal and am running out of time before boarding. Maybe it is a bug in Postman ??? As soon as the response contains an Umlaut in its message text (not in the headers, I didn't dare trying that), the request is not answered, but I also get no Exception...

This is strange...

Joachim




Am Freitag, 28. September 2018 15:46:32 UTC+2 schrieb Louis LaBrunda:
Hi Joachim,

I think the problem lies with the fact that SstByteMessage>contents is returning an #SstOSPtr and not a string as your overwritten code does.

contents
^SstOSPtr on: contents size: self contentsSize offset: self headerSize

Maybe you can return an #SstOSPtr that points to converted data?

Here is a completely untested wild ass guess:

 
contents
| utf8 |

utf8 := super contents asString convertFromCodePage: 'utf-8'.

^SstOSPtr on: utf8 size: utf8 size.


Lou


On Friday, September 28, 2018 at 3:42:49 AM UTC-4, Joachim Tuchel wrote:
Dear VAST community,

I am dusting off some old RESTful web services code I once wrote because our Web Application needs a public API.

Most things work out of the box now, but I am having a little problem with UTF-8 (i hear some readers sigh).

I'd like to subclass SstByteMessage in a way so that I can access the contents of an incoming request automatically converted from UTF-8 to the local code page (#convertFromCodePage:) and of course I'D like the setting of content of a response on the server side to automatically convert that to UTF-8 when I set it.

So the simplest thing that could possibly work is to sumply subclass SstByteMessage with my own and have it do the conversion in the #contents and #contents: methods.

My naive implementation lloks like this:

contents
  ^super contents asString convertFromCodePage: 'utf-8'

contents: str
  super contents: (str convertToCodePage: 'utf-8').


In theory, this should work. But it doesn't. Interestingly, the #contents: setter seems to work nicely. I use Postman as a development client and there I get correct utf-8 Umlauts in responses.
The #contents getter, however, must be remoived to make things work. As long as the overridden #contents method is in the image, a GET or POST request is never answered. I also get no Exception (http forwarding is disabled). I wonder why that is?

Am I on the right track anyways? One of the reasons why chose this approach is that this way the calculation of the content's size for the response is correctly taking the extra bytes for Umlauts into account....

If I am on the right track, I am facing another problem:

For now I implemented #messageClass in my SstHttpServer subclass to return my own SstByteMessage subclass. IIUC, this will often not work, because there are some places where the actual messageClass to instantiate is not retrieved from the server, but from the Transport configuration... I am not sure how to register  an individual Transport Configuration, however. The only references in the image are loaded or post-load intializers and seem to be global. I'd like to only set this for one Applicatipon context....

So I would be grateful for pointers and ideas.


Joachim







--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Looking for advice on SstByteMessage and UTF-8

Louis LaBrunda
Hi Joachim,

Sorry but I have run out of the low hanging fruit of suggestions.  Maybe you can see if the Pharo version of Seaside has solved this problem.

Lou

On Sunday, September 30, 2018 at 7:51:55 AM UTC-4, Joachim Tuchel wrote:
Lou,

I tried your suggestion. Unfortunately, it "somewhat works" - and I am really puzzled.

Funnily, using a curl -X POST seems to work , while Postman shows an error message. I am sitting at an airport terminal and am running out of time before boarding. Maybe it is a bug in Postman ??? As soon as the response contains an Umlaut in its message text (not in the headers, I didn't dare trying that), the request is not answered, but I also get no Exception...

This is strange...

Joachim




Am Freitag, 28. September 2018 15:46:32 UTC+2 schrieb Louis LaBrunda:
Hi Joachim,

I think the problem lies with the fact that SstByteMessage>contents is returning an #SstOSPtr and not a string as your overwritten code does.

contents
^SstOSPtr on: contents size: self contentsSize offset: self headerSize

Maybe you can return an #SstOSPtr that points to converted data?

Here is a completely untested wild ass guess:

 
contents
| utf8 |

utf8 := super contents asString convertFromCodePage: 'utf-8'.

^SstOSPtr on: utf8 size: utf8 size.


Lou


On Friday, September 28, 2018 at 3:42:49 AM UTC-4, Joachim Tuchel wrote:
Dear VAST community,

I am dusting off some old RESTful web services code I once wrote because our Web Application needs a public API.

Most things work out of the box now, but I am having a little problem with UTF-8 (i hear some readers sigh).

I'd like to subclass SstByteMessage in a way so that I can access the contents of an incoming request automatically converted from UTF-8 to the local code page (#convertFromCodePage:) and of course I'D like the setting of content of a response on the server side to automatically convert that to UTF-8 when I set it.

So the simplest thing that could possibly work is to sumply subclass SstByteMessage with my own and have it do the conversion in the #contents and #contents: methods.

My naive implementation lloks like this:

contents
  ^super contents asString convertFromCodePage: 'utf-8'

contents: str
  super contents: (str convertToCodePage: 'utf-8').


In theory, this should work. But it doesn't. Interestingly, the #contents: setter seems to work nicely. I use Postman as a development client and there I get correct utf-8 Umlauts in responses.
The #contents getter, however, must be remoived to make things work. As long as the overridden #contents method is in the image, a GET or POST request is never answered. I also get no Exception (http forwarding is disabled). I wonder why that is?

Am I on the right track anyways? One of the reasons why chose this approach is that this way the calculation of the content's size for the response is correctly taking the extra bytes for Umlauts into account....

If I am on the right track, I am facing another problem:

For now I implemented #messageClass in my SstHttpServer subclass to return my own SstByteMessage subclass. IIUC, this will often not work, because there are some places where the actual messageClass to instantiate is not retrieved from the server, but from the Transport configuration... I am not sure how to register  an individual Transport Configuration, however. The only references in the image are loaded or post-load intializers and seem to be global. I'd like to only set this for one Applicatipon context....

So I would be grateful for pointers and ideas.


Joachim







--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.