Truncation errors on TCP/IP socket messages

Hello Joel,

I think, that even if you send a message via a single send call from one machine you can not expect that you can read the whole message with one call from on another machine - this is a no-go programming under tcp/ip. Especially if the message is getting bigger you may hit problems you are never aware off.

You may have two possiblities:

a) You put your receiving code in a loop - and you have to be very careful about the loop termination condition. The difficult is to detect, when no more data can come (because all is received) or when an error condition is there or a timeout has been reached.

b) You change your protocol and you introduce the number of bytes your are expecting to get - then the end detection becomes simplier.

Marten

'Joel Zecher' via VA Smalltalk <<a onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="CRulWiyDAQAJ">va-sma...@...> hat am 2. November 2016 um 18:31 geschrieben:

Greetings,

I am running into an error with large (>1K) messages being sent to TCP/IP ports from one application to another on a single desktop. The main application is a large Smalltalk client app. It is conversing with a small .NET application written in C#. The problem we're having is only for messages coming from .NET to the Smalltalk application.

The code for starting up the listener socket for the Smalltalk client is shown below. I added some Transcript show messages for debugging purposes. For messages that are 1K or less in length, there is never a problem. For messages that are greater than 1K, we have an occasional problem.

startUpListenerSocket

 self listenerSocket: nil.
 self listenerSocket: (self serverSocketOnPort: self listeningPort).
 
 self listenerSocket isCommunicationsError not
  ifTrue: [ self listenerSocket bufferLength: 20480 ]
  ifFalse: [ Transcript cr;show: '<<<<<<<<<< Bad Socket: ', self listenerSocket asString ].

 [
  [ self listenerSocket notNil
    and: [ self listenerSocket isCommunicationsError not ] ]
   whileTrue: [ | acceptSocket receive priorReceive |
    (acceptSocket := self listenerSocket accept) isCommunicationsError not
     ifTrue: [

      Transcript cr;show: 'acceptSocket buffer length = ';show: acceptSocket buffer length asString.
      (receive := acceptSocket receive) isCommunicationsError not
       ifTrue: [

        Transcript cr;show: 'receive buffer length = ';show: receive length asString.
        self
         processSocket: acceptSocket
         data: receive data reference asString trimTrailingLineFeed ]
       ifFalse: [ acceptSocket disconnect ] ]
     ifFalse: [ "Do nothing" ] ]
 ] forkAt: Processor userInterruptPriority.

The method #serverSocketOnPort: sent in the above is a copy of AbtSocket class>>#serverSocketOnPort:. The only difference is that it uses the loopback address (127.0.0.1) instead of the local host.

A sampling of Transcript messages are shown below. You can see that the size of the buffer length for acceptSocket remains at 20480 (20K) throughout. A number of messages come through fine until the highlighted one which is something we call APPLICATIONCOLLECTION. It shows a buffer length of 1024 but the actual size of the message sent from .NET is around 1920. Everything in the message past the first 1024 bytes is truncated. Since it was an XML message, this causes a parsing error.

Now, here is where it gets weird. The following message for EMBEDDEDBOOKMARKCOLLECTION includes the entire 2086 bytes being sent from .NET.

acceptSocket buffer length = 20480

receive buffer length = 154

acceptSocket buffer length = 20480

receive buffer length = 109

acceptSocket buffer length = 20480

receive buffer length = 114

acceptSocket buffer length = 20480

receive buffer length = 115                             ACTIVEPAGE

acceptSocket buffer length = 20480

receive buffer length = 149                             NEWAPPLICATION

acceptSocket buffer length = 20480

receive buffer length = 115                             NIGO

acceptSocket buffer length = 20480

receive buffer length = 1024                           APPLICATIONCOLLECTION

acceptSocket buffer length = 20480

receive buffer length = 2086                           EMBEDDEDBOOKMARKCOLLECTION

acceptSocket buffer length = 20480

receive buffer length = 120                             OPENSUMMARY

acceptSocket buffer length = 20480

receive buffer length = 130

acceptSocket buffer length = 20480

receive buffer length = 237

acceptSocket buffer length = 20480

receive buffer length = 195                             CURRENTWINDOWSTATE

Now, I am not a socket scientist so I am at a loss at how to explain this. My first reaction was to blame the .NET application for sending me a truncated message since it was always the same one. The APPLICATION message is always truncated but the BOOKMARK message is always fine. A close review of that code and a number of tests from different points has pretty much eliminated that possibility.

We have done multiple iterations of tests. Originally my listener socket buffer length was only 5000 but I moved it up incrementally until it eventually reached 20K. We found that this increased the maximum size of some of the other messages but it would not fix the APPLICATION message. When we created test messages of lengths much larger than actually needed for our requirements, we found that the maximum that could get sent before being truncated was 8192 bytes (or 8K).

The last relevant piece of information I have is that the messages I am sending are just strings. Sure, they test is formatted into XML but they are still just strings. Formatted strings. There are no special characters other than the expected "<>/" punctuation characters. When I say that one message is the APPLICATION message, that just means that the message type="APPLICATION" within the XML framework. Still, a string.

In summary, is there some kind of limitation inherent with Port messages using AbtSocket or am I simply doing something wrong? Is there some other syntax I should be using to either set up my listener or prepping for the receive?

Many thanks,

Joel Zecher

 

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="CRulWiyDAQAJ">va-smalltalk...@googlegroups.com.
To post to this group, send email to <a onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="CRulWiyDAQAJ">va-sma...@....
Visit this group at <a onmousedown="this.href=&#39;https://groups.google.com/group/va-smalltalk&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/va-smalltalk&#39;;return true;" href="https://groups.google.com/group/va-smalltalk" target="_blank" rel="nofollow">https://groups.google.com/group/va-smalltalk.
For more options, visit <a onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" href="https://groups.google.com/d/optout" target="_blank" rel="nofollow">https://groups.google.com/d/optout.

b) "You change your protocol and you introduce the number of bytes your are expecting to get..." -  I am not sure how I would do this. I am using AbtSocket>>#receive to get the data. A quick perusal of the class browser reveals nothing obvious about how to interject an expected size. Also, I don't know the size until the message is sent (or should I say received). Would I have to get the .NET app to first send me a short message indicating the size of the next message and then I would somehow use that???

Hi Joel,

That should of course work but you may need to add a timeout in case '</DOCUMENT>' never shows up.

Lou

On Friday, November 4, 2016 at 1:50:11 PM UTC-4, Joel Zecher wrote:

Thank you to all who replied.

I finally fixed it by putting the "acceptSocket receive" into a method with a loop. Since the data coming back is a string in XML format, I simply test to see if it ends with '</DOCUMENT>' to see if I have it all. No syntax changes. No more truncation errors.