The Trunk: WebClient-Core-pre.117.mcz

> On 17.09.2018, at 17:02, [hidden email] wrote:
>
> Patrick Rein uploaded a new version of WebClient-Core to project The Trunk:
> http://source.squeak.org/trunk/WebClient-Core-pre.117.mcz
>
> ==================== Summary ====================
>
> Name: WebClient-Core-pre.117
> Author: pre
> Time: 17 September 2018, 4:01:07.501347 pm
> UUID: 7908aa6f-cf8c-dd40-b6f3-6efb2b8042d1
> Ancestors: WebClient-Core-ul.116
>
> Fixes an issue with chunked encoded WebMessages which resulted in already decoded content to be decoded again. While not perfect it is important to allow for larger GitHub repositories such as Metacello.
>
> =============== Diff against WebClient-Core-ul.116 ===============
>
> Item was added:
> + ----- Method: WebMessage>>streamDirectlyFrom:to:size:progress: (in category 'streaming') -----
> + streamDirectlyFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock
> + "Stream the content of srcStream to dstStream.
> + If a size is given, stream that many elements, otherwise stream up to the end."
> +
> + | buffer remaining size totalRead |
> + sizeOrNil = 0 ifTrue:[^self].
> +
> + buffer := (srcStream isBinary ifTrue:[ByteArray] ifFalse:[String]) new: 4096.
> + totalRead := 0.
> + size := sizeOrNil ifNil:[0].
> + [(sizeOrNil == nil and:[srcStream atEnd not]) or:[totalRead < size]] whileTrue:[
> + progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].
> + remaining := sizeOrNil ifNil:[99999] ifNotNil:[sizeOrNil - totalRead].
> + remaining > buffer size ifTrue:[remaining := buffer size].
> + buffer := srcStream next: remaining into: buffer startingAt: 1.
> + dstStream nextPutAll: (remaining < buffer size  
> + ifTrue:[(buffer copyFrom: 1 to: remaining)]
> + ifFalse:[buffer]).
> + totalRead := totalRead + buffer size.
> + ].
> + dstStream flush.
> + progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].!
>
> Item was changed:
>  ----- Method: WebMessage>>streamFrom:to:size:progress: (in category 'streaming') -----
>  streamFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock
> + "Stream the content of srcStream to dstStream applying any conversations necessary."
> - "Stream the content of srcStream to dstStream.
> - If a size is given, stream that many elements, otherwise stream up to the end."
>
> - | buffer totalRead remaining size |
>   (self headerAt: 'transfer-encoding') ifNotEmpty:[:encoding|
>   encoding = 'chunked'
> + ifTrue:[
> + self flag: #todo. " Ideally this would use the WebChunkedStream --pre"
> + ^self chunkFrom: srcStream to: dstStream progress: progressBlock]
> - ifTrue:[^self chunkFrom: srcStream to: dstStream progress: progressBlock]
>   ifFalse:[self error: 'Unknown transfer-encoding: ', encoding]].
>
> + ^ self streamDirectlyFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock.!
> - sizeOrNil = 0 ifTrue:[^self].
> -
> - buffer := (srcStream isBinary ifTrue:[ByteArray] ifFalse:[String]) new: 4096.
> - totalRead := 0.
> - size := sizeOrNil ifNil:[0].
> - [(sizeOrNil == nil and:[stream atEnd not]) or:[totalRead < size]] whileTrue:[
> - progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].
> - remaining := sizeOrNil ifNil:[99999] ifNotNil:[sizeOrNil - totalRead].
> - remaining > buffer size ifTrue:[remaining := buffer size].
> - buffer := srcStream next: remaining into: buffer startingAt: 1.
> - dstStream nextPutAll: (remaining < buffer size  
> - ifTrue:[(buffer copyFrom: 1 to: remaining)]
> - ifFalse:[buffer]).
> - totalRead := totalRead + buffer size.
> - ].
> - dstStream flush.
> - progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].!
>
> Item was changed:
>  ----- Method: WebMessage>>streamTo:size:progress: (in category 'streaming') -----
>  streamTo: dstStream size: size progress: aBlock
>   "Stream from the receiver's socket stream to the given destination stream.
>   Inbound. Can be used on both request/response depending on
>   whether it is utilized by WebClient or WebServer."
>   content ifNil:[
>   self streamFrom: stream to: dstStream size: size progress: aBlock
>   ] ifNotNil:[
> + self streamDirectlyFrom: content readStream to: dstStream size: size progress: aBlock
> - self streamFrom: content readStream to: dstStream size: size progress: aBlock
>   ].!
>
>

> Patrick Rein uploaded a new version of WebClient-Core to project The Trunk:
> http://source.squeak.org/trunk/WebClient-Core-pre.117.mcz
>
> ==================== Summary ====================
>
> Name: WebClient-Core-pre.117
> Author: pre
> Time: 17 September 2018, 4:01:07.501347 pm
> UUID: 7908aa6f-cf8c-dd40-b6f3-6efb2b8042d1
> Ancestors: WebClient-Core-ul.116
>
> Fixes an issue with chunked encoded WebMessages which resulted in already decoded content to be decoded again. While not perfect it is important to allow for larger GitHub repositories such as Metacello.
>
> =============== Diff against WebClient-Core-ul.116 ===============
>
> Item was added:
> + ----- Method: WebMessage>>streamDirectlyFrom:to:size:progress: (in category 'streaming') -----
> + streamDirectlyFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock
> + "Stream the content of srcStream to dstStream.
> + If a size is given, stream that many elements, otherwise stream up to the end."
> +
> + | buffer remaining size totalRead |
> + sizeOrNil = 0 ifTrue:[^self].
> +
> + buffer := (srcStream isBinary ifTrue:[ByteArray] ifFalse:[String]) new: 4096.
> + totalRead := 0.
> + size := sizeOrNil ifNil:[0].
> + [(sizeOrNil == nil and:[srcStream atEnd not]) or:[totalRead < size]] whileTrue:[
> + progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].
> + remaining := sizeOrNil ifNil:[99999] ifNotNil:[sizeOrNil - totalRead].
> + remaining > buffer size ifTrue:[remaining := buffer size].
> + buffer := srcStream next: remaining into: buffer startingAt: 1.
> + dstStream nextPutAll: (remaining < buffer size
> + ifTrue:[(buffer copyFrom: 1 to: remaining)]
> + ifFalse:[buffer]).
> + totalRead := totalRead + buffer size.
> + ].
> + dstStream flush.
> + progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].!
>
> Item was changed:
>  ----- Method: WebMessage>>streamFrom:to:size:progress: (in category 'streaming') -----
>  streamFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock
> + "Stream the content of srcStream to dstStream applying any conversations necessary."
> - "Stream the content of srcStream to dstStream.
> - If a size is given, stream that many elements, otherwise stream up to the end."
>
> - | buffer totalRead remaining size |
>   (self headerAt: 'transfer-encoding') ifNotEmpty:[:encoding|
>   encoding = 'chunked'
> + ifTrue:[
> + self flag: #todo. " Ideally this would use the WebChunkedStream --pre"
> + ^self chunkFrom: srcStream to: dstStream progress: progressBlock]
> - ifTrue:[^self chunkFrom: srcStream to: dstStream progress: progressBlock]
>   ifFalse:[self error: 'Unknown transfer-encoding: ', encoding]].
>
> + ^ self streamDirectlyFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock.!
> - sizeOrNil = 0 ifTrue:[^self].
> -
> - buffer := (srcStream isBinary ifTrue:[ByteArray] ifFalse:[String]) new: 4096.
> - totalRead := 0.
> - size := sizeOrNil ifNil:[0].
> - [(sizeOrNil == nil and:[stream atEnd not]) or:[totalRead < size]] whileTrue:[
> - progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].
> - remaining := sizeOrNil ifNil:[99999] ifNotNil:[sizeOrNil - totalRead].
> - remaining > buffer size ifTrue:[remaining := buffer size].
> - buffer := srcStream next: remaining into: buffer startingAt: 1.
> - dstStream nextPutAll: (remaining < buffer size
> - ifTrue:[(buffer copyFrom: 1 to: remaining)]
> - ifFalse:[buffer]).
> - totalRead := totalRead + buffer size.
> - ].
> - dstStream flush.
> - progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].!
>
> Item was changed:
>  ----- Method: WebMessage>>streamTo:size:progress: (in category 'streaming') -----
>  streamTo: dstStream size: size progress: aBlock
>   "Stream from the receiver's socket stream to the given destination stream.
>   Inbound. Can be used on both request/response depending on
>   whether it is utilized by WebClient or WebServer."
>   content ifNil:[
>   self streamFrom: stream to: dstStream size: size progress: aBlock
>   ] ifNotNil:[
> + self streamDirectlyFrom: content readStream to: dstStream size: size progress: aBlock
> - self streamFrom: content readStream to: dstStream size: size progress: aBlock

> > On 17.09.2018, at 17:02, [hidden email] wrote:
> >
> > Patrick Rein uploaded a new version of WebClient-Core to project The Trunk:
> > http://source.squeak.org/trunk/WebClient-Core-pre.117.mcz
> >
> > ==================== Summary ====================
> >
> > Name: WebClient-Core-pre.117
> > Author: pre
> > Time: 17 September 2018, 4:01:07.501347 pm
> > UUID: 7908aa6f-cf8c-dd40-b6f3-6efb2b8042d1
> > Ancestors: WebClient-Core-ul.116
> >
> > Fixes an issue with chunked encoded WebMessages which resulted in already decoded content to be decoded again. While not perfect it is important to allow for larger GitHub repositories such as Metacello.
> >
> > =============== Diff against WebClient-Core-ul.116 ===============
> >
> > Item was added:
> > + ----- Method: WebMessage>>streamDirectlyFrom:to:size:progress: (in category 'streaming') -----
> > + streamDirectlyFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock
> > + "Stream the content of srcStream to dstStream.
> > + If a size is given, stream that many elements, otherwise stream up to the end."
> > +
> > + | buffer remaining size totalRead |
> > + sizeOrNil = 0 ifTrue:[^self].
> > +
> > + buffer := (srcStream isBinary ifTrue:[ByteArray] ifFalse:[String]) new: 4096.
> > + totalRead := 0.
> > + size := sizeOrNil ifNil:[0].
> > + [(sizeOrNil == nil and:[srcStream atEnd not]) or:[totalRead < size]] whileTrue:[
> > + progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].
> > + remaining := sizeOrNil ifNil:[99999] ifNotNil:[sizeOrNil - totalRead].
> > + remaining > buffer size ifTrue:[remaining := buffer size].
> > + buffer := srcStream next: remaining into: buffer startingAt: 1.
> > + dstStream nextPutAll: (remaining < buffer size  
> > + ifTrue:[(buffer copyFrom: 1 to: remaining)]
> > + ifFalse:[buffer]).
> > + totalRead := totalRead + buffer size.
> > + ].
> > + dstStream flush.
> > + progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].!
> >
> > Item was changed:
> >  ----- Method: WebMessage>>streamFrom:to:size:progress: (in category 'streaming') -----
> >  streamFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock
> > + "Stream the content of srcStream to dstStream applying any conversations necessary."
> > - "Stream the content of srcStream to dstStream.
> > - If a size is given, stream that many elements, otherwise stream up to the end."
> >
> > - | buffer totalRead remaining size |
> >   (self headerAt: 'transfer-encoding') ifNotEmpty:[:encoding|
> >   encoding = 'chunked'
> > + ifTrue:[
> > + self flag: #todo. " Ideally this would use the WebChunkedStream --pre"
> > + ^self chunkFrom: srcStream to: dstStream progress: progressBlock]
> > - ifTrue:[^self chunkFrom: srcStream to: dstStream progress: progressBlock]
> >   ifFalse:[self error: 'Unknown transfer-encoding: ', encoding]].
> >
> > + ^ self streamDirectlyFrom: srcStream to: dstStream size: sizeOrNil progress: progressBlock.!
> > - sizeOrNil = 0 ifTrue:[^self].
> > -
> > - buffer := (srcStream isBinary ifTrue:[ByteArray] ifFalse:[String]) new: 4096.
> > - totalRead := 0.
> > - size := sizeOrNil ifNil:[0].
> > - [(sizeOrNil == nil and:[stream atEnd not]) or:[totalRead < size]] whileTrue:[
> > - progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].
> > - remaining := sizeOrNil ifNil:[99999] ifNotNil:[sizeOrNil - totalRead].
> > - remaining > buffer size ifTrue:[remaining := buffer size].
> > - buffer := srcStream next: remaining into: buffer startingAt: 1.
> > - dstStream nextPutAll: (remaining < buffer size  
> > - ifTrue:[(buffer copyFrom: 1 to: remaining)]
> > - ifFalse:[buffer]).
> > - totalRead := totalRead + buffer size.
> > - ].
> > - dstStream flush.
> > - progressBlock ifNotNil:[progressBlock value: sizeOrNil value: totalRead].!
> >
> > Item was changed:
> >  ----- Method: WebMessage>>streamTo:size:progress: (in category 'streaming') -----
> >  streamTo: dstStream size: size progress: aBlock
> >   "Stream from the receiver's socket stream to the given destination stream.
> >   Inbound. Can be used on both request/response depending on
> >   whether it is utilized by WebClient or WebServer."
> >   content ifNil:[
> >   self streamFrom: stream to: dstStream size: size progress: aBlock
> >   ] ifNotNil:[
> > + self streamDirectlyFrom: content readStream to: dstStream size: size progress: aBlock
> > - self streamFrom: content readStream to: dstStream size: size progress: aBlock
> >   ].!
> >
> >
>
>

>> + self streamDirectlyFrom: content readStream to: dstStream size: size progress: aBlock
>> - self streamFrom: content readStream to: dstStream size: size progress: aBlock
>
>It seems to me that this is the only real change. And if I'm not
>mistaken, that could simply be written as
>
> dstStream nextPutAll: content
>
>Which would be simpler and way more efficient.
>

>>> + self streamDirectlyFrom: content readStream to: dstStream size: size progress: aBlock
>>> - self streamFrom: content readStream to: dstStream size: size progress: aBlock
>>
>> It seems to me that this is the only real change. And if I'm not
>> mistaken, that could simply be written as
>>
>> dstStream nextPutAll: content
>>
>> Which would be simpler and way more efficient.
>>
>
> Yes, absolutely. My rational for that solution was to change as little as possible of the current control flow. After digging through various parts of the streamFrom:to: protocol (together with Tobias) we figured out that there are various inconsistent uses of the methods, in particular as WebChunkedStream exists but is not used. Thus, in order to not violate any assumptions of potential clients, I tried to preserve the control flow and added the flag for a potential refactoring after the 5.2 release.
>
> That being said and after looking at the streamDirectlyFrom:to:size:progress: message again carefully: I will change it to your proposed version :) It might take till the European afternoon though for me to upload it.
>
> Bests
> Patrick