Spur VM Bug for Chinese Characters?
Locked
12
12
Spur VM Bug for Chinese Characters?
 
Administrator
|
I serialized an object graph with lots of Chinese characters in Pharo 4.0. I have been able to rematerialize it back into Pharo 4.0. However, when I try to materialize into a Pharo 5.0 image, I get "PrimitiveFailed: primitive #value: in Character class failed" from `Character class>>#value: 1310876005`. The only difference immediately apparent is the VM (both via Launcher; details below).
More generally, I'm interested in debugging techniques for Fuel. How would I drill down and isolate the problem? Thanks! 5.0 VM: Virtual Machine --------------- /Applications/Pharo.app/Contents/MacOS/Pharo CoInterpreter VMMaker.oscog-eem.1855 uuid: d8e4a3c2-a3bf-4adc-b224-8012903a1ef4 May 4 2016 StackToRegisterMappingCogit VMMaker.oscog-eem.1855 uuid: d8e4a3c2-a3bf-4adc-b224-8012903a1ef4 May 4 2016 https://github.com/pharo-project/pharo-vm.git Commit: b8ec25a570d7539653e1d793e97609adb509aaed Date: 2016-05-04 11:14:22 +0200 By: Esteban Lorenzano <estebanlm@gmail.com> Jenkins build #589 Mac Cocoa Cog 5.8b12 21-Sep-10 >1B0534FA-246C-47C5-AB29-7A76C81CCDCB< VMMaker versionString https://github.com/pharo-project/pharo-vm.git Commit: b8ec25a570d7539653e1d793e97609adb509aaed Date: 2016-05-04 11:14:22 +0200 By: Esteban Lorenzano <estebanlm@gmail.com> Jenkins build #589 CoInterpreter VMMaker.oscog-eem.1855 uuid: d8e4a3c2-a3bf-4adc-b224-8012903a1ef4 May 4 2016 StackToRegisterMappingCogit VMMaker.oscog-eem.1855 uuid: d8e4a3c2-a3bf-4adc-b224-8012903a1ef4 May 4 2016 VS. 4.0 VM: Virtual Machine --------------- /Applications/Pharo.app/Contents/Resources/vm/40/Pharo.app/Contents/MacOS/Pharo NBCoInterpreter NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Apr 2 2015 NBCogit NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Apr 2 2015 https://github.com/pharo-project/pharo-vm.git Commit: 32d18ba0f2db9bee7f3bdbf16bdb24fe4801cfc5 Date: 2015-03-24 11:08:14 +0100 By: Esteban Lorenzano <estebanlm@gmail.com> Jenkins build #14904 Mac Cocoa Cog 5.8b12 21-Sep-10 >1B0534FA-246C-47C5-AB29-7A76C81CCDCB< VMMaker versionString https://github.com/pharo-project/pharo-vm.git Commit: 32d18ba0f2db9bee7f3bdbf16bdb24fe4801cfc5 Date: 2015-03-24 11:08:14 +0100 By: Esteban Lorenzano <estebanlm@gmail.com> Jenkins build #14904 NBCoInterpreter NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Apr 2 2015 NBCogit NativeBoost-CogPlugin-EstebanLorenzano.21 uuid: 4d9b9bdf-2dfa-4c0b-99eb-5b110dadc697 Apr 2 2015
Cheers,
Sean |
Re: Spur VM Bug for Chinese Characters?
|
|
Hi Sean,
On Fri, Oct 14, 2016 at 9:26 AM, Sean P. DeNigris <[hidden email]> wrote: I serialized an object graph with lots of Chinese characters in Pharo 4.0. I Spur allows 30-bit unsigned character codes, so the max character code is 1073741823, or 16r3FFFFFFF. So your character value is out of range. Character class>>#value: reds better commentary. The limit cannot be raised; there are only 30 bits available for immediate character values. Is that character code legal unicode? More generally, I'm interested in debugging techniques for Fuel. How would I _,,,^..^,,,_ best, Eliot |
Re: Spur VM Bug for Chinese Characters?
|
|
Hi Sean, You can debug the answer of #serialize:, which has information. If you suspect of certain objects you could configure the serializer before serialization with #when:substituteBy: to filter them or something like that. Martin On Fri, Oct 14, 2016 at 1:56 PM, Eliot Miranda <[hidden email]> wrote:
|
Re: Spur VM Bug for Chinese Characters?
|
|
In reply to this post by Sean P. DeNigris
On Fri, Oct 14, 2016 at 1:26 PM, Sean P. DeNigris <[hidden email]> wrote: I serialized an object graph with lots of Chinese characters in Pharo 4.0. I There is some info here: Thanks! |
Re: Spur VM Bug for Chinese Characters?
|
Administrator
|
In reply to this post by Eliot Miranda-2
No, but that turned out to be a red herring! This simple experiment seems to show a fundamental problem when serializing in 4.0 (non-spur) and materializing in 5.0 (spur): FileStream forceNewFileNamed: filename do: [:aFile | FLSerializer newDefault serialize: ($e) on: aFile binary ] . FLMaterializer materializeFromFileNamed: filename. Doing the serialization and materialization in the same 4.0 image works, but materializing in a 5.0 image throws an error (a different one this time - "MessageNotUnderstood: receiver of ""adaptToNumber:andSend:"" is nil")
Cheers,
Sean |
Re: Spur VM Bug for Chinese Characters?
|
|
AFAIU, Fuel is not capable working across versions, let alone over the non-spur/spur divide. It's a let down, but it is by design (to optimise for absolute speed).
> On 15 Oct 2016, at 05:46, Sean P. DeNigris <[hidden email]> wrote: > > Eliot Miranda-2 wrote >> Is that character code legal unicode? > > No, but that turned out to be a red herring! > > This simple experiment seems to show a fundamental problem when serializing > in 4.0 (non-spur) and materializing in 5.0 (spur): > > FileStream forceNewFileNamed: filename do: [:aFile | > FLSerializer newDefault > serialize: ($e) on: aFile binary ] . > > FLMaterializer materializeFromFileNamed: filename. > > Doing the serialization and materialization in the same 4.0 image works, but > materializing in a 5.0 image throws an error (a different one this time - > "MessageNotUnderstood: receiver of ""adaptToNumber:andSend:"" is nil") > > > > ----- > Cheers, > Sean > -- > View this message in context: http://forum.world.st/Spur-VM-Bug-for-Chinese-Characters-tp4918883p4918909.html > Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com. > |
Re: Spur VM Bug for Chinese Characters?
|
Administrator
|
Lack of portability between Fuel versions is an annoyance, but is easy to overcome by updating Fuel on the source image to the same version as the target image before porting. But I don't see an obvious workaround for the non-spur -> spur breakdown. How can we leave users in a situation where it seems impossible to migrate data between Pharo versions?!
Cheers,
Sean |
Re: Spur VM Bug for Chinese Characters?
|
|
Sean,
fuel relies on the fact that the environment for materialization is the same as where it has been serialized. So classes and methods need to be mostly the same. A non-spur to spur change is likely to make it fail. These changes are rare but sonetimes the need to be done. If you want to move data between different environments you are better off using STON. Norbert > Am 15.10.2016 um 23:28 schrieb Sean P. DeNigris <[hidden email]>: > > Sven Van Caekenberghe-2 wrote >> AFAIU, Fuel is not capable working across versions, let alone over the >> non-spur/spur divide. It's a let down, but it is by design (to optimise >> for absolute speed). > > Lack of portability between Fuel versions is an annoyance, but is easy to > overcome by updating Fuel on the source image to the same version as the > target image before porting. But I don't see an obvious workaround for the > non-spur -> spur breakdown. How can we leave users in a situation where it > seems impossible to migrate data between Pharo versions?! > > > > ----- > Cheers, > Sean > -- > View this message in context: http://forum.world.st/Spur-VM-Bug-for-Chinese-Characters-tp4918883p4918943.html > Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com. > |
Re: Spur VM Bug for Chinese Characters?
|
Administrator
|
Thankfully :) But I thought STON does not preserve references. That wouldn't work for all but the simplest models, no?
Cheers,
Sean |
Re: Spur VM Bug for Chinese Characters?
|
|
> On 16 Oct 2016, at 23:02, Sean P. DeNigris <[hidden email]> wrote: > > NorbertHartl wrote >> These changes are rare > > Thankfully :) > > > NorbertHartl wrote >> you are better off using STON > > But I thought STON does not preserve references. That wouldn't work for all > but the simplest models, no? Of course it does, it wouldn't be very good if it didn't ;-) See #testReferenceSharing and #testReferenceCycle for example > > ----- > Cheers, > Sean > -- > View this message in context: http://forum.world.st/Spur-VM-Bug-for-Chinese-Characters-tp4918883p4919011.html > Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com. > |
Re: Spur VM Bug for Chinese Characters?
|
Administrator
|
Great! I fell down on "STONWriterError: custom #stonOn: implementation needed for variable/indexable class" for AbsolutePath. Is there an example of how to handle this? I read the paper and googled, but didn't see one...
Cheers,
Sean |
Re: Spur VM Bug for Chinese Characters?
|
|
I am in the other boat for reading OrderPreservingDictionary. Phil On Mon, Oct 17, 2016 at 3:43 AM, Sean P. DeNigris <[hidden email]> wrote: Sven Van Caekenberghe-2 wrote |
Re: Spur VM Bug for Chinese Characters?
|
|
> On 17 Oct 2016, at 07:36, [hidden email] wrote: > > I am in the other boat for reading OrderPreservingDictionary. Both OrderedDictionary and OrderedIdentityDictionary are handled well, see #testOrderedDictionary. I believe OrderPreservingDictionary is not a standard Pharo part, it is part of PharoExtras. But it is in my main image too. The solution is probably to copy over #stonOn: and #fromSton: from [Ordered]Dictionary to StandardOrderedDictionary. > Phil > > On Mon, Oct 17, 2016 at 3:43 AM, Sean P. DeNigris <[hidden email]> wrote: > Sven Van Caekenberghe-2 wrote > > Of course it does, it wouldn't be very good if it didn't ;-) > > Great! > > I fell down on "STONWriterError: custom #stonOn: implementation needed for > variable/indexable class" for AbsolutePath. Is there an example of how to > handle this? I read the paper and googled, but didn't see one... > > > > ----- > Cheers, > Sean > -- > View this message in context: http://forum.world.st/Spur-VM-Bug-for-Chinese-Characters-tp4918883p4919031.html > Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com. > > |
Re: Spur VM Bug for Chinese Characters?
|
|
> On 17 Oct 2016, at 09:17, Sven Van Caekenberghe <[hidden email]> wrote: > > >> On 17 Oct 2016, at 07:36, [hidden email] wrote: >> >> I am in the other boat for reading OrderPreservingDictionary. > > Both OrderedDictionary and OrderedIdentityDictionary are handled well, see #testOrderedDictionary. > > I believe OrderPreservingDictionary is not a standard Pharo part, it is part of PharoExtras. But it is in my main image too. > I think OrderedDictionary is just a copy of OrderPreservingDictionary, then improved. The idea was that we did not want to break peoples code that relied on OrderPreservingDictionary and thus used another (and better) name. Marcus |
Re: Spur VM Bug for Chinese Characters?
|
|
In reply to this post by Sean P. DeNigris
> On 17 Oct 2016, at 03:43, Sean P. DeNigris <[hidden email]> wrote: > > Sven Van Caekenberghe-2 wrote >> Of course it does, it wouldn't be very good if it didn't ;-) > > Great! > > I fell down on "STONWriterError: custom #stonOn: implementation needed for > variable/indexable class" for AbsolutePath. Is there an example of how to > handle this? I read the paper and googled, but didn't see one... Hmm, Path (the superclass) is handled, it has custom #stonOn: and #fromSton: implementations. Do you have them in your image ? Here is an example that contains embedded absolute paths: STON toStringPretty: (FileLocator desktop / 'foo.txt') resolve. 'FileReference { #filesystem : FileSystem { #workingDirectory : AbsolutePath [ ''Users'', ''sven'', ''Develop'', ''Pharo'' ], #store : MacStore { #maxFileNameLength : ''255'' } }, #path : AbsolutePath [ ''Users'', ''sven'', ''Desktop'', ''foo.txt'' ] }' What does not work for you ? > ----- > Cheers, > Sean > -- > View this message in context: http://forum.world.st/Spur-VM-Bug-for-Chinese-Characters-tp4918883p4919031.html > Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com. > |
Re: Spur VM Bug for Chinese Characters?
|
Administrator
|
They are not included in the stable version from either the Pharo 4.0 MetaC Browser, or Pharo 5.0 Catalog browser, but I copy pasted them from the latest STON package. Thanks.
Cheers,
Sean |
Re: Spur VM Bug for Chinese Characters?
|
Administrator
|
Ugh. FileLocator home / 'Downloads' / 'myfile.ston' writeStreamDo: [ :s | STON put: myDB onStream: s]. and then FileLocator home / 'Downloads' / 'myfile.ston' readStreamDo: [ :s | STON fromStream: s ] resulted in: "STONReaderError: At character 978274: unexpected property name type" Any thoughts on how to debug?
Cheers,
Sean |
Re: Spur VM Bug for Chinese Characters?
|
|
> On 17 Oct 2016, at 21:14, Sean P. DeNigris <[hidden email]> wrote: > > Sean P. DeNigris wrote >> I copy pasted them from the latest STON package. Thanks. > > Ugh. > > FileLocator home / 'Downloads' / 'myfile.ston' writeStreamDo: [ :s | > STON put: myDB onStream: s]. > > and then > > FileLocator home / 'Downloads' / 'myfile.ston' readStreamDo: [ :s | > STON fromStream: s ] > > resulted in: "STONReaderError: At character 978274: unexpected property name > type" > > Any thoughts on how to debug? That obviously depends on what 'myDB' contains, the basic call is OK though: FileLocator home / 'Downloads' / 'myfile.ston' writeStreamDo: [ :s | STON put: STONTestUser dummy onStream: s ]. FileLocator home / 'Downloads' / 'myfile.ston' readStreamDo: [ :s | STON fromStream: s ]. The error you get basically means that something non-simple was used as property name. You could try doing: FileLocator home / 'Downloads' / 'myfile.ston' readStreamDo: [ :s | (STON reader on: s) allowComplexMapKeys: true; next ]. From the class comment of STONReader - allowComplexMapKeys <Boolean> default is false if true, any object is allowed as map key if false, only strings, symbols and numbers are allowed as map keys > ----- > Cheers, > Sean > -- > View this message in context: http://forum.world.st/Spur-VM-Bug-for-Chinese-Characters-tp4918883p4919109.html > Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com. > |
Re: Spur VM Bug for Chinese Characters?
|
Administrator
|
Hmm, if the default writing may use complex keys, shouldn't the default reading do the same? I found it jarring to write using the defaults, and then have the default read fail!
Cheers,
Sean |
Re: Spur VM Bug for Chinese Characters?
|
Administrator
|
In reply to this post by Sven Van Caekenberghe-2
BTW that worked. Thanks!
Cheers,
Sean |
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |