Storing test data in methods
Locked
 
|
I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in.
In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it. Any ideas? Norbert |
|
|
Have a look at InstanceEncoder
http://www.smalltalkhub.com/#!/~hernan/InstanceEncoder Hernán El 23/07/2013 7:47, Norbert Hartl escribió: > I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in. > > In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it. > > Any ideas? > > Norbert > |
|
|
In reply to this post by NorbertHartl
Norbert,
for similar purposes (INI-File contents etc.), in order to keep them under version control in envy, I wrote methods that produced a String that could recreate the objects when evaluated. Such a piece of text makes for a great class or instance method that you can compile into some Smalltalk class and get it under version control together with the code that uses the objects. Example: IniFileEntry with iVars #key and #value instance method: creationStringOn: aStream aStream nextPutAll: self class name; nextPut: $=; nextPutAll: self value asString. Thus you can ask all your IniFileEntries to write their own creation string onto a Stream and compile that into a method. You can see me talking about this and other related stuff here: http://www.objektfabrik.de/Downloads/VASTForum2010/VFM10-04-GetBackToShapre-Tuchel.mp4 (chapter "Get Everything under Version control, ca. 1/3 through the recording) HTH, Joachim m 23.07.13 12:47, schrieb Norbert Hartl: > I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in. > > In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it. > > Any ideas? > > Norbert > |
|
|
HI again,
I just saw the link I posted doesn't work right now. The objektfabrik site moved to a new server a while agow and it seems a few files didn't make it. The upload is currently in process, so please check back in an hour or so. Joachim Am 23.07.13 13:25, schrieb [hidden email]: > Norbert, > > for similar purposes (INI-File contents etc.), in order to keep them > under version control in envy, I wrote methods that produced a String > that could recreate the objects when evaluated. Such a piece of text > makes for a great class or instance method that you can compile into > some Smalltalk class and get it under version control together with > the code that uses the objects. > > Example: IniFileEntry with iVars #key and #value > instance method: > > creationStringOn: aStream > aStream nextPutAll: self class name; nextPut: $=; nextPutAll: self > value asString. > > Thus you can ask all your IniFileEntries to write their own creation > string onto a Stream and compile that into a method. You can see me > talking about this and other related stuff here: > http://www.objektfabrik.de/Downloads/VASTForum2010/VFM10-04-GetBackToShapre-Tuchel.mp4 > (chapter "Get Everything under Version control, ca. 1/3 through the > recording) > > HTH, > > Joachim > > > m 23.07.13 12:47, schrieb Norbert Hartl: >> I try to untangle my modules. I have a parser that creates a model >> and a tool that uses that model. For testing it is convenient to just >> generate the model and work with it. But this way I introduce >> dependencies which always pulls everything in. >> >> In order to untangle I would need a way to easily serialize the >> runtime model into methods or something similar. Anything that can be >> transported via monticello is ok. I just don't want to have >> dependencies to present of external files. That would just change the >> problem but not solving it. >> >> Any ideas? >> >> Norbert >> > > > |
Re: Storing test data in methods
|
|
In reply to this post by hernanmd
Am 23.07.2013 um 13:20 schrieb Hernán Morales Durand <[hidden email]>: > Have a look at InstanceEncoder > > http://www.smalltalkhub.com/#!/~hernan/InstanceEncoder > Thanks, how would you say it works compared to store an Fuel ByteArray? Norbert > > El 23/07/2013 7:47, Norbert Hartl escribió: >> I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in. >> >> In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it. >> >> Any ideas? >> >> Norbert >> > > |
Re: Storing test data in methods
|
|
In reply to this post by jtuchel
Joachim,
Am 23.07.2013 um 13:25 schrieb "[hidden email]" <[hidden email]>: > Norbert, > > for similar purposes (INI-File contents etc.), in order to keep them under version control in envy, I wrote methods that produced a String that could recreate the objects when evaluated. Such a piece of text makes for a great class or instance method that you can compile into some Smalltalk class and get it under version control together with the code that uses the objects. > > Example: IniFileEntry with iVars #key and #value > instance method: > > creationStringOn: aStream > aStream nextPutAll: self class name; nextPut: $=; nextPutAll: self value asString. > > Thus you can ask all your IniFileEntries to write their own creation string onto a Stream and compile that into a method. You can see me talking about this and other related stuff here: http://www.objektfabrik.de/Downloads/VASTForum2010/VFM10-04-GetBackToShapre-Tuchel.mp4 (chapter "Get Everything under Version control, ca. 1/3 through the recording) > I think I have to options: 1. The most generic one is serializing into Fuel and compile the ByteArray in a method. 2. The most readable one will be using STON and compiling the STON string into a method Maybe I should try both. thanks, Norbert > > m 23.07.13 12:47, schrieb Norbert Hartl: >> I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in. >> >> In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it. >> >> Any ideas? >> >> Norbert >> > > |
|
|
Norbert,
Am 23.07.13 14:07, schrieb Norbert Hartl: > Joachim, > > I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :) We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless. > I think I have to options: > > 1. The most generic one is serializing into Fuel and compile the ByteArray in a method. > 2. The most readable one will be using STON and compiling the STON string into a method > > Maybe I should try both. I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such. Joachim |
Re: Storing test data in methods
|
|
Am 23.07.2013 um 14:23 schrieb "[hidden email]" <[hidden email]>: > Norbert, > > Am 23.07.13 14:07, schrieb Norbert Hartl: >> Joachim, >> >> I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :) > > We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless. >> I think I have to options: >> >> 1. The most generic one is serializing into Fuel and compile the ByteArray in a method. >> 2. The most readable one will be using STON and compiling the STON string into a method >> >> Maybe I should try both. > I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such. > I'll report further findings, Norbert |
Re: Storing test data in methods
|
|
In reply to this post by jtuchel
Am 23.07.2013 um 14:23 schrieb "[hidden email]" <[hidden email]>: > Norbert, > > Am 23.07.13 14:07, schrieb Norbert Hartl: >> Joachim, >> >> I'm aware that you can do serialization/deserialization yourself. I was asking for something to help along the way because a have quite some model classes to cover. I want a more generalized way of doing it. As often I really think about things _after_ I've sent the mail :) > > We seem to have a lot in common here. Regard it this way: sending the mail made you think again. So the mail itself was all but useless. >> I think I have to options: >> >> 1. The most generic one is serializing into Fuel and compile the ByteArray in a method. >> 2. The most readable one will be using STON and compiling the STON string into a method >> >> Maybe I should try both. > I'd be very interested in what you find. I guess Fuel is smaller and maybe even faster. STON, OTOH, is much friendlier for comparing versions and such. > (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) size is 237445 Bytes. Serializing [ MAPExampleModels class compile: 'tcap ^ ', (ByteArray streamContents: [:stream| FLSerializer serialize: TCAP current model on: stream ]) storeString ] timeToRun took 647 ms but [FLMaterializer materializeFromByteArray: MAPExampleModels tcap] timeToRun took only 30ms. Basically I'm not fond of putting too much binary stuff into compiled methods but it is the most easy and reliable way of providing tests with test data. The whole grammar is stored in methods as well. It is text but…. I think I stay with it for a while. Norbert |
Re: Storing test data in methods
|
|
Norbert, does the model have lots of strings? Because if so you can try using the compression. It may increase speed. There is always a trade off between the additional time of compressing/uncompressing. But if the model has quite an amount of strings, since writing to disk is also slow and the final stream would be smaller, it could be faster. Cheers, On Tue, Jul 23, 2013 at 9:40 AM, Norbert Hartl <[hidden email]> wrote:
Mariano http://marianopeck.wordpress.com |
Re: Storing test data in methods
|
|
Mariano,
Am 23.07.2013 um 14:43 schrieb Mariano Martinez Peck <[hidden email]>: thanks. Yes, the model is a lot of Dictionaries having string keys. And there are lots more. The usage of the model is write once read many. And 30ms is good enough for me. So I might try compression only to see the difference. Well, and maybe to reduce storage size. The mcz is now 261 kb. Thanks, I'll see but I am already satisfied with the current approach. Norbert
|
Re: Storing test data in methods
|
|
On Tue, Jul 23, 2013 at 9:48 AM, Norbert Hartl <[hidden email]> wrote:
Mmmm I thought it was ready, sorry my bad. Seems it was not ready nor integrated: https://code.google.com/p/fuel/issues/detail?id=160
But I might be wrong. Max? Cheers,
Mariano http://marianopeck.wordpress.com |
|
|
In reply to this post by NorbertHartl
El 23/07/2013 9:01, Norbert Hartl escribió:
> > Am 23.07.2013 um 13:20 schrieb Hernán Morales Durand <[hidden email]>: > >> Have a look at InstanceEncoder >> >> http://www.smalltalkhub.com/#!/~hernan/InstanceEncoder >> > Thanks, how would you say it works compared to store an Fuel ByteArray? > This is not difficult to test. First save your image and download this file, is the chromosome 22 of a human reference genome (about 10mb of gzip file): | client ttr object fileName fStream | fileName := 'hs_alt_HuRef_chr22.fa.gz'. [ client := (FTPClient openOnHostNamed: 'ftp.ncbi.nlm.nih.gov') loginUser: 'anonymous' password: ''; binary; changeDirectoryTo: 'genomes/H_sapiens/CHR_22'. (FileStream newFileNamed: fileName) binary; nextPutAll: (client getFileNamed: fileName); close ] on: NetworkError, LoginFailedException do: [ : ex | self error: 'Connection failed' ]. Now storing with InstanceEncoder: fileName := 'hs_alt_HuRef_chr22.fa.gz'. ttr := [ Object store: fileName asFileReference contents asSelector: #storedObjectInstanceEncoder ] timeToRun. (Duration milliSeconds: ttr) asSeconds. And with Fuel Smalltalk garbageCollect. fileName := 'hs_alt_HuRef_chr22.fa.gz'. ttr := [ Object class compile: 'storedObjectFuel ^ ', (ByteArray streamContents: [ : stream | FLSerializer serialize: fileName asFileReference contents on: stream ]) storeString ] timeToRun. (Duration milliSeconds: ttr) asSeconds. Results for writing =================== InstanceEncoder avg: 8 sec. Fuel avg: 45 sec. And reading scripts: [ Object storedObjectInstanceEncoder ] timeToRun. [ FLMaterializer materializeFromByteArray: Object storedObjectFuel ] timeToRun Results for reading =================== InstanceEncoder avg: 3955 msec Fuel avg: 70 msec And the final cleaning of course :) Object class removeSelector: #storedObjectFuel. Object class removeSelector: #storedObjectInstanceEncoder. Cheers, Hernán > Norbert >> >> El 23/07/2013 7:47, Norbert Hartl escribió: >>> I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in. >>> >>> In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it. >>> >>> Any ideas? >>> >>> Norbert >>> >> >> > > > . > |
Re: Storing test data in methods
|
|
Hernán,
Am 23.07.2013 um 16:13 schrieb Hernán Morales Durand <[hidden email]>: > El 23/07/2013 9:01, Norbert Hartl escribió: >> >> Am 23.07.2013 um 13:20 schrieb Hernán Morales Durand <[hidden email]>: >> >>> Have a look at InstanceEncoder >>> >>> http://www.smalltalkhub.com/#!/~hernan/InstanceEncoder >>> >> Thanks, how would you say it works compared to store an Fuel ByteArray? >> > > This is not difficult to test. First save your image and download this file, is the chromosome 22 of a human reference genome (about 10mb of gzip file): > > | client ttr object fileName fStream | > fileName := 'hs_alt_HuRef_chr22.fa.gz'. > [ client := (FTPClient openOnHostNamed: 'ftp.ncbi.nlm.nih.gov') > loginUser: 'anonymous' password: ''; > binary; > changeDirectoryTo: 'genomes/H_sapiens/CHR_22'. > (FileStream newFileNamed: fileName) > binary; > nextPutAll: (client getFileNamed: fileName); > close ] > on: NetworkError, LoginFailedException > do: [ : ex | self error: 'Connection failed' ]. > > Now storing with InstanceEncoder: > > fileName := 'hs_alt_HuRef_chr22.fa.gz'. > ttr := [ Object store: fileName asFileReference contents asSelector: #storedObjectInstanceEncoder ] timeToRun. > (Duration milliSeconds: ttr) asSeconds. > > And with Fuel > > Smalltalk garbageCollect. > fileName := 'hs_alt_HuRef_chr22.fa.gz'. > ttr := [ Object class compile: 'storedObjectFuel > ^ ', (ByteArray streamContents: [ : stream | FLSerializer serialize: fileName asFileReference contents on: stream ]) storeString ] timeToRun. > (Duration milliSeconds: ttr) asSeconds. > > Results for writing > =================== > InstanceEncoder avg: 8 sec. > Fuel avg: 45 sec. > > And reading scripts: > > [ Object storedObjectInstanceEncoder ] timeToRun. > [ FLMaterializer materializeFromByteArray: Object storedObjectFuel ] timeToRun > > Results for reading > =================== > InstanceEncoder avg: 3955 msec > Fuel avg: 70 msec > > And the final cleaning of course :) > > Object class removeSelector: #storedObjectFuel. > Object class removeSelector: #storedObjectInstanceEncoder. > Norbert >> Norbert >>> >>> El 23/07/2013 7:47, Norbert Hartl escribió: >>>> I try to untangle my modules. I have a parser that creates a model and a tool that uses that model. For testing it is convenient to just generate the model and work with it. But this way I introduce dependencies which always pulls everything in. >>>> >>>> In order to untangle I would need a way to easily serialize the runtime model into methods or something similar. Anything that can be transported via monticello is ok. I just don't want to have dependencies to present of external files. That would just change the problem but not solving it. >>>> >>>> Any ideas? >>>> >>>> Norbert >>>> >>> >>> >> >> >> . >> > > |
|
|
In reply to this post by Mariano Martinez Peck
On 23.07.2013, at 15:32, Mariano Martinez Peck <[hidden email]> wrote:
No, that issue is still open. You're welcome to use the experimental version though (Fuel-MaxLeske.757.mcz in http://www.squeaksource.com/FuelExperiments) but keep in mind that this is a few months behind stable. The other idea however, that Mariano describes in the linked issue, is available but probably not desirable: simply pass a compressing stream to Fuel (see FLGZippedBasicSerializationTest for an example on how to do that). That will compress *all* data, not only strings (which is the idea outlined in the linked issue). Cheers, Max (Note: I don't read the users list so please put me on the CC explicitly if you want me to read on)
|
Re: Storing test data in methods
|
|
Max,
Am 23.07.2013 um 17:27 schrieb Max Leske <[hidden email]>: I'm not sure the priority is high enough to try. I don't think that will gain anything. A monticello package is written as zip. So I wouldn't expect much from an additional compression. thanks, Norbert
|
Re: Storing test data in methods
|
|
On Jul 23, 2013, at 5:33 , Norbert Hartl <[hidden email]> wrote: Unless MAPExampleModels tcap is something like ^'mySerializedTCAP.bin' asFile readStream binary contents instead of tcap ^#( 0 0 0 … ), compressing the fuel binary output would save you some runtime memory (method literal size) / .changes space (method source), though for test resources I guess that's a less relevant aspect than for other resources. Cheers, Henry
|
Re: Storing test data in methods
|
|
Right! They difference is in the image. I'll check that. It might come in handy when I have more of these methods. But then I need to try because every method is used multiple times and having it compressed in the image will add some time to every test run. And then it is measuring penalties (again) thanks, Norbert
|
«
Return to Pharo Smalltalk Users
|
1 view|%1 views
| Free forum by Nabble | Edit this page |