Smalltalk › Pharo › Pharo Smalltalk Users

best practices for using external files for testing

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

7 messages Options

Peter Uhnak

best practices for using external files for testing

Hi,

is there a common/best practice for using external files in tests?

In my specific case I am interested in git-based projects, where I have a big (~1MB) file stored in repository and I would like to use it in my tests.

For GitFileTree project I could presumably use the following to access it:

'OP-XMI' asPackage mcPackage workingCopy repositoryGroup remotes first directory / 'tests' / 'my-test-file.xmi'

This will retrieve the MCPackage of the Package and then retireve where it the repo is actually stored on the disk.

Are there better ways to do this? Could something similar be done with IceBerg?

(p.s. in theory I could compile the entire file (e.g. 1MB) to a method, but that is very ugly to me)

Thanks,
Peter

hernanmd

Re: best practices for using external files for testing

Hi Peter,

In BioSmalltalk I download and extract all test files in the Configuration and then use a method in abstract test class to access test files.

Cheers,

Hernán

2017-04-15 13:52 GMT-03:00 Peter Uhnak <[hidden email]>:

Hi,

is there a common/best practice for using external files in tests?

In my specific case I am interested in git-based projects, where I have a big (~1MB) file stored in repository and I would like to use it in my tests.

For GitFileTree project I could presumably use the following to access it:

'OP-XMI' asPackage mcPackage workingCopy repositoryGroup remotes first directory / 'tests' / 'my-test-file.xmi'

This will retrieve the MCPackage of the Package and then retireve where it the repo is actually stored on the disk.

Are there better ways to do this? Could something similar be done with IceBerg?

(p.s. in theory I could compile the entire file (e.g. 1MB) to a method, but that is very ugly to me)

Thanks,
Peter

kilon.alios

Re: best practices for using external files for testing

In reply to this post by Peter Uhnak

1 mb is not big for git , 1 gb is. It's small.

I version control blender files of 100-500mbs with ease . Git is always very fast.

I have used image files (PNG) for my project ChronosManager ,it fetches them together with the repo, they sit in their own image folder inside the repo folder that contains the code . I then open them with a path I get using filesystem currentdirectory method and then I navigate inside the folder searching using the name of folderminside the github-cache subfolder. You can find the code in that project.

Don't know if that is the best way to do this , it works for me. I also check to make sure that images are not already loaded in the image because git may be fast but Pharo is very slow when loading PNG files. I load them once and save the image in them and I reload only if I change one of the PNGs.

I have also created a mechanism to detect whether there is an update in the github repo and only then pull the git repo. It's a cheap hack where I put version info in the README which I access online and parse. I have created this way an auto update API. I could fine tune it to provide precise meta data of which png is updated and fetch only that but I have not felt the need so far to do this.

On Sat, 15 Apr 2017 at 19:52, Peter Uhnak <[hidden email]> wrote:

Hi,

is there a common/best practice for using external files in tests?

In my specific case I am interested in git-based projects, where I have a big (~1MB) file stored in repository and I would like to use it in my tests.

For GitFileTree project I could presumably use the following to access it:

'OP-XMI' asPackage mcPackage workingCopy repositoryGroup remotes first directory / 'tests' / 'my-test-file.xmi'

This will retrieve the MCPackage of the Package and then retireve where it the repo is actually stored on the disk.

Are there better ways to do this? Could something similar be done with IceBerg?

(p.s. in theory I could compile the entire file (e.g. 1MB) to a method, but that is very ugly to me)

Thanks,
Peter

Peter Uhnak

Re: best practices for using external files for testing

On Mon, Apr 17, 2017 at 12:56:42PM +0000, Dimitris Chloupis wrote:
> 1 mb is not big for git , 1 gb is. It's small.

I meant big for Pharo/to be compiled as a method source code, i.e. Something>>xmlFile ^ '<xml> ... 1MB of string...</xml>'

>
> I version control blender files of 100-500mbs with ease . Git is always
> very fast.
>
> I have used image files (PNG) for my project ChronosManager ,it fetches
> them together with the repo, they sit in their own image folder inside the
> repo folder that contains the code . I then open them with a path I get
> using filesystem currentdirectory method and then I navigate inside the
> folder searching using the name of folderminside the github-cache
> subfolder. You can find the code in that project.
>
> Don't know if that is the best way to do this , it works for me. I also
> check to make sure that images are not already loaded in the image because
> git may be fast but Pharo is very slow when loading PNG files. I load them
> once and save the image in them and I reload only if I change one of the
> PNGs.
>
> I have also created a mechanism to detect whether there is an update in the
> github repo and only then pull the git repo. It's a cheap hack where I put
> version info in the README which I access online and parse. I have created
> this way an auto update API. I could fine tune it to provide precise meta
> data of which png is updated and fetch only that but I have not felt the
> need so far to do this.
> On Sat, 15 Apr 2017 at 19:52, Peter Uhnak <[hidden email]> wrote:
>
> > Hi,
> >
> > is there a common/best practice for using external files in tests?
> >
> > In my specific case I am interested in git-based projects, where I have a
> > big (~1MB) file stored in repository and I would like to use it in my tests.
> >
> > For GitFileTree project I could presumably use the following to access it:
> >
> > 'OP-XMI' asPackage mcPackage workingCopy repositoryGroup remotes first
> > directory / 'tests' / 'my-test-file.xmi'
> >
> > This will retrieve the MCPackage of the Package and then retireve where it
> > the repo is actually stored on the disk.
> >
> > Are there better ways to do this? Could something similar be done with
> > IceBerg?
> >
> > (p.s. in theory I could compile the entire file (e.g. 1MB) to a method,
> > but that is very ugly to me)
> >
> > Thanks,
> > Peter
> >
> >

kilon.alios

Re: best practices for using external files for testing

ah ok that changes everything, I agree. Why use an XML file instead of a ST source file ? maybe using ST may be faster ?

On Mon, Apr 17, 2017 at 7:02 PM Peter Uhnak <[hidden email]> wrote:

On Mon, Apr 17, 2017 at 12:56:42PM +0000, Dimitris Chloupis wrote:
> 1 mb is not big for git , 1 gb is. It's small.

I meant big for Pharo/to be compiled as a method source code, i.e. Something>>xmlFile ^ '<xml> ... 1MB of string...</xml>'

>
> I version control blender files of 100-500mbs with ease . Git is always
> very fast.
>
> I have used image files (PNG) for my project ChronosManager ,it fetches
> them together with the repo, they sit in their own image folder inside the
> repo folder that contains the code . I then open them with a path I get
> using filesystem currentdirectory method and then I navigate inside the
> folder searching using the name of folderminside the github-cache
> subfolder. You can find the code in that project.
>
> Don't know if that is the best way to do this , it works for me. I also
> check to make sure that images are not already loaded in the image because
> git may be fast but Pharo is very slow when loading PNG files. I load them
> once and save the image in them and I reload only if I change one of the
> PNGs.
>
> I have also created a mechanism to detect whether there is an update in the
> github repo and only then pull the git repo. It's a cheap hack where I put
> version info in the README which I access online and parse. I have created
> this way an auto update API. I could fine tune it to provide precise meta
> data of which png is updated and fetch only that but I have not felt the
> need so far to do this.
> On Sat, 15 Apr 2017 at 19:52, Peter Uhnak <[hidden email]> wrote:
>
> > Hi,
> >
> > is there a common/best practice for using external files in tests?
> >
> > In my specific case I am interested in git-based projects, where I have a
> > big (~1MB) file stored in repository and I would like to use it in my tests.
> >
> > For GitFileTree project I could presumably use the following to access it:
> >
> > 'OP-XMI' asPackage mcPackage workingCopy repositoryGroup remotes first
> > directory / 'tests' / 'my-test-file.xmi'
> >
> > This will retrieve the MCPackage of the Package and then retireve where it
> > the repo is actually stored on the disk.
> >
> > Are there better ways to do this? Could something similar be done with
> > IceBerg?
> >
> > (p.s. in theory I could compile the entire file (e.g. 1MB) to a method,
> > but that is very ugly to me)
> >
> > Thanks,
> > Peter
> >
> >

monty-3

Re: best practices for using external files for testing

In reply to this post by Peter Uhnak

XMLParser's XML-Tests-Conformance project, which is automatically generated from the W3C's Conformance Test Suites project (https://www.w3.org/XML/Test/), stores the contents of its files in class methods. This way it's self-contained, portable, and the actual files only need to be downloaded and unzipped to regenerate the TestCases.

Consider extracting it into a separate project with a separate CI job if it gets too big.

> Sent: Saturday, April 15, 2017 at 12:52 PM
> From: "Peter Uhnak" <[hidden email]>
> To: [hidden email]
> Subject: [Pharo-users] best practices for using external files for testing
>
> Hi,
>
> is there a common/best practice for using external files in tests?
>
> In my specific case I am interested in git-based projects, where I have a big (~1MB) file stored in repository and I would like to use it in my tests.
>
> For GitFileTree project I could presumably use the following to access it:
>
> 'OP-XMI' asPackage mcPackage workingCopy repositoryGroup remotes first directory / 'tests' / 'my-test-file.xmi'
>
> This will retrieve the MCPackage of the Package and then retireve where it the repo is actually stored on the disk.
>
> Are there better ways to do this? Could something similar be done with IceBerg?
>
> (p.s. in theory I could compile the entire file (e.g. 1MB) to a method, but that is very ugly to me)
>
> Thanks,
> Peter
>
>

Peter Uhnak

Re: best practices for using external files for testing

In reply to this post by hernanmd

I think the best way would be to profile it I guess, or try all options...

my idea was that I will keep with the repo (because I don't want to pull off-site resources every time the test suite runs (e.g. on travis), and at the same time I didn't want to make the source code big... but I guess I could also download the file from the repo itself (if it won't be accessible locally...) which I guess would make most sense.

Thank you all for ideas. :)

Peter

On Mon, Apr 17, 2017 at 09:09:15AM -0300, Hernán Morales Durand wrote:

> Hi Peter,
>
> In BioSmalltalk I download and extract all test files in the Configuration
> and then use a method in abstract test class to access test files.
> Cheers,
>
> Hernán
>
>
> 2017-04-15 13:52 GMT-03:00 Peter Uhnak <[hidden email]>:
>
> > Hi,
> >
> > is there a common/best practice for using external files in tests?
> >
> > In my specific case I am interested in git-based projects, where I have a
> > big (~1MB) file stored in repository and I would like to use it in my tests.
> >
> > For GitFileTree project I could presumably use the following to access it:
> >
> > 'OP-XMI' asPackage mcPackage workingCopy repositoryGroup remotes first
> > directory / 'tests' / 'my-test-file.xmi'
> >
> > This will retrieve the MCPackage of the Package and then retireve where it
> > the repo is actually stored on the disk.
> >
> > Are there better ways to do this? Could something similar be done with
> > IceBerg?
> >
> > (p.s. in theory I could compile the entire file (e.g. 1MB) to a method,
> > but that is very ugly to me)
> >
> > Thanks,
> > Peter
> >
> >