'VOMongoConnectionError' when dowloading mcz from smalltalkhub.com

Christophe,

I appreciate your effort, but I assume that you aren't finished yet?

When I use the url from your message I get a `Not Found` error (note that I explicitly included a trailing slash in the url):

This is a bug (or omission) as well, since other Monticello sites produce the expected HTML output with or without a trailing slash.

Without the trailing slash I get the same (non-html) response that was causing a problem before ... although something must have changed, because the GemStone command is now producing an error instead of an empty list ... Once you get the static site to produce the expected output, I'm confident that the GemStone errors will go away (this code has been running for about a decade on all Monticello html repositories).

I suggest that you look at what is produced on SqueakSource (http://www.squeaksource.com/MooseSQL/) as an example of expected output of the mcz file listing ... this html page format has been used since 2003 for ALL (valid) monticello repositories.

If it is not clear, the `?format=raw` option is a recent Pharo only option and when the dynamic site was running and the `?format-raw` option was OMITTED it produced output compatible with (http://www.squeaksource.com/MooseSQL/) ...

Restoring HTML is *REQUIRED* for the static site to be a faithful replacement of the dynamic site... the format that is produced today is only compatible with the Pharo only `?format=raw` option.

Bug or omission ... the static site is currently in worse shape than the dynamic site before the swap ...

Dale

Le 28 août 2020 à 10:15, Sven Van Caekenberghe [hidden email] a écrit :

On 28 Aug 2020, at 10:13, Christophe Demarey [hidden email] wrote:

Smalltalkhub is now to able to distinguish between raw and not raw mcz listing requests.
Ex: 
	http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main?format=raw
	http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main/

I already use the apache directory index for another page so I will not be able to modify this one. I guess the current index will «  not work » because glass expect some HTML structure.
Could you confirm and tell me what is the expected structure?

See my earlier mail with code.

Thanks Sven

Christophe,

Again, I appreciate your effort, but you are not quite there.

If you look at SqueakSource or GemSource html (attached), the html uses <br /> to create newlines on the page and in the Smalltalkhub page, you've inserted a newline as part of the href filename, while SqueakSource and GemSource (and the other monticello sites) do not include a newline in the href, so for better or worse, in GemStone the method MCReader>>canReadFileNamed: expects the filename to end with `.mcz` not a newline and all of the filenames parsed from the SmalltalkHub page are discarded as non-Monticello files.

So if you eliminate the trailing newline in the href, the GemStone code should finally be happy.

Thanks for your effort!

Dale

I understand that he new site is unusable if tools do not get the expected input. Sorry for the inconvenience. I was just not aware of this as Pharo uses the raw format.

I had to set up a CGI to produce an html listing from the raw file.

It should now be ok. Could you tell me if it works in GemStone now?

Christophe

Le 28 août 2020 à 19:21, Dale Henrichs <[hidden email]> a écrit :

Christophe,

I appreciate your effort, but I assume that you aren't finished yet?

When I use the url from your message I get a `Not Found` error (note that I explicitly included a trailing slash in the url):

<dkbjiegjmlbelgio.png>

This is a bug (or omission) as well, since other Monticello sites produce the expected HTML output with or without a trailing slash.

Without the trailing slash I get the same (non-html) response that was causing a problem before ... although something must have changed, because the GemStone command is now producing an error instead of an empty list ... Once you get the static site to produce the expected output, I'm confident that the GemStone errors will go away (this code has been running for about a decade on all Monticello html repositories).

I suggest that you look at what is produced on SqueakSource (http://www.squeaksource.com/MooseSQL/) as an example of expected output of the mcz file listing ... this html page format has been used since 2003 for ALL (valid) monticello repositories.

If it is not clear, the `?format=raw` option is a recent Pharo only option and when the dynamic site was running and the `?format-raw` option was OMITTED it produced output compatible with (http://www.squeaksource.com/MooseSQL/) ...

Restoring HTML is *REQUIRED* for the static site to be a faithful replacement of the dynamic site... the format that is produced today is only compatible with the Pharo only `?format=raw` option.

Bug or omission ... the static site is currently in worse shape than the dynamic site before the swap ...

Dale

On 8/28/20 2:12 AM, Christophe Demarey wrote:

Le 28 août 2020 à 10:15, Sven Van Caekenberghe [hidden email] a écrit :

On 28 Aug 2020, at 10:13, Christophe Demarey [hidden email] wrote:

Smalltalkhub is now to able to distinguish between raw and not raw mcz listing requests.
Ex: 
	http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main?format=raw
	http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main/

I already use the apache directory index for another page so I will not be able to modify this one. I guess the current index will «  not work » because glass expect some HTML structure.
Could you confirm and tell me what is the expected structure?

See my earlier mail with code.

Thanks Sven

squeaksource.html (10K) Download Attachment
gemsource.html (127K) Download Attachment

Sooooo close, the interactive tests are passing, but there is a use case that is popping up in a travis job[1] where this url http://smalltalkhub.com/mc/Swazoo/Swazoo/main is being used (no trailing slash) and this form of the url still produces the `raw` listing, on SqueakSource (and other sites) the trailing slash is not required to get the html-based listing ...

Again, I appreciate your effort, I hopeful that this is the last detail,

Dale

[1] https://travis-ci.org/github/GsDevKit/GsDevKit_home/jobs/721523220

I hope this time it will be fine. I removed the newline characters.

I will also check if I can keep the original url in the browser rather than the redirected one.

Regards,

Christophe.

Le 29 août 2020 à 18:41, Dale Henrichs <[hidden email]> a écrit :

Christophe,

Again, I appreciate your effort, but you are not quite there.

If you look at SqueakSource or GemSource html (attached), the html uses <br /> to create newlines on the page and in the Smalltalkhub page, you've inserted a newline as part of the href filename, while SqueakSource and GemSource (and the other monticello sites) do not include a newline in the href, so for better or worse, in GemStone the method MCReader>>canReadFileNamed: expects the filename to end with `.mcz` not a newline and all of the filenames parsed from the SmalltalkHub page are discarded as non-Monticello files.

So if you eliminate the trailing newline in the href, the GemStone code should finally be happy.

Thanks for your effort!

Dale

On 8/28/20 3:05 PM, Christophe Demarey wrote:

I understand that he new site is unusable if tools do not get the expected input. Sorry for the inconvenience. I was just not aware of this as Pharo uses the raw format.

I had to set up a CGI to produce an html listing from the raw file.

It should now be ok. Could you tell me if it works in GemStone now?

Christophe

Le 28 août 2020 à 19:21, Dale Henrichs <[hidden email]> a écrit :

Christophe,

I appreciate your effort, but I assume that you aren't finished yet?

When I use the url from your message I get a `Not Found` error (note that I explicitly included a trailing slash in the url):

<dkbjiegjmlbelgio.png>

This is a bug (or omission) as well, since other Monticello sites produce the expected HTML output with or without a trailing slash.

Without the trailing slash I get the same (non-html) response that was causing a problem before ... although something must have changed, because the GemStone command is now producing an error instead of an empty list ... Once you get the static site to produce the expected output, I'm confident that the GemStone errors will go away (this code has been running for about a decade on all Monticello html repositories).

I suggest that you look at what is produced on SqueakSource (http://www.squeaksource.com/MooseSQL/) as an example of expected output of the mcz file listing ... this html page format has been used since 2003 for ALL (valid) monticello repositories.

If it is not clear, the `?format=raw` option is a recent Pharo only option and when the dynamic site was running and the `?format-raw` option was OMITTED it produced output compatible with (http://www.squeaksource.com/MooseSQL/) ...

Restoring HTML is *REQUIRED* for the static site to be a faithful replacement of the dynamic site... the format that is produced today is only compatible with the Pharo only `?format=raw` option.

Bug or omission ... the static site is currently in worse shape than the dynamic site before the swap ...

Dale

On 8/28/20 2:12 AM, Christophe Demarey wrote:

Le 28 août 2020 à 10:15, Sven Van Caekenberghe [hidden email] a écrit :

On 28 Aug 2020, at 10:13, Christophe Demarey [hidden email] wrote:

Smalltalkhub is now to able to distinguish between raw and not raw mcz listing requests.
Ex: 
	http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main?format=raw
	http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main/

I already use the apache directory index for another page so I will not be able to modify this one. I guess the current index will «  not work » because glass expect some HTML structure.
Could you confirm and tell me what is the expected structure?

See my earlier mail with code.

Thanks Sven

<squeaksource.html><gemsource.html>

Hi Dale,

Le 29 août 2020 à 19:40, Dale Henrichs <[hidden email]> a écrit :

Sooooo close, the interactive tests are passing, but there is a use case that is popping up in a travis job[1] where this url http://smalltalkhub.com/mc/Swazoo/Swazoo/main is being used (no trailing slash) and this form of the url still produces the `raw` listing,

Are you sure? For me, http://smalltalkhub.com/mc/Swazoo/Swazoo/main produces the html listing

I am sure that the travis build is still failing[1] after having passed 2 weeks ago, before the dynamic site was switched out.

I am sure the a manual test (different code path than the travis site) that was failing with the new site is now passing.

I did discovered that that the reason I _thought_, that the html page wasn't being displayed was because the old page for that url was cached in my web browser:( Sorry for the red herring.

Tomorrow I will dig a little deeper and let you know what is causing the travis test to fail ... I've seen this particular error with the Swazoo repository show up in production tests that I run locally, so I should be able to identify the cause of the failures.

Dale

[1] https://travis-ci.org/github/GsDevKit/GsDevKit_home/jobs/721523220#L2354

Okay, I've finally sorted things out ... and it gets down to the difference between MCHttpRepository>>parseFileNamesFromStream: returning an empty list (the original problem) and returning a list of the wrong thing the current problem). When I reported that the "manual test" was passing, that was true, but instead of listing the names of the packages, the "manual test" was returning a list of urls, and not a list of packageNames and I didn't recognize the difference --- but the code that is executing in the travis test does recognize the difference.

The GemStone code does not special case SmalltalkHub, while Pharo does handle SmalltalkHub as a special case and to actually see the difference between the two, you have to use the following expression (the  MCHttpRepository class>>location:user:password: is intercepted and creates an MCSqueaksourceRepository instance, which has different implementation of parseFileNamesFromStream:):

(MCHttpRepository new
    location: 'http://smalltalkhub.com/mc/Swazoo/Swazoo/main';
    user: '';
    password: '';
    yourself) allFileNames

Both Pharo and GemStone, use the pretty much same code for MCHttpRepository>>parseFileNamesFromStream: (this is the GemStone version):

parseFileNamesFromStream: aStream
  | names fullName |
  names := OrderedCollection new.
  [ aStream atEnd ]
    whileFalse: [ 
      [ 
      aStream upTo: $<.
      aStream atEnd
        ifTrue: [ true ]
        ifFalse: [ 
          {$a.
          $A.
          nil} includes: aStream next ] ]
        whileFalse.
      aStream upTo: $".
      aStream atEnd
        ifFalse: [ 
          fullName := aStream upTo: $".
          names add: fullName unescapePercents ] ].
  ^ names

and both return the same result when pointed at http://www.squeaksource.com/MooseSQL (you should recognize the EyeCollectionInspector from Pharo3.0:) and the other inspector is from tODE ... Note that both are producing a list of .mcz file names:

(MCHttpRepository new
    location: 'http://smalltalkhub.com/mc/Swazoo/Swazoo/main';
    user: '';
    password: '';
    yourself) allFileNames

is used, both Pharo and GemStone produce the same (wrong) list of package names from SmalltalkHub:

The travis test is trying to scan for ConfigurationOfSwazoo2 package names in the list and that algorithm is failing because it expects only a list of .mcz file names as produced like that produced for http://www.squeaksource.com/MooseSQL ...

On 8/30/20 3:07 AM, Christophe Demarey wrote:

Hi Dale,

Le 29 août 2020 à 19:40, Dale Henrichs <[hidden email]> a écrit :

Sooooo close, the interactive tests are passing, but there is a use case that is popping up in a travis job[1] where this url http://smalltalkhub.com/mc/Swazoo/Swazoo/main is being used (no trailing slash) and this form of the url still produces the `raw` listing,

Are you sure? For me, http://smalltalkhub.com/mc/Swazoo/Swazoo/main produces the html listing

I am sure that the travis build is still failing[1] after having passed 2 weeks ago, before the dynamic site was switched out.

I am sure the a manual test (different code path than the travis site) that was failing with the new site is now passing.

I did discovered that that the reason I _thought_, that the html page wasn't being displayed was because the old page for that url was cached in my web browser:( Sorry for the red herring.

Tomorrow I will dig a little deeper and let you know what is causing the travis test to fail ... I've seen this particular error with the Swazoo repository show up in production tests that I run locally, so I should be able to identify the cause of the failures.

Dale

[1] https://travis-ci.org/github/GsDevKit/GsDevKit_home/jobs/721523220#L2354