NetClients.HTTP

Hi all,

I wish to monitor a web site with a script so I know if it goes down.

I hunted around in the source in packages/net, and got this to work to
an onsite web server...

#!/usr/local/bin/gst -f
   PackageLoader fileInPackage: 'NetClients'.
   siteurl := 'http://localhost/index.html'.
   request := NetClients.URIResolver openStreamOn: siteurl ifFail: [
Transcript show: 'Request failed'].
   Transcript show: request contents.

Now the host to be monitored is a Moodle server, but when I use the same
code with the URL of the Moodle server I get an error. For instance,
with code above and the siteurl below:-

siteurl := 'http://moodle.org/'.   "moodle.org shows same error as our site"

this is the output...
Request failedObject: TextCollector new "<0x40372180>" error: method is
responsibility of a subclass
SystemExceptions.SubclassResponsibility(Exception)>>signal
(ExcHandling.st:254)
SystemExceptions.SubclassResponsibility class(Exception class)>>signal
(ExcHandling.st:151)
TextCollector(Object)>>subclassResponsibility (Object.st:1365)
TextCollector(Stream)>>atEnd (Stream.st:324)
TextCollector(Stream)>>nextPutAllOn: (Stream.st:308)
TextCollector(Stream)>>upToEnd (Stream.st:203)
TextCollector(Stream)>>contents (Stream.st:192)
UndefinedObject>>executeStatements (sitemonitor.st:10)

Help to get the body of the http response from moodle.org would be much
appreciated.

Thanks
Stephen

_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

On 03/28/2011 09:52 PM, Stephen wrote:
>    request := NetClients.URIResolver openStreamOn: siteurl ifFail: [
> Transcript show: 'Request failed'].
>    Transcript show: request contents.

The problem is in the failure.  The error happens simply because your
ifFail block is returning the transcript itself.

The first step in analyzing the failure is to lift the abstraction
provided by URIResolver: this

NetClients.HTTP.HTTPClient exampleURL: 'http://moodle.org/' host: 'moodle.org' port: 80

provides a better error:

Object: MimeScanner new "<0x2b3b619fb4e0>" error: cannot step back twice
Error(Exception)>>signal (ExcHandling.st:254)
Error(Exception)>>signal: (ExcHandling.st:264)
NetClients.MIME.MimeScanner(Object)>>error: (SysExcept.st:1415)
NetClients.MIME.MimeScanner(NetClients.MIME.SimpleScanner)>>stepBack (NetClients.star#VFS.ZipFile/MIME.st:432)
NetClients.MIME.MimeScanner(NetClients.MIME.SimpleScanner)>>scanWhile: (NetClients.star#VFS.ZipFile/MIME.st:421)
NetClients.MIME.MimeScanner(NetClients.MIME.RFC822Scanner)>>skipWhiteSpace (NetClients.star#VFS.ZipFile/MIME.st:2790)
NetClients.MIME.HeaderField class>>readFieldNameFrom: (NetClients.star#VFS.ZipFile/MIME.st:1442)
NetClients.MIME.HeaderField class>>readFrom: (NetClients.star#VFS.ZipFile/MIME.st:1449)
NetClients.MIME.MimeEntity>>parseFieldFrom: (NetClients.star#VFS.ZipFile/MIME.st:1142)
NetClients.MIME.MimeEntity>>parseFieldsFrom: (NetClients.star#VFS.ZipFile/MIME.st:1151)
NetClients.HTTP.HTTPResponse>>parseResponse: (NetClients.star#VFS.ZipFile/HTTP.st:520)

So the next step is grabbing the response with wireshark.  You can put
it into a file and "serve it" with

nc -l localhost 8080 < g

if you change the gst script to

NetClients.HTTP.HTTPClient exampleURL: 'http://moodle.org/' host: 'localhost' port: 8080

After some attempts it looks like it's choking on a header "Expires: ".
I think this fixes it, but it needs more testing:

diff --git a/packages/net/MIME.st b/packages/net/MIME.st
index 0460bc2..3bbc8da 100644
--- a/packages/net/MIME.st
+++ b/packages/net/MIME.st
@@ -588,6 +588,7 @@ Object subclass: SimpleScanner [
  <category: 'stream interface -- reading'>
  lookahead notNil ifTrue: [^lookahead].
  self atEnd ifTrue: [^nil].
+ hereChar := nil.
  lookahead := source next.
  ^lookahead
     ]
@@ -1144,8 +1145,11 @@ MessageElement subclass: MimeEntity [
  <category: 'parsing'>
  | cr nl |
 
- [(cr := rfc822Stream peekFor: Character cr)
-    | (nl := rfc822Stream peekFor: Character nl)]
+ [(cr := rfc822Stream hereChar == Character cr)
+             ifTrue: [rfc822Stream step].
+ (nl := rfc822Stream hereChar == Character nl)
+             ifTrue: [rfc822Stream step].
+        cr or: [nl]]
  whileFalse: [self parseFieldFrom: rfc822Stream]
     ]
 
@@ -2863,14 +2867,13 @@ MailScanner subclass: RFC822Scanner [
  <category: 'private'>
  | char |
  self atEnd ifTrue: [^false].
- char := source next.
+ char := self peek.
  ^((self classificationMaskFor: char) anyMask: WhiteSpaceMask)
     ifFalse:
- [lookahead := char.
- self resetToken.
+ [self resetToken.
  false]
     ifTrue:
- [self sourceTrailNextPut: char.
+ [self next. self sourceTrailNextPut: char.
  true]
     ]
 

Travis, do you know who wrote the MIME/RFC822 packages for VW?
Cincom released them as open source, so GNU Smalltalk's packages
are based on those.

Paolo

_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

Hi

Thanks for looking into this, Paolo, and for showing how the code could
best be tested.

I have been away with work for a week with work and hence the delay
responding.

I've applied your patch to latest GST from Git today (3.2.90-3cd3a73),
however there is still an error.

~/gst-devel $ ./sitemonitor.st
Object: MimeScanner new "<0x4039f568>" error: Invalid Field (Missing colon)
Error(Exception)>>signal (ExcHandling.st:254)
Error(Exception)>>signal: (ExcHandling.st:264)
NetClients.MIME.MimeScanner(Object)>>error: (SysExcept.st:1415)
NetClients.MIME.MimeScanner(NetClients.MIME.SimpleScanner)>>notify:
(NetClients.star#VFS.ZipFile/MIME.st:356)
NetClients.MIME.MimeScanner(NetClients.MIME.SimpleScanner)>>mustMatch:notify:
(NetClients.star#VFS.ZipFile/MIME.st:386)
NetClients.MIME.HeaderField class>>readFieldNameFrom:
(NetClients.star#VFS.ZipFile/MIME.st:1443)
NetClients.MIME.HeaderField class>>readFrom:
(NetClients.star#VFS.ZipFile/MIME.st:1451)
NetClients.MIME.MimeEntity>>parseFieldFrom:
(NetClients.star#VFS.ZipFile/MIME.st:1141)
NetClients.MIME.MimeEntity>>parseFieldsFrom:
(NetClients.star#VFS.ZipFile/MIME.st:1153)
NetClients.HTTP.HTTPResponse>>parseResponse:
(NetClients.star#VFS.ZipFile/HTTP.st:520)
NetClients.HTTP.HTTPResponse class(NetClients.NetResponse
class)>>fromClient: (NetClients.star#VFS.ZipFile/Base.st:951)
NetClients.HTTP.HTTPProtocolInterpreter(NetClients.NetProtocolInterpreter)>>getResponse
(NetClients.star#VFS.ZipFile/Base.st:792)
NetClients.HTTP.HTTPProtocolInterpreter>>readResponseInto:
(NetClients.star#VFS.ZipFile/HTTP.st:207)
NetClients.HTTP.HTTPProtocolInterpreter>>get:requestHeaders:into:
(NetClients.star#VFS.ZipFile/HTTP.st:150)
NetClients.HTTP.HTTPClient>>get:requestHeaders:into:
(NetClients.star#VFS.ZipFile/HTTP.st:76)
[] in NetClients.HTTP.HTTPClient class>>exampleURL:host:port:
(NetClients.star#VFS.ZipFile/HTTP.st:63)
BlockClosure>>ensure: (BlkClosure.st:269)
NetClients.HTTP.HTTPClient class>>exampleURL:host:port:
(NetClients.star#VFS.ZipFile/HTTP.st:68)
UndefinedObject>>executeStatements (sitemonitor.st:9)



And as a check, here is a diff on MIME.st after I had applied your patch...

diff --git a/packages/net/MIME.st b/packages/net/MIME.st
index 0460bc2..7a4c199 100644
--- a/packages/net/MIME.st
+++ b/packages/net/MIME.st
@@ -588,6 +588,7 @@ Object subclass: SimpleScanner [
  <category: 'stream interface -- reading'>
  lookahead notNil ifTrue: [^lookahead].
  self atEnd ifTrue: [^nil].
+        hereChar := nil.
  lookahead := source next.
  ^lookahead
      ]
@@ -1144,9 +1145,12 @@ MessageElement subclass: MimeEntity [
  <category: 'parsing'>
  | cr nl |
 
- [(cr := rfc822Stream peekFor: Character cr)
-    | (nl := rfc822Stream peekFor: Character nl)]
- whileFalse: [self parseFieldFrom: rfc822Stream]
+       [(cr := rfc822Stream hereChar == Character cr)
+                    ifTrue: [rfc822Stream step].
+        (nl := rfc822Stream hereChar == Character nl)
+                    ifTrue: [rfc822Stream step].
+         cr or: [nl]]
+                whileFalse: [self parseFieldFrom: rfc822Stream]
      ]

      parseMultipartBodyFrom: rfc822Stream [
@@ -2863,14 +2867,13 @@ MailScanner subclass: RFC822Scanner [
  <category: 'private'>
  | char |
  self atEnd ifTrue: [^false].
- char := source next.
+ char := self peek.
  ^((self classificationMaskFor: char) anyMask: WhiteSpaceMask)
     ifFalse:
- [lookahead := char.
- self resetToken.
+ [self resetToken.
  false]
     ifTrue:
- [self sourceTrailNextPut: char.
+ [self next. self sourceTrailNextPut: char.
  true]
      ]

Thanks
Stephen

On 29/03/11 9:24 PM, Paolo Bonzini wrote:

_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

On 04/07/2011 02:50 PM, Stephen wrote: Try this one.  It works on the full moodle.org site, not just my toy
testcase.

(I wasn't too sure of the other patch, which is why I hadn't pushed it yet).

Paolo

_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

> Try this one. It works on the full moodle.org site,
>
> Paolo

Hi and just to confirm that the patch works. I've compiled from latest
sources and can monitor our Moodle site using Netclient.HTTP. Fantastic.

Many thanks
Stephen

_______________________________________________
help-smalltalk mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-smalltalk