Mystery ... URLMON Call Suddenly stopped working ?!?

Hello Dolphin Users...

Has anyone come across this problem?

I have some code that worked flawlessly for a few years and now is
failing.  It looks something like this:

URLMonLibrary open
        urlDownload:  'http://www.tradeperformance.com'
        toFile: (TPSystem defaultDirectory, 'aaaaaa.txt').

The funny thing is it just stopped working on my Windows 2000 machine
about three days ago.

It appears to go off to URLMON and never comes back.
Also if I trace it in the debugger when I get to
urlDownload: url toFile: aStringFilename

this is where it falls off the earth.  before executing the code there
_stack0 is some number after the failure it gets
index out of bounds 221.


Thanks for any help in advance .


Steve

> URLMonLibrary open
> urlDownload:  'http://www.tradeperformance.com'
> toFile: (TPSystem defaultDirectory, 'aaaaaa.txt').
>
> The funny thing is it just stopped working on my Windows 2000 machine
> about three days ago.

It's a long shot, but have you told IE or OE to "work offline" by any chance ?
IIRC that stops the URLMon stuff working.

Alternatively, perhaps your temporary file space is exhausted, or otherwise
screwed up.  Can you still do file downloads in IE ?

    -- chris

Another long shot.

You didn't recently use Windows update to get the latest
InternetExplorer cumulative update (Q832894) did you.  Some of the
changes it made have had some very strange side effects.

--
Ian

Use the Reply-To address to contact me.
Mail sent to the From address is ignored.

Ian,

> the latest
> InternetExplorer cumulative update (Q832894) did you.  Some of the
> changes it made have had some very strange side effects.

Now you've made me nervous.

I was just thinking that it's time I allowed a Windows Update again, but...

;-)


    -- chris

What is most interesting is that I went back to my old Win 98 machine
and the download no longer work there either...

Is it possible the ISP has done something?

Does anyone know what else uses URLMON?  This so I can see if that
works.  Dolphin or otherwise?

Also, a handful of my customers are also complaining of this problem,
(which is how I got onto this in the first place)


The only thing I have done to this machine is install the new Mozilla.
This browser has a built in pop-up blocker...not sure if this matters
But that was about three weeks before the URLMON started failing

Any additional clues are greatly appreciated.


Thanks,
Steve



Chris Uppal wrote:

Steve,

> What is most interesting is that I went back to my old Win 98 machine
> and the download no longer work there either...

FWIW, I just tried your code fragment and it downloaded to the text file
with
no apparent problems.

Windows XP with all the patches and Dolphin 5.1.3

--
Ian

Use the Reply-To address to contact me.
Mail sent to the From address is ignored.

Ian,

> You didn't recently use Windows update to get the latest
> InternetExplorer cumulative update (Q832894) did you.  Some of the
> changes it made have had some very strange side effects.

Please list them, if you are comfortable doing so.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Chris,

> > the latest
> > InternetExplorer cumulative update (Q832894) did you.  Some of the
> > changes it made have had some very strange side effects.
>
> Now you've made me nervous.
>
> I was just thinking that it's time I allowed a Windows Update again,
but...
>
> ;-)

Humor aside, one has no choice but to apply many of the patches.  Rather
than use Windows Update, I download them, and use InnoSetup to wrap them
into a single installer (one per month at present).  Automatically updating
is out of the question, because many of the machines are in use 7x24, so I
need to know when they are down.  FWIW, I am finding the "wrapping" to be a
little more error prone than I would like, and hope to make some time to get
Dolphin to write the installer scripts for me.

Of course, it is inevitable that the patches are adding new bugs, and only a
matter of time before features start breaking.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Steve,

> What is most interesting is that I went back to my old Win 98 machine
> and the download no longer work there either...
>
> Is it possible the ISP has done something?

I would hesitate to rule it out w/o doing some experiments.  However, I am
assuming that you can see we pages from the affected server.  Do you by
chance put a user name and password in the URL?  If so, you will find that
some recent patches disallow this behavior (not necessarily a bad idea).
There is supposed to be a registry value that you can set to leave the
feature enabled, but I could not get it to work.  I should confess that
having refused to build anything mission critical on top of the URLMoniker
library, I wasn't terribly motivated to investigate.


> Does anyone know what else uses URLMON?  This so I can see if that
> works.  Dolphin or otherwise?

Perhaps a smoke screen, I have found that IE refuses to save pictures from
some web sites (www.boortz.com) is a prime example), and very frequently
refuses to save .jpg files, offering only .bmp as a file type.  The latter
can sometimes be fixed by opening a new browser on the same URL, sometimes
by closing all browsers, sometimes by waiting a while, and sometimes only a
reboot seems to take care of it - all on win2k.  One particular win2k
machine seems to randomly loose the ability to talk to its DNS servers; that
takes a reboot to fix.  All of this is fairly recent (past six months or
so).


> Also, a handful of my customers are also complaining of this problem,
> (which is how I got onto this in the first place)

Chris' suggestion of "browse offline" is a something to check.


> The only thing I have done to this machine is install the new Mozilla.
> This browser has a built in pop-up blocker...not sure if this matters
> But that was about three weeks before the URLMON started failing
>
> Any additional clues are greatly appreciated.

<somebodyHasToSayIt>
There is little reason to let Microsoft do this work for you.  There is a
very comprehensive HTTP client (on Steve Waring's site??) for Dolphin.  For
various reasons, I chose to write my own.  The latter is tied into some code
that I cannot release at present, but I could probably separate it if the
other choices are not to your liking.  Once you have control of the code,
you will not be subject to the quirks of a crumbling black box.
</somebodyHasToSayIt>

Squeak is a great source of HTTP wisdom.  Also, you can look for OpenSSL
course samples from Carnegie Mellon.

Good luck in solving your problem!

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Bill,

> Please list them, if you are comfortable doing so.

I can't give you a list, it's more of a case of things that just stop
working after the IE update is installed.  I participate in a
distributed computing project (www.climateprediction.net) and after
installing the update the client just couldn't connect to the project
server, no error messages or anything.  The people who wrote the client
claimed they had gone by the original IE api rules, and I have no reason
to doubt them, but have had to rewrite the client, using some other
library, to get the project working again.  In the project's discussion
forum there were mention of other applications that failed when the
update was installed but I have no direct experience of them.

I have also seen the removal of the url/password shortcut that you
mentioned in another post being blamed as a cause of some subtle
failures.

--
Ian

Use the Reply-To address to contact me.
Mail sent to the From address is ignored.

Steve,

> What is most interesting is that I went back to my old Win 98 machine
> and the download no longer work there either...
>
> Is it possible the ISP has done something?

It's *possible*, but I'd have thought it unlikely that the ISP would affect
your customers too.  If they *have* done something, then it *could* be to do
with proxy settings.  If you are picking up the proxy settings by DHCP (or
similar) from the ISP's servers then something *might* have gone wrong with
that.


> Does anyone know what else uses URLMON?  This so I can see if that
> works.  Dolphin or otherwise?

As far as I know IE uses URLMon -- MS have a fairly bad record of not using
their own public APIs in their own products, but AFAIK IE does use the public
API.  The "dolphin update" feature also uses URLMon, though that's Dolphin
code, and will presumably suffer from the same problem -- whatever it is...


> Also, a handful of my customers are also complaining of this problem,
> (which is how I got onto this in the first place)

That makes it sound more like a problem with that server (unless the affected
users are all with your ISP too).  Is the URL you used as an example actually
one that is known to be giving problems ?

As Bill has suggested, switching to a Smalltalk HTTP client (such as Steve
Waring's -- which seems very good, although I have only used it a little bit)
would insulate you from MS-sourced problems (such as URLMon not working when
the user is "off-line").  Even if you don't want to switch to that (or
something home-grown) for your real application, experimenting with a Smalltalk
implementation would at a least give you a better chance to find out what's
going wrong.

Another possibility would be to install a network sniffer such as ethereal and
see what's going up and down the connection to the server.  That's what I would
probably do first, but it does depend on being reasonably comfortable reading
the HTTP protocol.

    -- chris


P.S  I just tried the URL you quoted and *did* get an error, but I don't know
if it's related to what you are seeing.  FWIW, the ethereal showed that the
data was coming back from the server OK, but URLMon library seemed to be
unhappy with it for some reason that I have not fathomed.

Hi Steve,

> > Does anyone know what else uses URLMON?  This so I can see if that
> > works.  Dolphin or otherwise?
>
> <somebodyHasToSayIt>
> There is little reason to let Microsoft do this work for you.  There is a
> very comprehensive HTTP client (on Steve Waring's site??) for Dolphin.
For
> various reasons, I chose to write my own.  The latter is tied into some
code
> that I cannot release at present, but I could probably separate it if the
> other choices are not to your liking.  Once you have control of the code,
> you will not be subject to the quirks of a crumbling black box.
> </somebodyHasToSayIt>
>

I've got a number of applications that need internet access. I started using
URLMoniker but found it unreliable on several occasions (similar problems to
yours). So I reimplemented it in Steve Waring's HTTPClient
(http://www.dolphinharbor.org/dh/projects/httpclient/index.html) and had no
problem since. At first I built and ran the same apps with URLMon and
Steve's in parrallel, but at some point I dropped the URLMon completely, as
Steve's appeared far more reliable. (With the added benefit that it's all in
Smalltalk, so if something goes wrong then you can at least debug properly).

Hope that helps,

Ted

I wrote:

> P.S  I just tried the URL you quoted and *did* get an error, but I don't
> know if it's related to what you are seeing.  FWIW, the ethereal showed
> that the data was coming back from the server OK, but URLMon library
> seemed to be unhappy with it for some reason that I have not fathomed.

A bit of a follow-up.  I've tried this now on two machines.  One is a W2K box
with no Mozilla and which is up to date with MS patches, the other is an XP box
with Mozilla (as the default browser) and is about a month behind on MS
patches.  The two boxes have different virus checkers (albeit McAfee and Dr
Solomon's which may share a lot of code).  Both fail in the same way when I
try:

    URLMonLibrary open
        urlDownload:  'http://www.tradeperformance.com'
        toFile: 'C:\temp\xxx'.

I now believe this to be some sort of bug in URLMon, or in the interaction
between URLMon, compression, and Dolphin's overlapped calls.  URLMon asks for
the data from the server, gets if back OK (it's gzip compressed), stores it in
its cache, and then throws an error saying 'invalid syntax, FACILITY_ITF'.   As
far as I can see the data that comes back from the server is entirely OK (e.g.
I've decompressed it with gzip).  What's more URLMon seems to think so too --
if I ask it a second time.  If I re-evaluate the snippet above, then everything
works fine, but URLMon is working with exactly the same data!  It has pulled it
from its cache rather than going back to the server (ethereal shows that it
didn't go to the network at all), and *this* time it's perfectly happy with
what it finds.

I've also tried it with a few other sites and it *seems* to happen whenever
they use a gzip transfer-encoding.  E.g:
http://groups.google.com/groups?selm=eDfXb.10312%24%25d3.652934%40twister.southeast.rr.com

One thing that I have noticed is that if I use the non-overlapped form of the
underlying call:

    URLMonLibrary default urlDownloadToFile: nil
        szURL: 'http://www.tradeperformance.com'
        szFilename: 'C:\Temp\xxx.html'
        dwReserved: 0
        pBSC: nil.

then it seems to work fine.

Just to add to the weirdness, as I was writing this it suddenly started
working!  After a bit more experimentation, it seems that using the
non-overlapped version a few times will make the overlapped version start
working until Dolphin is restarted.

Very strange...

    -- chris

Chris,

> I now believe this to be some sort of bug in URLMon, or in the
> interaction between URLMon, compression, and Dolphin's overlapped
> calls.  URLMon asks for the data from the server, gets if back OK
> (it's gzip compressed),

I'm confused now.  When I evaluated that bit of code I got, and still do
get, the contents of a a html page returned - no sign of any compression
at all.  Am I missing something?

--
Ian

Use the Reply-To address to contact me.
Mail sent to the From address is ignored.

Ian,

> I'm confused now.  When I evaluated that bit of code I got, and still do
> get, the contents of a a html page returned - no sign of any compression
> at all.  Am I missing something?

Possibly, or maybe your system just Works Properly (tm).

Anyway, here's what I'm doing -- in wearying detail ;-):

First I tell IE to clear its caches -- if I don't do that then the code snippet
will always just get the data from cache and so will always seem to work.

Then I start my network sniffer (ethereal).

Then I evaluate the code snippet, resulting in the error I described.

Then I look to see what really happened on the network.  It may be
configuration dependent in some way (I've not been able to find the options,
but I suspect they're there somewhere), but when URLMon connects to the server
it says that it is willing to accept "Content-Encoding" of (from memory) "gzip,
deflate".  As a result the server sends back gzip-compressed data in the HTTP
response which URLMon seems to put in the cache even though it's claiming
there's some sort of error.

Then I restart the sniffer, and re-evaluate the snippet.

This time ethereal shows that no data was transfered over the network, but
URLMon nevertheless has pulled the data out of the cache, decompressed it
(silently) and correctly copied the data to the destination file.

I can do that as many times as I like (but it does get a bit boring after a
while).

The compression is only used as part of the on-the-wire protocol, the client
s/w is supposed to do the decompression transparently (or, if it doesn't know
how to do that, then it shouldn't state that it will accept "gzip" as a
Content-Encoding).

Incidently, sometimes clients and servers get confused when they are
transfering a gzipped file. That *should* be a transfer of data of type
application/binary (or something like that), using no special Content-Encoding,
which would result in the file being copied, still gzipped, to the client
machine.  But quite often something goes wrong, and the client thinks that it
is supposed to do the decompression itself (presumably because the server tells
it the Content-Encoding is "gzip") so you end up with a file that is called
something like stuff.ps.gz but which has already been magically decompressed.

    -- chris

Wow...  What a weird trail this bug had led us on...

I tried it on 2 diffrent ISPs, I completely trashed and deleted my IE
cache, deleted all cookies. defragged my hard drive. I ran it under the
Dependency Walker, I re-registered all the IE DLLs.  I also scanned for
spyware, downloaded a new virus scanner, and turned off the firewall.
all without any success.

Then... After one of your posts I started trying different URLs and
found some of them work but most did not.  So I figured it
had something to do with software upgrades to the Internet servers
involved.  (Maybe to prevent denial of service attacks...)

This led me to Upgrade to the latest Internet Explorer V6 SP1.
It still failed.

Last Night I discovered that Dolphin uses  IStream >> onURL:
this seems to work flawlessly even though it also goes through
URLMONLibrary.dll  , but not through a file or the cache...just a byte
array. And as you found also, once it got data back correctly the
failing code seemed to start working again).

I changed my code to use IStream onURL:.  A number of posters suggested
I get away from the Microsoft implementation but I have customers hung
up on this and need a fix now.  If it blows up again, I will have to
rol up my sleeves and go the other way.


Many thanks to everyone for all the help and suggesstion and especially
to Chris for for the low level debugging in the http cache...



Steve

Steve,

> I tried it on 2 diffrent ISPs, I completely trashed and deleted my IE
> cache, deleted all cookies. defragged my hard drive. I ran it under the
> Dependency Walker, I re-registered all the IE DLLs.  I also scanned for
> spyware, downloaded a new virus scanner, and turned off the firewall.
> all without any success.

You didn't mention sacrificing a black goat at midnight...

;-)

    -- chris