Smalltalk › Cincom › VisualWorks

Filename delete problem? maybe? ...

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

19 messages Options

Dennis smith-4

Filename delete problem? maybe? ...

I am having trouble tracking this one down so I wondered if anyone has seen
anything like it.

Running VW7.7 on Win-XP.

I say
path asFilename delete.
path asFilename exists ifTrue: [self halt].
... wait a bit ...
path asFilename exists ifTrue: [self halt].

Every so often, the second halt is executed -- never (in my testing) the
first one.
In addition, if I look the file IS there but its empty!

I cannot reproduce this using the above simple code, but its the
sequence that is
being run in my production code.

Its as if some forked process is running my production code so that the
file in fact
gets re-created -- but I have pretty much ruled that out.

Anyone??

--
Dennis Smith +1 416.798.7948
Cherniak Software Development Corporation Fax: +1 416.798.0948
509-2001 Sheppard Avenue East [hidden email]
Toronto, ON M2J 4Z8 sip:[hidden email]
Canada http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

mkobetic

Re: Filename delete problem? maybe? ...

"Dennis Smith"<[hidden email]> wrote:
> I cannot reproduce this using the above simple code, but its the
> sequence that is
> being run in my production code.
>
> Its as if some forked process is running my production code so that the
> file in fact
> gets re-created -- but I have pretty much ruled that out.
>
> Anyone??

It is enough to open a write stream on a file to have it recreated, you don't have to write anything into it. Possibly if there's a write stream to the file sticking around it might be triggering recreation under some circumstances, e.g. I believe if you tickle even a closed stream the right way, the built in reopening mechanisms could kick in.

Just a guess, HTH!

Martin

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dennis smith-4

Re: Filename delete problem? maybe? ...

Interesting -- just 5min before seeing this, I went and tried a few
things to check for this
ExternalWriteStream allInstances (just 1, the .cha file)
use a different location for my files -- same results

but it still sounds like something a tiny bit promising, so I will
continue to look for something like that,
thanks

On Mar 7/11 1:53 PM, [hidden email] wrote:

>
> "Dennis Smith"<[hidden email]> wrote:
>> I cannot reproduce this using the above simple code, but its the
>> sequence that is
>> being run in my production code.
>>
>> Its as if some forked process is running my production code so that the
>> file in fact
>> gets re-created -- but I have pretty much ruled that out.
>>
>> Anyone??
> It is enough to open a write stream on a file to have it recreated, you don't have to write anything into it. Possibly if there's a write stream to the file sticking around it might be triggering recreation under some circumstances, e.g. I believe if you tickle even a closed stream the right way, the built in reopening mechanisms could kick in.
>
> Just a guess, HTH!
>
> Martin

--
Dennis Smith +1 416.798.7948
Cherniak Software Development Corporation Fax: +1 416.798.0948
509-2001 Sheppard Avenue East [hidden email]
Toronto, ON M2J 4Z8 sip:[hidden email]
Canada http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andres Valloud-6

Re: Filename delete problem? maybe? ...

In reply to this post by Dennis smith-4

On Windows, primitive 1601 uses DeletFile() to delete files. MSDN
states (http://msdn.microsoft.com/en-us/library/aa363915%28VS.85%29.aspx):

=====================
The DeleteFile function marks a file for deletion on close. Therefore,
the file deletion does not occur until the last handle to the file is
closed. Subsequent calls to CreateFile to open the file fail with
ERROR_ACCESS_DENIED.
=====================

Maybe the code is trying to delete a file that has not been closed? Or
perhaps Windows' implementation takes a little time to actually delete
the file once the last handle is closed?

Instead of trying the exists check, if the file exists can you try
recreating it? If you get file creation to fail with
ERROR_ACCESS_DENIED after you delete it, that would be an interesting
piece of the puzzle.

Andres.

On 3/7/2011 8:17 AM, Dennis Smith wrote:

> I am having trouble tracking this one down so I wondered if anyone has seen
> anything like it.
>
> Running VW7.7 on Win-XP.
>
> I say
> path asFilename delete.
> path asFilename exists ifTrue: [self halt].
> ... wait a bit ...
> path asFilename exists ifTrue: [self halt].
>
> Every so often, the second halt is executed -- never (in my testing) the
> first one.
> In addition, if I look the file IS there but its empty!
>
> I cannot reproduce this using the above simple code, but its the
> sequence that is
> being run in my production code.
>
> Its as if some forked process is running my production code so that the
> file in fact
> gets re-created -- but I have pretty much ruled that out.
>
> Anyone??
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dennis smith-4

Re: Filename delete problem? maybe? ...

Interesting thought -- here is what I did but the results were unexpected

We have a wrapper on Filename called PPfilename -- its main job in life is
to wrap Filename on VW and GsFile handling on Gemstone and make them
look almost the same.

So "delete" in our wrapper is
delete
^self filename delete

I change it to be
delete
self filename delete.
self filename writeStream close.
self filename delete

The problem went away. I MIGHT have expected the writeStream or the the
close to fail but to
hide the problem totally -- I guess I will spend more of the afternoon
digging -- thanks for this suggestion.

On Mar 7/11 2:16 PM, Andres Valloud wrote:

> On Windows, primitive 1601 uses DeletFile() to delete files. MSDN
> states (http://msdn.microsoft.com/en-us/library/aa363915%28VS.85%29.aspx):
>
> =====================
> The DeleteFile function marks a file for deletion on close. Therefore,
> the file deletion does not occur until the last handle to the file is
> closed. Subsequent calls to CreateFile to open the file fail with
> ERROR_ACCESS_DENIED.
> =====================
>
> Maybe the code is trying to delete a file that has not been closed? Or
> perhaps Windows' implementation takes a little time to actually delete
> the file once the last handle is closed?
>
> Instead of trying the exists check, if the file exists can you try
> recreating it? If you get file creation to fail with
> ERROR_ACCESS_DENIED after you delete it, that would be an interesting
> piece of the puzzle.
>
> Andres.
>
> On 3/7/2011 8:17 AM, Dennis Smith wrote:
>> I am having trouble tracking this one down so I wondered if anyone has seen
>> anything like it.
>>
>> Running VW7.7 on Win-XP.
>>
>> I say
>> path asFilename delete.
>> path asFilename exists ifTrue: [self halt].
>> ... wait a bit ...
>> path asFilename exists ifTrue: [self halt].
>>
>> Every so often, the second halt is executed -- never (in my testing) the
>> first one.
>> In addition, if I look the file IS there but its empty!
>>
>> I cannot reproduce this using the above simple code, but its the
>> sequence that is
>> being run in my production code.
>>
>> Its as if some forked process is running my production code so that the
>> file in fact
>> gets re-created -- but I have pretty much ruled that out.
>>
>> Anyone??
>>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dennis smith-4

Re: Filename delete problem? maybe? ...

In reply to this post by Andres Valloud-6

Further to my last post (to this email), I made a different change. I
made delete do this
delete
self filename delete
(Delay forMilliseconds: 200) wait

I have since tried 1000 runs of my test (takes a longer time with that
delay), and "zero" failures.
Before it would fail after 4 or 5.

The code does (among many other things)
- delete file in directory "X"
- delete directory "X"

and the delete of the directory would sometimes fail (not empty).

Note that a delay of 100ms helps but does not prevent it totally.

Note also that
delete
self filename delete.
self filename exists ifFalse: [self halt]
never halts. So its something a bit strange.

On Mar 7/11 2:16 PM, Andres Valloud wrote:

Andres Valloud-6

Re: Filename delete problem? maybe? ...

In reply to this post by Andres Valloud-6

> =====================
> The DeleteFile function marks a file for deletion on close. Therefore,
> the file deletion does not occur until the last handle to the file is
> closed. Subsequent calls to CreateFile to open the file fail with
> ERROR_ACCESS_DENIED.
> =====================

Huh... so what happens if an antivirus opens the file you try to delete
before the file is deleted? Then the file will be marked for deletion
but it won't be deleted until the antivirus is done with it.

Do you have an antivirus running on the machine?

Could other programs entice similar behavior? Defragmenting comes to
mind... if the file is being defragmented and you try to delete it, what
happens? Is the file defragmented and then deleted, or is the
defragmentation canceled (which may take some time anyway) and then the
file is deleted, or the defragmentation fails immediately because the
file was deleted?

Andres.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Mark Roberts

Re: Filename delete problem? maybe? ...

I have also seen problems with deleting files on Windows XP SP3, and very annoying because they were intermittent.

The failure was always in code that attempts to quickly delete a bunch of files at once. E.g.:

    dContents := SortedCollection sortBlock: [:a :b | a > b].
    dContents
        addAll: someDirectory asFilename directoryContentsRecursively.
    dContents
        do: [:fName | fName asFilename delete].

I found that even though I had closed all of the file streams, I would still get primitive failures on #delete with ERROR_SHARING_VIOLATION.

I tried adding (Delay forSeconds: 1) wait after closing the streams and before deleting the files, and that seemed to help make it the exception strike less often, but didn't totally resolve the problem. Without the delay, I would see failures over 50% of the time. Looking in Filename>>delete, I found the return value was a SystemError(#'io error', 32).

I can rule out an anti-virus app or something else because there was no AV running on this box and no other applications were sharing the files.

M

On 3/8/2011 5:17 AM, Andres Valloud wrote:

=====================
The DeleteFile function marks a file for deletion on close. Therefore,
the file deletion does not occur until the last handle to the file is
closed. Subsequent calls to CreateFile to open the file fail with
ERROR_ACCESS_DENIED.
=====================

Huh... so what happens if an antivirus opens the file you try to delete 
before the file is deleted?  Then the file will be marked for deletion 
but it won't be deleted until the antivirus is done with it.

Do you have an antivirus running on the machine?

Could other programs entice similar behavior?  Defragmenting comes to 
mind... if the file is being defragmented and you try to delete it, what 
happens?  Is the file defragmented and then deleted, or is the 
defragmentation canceled (which may take some time anyway) and then the 
file is deleted, or the defragmentation fails immediately because the 
file was deleted?

Andres.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andres Valloud-6

Re: Filename delete problem? maybe? ...

Error number 32 is ERROR_SHARING_VIOLATION:

=================
The process cannot access the file because it is being used by another
process.
=================

Ok, so the file was being used by another process, not the VM. Looking
at this from here, it's hard to figure out which one was it. However,
from the error description, it looks like something other than the VM is
also looking at files.

http://msdn.microsoft.com/en-us/library/ms681382%28v=VS.85%29.aspx

Andres.

On 3/7/2011 4:59 PM, Mark D. Roberts wrote:

> I have also seen problems with deleting files on Windows XP SP3, and
> very annoying because they were intermittent.
>
> The failure was always in code that attempts to quickly delete a bunch
> of files at once. E.g.:
>
> dContents := SortedCollection sortBlock: [:a :b | a > b].
> dContents
> addAll: someDirectory asFilename directoryContentsRecursively.
> dContents
> do: [:fName | fName asFilename delete].
>
> I found that even though I had closed all of the file streams, I would
> still get primitive failures on #delete with ERROR_SHARING_VIOLATION.
>
> I tried adding (Delay forSeconds: 1) wait after closing the streams and
> before deleting the files, and that seemed to help make it the exception
> strike less often, but didn't totally resolve the problem. Without the
> delay, I would see failures over 50% of the time. Looking in
> Filename>>delete, I found the return value was a SystemError(#'io
> error', 32).
>
> I can rule out an anti-virus app or something else because there was no
> AV running on this box and no other applications were sharing the files.
>
> M
>
> On 3/8/2011 5:17 AM, Andres Valloud wrote:
>>> =====================
>>> The DeleteFile function marks a file for deletion on close. Therefore,
>>> the file deletion does not occur until the last handle to the file is
>>> closed. Subsequent calls to CreateFile to open the file fail with
>>> ERROR_ACCESS_DENIED.
>>> =====================
>> Huh... so what happens if an antivirus opens the file you try to delete
>> before the file is deleted? Then the file will be marked for deletion
>> but it won't be deleted until the antivirus is done with it.
>>
>> Do you have an antivirus running on the machine?
>>
>> Could other programs entice similar behavior? Defragmenting comes to
>> mind... if the file is being defragmented and you try to delete it, what
>> happens? Is the file defragmented and then deleted, or is the
>> defragmentation canceled (which may take some time anyway) and then the
>> file is deleted, or the defragmentation fails immediately because the
>> file was deleted?
>>
>> Andres.
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Mark Roberts

Re: Filename delete problem? maybe? ...

Thanks, Andres.

Again, I find it very hard to believe that any other process would be
looking at these files. They are created and destroyed by the application.

Are there any windows tools that I could use to gather data on this?
What would you suggest?

M

On 3/8/2011 10:55 AM, Andres Valloud wrote:

> Error number 32 is ERROR_SHARING_VIOLATION:
>
> =================
> The process cannot access the file because it is being used by another
> process.
> =================
>
> Ok, so the file was being used by another process, not the VM. Looking
> at this from here, it's hard to figure out which one was it. However,
> from the error description, it looks like something other than the VM is
> also looking at files.
>
> http://msdn.microsoft.com/en-us/library/ms681382%28v=VS.85%29.aspx
>
> Andres.
>
> On 3/7/2011 4:59 PM, Mark D. Roberts wrote:
>> I have also seen problems with deleting files on Windows XP SP3, and
>> very annoying because they were intermittent.
>>
>> The failure was always in code that attempts to quickly delete a bunch
>> of files at once. E.g.:
>>
>> dContents := SortedCollection sortBlock: [:a :b | a> b].
>> dContents
>> addAll: someDirectory asFilename directoryContentsRecursively.
>> dContents
>> do: [:fName | fName asFilename delete].
>>
>> I found that even though I had closed all of the file streams, I would
>> still get primitive failures on #delete with ERROR_SHARING_VIOLATION.
>>
>> I tried adding (Delay forSeconds: 1) wait after closing the streams and
>> before deleting the files, and that seemed to help make it the exception
>> strike less often, but didn't totally resolve the problem. Without the
>> delay, I would see failures over 50% of the time. Looking in
>> Filename>>delete, I found the return value was a SystemError(#'io
>> error', 32).
>>
>> I can rule out an anti-virus app or something else because there was no
>> AV running on this box and no other applications were sharing the files.
>>
>> M
>>
>> On 3/8/2011 5:17 AM, Andres Valloud wrote:
>>>> =====================
>>>> The DeleteFile function marks a file for deletion on close. Therefore,
>>>> the file deletion does not occur until the last handle to the file is
>>>> closed. Subsequent calls to CreateFile to open the file fail with
>>>> ERROR_ACCESS_DENIED.
>>>> =====================
>>> Huh... so what happens if an antivirus opens the file you try to delete
>>> before the file is deleted? Then the file will be marked for deletion
>>> but it won't be deleted until the antivirus is done with it.
>>>
>>> Do you have an antivirus running on the machine?
>>>
>>> Could other programs entice similar behavior? Defragmenting comes to
>>> mind... if the file is being defragmented and you try to delete it, what
>>> happens? Is the file defragmented and then deleted, or is the
>>> defragmentation canceled (which may take some time anyway) and then the
>>> file is deleted, or the defragmentation fails immediately because the
>>> file was deleted?
>>>
>>> Andres.
>>> _______________________________________________
>>> vwnc mailing list
>>> [hidden email]
>>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>
>>
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Reinout Heeck-2

Re: Filename delete problem? maybe? ...

In reply to this post by Andres Valloud-6

On 3/8/2011 2:55 AM, Andres Valloud wrote:
> Error number 32 is ERROR_SHARING_VIOLATION:
>
> =================
> The process cannot access the file because it is being used by another
> process.
> =================
>

Are you watching the deletion process with a windows explorer? (Don't,
it seems to radically changes semantics of the filesystem).

Is this a local drive or networked?
is it shared/sharable?

Do you have search tools installed like Google toolbar,
or is the windows search setup to scan and create indexes?

etc... etc...

R
-

> Ok, so the file was being used by another process, not the VM. Looking
> at this from here, it's hard to figure out which one was it. However,
> from the error description, it looks like something other than the VM is
> also looking at files.
>
> http://msdn.microsoft.com/en-us/library/ms681382%28v=VS.85%29.aspx
>
> Andres.
>
> On 3/7/2011 4:59 PM, Mark D. Roberts wrote:
>> I have also seen problems with deleting files on Windows XP SP3, and
>> very annoying because they were intermittent.
>>
>> The failure was always in code that attempts to quickly delete a bunch
>> of files at once. E.g.:
>>
>> dContents := SortedCollection sortBlock: [:a :b | a> b].
>> dContents
>> addAll: someDirectory asFilename directoryContentsRecursively.
>> dContents
>> do: [:fName | fName asFilename delete].
>>
>> I found that even though I had closed all of the file streams, I would
>> still get primitive failures on #delete with ERROR_SHARING_VIOLATION.
>>
>> I tried adding (Delay forSeconds: 1) wait after closing the streams and
>> before deleting the files, and that seemed to help make it the exception
>> strike less often, but didn't totally resolve the problem. Without the
>> delay, I would see failures over 50% of the time. Looking in
>> Filename>>delete, I found the return value was a SystemError(#'io
>> error', 32).
>>
>> I can rule out an anti-virus app or something else because there was no
>> AV running on this box and no other applications were sharing the files.
>>
>> M
>>
>> On 3/8/2011 5:17 AM, Andres Valloud wrote:
>>>> =====================
>>>> The DeleteFile function marks a file for deletion on close. Therefore,
>>>> the file deletion does not occur until the last handle to the file is
>>>> closed. Subsequent calls to CreateFile to open the file fail with
>>>> ERROR_ACCESS_DENIED.
>>>> =====================
>>> Huh... so what happens if an antivirus opens the file you try to delete
>>> before the file is deleted? Then the file will be marked for deletion
>>> but it won't be deleted until the antivirus is done with it.
>>>
>>> Do you have an antivirus running on the machine?
>>>
>>> Could other programs entice similar behavior? Defragmenting comes to
>>> mind... if the file is being defragmented and you try to delete it, what
>>> happens? Is the file defragmented and then deleted, or is the
>>> defragmentation canceled (which may take some time anyway) and then the
>>> file is deleted, or the defragmentation fails immediately because the
>>> file was deleted?
>>>
>>> Andres.
>>> _______________________________________________
>>> vwnc mailing list
>>> [hidden email]
>>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>
>>
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

--
*********************************************************************

Dit e-mailbericht is alleen bestemd voor de geadresseerde(n).

Gebruik door anderen is niet toegestaan. Indien u niet degeadresseerde(n) bent wordt u verzocht de verzender hiervan op de hoogte te stellen en het bericht te verwijderen. Door de elektronische verzending kunnen aan de inhoud van dit bericht geen rechten worden ontleend.

Soops B.V. is gevestigd te Amsterdam, Nederland, en is geregistreerd bij de Kamer van Koophandel onder nummer 33240368.
Soops B.V. levert volgens de Fenit voorwaarden, gedeponeerd te Den Haag op 8 december 1994 onder nummer 1994/189.
**********************************************************************

This e-mail message is intended to be exclusively for the addressee.

If you are not the intended recipient you are kindly requested not to make any use whatsoever of the contents and to notify the sender immediately by returning this e-mail message. No rights can be derived from this message.

Soops B.V. is a private limited liability company and has its seat at Amsterdam, The Netherlands and is registered with the Trade Registry of the Chamber of Commerce and Industry under number 33240368.
Soops B.V. delivers according to the General Terms and Conditions of Business of Fenit, registered at The Hague, The Netherlands on December 8th, 1994, under number 1994/189
**********************************************************************

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Thomas Brodt-2

Re: Filename delete problem? maybe? ...

In reply to this post by Mark Roberts

Maybe you can try handle.exe of the SysInternals Package. Maybe there
are others, too, like the process explorer from the same package that
shows a lot more infos than the windows task manager.

Thomas

Am 08.03.2011 03:52, schrieb Mark D. Roberts:

> Thanks, Andres.
>
> Again, I find it very hard to believe that any other process would be
> looking at these files. They are created and destroyed by the application.
>
> Are there any windows tools that I could use to gather data on this?
> What would you suggest?
>
> M
>
> On 3/8/2011 10:55 AM, Andres Valloud wrote:
>> Error number 32 is ERROR_SHARING_VIOLATION:
>>
>> =================
>> The process cannot access the file because it is being used by another
>> process.
>> =================
>>
>> Ok, so the file was being used by another process, not the VM. Looking
>> at this from here, it's hard to figure out which one was it. However,
>> from the error description, it looks like something other than the VM is
>> also looking at files.
>>
>> http://msdn.microsoft.com/en-us/library/ms681382%28v=VS.85%29.aspx
>>
>> Andres.
>>
>> On 3/7/2011 4:59 PM, Mark D. Roberts wrote:
>>> I have also seen problems with deleting files on Windows XP SP3, and
>>> very annoying because they were intermittent.
>>>
>>> The failure was always in code that attempts to quickly delete a bunch
>>> of files at once. E.g.:
>>>
>>> dContents := SortedCollection sortBlock: [:a :b | a> b].
>>> dContents
>>> addAll: someDirectory asFilename directoryContentsRecursively.
>>> dContents
>>> do: [:fName | fName asFilename delete].
>>>
>>> I found that even though I had closed all of the file streams, I would
>>> still get primitive failures on #delete with ERROR_SHARING_VIOLATION.
>>>
>>> I tried adding (Delay forSeconds: 1) wait after closing the streams and
>>> before deleting the files, and that seemed to help make it the exception
>>> strike less often, but didn't totally resolve the problem. Without the
>>> delay, I would see failures over 50% of the time. Looking in
>>> Filename>>delete, I found the return value was a SystemError(#'io
>>> error', 32).
>>>
>>> I can rule out an anti-virus app or something else because there was no
>>> AV running on this box and no other applications were sharing the files.
>>>
>>> M
>>>
>>> On 3/8/2011 5:17 AM, Andres Valloud wrote:
>>>>> =====================
>>>>> The DeleteFile function marks a file for deletion on close. Therefore,
>>>>> the file deletion does not occur until the last handle to the file is
>>>>> closed. Subsequent calls to CreateFile to open the file fail with
>>>>> ERROR_ACCESS_DENIED.
>>>>> =====================
>>>> Huh... so what happens if an antivirus opens the file you try to delete
>>>> before the file is deleted? Then the file will be marked for deletion
>>>> but it won't be deleted until the antivirus is done with it.
>>>>
>>>> Do you have an antivirus running on the machine?
>>>>
>>>> Could other programs entice similar behavior? Defragmenting comes to
>>>> mind... if the file is being defragmented and you try to delete it, what
>>>> happens? Is the file defragmented and then deleted, or is the
>>>> defragmentation canceled (which may take some time anyway) and then the
>>>> file is deleted, or the defragmentation fails immediately because the
>>>> file was deleted?
>>>>
>>>> Andres.
>>>> _______________________________________________
>>>> vwnc mailing list
>>>> [hidden email]
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>>
>>> _______________________________________________
>>> vwnc mailing list
>>> [hidden email]
>>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steven Kelly

Re: Filename delete problem? maybe? ...

In reply to this post by Mark Roberts

We’ve seen this for a long time too. The errors are serious in that they often lead to data loss. They aren’t common: less than once a week per user, using our application for a good portion of the day. They are however consistently there, going back to 1997 – and so across multiple Windows versions and VW versions (2.0 – 7.4.1, the newest we’ve deployed). The problems occur (only?) on Windows. Some cases involve files that could be shared by another user, but some involve files that could only be used by one VW process. No Windows Explorer windows are open on that directory. The file is often on a network disk, mounted or used via a UNC.

We’re normally iterating over #directoryContents, closing any open streams on that file (by looking at OpenStreams), then deleting. Sometimes we’re just deleting a single file.

Using Sysinternals Process Explorer I’ve noticed that sometimes the VM holds on to file handles, although that file is not in OpenStreams. It can have several handles on the same file (which can be on a network or local disk). I can reproduce that, but not consistently. The handles stay open forever, and can’t be freed e.g. by opening and closing a new stream on that file.

Steve

From: [hidden email] [mailto:[hidden email]] On Behalf Of Mark D. Roberts
Sent: 8. maaliskuuta 2011 3:00
To: [hidden email]
Subject: Re: [vwnc] Filename delete problem? maybe? ...

=====================

The DeleteFile function marks a file for deletion on close. Therefore,

the file deletion does not occur until the last handle to the file is

closed. Subsequent calls to CreateFile to open the file fail with

ERROR_ACCESS_DENIED.

=====================

Huh... so what happens if an antivirus opens the file you try to delete

before the file is deleted?  Then the file will be marked for deletion

but it won't be deleted until the antivirus is done with it.

Do you have an antivirus running on the machine?

Could other programs entice similar behavior?  Defragmenting comes to

mind... if the file is being defragmented and you try to delete it, what

happens?  Is the file defragmented and then deleted, or is the

defragmentation canceled (which may take some time anyway) and then the

file is deleted, or the defragmentation fails immediately because the

file was deleted?

Andres.

_______________________________________________

vwnc mailing list

[hidden email]

http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Mark Roberts

Re: Filename delete problem? maybe? ...

In reply to this post by Thomas Brodt-2

Thanks, all, for your suggestions.

Out of curiosity, I made a few more tests. I don't have any google search tools installed, AV is turned off, and I disabled the Windows Indexing Service for the entire folder structure where I am creating the files in question. The files are all on the local disk.

With a Windows File Explorer open on the parent directory, I get frequent sharing violations, but when I check the file handles using handle.exe, there are no other processes that have a handle on the file that gives the sharing violation. File Explorer is listed as having one handle on a parent directory, but it is not viewing the directory where I am creating the files that get the violation. Also, I am trying to delete about a dozen files at once, and most of them can be deleted with no problem. If the open file handle on the parent directory is a problem, why would it only fail on *some* of the files in a subdirectory? That seems odd.

Next, I tried closing the Windows File Explorer and creating/deleting files again. This time, there are fewer sharing violations, but they still happen 15-20% of the time. Handle.exe says there are no other processes with handles to the directory where I am creating and deleting these files. Still, I am getting sharing violations.

Based upon what I'm seeing here, either Windows is raising spurious sharing violations, or there is some kind of bug in our VM that is causing these errors.

M

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steven Kelly

Re: Filename delete problem? maybe? ...

Interesting! Some questions come to mind:

- Does VW have a Windows file handle on the file it is deleting?

- Is there an OSHandle in the image corresponding to that?

- What is your OS version?

- Is the disk NTFS or something else?

Maybe ERROR_SHARING_VIOLATION should be treated like a transient error, with automatic retries? (Last time I looked at MSDN, it wasn't meant to be retried, but perhaps in practice this would be a useful workaround.)

Steve

From: [hidden email] on behalf of Mark D. Roberts
Sent: Tue 08/03/2011 18:26
To: [hidden email]
Subject: Re: [vwnc] Filename delete problem? maybe? ...

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Mark Roberts

Re: Filename delete problem? maybe? ...

On 3/9/2011 8:21 AM, Steven Kelly wrote:

Interesting! Some questions come to mind:

- Does VW have a Windows file handle on the file it is deleting?

Not sure how to test this.

- Is there an OSHandle in the image corresponding to that?

Not sure how to test this.

- What is your OS version?

Win XP SP3

- Is the disk NTFS or something else?

NTFS

Maybe ERROR_SHARING_VIOLATION should be treated like a transient error, with automatic retries? (Last time I looked at MSDN, it wasn't meant to be retried, but perhaps in practice this would be a useful workaround.)

I haven't tried to retry. I guess that would be a next step.

Steve

From: [hidden email] on behalf of Mark D. Roberts
Sent: Tue 08/03/2011 18:26
To: [hidden email]
Subject: Re: [vwnc] Filename delete problem? maybe? ...

Thanks, all, for your suggestions.

Out of curiosity, I made a few more tests. I don't have any google search tools installed, AV is turned off, and I disabled the Windows Indexing Service for the entire folder structure where I am creating the files in question. The files are all on the local disk.

With a Windows File Explorer open on the parent directory, I get frequent sharing violations, but when I check the file handles using handle.exe, there are no other processes that have a handle on the file that gives the sharing violation. File Explorer is listed as having one handle on a parent directory, but it is not viewing the directory where I am creating the files that get the violation. Also, I am trying to delete about a dozen files at once, and most of them can be deleted with no problem. If the open file handle on the parent directory is a problem, why would it only fail on *some* of the files in a subdirectory? That seems odd.

Next, I tried closing the Windows File Explorer and creating/deleting files again. This time, there are fewer sharing violations, but they still happen 15-20% of the time. Handle.exe says there are no other processes with handles to the directory where I am creating and deleting these files. Still, I am getting sharing violations.

Based upon what I'm seeing here, either Windows is raising spurious sharing violations, or there is some kind of bug in our VM that is causing these errors.

M

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steven Kelly

Re: Filename delete problem? maybe? ...

> - Does VW have a Windows file handle on the file it is deleting?
> Not sure how to test this.

Using handle.exe (or Process Explorer). I ask because you said "no OTHER
process had a handle on the file", which leaves unclear whether you saw
a handle used by the VW process.

> - Is there an OSHandle in the image corresponding to that?
> Not sure how to test this.

Me neither :-). It would be a PCDiskFileAccessor, and not used by any
stream in OpenStreams. You can ask a stream for its ioConnection, and
ask that for its input and output, which are PCDiskFileAccessors. Here's
code that I think should do this:

allAccessors := PCDiskFileAccessor allInstances.
openStreams := ExternalStream.OpenStreams copy.
usedAccessors := OrderedCollection new.
openStreams do: [:s |
s ioConnection input ifNotNil: [:accessor | usedAccessors add:
accessor].
s ioConnection output ifNotNil: [:accessor | usedAccessors add:
accessor]].
unusedAccessors := allAccessors reject: [:accessor | usedAccessors
includes: accessor].

Running that in visual.im shows no unused accessors. However, running it
in my dev image shows 4, even after global GC - they are held onto by
KnownFDs. Changing the last line to reject PCDiskFileAccessors with the
same handle ByteArray as an open stream (ByteArray == rather than
PCDiskFileAccessor ==) gets rid of these:

unusedAccessors := allAccessors reject: [:accessor | usedAccessors
anySatisfy: [:a | a key = accessor key]].

If Process Explorer says VW has a handle on the file, but the file isn't
mentioned in OpenStreams, and unusedAccessors is empty, I guess we can
be sure there is no image reference to the handle, which would imply the
VM may be holding onto it unnecessarily. But that's a lot of "ifs" :-).

Maybe you could post your testing code, if it's runnable in visual.im?

Steve

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Mark Roberts

Re: Filename delete problem? maybe? ...

Steve,

Thanks for the code snippet. I did a few more experiments and found
there are many instances of PCDiskFileAccessor in the image, many of
which look stale.

So, I am now inclined to think that is in fact the problem. I need to
figure out why these are not getting garbage collected.

My testing code is part of an old application, and has too many
dependencies to export in a simple manner.

If I eliminate this variable and it still looks like there is a VM or OS
issue, I'll try to write a simple test that triggers it.

In any case, I hope this thread includes some useful information for
anybody else trying to debug problems with file handles. I definitely
learned a few tricks.

M

On 3/9/2011 6:23 PM, Steven Kelly wrote:

>> - Does VW have a Windows file handle on the file it is deleting?
>> Not sure how to test this.
> Using handle.exe (or Process Explorer). I ask because you said "no OTHER
> process had a handle on the file", which leaves unclear whether you saw
> a handle used by the VW process.
>
>> - Is there an OSHandle in the image corresponding to that?
>> Not sure how to test this.
> Me neither :-). It would be a PCDiskFileAccessor, and not used by any
> stream in OpenStreams. You can ask a stream for its ioConnection, and
> ask that for its input and output, which are PCDiskFileAccessors. Here's
> code that I think should do this:
>
> allAccessors := PCDiskFileAccessor allInstances.
> openStreams := ExternalStream.OpenStreams copy.
> usedAccessors := OrderedCollection new.
> openStreams do: [:s |
> s ioConnection input ifNotNil: [:accessor | usedAccessors add:
> accessor].
> s ioConnection output ifNotNil: [:accessor | usedAccessors add:
> accessor]].
> unusedAccessors := allAccessors reject: [:accessor | usedAccessors
> includes: accessor].
>
> Running that in visual.im shows no unused accessors. However, running it
> in my dev image shows 4, even after global GC - they are held onto by
> KnownFDs. Changing the last line to reject PCDiskFileAccessors with the
> same handle ByteArray as an open stream (ByteArray == rather than
> PCDiskFileAccessor ==) gets rid of these:
>
> unusedAccessors := allAccessors reject: [:accessor | usedAccessors
> anySatisfy: [:a | a key = accessor key]].
>
> If Process Explorer says VW has a handle on the file, but the file isn't
> mentioned in OpenStreams, and unusedAccessors is empty, I guess we can
> be sure there is no image reference to the handle, which would imply the
> VM may be holding onto it unnecessarily. But that's a lot of "ifs" :-).
>
> Maybe you could post your testing code, if it's runnable in visual.im?
>
> Steve

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steven Kelly

Re: Filename delete problem? maybe? ...

It hasn't been explicitly mentioned yet, so it bears saying that before
deleting, you should do "ExternalStream closeOpenStreamsNamed:
myFilename". That's better than just closing your known streams for the
file, because after an error or messing around in a workspace you may
end up with open streams that aren't held by your application. You can
see all open streams in ExternalStream.OpenStreams. Always wrapping your
"stream close" in an ensure: block helps avoid most problems with
errors, but there are still corner cases.

One problem in that unlike most similar OS/VM-related collections,
OpenStreams isn't thread safe. If you have more than one process,
particularly at different priorities, it's positively dangerous. One
process can do "ExternalStream closeOpenStreamsNamed: myFilename.
myFilename delete", and any time during the first command, a higher
priority process can open a new stream, and that will not be closed by
the first command, so the #delete will fail.

OpenStreams also holds its elements strongly rather than weakly - so if
you say "myFilename writeStream" in a workspace, without storing it
anywhere, it never gets garbage collected. It stays open forever, and
you cannot delete that file. (If you use #closeOpenStreamsNamed: to
remove the stream from OpenStreams, the stream is explicitly closed, and
deletion is possible.)

It looks like OpenStreams could be made weak: Even if you simply remove
the element from OpenStreams, the finalize mechanism in IOAccessor takes
care of closing the handle, allowing deletion of that file (whenever the
gc and finalize mechanism gets around to it). There's obviously more to
think about there first though.

All the best,
Steve

> -----Original Message-----
> From: Mark D. Roberts [mailto:[hidden email]]
> Sent: 11. maaliskuuta 2011 2:53
> To: Steven Kelly
> Cc: [hidden email]
> Subject: Re: [vwnc] Filename delete problem? maybe? ...
>
> Steve,
>
> Thanks for the code snippet. I did a few more experiments and found
> there are many instances of PCDiskFileAccessor in the image, many of
> which look stale.
>
> So, I am now inclined to think that is in fact the problem. I need to
> figure out why these are not getting garbage collected.
>
> My testing code is part of an old application, and has too many
> dependencies to export in a simple manner.
>
> If I eliminate this variable and it still looks like there is a VM or
> OS
> issue, I'll try to write a simple test that triggers it.
>
> In any case, I hope this thread includes some useful information for
> anybody else trying to debug problems with file handles. I definitely
> learned a few tricks.
>
> M
>
> On 3/9/2011 6:23 PM, Steven Kelly wrote:
> >> - Does VW have a Windows file handle on the file it is deleting?
> >> Not sure how to test this.
> > Using handle.exe (or Process Explorer). I ask because you said "no
> OTHER
> > process had a handle on the file", which leaves unclear whether you
> saw
> > a handle used by the VW process.
> >
> >> - Is there an OSHandle in the image corresponding to that?
> >> Not sure how to test this.
> > Me neither :-). It would be a PCDiskFileAccessor, and not used by

any
> > stream in OpenStreams. You can ask a stream for its ioConnection,
and

> > ask that for its input and output, which are PCDiskFileAccessors.
> Here's
> > code that I think should do this:
> >
> > allAccessors := PCDiskFileAccessor allInstances.
> > openStreams := ExternalStream.OpenStreams copy.
> > usedAccessors := OrderedCollection new.
> > openStreams do: [:s |
> > s ioConnection input ifNotNil: [:accessor | usedAccessors add:
> > accessor].
> > s ioConnection output ifNotNil: [:accessor | usedAccessors add:
> > accessor]].
> > unusedAccessors := allAccessors reject: [:accessor | usedAccessors
> > includes: accessor].
> >
> > Running that in visual.im shows no unused accessors. However,

running
> it
> > in my dev image shows 4, even after global GC - they are held onto
by

> > KnownFDs. Changing the last line to reject PCDiskFileAccessors with
> the
> > same handle ByteArray as an open stream (ByteArray == rather than
> > PCDiskFileAccessor ==) gets rid of these:
> >
> > unusedAccessors := allAccessors reject: [:accessor | usedAccessors
> > anySatisfy: [:a | a key = accessor key]].
> >
> > If Process Explorer says VW has a handle on the file, but the file
> isn't
> > mentioned in OpenStreams, and unusedAccessors is empty, I guess we
> can
> > be sure there is no image reference to the handle, which would imply
> the
> > VM may be holding onto it unnecessarily. But that's a lot of "ifs"

:-
> ).
> >
> > Maybe you could post your testing code, if it's runnable in
visual.im?
> >
> > Steve

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc