[Q] how to write an Editor for very large files

Hi,

I'd like to write a, for the moment read only, binary editor for very
large files.

Does anybody here know of some sample code, preferably in Smalltalk, or
a goodie.

I'm curious on how one would go about the buffering.

Günther

Günther,

> I'd like to write a, for the moment read only, binary editor for very
> large files.

How large is "very large"?

> Does anybody here know of some sample code, preferably in Smalltalk, or a
> goodie.

One of my goodies, "IDB File Browser" has a hex tab that renders the
selected file as a series of hex numbers, along with their ASCII values.  It
is not buffered at all and is very slow with large files, which is why the
hex option is turned off by default.  It is only a reader though - you can't
edit the hex values.

> I'm curious on how one would go about the buffering.

I wouldn't have thought it too bad - you let Dolphin do it :-).  You can
position a FileStream to a certain point in the file so just keep track of
the first position being displayed, position the stream appropriately and
just read the next n bytes from there.  Writing back changes would work the
same way.

--
Ian

Use the Reply-To address to contact me.
Mail sent to the From address is ignored.

Ian,

Ian Bartholomew wrote:
> Günther,
>
>
>>I'd like to write a, for the moment read only, binary editor for very
>>large files.
>
>
> How large is "very large"?

1 - 2 GB.

ok. I'll see how it goes.

Well I was in particular wondering about the display buffering. As it is
not feasable to keep the streams data in memory I was wondering how to
do the display buffering in a scrollable (text) view.

At this moment editing is not really a problem, yet, the buffering is
the more interessting part.

My current goal is actually to write an IRC server, I've just been
playing around with BlockingCallMonitor, extremly useful btw., but I see
a problem coming up when the data gets too much, that's why I'm
interessted in a buffering technique.

Günther

Ian,

sorry, your File Browser folded at a mere 1.2 GB!

Booooh! :-)

I guess buffering still remains an issue.

Günther

Günther Schmidt wrote:
> I'm curious on how one would go about the buffering.
I see two ways.

The Smalltalk Way:
Implement an extended ByteArray which keeps a refrence to the file it
should represent and takes care of the buffering issue. When getting a
SubSollection this subcollection needs a refernce to the originating
collection and it's postion in it if you want to implement writing back
changes. The challenges I see are implementing the random-access
buffering and writing back.

The M$ Way:
Use Memory Mappen files
(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dngenlib/html/msdn_manamemo.asp).
MemoryMapped files should take care of all the buffering/write-back
details. Once the files are mapped access from DS should be easy as it's
just a MemoryAccess.

Hope this helps.

CU,

Udo

Udo,

thanks, it does.

I think at the moment I'd prefer the Smalltalk way.
Once again however I should think this is indeed a very generic problem,
  so I'm sure there must be some code (or examples) in some ST archive.

I think I'll also post this to the general smalltalk newsgroup.

Günther

Wow, I guess viewing huge files *is* a problem after all.

I just tried XVI32
(http://www.chmaas.handshake.de/delphi/freeware/xvi32/xvi32.htm)
which claims it is suited for handling large files, on a 1.2 GB file and
it folded too.

Bugger!

Günther

Günther,

> sorry, your File Browser folded at a mere 1.2 GB!

I'm not surprised, you must have run out of memory/swap space.

Have a look at the, very very crude, Dolphin 5 package I've posted at

http://www.idb.me.uk/files/bfb.pac  (~8 KiB)

It has no problem at all on a 600 MiB file so should be OK on yours.

It does nothing clever at all, just uses the normal Dolphin FileStream
buffering.

--
Ian

Use the Reply-To address to contact me.
Mail sent to the From address is ignored.

Ian,

thanks, that helped quite a bit!

Günther

Hi Gunther,

Günther Schmidt wrote:
Have you had a look at how Dolphin copes with its own sources and
changes files? They can become quite large and never seem to cause problems.

HTH

Ted