[Fwd: Image file loading]

>
> Igor Stasenko wrote:
>>
>> Why one should wait , downloading the whole image, only after that to
>> discover that image format is not compatible
>> with interpreter, or not enough memory etc etc?
>
> Because your use case is purely hypothetical. You are talking about this as
> if it was the most common thing to run incompatible images over slow network
> connections. But it's not. The use case you're describing simply doesn't
> exist and won't for any foreseeable time.
>
> I'm more than willing to revisit this issue once we have a *some* evidence
> that this is an actual issue but up until that point let's avoid solving
> problems we don't actually have.
>

>
> 2010/1/21 Andreas Raab <[hidden email]>:
>>
>> Igor Stasenko wrote:
>>>
>>> Why one should wait , downloading the whole image, only after that to
>>> discover that image format is not compatible
>>> with interpreter, or not enough memory etc etc?
>>
>> Because your use case is purely hypothetical. You are talking about this as
>> if it was the most common thing to run incompatible images over slow network
>> connections. But it's not. The use case you're describing simply doesn't
>> exist and won't for any foreseeable time.
>>
>> I'm more than willing to revisit this issue once we have a *some* evidence
>> that this is an actual issue but up until that point let's avoid solving
>> problems we don't actually have.
>>
>
> Reading a 10Mb file from flash takes more that 1 second.
> Reading 64bytes(header) from flash stick, takes much less time. Same
> for networked mounted drive etc etc.
> Take in account a failures, like broken image file, or supplying wrong
> file etc etc.
> While all of it is obscure cases, it should be handled effectively.
>
> Also, i'm not a fan of burning CPU cycles & system resources in
> advance, if there's a possibility for shortcut.
> I am sympathizing a green-life movement ;)
>
>
>> Cheers,
>>  - Andreas
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.

>
> Apologies, accidentally posted to squeak-dev instead of vm-dev.
>
>   - A.
>
> -------- Original Message --------
> Subject: Image file loading
> Date: Tue, 19 Jan 2010 14:06:00 -0800
> From: Andreas Raab <[hidden email]>
> Reply-To: The general-purpose Squeak developers list
> <[hidden email]>
> To: The general-purpose Squeak developers list
> <[hidden email]>
> Newsgroups: gmane.comp.lang.smalltalk.squeak.general
>
> Folks -
>
> I think that one of the more clever hacks I did for the Android VM
> deserves proper generalization in the Interpreter. Since on Android the
> image is part of the (compressed) application package it's not easily
> possible to pass a file handle into the VM and have the VM read the
> image as an external file.
>
> Instead, I'm preloading the image file into memory and then hacked an
> implementation of the sqImageFile* and sqAlloc* functions that would
> basically operate on the image in memory. I think we should formalize
> this idea for two reasons:
>
> a) It allows preloading the image into memory on platforms where the
> image may not be in an external file.
>
> b) It trivially allows mmap()ing the entire image into memory (I think
> John does that on the iPhone).
>
> What I'm proposing is an additional VM entry point, called
>
>   Interpreter>>readImageFromHeap: heapStart size: heapSize
>
> which takes to arguments: The pointer to the beginning of the
> (pre-allocated) memory portion where the image file has been loaded or
> mapped and a size for the allocated memory portion. The VM then assumes
> that beginning at heapStart there is an image provided that it needs to
> properly prepare (i.e., perform byte-reversal, pointer adjustment etc).
>
> The VMs would generally use this along the lines of:
>
>   memStart = self malloc(heapSize);
>   fread(imageFile, memStart, fsize(imageFile));
>   readImageFromHeapSize(memStart, heapSize);
>
> and of course a backwards compatible version of
> readImageFromFile:HeapSize:StartingAt: could do the same (although I'd
> rather drop that function altogether since it removes the need for
> supporting various sqImageFile* functions).
>
> I'd also say that the same pattern should be used for *writing* image
> files, that is we assemble the entire image in memory then call
> sqImageFileWrite(fileName, heapStart, heapSize) and sqImageFileWrite is
> really just a combo of fopen(); fwrite(); fclose();
>
> I think this scheme is much nicer, less complicated and offers more
> options for implementing image loading than what we currently have.
>
> Comments?
>
> Cheers,
>   - Andreas
>
>

>
> Interpreter provides a single API function:
>
> Interpreter>>readImageFrom: handle,
>
> where handle is just a (void*) pointer, which only
> role is to serve as an argument , which interpreter should use to
> identify this resource, when
> calling a service functions, provided by platform code.
>
> In our specific case, the only service function, which we need is to
> ask platform code to read a next N bytes from
> given resource and store result at specified location:
>
> Platform>>readFrom: object into: memoryLocation size: bytesToRead
>
> here, the memoryLocation is a pointer to buffer, which Interpreter can
> obtain by allocating the memory
> or just a temporary buffer on stack.
>
> Now, it should be more clear, what Interpreter>>readImageFrom: handle, does:
>
> Interpreter>>readImageFrom: handle
>  | header |
>  self var: 'header' declareC: 'char[64]'.
>
>  "first, read first 64 bytes - an image header'
>  (Platform readFrom: handle into: header size: 64) ~~ 64 ifTrue: [ ^ nil ].
>
>  "parse the header, and check if it is valid etc"
>  ((self badHeaderIn: header ) ifTrue:  [ ^ nil ]. "Fail sooner"
>
>  memsize := header longAt: xxx.
>  (self isNotEnoughMemoryFor: memsize ) ifTrue: [ ^ nil ].
>  ....
>
>  "now, we're ready to allocate the memory for image"
>  memory := Platform allocate: memsize + self breathingRoom.
>
>  "now read the image into allocated area"
>  Platform readFrom: handle into: memory size: memsize.
>
> ----------------
> Since allocation and reading is both controlled by platform code,
> one could do any tricks with it, like reserve an address space, but
> not commit it into physical memory,
> but instead lazily read only pages which is not yet in memory using a
> page-fault mechanism etc.
>
>