Another Memory Problem

Hi,

I'm no expert in crashing the VM so maybe it's me doing something stupid.
I have a 157 MB semicolon delimited file from which I basically want to
create 1.5 million objects.
Around 1.37 million I get the attached crash dump. Memory is around 530M
Everything works fine if I only read the second half of the file.
I tried croquet.exe -memory: 1024 without success.

I read the whole file into a String and before I sprinkled the method
with debugging aids it looked like:

initFrom: aString
     "read the protocol file via feeding its lines into a state machine
each performed method returns the state for the next line"
     |status|
     status := #initialHeaders:.
     aString linesDo: [:line|
         status := self perform: status with: line].

Latest Cog VM, image updated to 12333.

Cheers,

Herbert

If you try just allocating one really big String or ByteArray equivalent to those 1.5 million objects, say 600 or 700MB, what happens?

Cheers,
Bob

Hi,

I'm no expert in crashing the VM so maybe it's me doing something stupid.
I have a 157 MB semicolon delimited file from which I basically want to create 1.5 million objects.
Around 1.37 million I get the attached crash dump. Memory is around 530M
Everything works fine if I only read the second half of the file.
I tried croquet.exe -memory: 1024 without success.

I read the whole file into a String and before I sprinkled the method with debugging aids it looked like:

initFrom: aString
    "read the protocol file via feeding its lines into a state machine each performed method returns the state for the next line"
    |status|
    status := #initialHeaders:.
    aString linesDo: [:line|
        status := self perform: status with: line].

Latest Cog VM, image updated to 12333.

Cheers,

Herbert

I know on the Mac there's a limit to image size which is specified IIRC in the plist file. Not sure about other OSes. The only other limit I know about is a 32-bit address space.

Hi Herbert,

btw, have you looked at NeoCSV by Sven Van Caekenberghe. 

http://mc.stfx.eu/Neo/
https://github.com/svenvc/docs/blob/master/neo/neo-csv-paper.md

Herbert König wrote:

Hi,

I'm no expert in crashing the VM so maybe it's me doing something stupid.
I have a 157 MB semicolon delimited file from which I basically want to create 1.5 million objects.
Around 1.37 million I get the attached crash dump. Memory is around 530M
Everything works fine if I only read the second half of the file.
I tried croquet.exe -memory: 1024 without success.

I read the whole file into a String and before I sprinkled the method with debugging aids it looked like:

initFrom: aString
    "read the protocol file via feeding its lines into a state machine each performed method returns the state for the next line"
    |status|
    status := #initialHeaders:.
    aString linesDo: [:line|
        status := self perform: status with: line].

Latest Cog VM, image updated to 12333.

Cheers,

Herbert

On Mon, 18 Mar 2013, Herbert König wrote:
That's pretty likely the cause of the problem, your image is probably
running out of memory. You can't use more than 512 MB on windows.

You should rewrite your code to read the file line by line (it should be a
lot faster this way). E.g.:

FileStream readOnlyFileNamed: 'yourBigFile.csv' do: [ :file |
  [ file atEnd ] whileFalse: [
  | line |
  line := file nextLine.
  "process the line" ] ].

If your file only contains ASCII characters, then you can optimize it
even further by using StandardFileStream instead of FileStream, which will
save you the time spent with encoding the characters.


Levente

Hi all,

I will have time to look further tonight.

I have Win 7 4G of RAM.

I can allocate a ByteArray of 400M but not one of 500M regardless if I start with the -memory: 1024 option.

I also cannot allocate 400M and then 100M. I get the low space warning which repeats constantly if I click "Proceed".

Thanks to all, I will try to upload an image and changes for Eliot tonight.

Cheers,

Herbert

Hi,

I'm no expert in crashing the VM so maybe it's me doing something stupid.
I have a 157 MB semicolon delimited file from which I basically want to create 1.5 million objects.
Around 1.37 million I get the attached crash dump. Memory is around 530M
Everything works fine if I only read the second half of the file.
I tried croquet.exe -memory: 1024 without success.

I read the whole file into a String and before I sprinkled the method with debugging aids it looked like:

initFrom: aString
    "read the protocol file via feeding its lines into a state machine each performed method returns the state for the next line"
    |status|
    status := #initialHeaders:.
    aString linesDo: [:line|
        status := self perform: status with: line].

Latest Cog VM, image updated to 12333.

Cheers,

Herbert

Hi Levente,

I read the whole file into a String and before I sprinkled the method with debugging aids it looked like:

That's pretty likely the cause of the problem, your image is probably running out of memory. You can't use more than 512 MB on windows.

You should rewrite your code to read the file line by line (it should be a lot faster this way). E.g.:

I thought of this too but assumed it to be slower as I have to go to the OS more often. Will check and report.

I wasn't aware of the hard 512M limit on Win. The normal case would be 1000 files of 160k each. Each file represents the production test of one amplifier and in this special case I tested one Amp a thousand times.

FileStream readOnlyFileNamed: 'yourBigFile.csv' do: [ :file |
    [ file atEnd ] whileFalse: [
        | line |
        line := file nextLine.
        "process the line" ] ].

If your file only contains ASCII characters, then you can optimize it even further by using StandardFileStream instead of FileStream, which will save you the time spent with encoding the characters.

Thanks for the tip.

Cheers

Herbert

Levente

initFrom: aString
   "read the protocol file via feeding its lines into a state machine each performed method returns the state for the next line"
   |status|
   status := #initialHeaders:.
   aString linesDo: [:line|
       status := self perform: status with: line].

Latest Cog VM, image updated to 12333.

Cheers,

Herbert

Hi,

>     can you supply the csv file?  If so I could take a look.  If you
> can supply the csv can you try and make the system crash by saving an
> image via a doit that continues to read the file?  e.g.

I sent everything to Eliot.

If somebody else is interested to play, I can send you the download
link, 30MB zip.

Cheers

Herbert

Hi,

That's pretty likely the cause of the problem, your image is probably running out of memory. You can't use more than 512 MB on windows.

You should rewrite your code to read the file line by line (it should be a lot faster this way). E.g.:

FileStream readOnlyFileNamed: 'yourBigFile.csv' do: [ :file |
    [ file atEnd ] whileFalse: [
        | line |
        line := file nextLine.
        "process the line" ] ].

biggest gain is the reduction from 470 MB of peak memory to 352 MB. Time went down from 2:40 to 2:15 it seems most of the time is spent in initializing the objects. Changing more ivars to symbols brought the time back up to 2:25 but memory down to 227 MB.  Never thought of symbols as memory savers before :-)

Thanks for the tip,

Herbert

If your file only contains ASCII characters, then you can optimize it even further by using StandardFileStream instead of FileStream, which will save you the time spent with encoding the characters.


Levente

initFrom: aString
   "read the protocol file via feeding its lines into a state machine each performed method returns the state for the next line"
   |status|
   status := #initialHeaders:.
   aString linesDo: [:line|
       status := self perform: status with: line].

Latest Cog VM, image updated to 12333.

Cheers,

Herbert