OO Database question with FLASH drive

Hi everyone,

I'm in the process of starting the next version of my autonomous
controller, which is going to be much more like a brain than anything I've
done before. I want to use an OO database to store persistent "memory" type
data, but the first version of this brain is going to run on a gumstix.

The gumstix uses a MMC FLASH card as the file system (under linux), so
there are issues with writing data to the drive many times (FLASH has write
limits). I haven't looked too closely at any of the OO databases that are
available for Squeak, but I need one that doesn't write to the file system
unless I explicitly tell it to commit a transaction.

This may end up being a non-issue, but I'd like to hear from the developers
or peope in the know if I need to be concerned.

Thanks,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

have you looked into image segments?

On Nov 14, 2006, at 1:54 PM, Jon Hylands wrote:

On Tue, 14 Nov 2006 14:05:50 -0800, Todd Blanchard <[hidden email]>
wrote:

> have you looked into image segments?

Well, I've thought about it, but I really want to keep the state saved in
"real time" which means I need incremental updates of the data. While
ImageSegments could be used that way, they would be very wasteful of both
FLASH write cycles and processor time, both of which are in short supply on
a machine like the gumstix.

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

Jon Hylands wrote: Are you going to write more than a million times? Isn't that the current
known limitation regarding writing?
BTW, are you using a special Linux that can remap areas of the FLASH
that have been written too much? Is there such a beast?

brad

Jon,

Not to mention slow! On a 200-400 MHz PDAish machine like the  
gumstix, saving stuff out as an image segment takes a few seconds,  
even if it's just something simple. But saving something in Magma or  
MinneStore are pretty much instant and transparent.

I think some sort of object database does indeed make sense. The two  
options that would be useful for you are Magma or MinneStore.  
MinneStore is older and might not even work on newer versions of  
Squeak.  I'm not 100%, but I recall that Magma doesn't write until  
you commit.  Magma does indeed have sessions, where do you do a  
commit after changing/adding stuff to the root object.

What is extra cool about Magma is that the database is basically just  
a Dictionary object that you can use like a regular dictionary, and  
then say 'commit' when you want that to be saved. You have to talk to  
Chris Muller or try it out yourself to find out for sure.

I used to use MinneStore and currently use Magma on Linux and WinCE  
PDAs. I've been impressed with their performance. I'm not saving huge  
amounts of information- notes, PIM data, etc- but it's a lot faster  
than ImageSegments or ReferenceStreams.

Regards,
Aaron

On Nov 14, 2006, at 4:49 PM, Jon Hylands wrote:

On Tue, 14 Nov 2006 15:10:33 -0800, Brad Fuller <[hidden email]> wrote:

> Are you going to write more than a million times? Isn't that the current
> known limitation regarding writing?

Well, it depends entirely on the FLASH device. Most of the ones I've seen
are closer to 10,000 - 100,000 times. At 100,000, if I'm writing once per
second while the thing is operating, that's only 27 hours of use.

> BTW, are you using a special Linux that can remap areas of the FLASH
> that have been written too much? Is there such a beast?

Linux uses JFFS2, which is very good at avoiding this problem:

http://en.wikipedia.org/wiki/JFFS2

However, if you're writing enough data, you can eventually run into it,
even with something like JFFS2.

I just want to make sure I don't run into any problems that are caused by
some journaling sub-system of the OO database that needs to write stuff
every time you read something, or something like that.

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

On Tue, 14 Nov 2006 17:18:06 -0600, Aaron Reichow <[hidden email]>
wrote:

> I used to use MinneStore and currently use Magma on Linux and WinCE  
> PDAs. I've been impressed with their performance. I'm not saving huge  
> amounts of information- notes, PIM data, etc- but it's a lot faster  
> than ImageSegments or ReferenceStreams.

Cool, thanks Aaron. I'll get in touch with Chris and see what he thinks...

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

Hi Jon,

> Linux uses JFFS2, which is very good at avoiding this problem:
>
> http://en.wikipedia.org/wiki/JFFS2
>
> However, if you're writing enough data, you can eventually run into it,
> even with something like JFFS2.

JFFS2 is only used for the directly accessible flash, i.e. the 4Mb or
16 Mb of flash that's part of the gumstix. The flash stored on the MMC
card is written using a FAT file system by default, although you could
change it to use another (but you can't use JFFS2 on an MMC card).

Many people write to RAM disk (/tmp on the gumstix) for the frequent
stuff, and then less frequently roll it out to the MMC card.

--
Dave Hylands
Vancouver, BC, Canada
http://www.DaveHylands.com/

On Tue, 14 Nov 2006 17:38:52 -0800, "Dave Hylands" <[hidden email]>
wrote:

> Many people write to RAM disk (/tmp on the gumstix) for the frequent
> stuff, and then less frequently roll it out to the MMC card.

Interesting, although I'm not sure it will help me. I will have the data in
the image, so writing it to a RAM disk doesn't really buy me anything. I'm
trying to make it so that I can shut this robot off fairly easily without
having to worry about losing a lot of data that hasn't been flushed to
FLASH.

I guess I'm really after two conflicting things - maximizing my FLASH
drive's life, while still writing fairly frequently in small batches.

Given the price of 128 MB MMC cards, I suppose it really won't matter much.

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

On Nov 15, 2006, at 2:38 , Dave Hylands wrote:

What if you make a huge file on the MMC card, and put a jffs2 into it  
via loopback?

- Bert -

Hi Bert,

> What if you make a huge file on the MMC card, and put a jffs2 into it
> via loopback?

It's not that simple. jffs2 is layered on top of MTD rather than just
being a block device.

I'm sure you could write a driver which worked with a loopback, but
I'm pretty sure that functionality doesn't currently exist.

I'm also pretty sure that if you could do such a thing you would just
be creating a false sense of security. The weakest link would still be
the underlying FAT file system.

--
Dave Hylands
Vancouver, BC, Canada
http://www.DaveHylands.com/

Flash Drives don't deserves such confidence.
Google 'flash drives failing'
Or read this:  
http://reseller.co.nz/reseller.nsf/news/C3AFEC33918A8EE2CC2571EC00091A66

        I hope this prevent data looses ;-)

        regards,

Sebastian Sastre
 
[hidden email]
Seaswork
Special Software Solutions


 

On Fri, 17 Nov 2006 10:30:44 -0200, Sebastián Sastre
<[hidden email]> wrote:

> Or read this:  
> http://reseller.co.nz/reseller.nsf/news/C3AFEC33918A8EE2CC2571EC00091A66

That is one of the silliest things I've ever read, re: fragmentation on a
flash drive. Fragmentation is only a problem when you have a physical read
head that has to traverse different areas of the media to read the data.
When its just reading electronics, it shouldn't make an iota of difference.

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

Perhaps, but that opinion could be partialist if you dont evaluate that there are also a
mention to other problems. Google a little and you will find out other approaches. Just to
illustrate I've lost data myself just after entering in an airplane in a travel with a
device like those completely umpowered. After that unhappy fact I've heared that static
can make crazy things in this devices and some devices relying in those technologies where
being discontinued. To feel the fact: who can categorically deny it and sign a contract
compromising it's own house saying that the technology is reliable and fact like that was
not because of the airport electromagnetic interference nor the static in the airplane?
I'm pretty sure that nor the vendors nor the manufacturers will do a crazy thing like
that.

All I'm saying is that that techology is too young (unmature) to be reliable to store data
not to mention to be reliable to store an odb. We can just use the mature ones and wait
they do better until we can rely on it.

I'm also saying I don't want airplanes with components relying in such unreliable (by now)
tech ;-)

        regards,

Sebastian Sastre
 
[hidden email]
Seaswork
Special Software Solutions
 
Este mensaje y sus adjuntos son confidenciales y de uso exclusivo para el usuario a quien
esta dirigido. Puede contener información amparada por el secreto profesional. Si Ud. no
es el destinatario especificado no debe copiar, enviar o utilizar ninguna parte del mismo
y/o de sus adjuntos por ningún medio tecnológico. Las opiniones vertidas son
responsabilidad del autor y no son emitidas ni avaladas por SEASWORK a menos que se
indique claramente lo contrario y que la identidad y autoridad del autor, para comprometer
a nuestra empresa, puedan ser verificados. No se garantiza la integridad de los mensajes
enviados por e-mail ni que los mismos sean enviados en termino, o que no contengan errores
o virus. El emisor no aceptara responsabilidad por los errores, modificaciones u omisiones
que resulten en el mensaje, bajo la hipótesis de que pudo ser modificado.

 

Hi Jon

> That is one of the silliest things I've ever read, re: fragmentation on a
> flash drive. Fragmentation is only a problem when you have a physical read
> head that has to traverse different areas of the media to read the data.
> When its just reading electronics, it shouldn't make an iota of difference.

It's a real issue. The issue is this, when you deal with flash memory,
you hve an erase size and a page size. The erase size is often quite
large, like 128K and the page size is usually 512 or 2048 bytes.

So the flash file systems use a model where they erase a page (clears
it to all 0xff's) and then write out pages. The pages may come from
different files and the fileystem deals with the fact that some of the
pages contain old data. When you say write the forst page of a file,
and then rewrite it, it adds a new page somewhere else on the
filesystem, and doesn't typically replace the old one. The old one
will eventually get garbage collected.

So you wind up with fragmentation because you've got lots of good data
and lots of useless data all intermingled together.

JFFS2 takes care of garbage collecting all of this stuff together and
doing the consolidation. It may be that flash drives don't.

Also, NAND chips can only take about N writes to an erase block before
you need to erase and rewrite (N is a small number probably around 10
- depends on the flash chip), Whereas with NOR chips you can writes
pages until you fill the erase block.

For more background, the "How does it work" section is a really good read:
http://sourceware.org/jffs2/

--
Dave Hylands
Vancouver, BC, Canada
http://www.DaveHylands.com/