Recovering PNG/JPG images that were automatically serialized as FUEL when storing into external database

Hi,

I was storing Pharo objects into an external SQLite3 database. It went fine, but now all my png/jpg images are serialized with FUEL, when I try to see them inside SQLite and when I import them back to the image, they're long byte streams. There is any way to recover the images as png/jpg morphs that visible inside the image again? How can I disable this automatic serialization for making the database portable without using FUEL to recover back the PNG/JPG images?

Thanks,

Offray

Hi,

I have not solved this issue, but I'm using Fuel for my persistence model instead of SQLite in this project and is working like a charm. Serialization and materialization of objects, including png/jpeg images in external files is pretty fast and space efficient.

Cheers,

Offray

Hi,

I was storing Pharo objects into an external SQLite3 database. It went fine, but now all my png/jpg images are serialized with FUEL, when I try to see them inside SQLite and when I import them back to the image, they're long byte streams. There is any way to recover the images as png/jpg morphs that visible inside the image again? How can I disable this automatic serialization for making the database portable without using FUEL to recover back the PNG/JPG images?

Thanks,

Offray

Thanks Bernardo,

I can read the blobs from SQLite. The problem is that they're stored as FUEL inside SQLite and I don't know how to read them back to recover the png/jpeg image that was serialized that way.

Anyway, using Fuel instead of SQLite solved the external persistency problem and I can read and store them without any problem.

Cheers,

Offray

i found this 

http://stackoverflow.com/questions/625029/how-do-i-store-and-retrieve-a-blob-from-sqlite

hth

On Sat, Jul 2, 2016 at 11:03 PM, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:

Hi,

I have not solved this issue, but I'm using Fuel for my persistence model instead of SQLite in this project and is working like a charm. Serialization and materialization of objects, including png/jpeg images in external files is pretty fast and space efficient.

Cheers,

Offray

On 02/07/16 16:59, Offray Vladimir Luna Cárdenas wrote:

Hi,

I was storing Pharo objects into an external SQLite3 database. It went fine, but now all my png/jpg images are serialized with FUEL, when I try to see them inside SQLite and when I import them back to the image, they're long byte streams. There is any way to recover the images as png/jpg morphs that visible inside the image again? How can I disable this automatic serialization for making the database portable without using FUEL to recover back the PNG/JPG images?

Thanks,

Offray

--

Bernardo E.C.

Sent from a cheap desktop computer in South America.

On Sat, Jul 02, 2016 at 10:01:14PM -0500, Offray Vladimir Luna Cárdenas wrote:
> I can read the blobs from SQLite. The problem is that they're stored
> as FUEL inside SQLite and I don't know how to read them back to
> recover the png/jpeg image that was serialized that way.

UDBCSQLite uses Fuel to serialize/materialize objects that aren't integers,
real numbers and strings. Sounds like you've hit a bug, whereby your png/jpg
files are serialized but aren't being materialized when read back, so you're
getting back raw byte arrays.

Can you provide simple schema and code fragment to reproduce the problem
please.

Pierce

Hi Pierce,

I was my fault. I was not understanding properly the way of materialize back the png/jpeg images. With Max's help in the Slack chat channel it was clearer. Now is working properly and I have updated the database query messages accordingly. 

This combination of Fuel + SQLite seems very promising. For the moment I'm using mostly Fuel with the same efficiency of SQLite for external storage: 27 Mb both, the sqlite file and the fuel one, and almost the same reading time and without the impedance of object object-relational mapping. I imagine that the more we start to query the data, the more we're going to rely on SQLite bindings.

Cheers,

Offray

On Sat, Jul 02, 2016 at 10:01:14PM -0500, Offray Vladimir Luna Cárdenas wrote:

I can read the blobs from SQLite. The problem is that they're stored
as FUEL inside SQLite and I don't know how to read them back to
recover the png/jpeg image that was serialized that way.

UDBCSQLite uses Fuel to serialize/materialize objects that aren't integers,
real numbers and strings. Sounds like you've hit a bug, whereby your png/jpg
files are serialized but aren't being materialized when read back, so you're
getting back raw byte arrays.

Can you provide simple schema and code fragment to reproduce the problem
please. 

Pierce

Bernardo,

To serialize objects in a SQLite data base you do:

================

    | db |
    db := UDBCSQLite3Connection on: self database
    db open.
    db execute:
        'CREATE TABLE IF NOT EXISTS interaction_profiles (
            screenName text PRIMARY KEY,
            avatar blob,
            tweets integer,
            retweets integer
        );'.
    self interactionProfiles do: [ :ip |
        db
            execute: 'INSERT INTO interaction_profiles values (?, ?, ?, ?);'
            with: { ip screenName . ip avatar . ip tweets . ip retweets}
    ].
    db close.

================

To materialize data you can do:

================

    | query answer |
    query := 'SELECT * FROM interaction_profiles;'.
    answer := (self database open execute: query) rows collect: [ :each | each data ].
    self database isOpen ifTrue: [ self database close ].
    answer do: [ :each |
        self interactionProfiles add:
            (TwitterInteractionProfile new
                screenName: (each at: 'screenName');
                avatar: (FLMaterializer materializeFromByteArray: (each at: 'avatar'));
                tweets: (each at: 'tweets');
                retweets: (each at: 'retweets')
                )
    ]

================

where "self database" gives your sqlite database location. You can find especific details on how I'm using this in the Dataviz package:

http://smalltalkhub.com/#!/~Offray/Dataviz

Cheers,

Offray

Offray,

   could you post a snippet of the code to serialize and materialize an object and insert it into the sqlite database, please?

  just what's needed in order to get the idea

thanks;

On Wed, Jul 6, 2016 at 3:00 PM, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:

Hi Pierce,

I was my fault. I was not understanding properly the way of materialize back the png/jpeg images. With Max's help in the Slack chat channel it was clearer. Now is working properly and I have updated the database query messages accordingly. 

This combination of Fuel + SQLite seems very promising. For the moment I'm using mostly Fuel with the same efficiency of SQLite for external storage: 27 Mb both, the sqlite file and the fuel one, and almost the same reading time and without the impedance of object object-relational mapping. I imagine that the more we start to query the data, the more we're going to rely on SQLite bindings.

Cheers,

Offray

On 04/07/16 22:51, Pierce Ng wrote:

On Sat, Jul 02, 2016 at 10:01:14PM -0500, Offray Vladimir Luna Cárdenas wrote:

I can read the blobs from SQLite. The problem is that they're stored
as FUEL inside SQLite and I don't know how to read them back to
recover the png/jpeg image that was serialized that way.

UDBCSQLite uses Fuel to serialize/materialize objects that aren't integers,
real numbers and strings. Sounds like you've hit a bug, whereby your png/jpg
files are serialized but aren't being materialized when read back, so you're
getting back raw byte arrays.

Can you provide simple schema and code fragment to reproduce the problem
please. 

Pierce

--

Bernardo E.C.

Sent from a cheap desktop computer in South America.