Smalltalk › Gemtalk › GLASS

[Glass] large blobs

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

4 messages Options

johnmci

[Glass] large blobs

I'm collecting thoughts on using gemstone to store large chunks of data (say JPGS). Say 8 to 16MB in size. I have another case of blob data maybe upwards of 200MB but I'd store that in chunks just to enable better restart capabilities when pushing/pulling to a mobile device as that would need to restart a previously aborted transmission.

I'm wondering if anyone has had any experience with this?

--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd. Twitter: squeaker68882
===========================================================================

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Tobias Pape

Re: [Glass] large blobs

On 16.06.2014, at 22:46, John McIntosh <[hidden email]> wrote:

> I'm collecting thoughts on using gemstone to store large chunks of data (say JPGS). Say 8 to 16MB in size. I have another case of blob data maybe upwards of 200MB but I'd store that in chunks just to enable better restart capabilities when pushing/pulling to a mobile device as that would need to restart a previously aborted transmission.
>
> I'm wondering if anyone has had any experience with this?
>

For SqueakSource3, currently all mcz files are stored in the stone (for whatever reason).
The only thing to look out for I encountered was to avoid constant 'kind-change',
like repeated #asString -> #asByteArray. We had that and this costs memory and time.
Apart from that I cannot remember problem regarding the mcz's size. However, the biggest
did not exceed 8MB or so.

Best
-Tobias

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

signature.asc (1K) Download Attachment

Dale Henrichs-3

Re: [Glass] large blobs

In reply to this post by johnmci

John,

You might ask your question on the gemstone mailing list ... most of the commercial customers do not subscribe to the GLASS list and the commercial guys are likely to have some experience there ... and they might be willing to share:)

I assume you know that your blobs will be broken up into 16Kbyte pages for storage on disk in GemStone so your i/o won't necessarily be as smooth as sucking large contiguous chunks of bytes directly into the disk buffer ... that may mean that there could be an advantage to chunking the data yourself in page sized chucks and building a little infrastructure on top so that you can randomly access chunks efficiently ... but I haven't actually tried anything like this...

Dale

[1] http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk

On Mon, Jun 16, 2014 at 1:46 PM, John McIntosh <[hidden email]> wrote:

I'm collecting thoughts on using gemstone to store large chunks of data (say JPGS). Say 8 to 16MB in size. I have another case of blob data maybe upwards of 200MB but I'd store that in chunks just to enable better restart capabilities when pushing/pulling to a mobile device as that would need to restart a previously aborted transmission.

I'm wondering if anyone has had any experience with this?

--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd. Twitter: squeaker68882
===========================================================================

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Paul DeBruicker

Re: [Glass] large blobs

In reply to this post by johnmci

I don't have any experience with large blobs in GemStone.

But I did wonder why client requests with the HTTP Range: header wouldn't help with your problem. See

http://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7233.html

Presumably the client could/would know exactly how much of the blob data it already has and then form appropriate requests to just get the missing part from the filesystem.

Why prefer storing in GemStone vs the Range: scheme?

johnmci wrote

I'm collecting thoughts on using gemstone to store large chunks of data
(say JPGS). Say 8 to 16MB in size. I have another case of blob data maybe
upwards of 200MB but I'd store that in chunks just to enable better restart
capabilities when pushing/pulling to a mobile device as that would need to
restart a previously aborted transmission.

I'm wondering if anyone has had any experience with this?

--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd. Twitter: squeaker68882
===========================================================================

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass