Who is using Fuel?
Locked
 
|
Of course, not the fuel you use to go to work every day ;) I am talking about Fuel serializer.
In particular, I wonder if our page deserves an update: rmod.lille.inria.fr/web/pier/software/Fuel/Software-using-Fuel Let me know if there is something I should add. Thanks, -- Mariano http://marianopeck.wordpress.com |
Re: [Pharo-users] Who is using Fuel?
|
|
Am 28.02.2012 um 16:41 schrieb Mariano Martinez Peck: Of course, not the fuel you use to go to work every day ;) I am talking about Fuel serializer.We are using it in I first used image persistence but the image grow to large. Then I added fuel as journal to be written before image save in order to be able to recover from an emergency. Now we redo the persistence part. An account object graph is dissected into configuration part and payload part. The payload is written directly to mongo db in a Json format. The configuration part will be written as fuel blob to mongo. Planned is that at startup a configured amount of account fuel blobs are loaded and started. Those configurations have their own processes that write back payload to mongo. Etc. Just in case you want to know :) Norbert |
Re: [Pharo-users] Who is using Fuel?
|
|
On Tue, Feb 28, 2012 at 5:03 PM, Norbert Hartl <[hidden email]> wrote:
I didn't undersantd this one. What is what you are serializing that you want to be able to recover in case of an emergency?
Interested indeed :) I added it under "Custom applications" Thanks! -- Mariano http://marianopeck.wordpress.com |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by NorbertHartl
On 28 Feb 2012, at 17:03, Norbert Hartl wrote: > Am 28.02.2012 um 16:41 schrieb Mariano Martinez Peck: > >> Of course, not the fuel you use to go to work every day ;) I am talking about Fuel serializer. >> In particular, I wonder if our page deserves an update: rmod.lille.inria.fr/web/pier/software/Fuel/Software-using-Fuel >> >> Let me know if there is something I should add. >> > We are using it in > > http://www.2denker.de/product/apitester > > I first used image persistence but the image grow to large. Then I added fuel as journal to be written before image save in order to be able to recover from an emergency. > Now we redo the persistence part. An account object graph is dissected into configuration part and payload part. The payload is written directly to mongo db in a Json format. The configuration part will be written as fuel blob to mongo. Planned is that at startup a configured amount of account fuel blobs are loaded and started. Those configurations have their own processes that write back payload to mongo. Etc. > > Just in case you want to know :) > > Norbert Cool and interesting. Yes, we want to know about stuff like that ! Sven |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Mariano Martinez Peck
Am 28.02.2012 um 17:23 schrieb Mariano Martinez Peck:
Now the big parts of the data are in mongo so I don't expect it to happen real soon again. thanks, Norbert |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by NorbertHartl
On 28/02/12 11:03 AM, Norbert Hartl wrote:
> > I first used image persistence but the image grow to large. Then I added > fuel as journal to be written before image save in order to be able to > recover from an emergency. > Now we redo the persistence part. An account object graph is dissected > into configuration part and payload part. The payload is written > directly to mongo db in a Json format. The configuration part will be > written as fuel blob to mongo. Planned is that at startup a configured > amount of account fuel blobs are loaded and started. Those > configurations have their own proces ses that write back payload to > mongo. Etc. That's similar to what I do, except no image save and no mongo db. Here's how it goes: - on restart the image reconstructs a Pier kernel saved via Fuel to a file - included in the Pier kernel is what you're calling the "configuration part" - there are custom (non-std Pier) components here. - the components in the reconstructed image can read/write a "payload" part, which is serialized to files using Fuel and SandstoneDb. - additionally, a "sub-tree" of the Pier kernel can be exported to a file using Fuel serialization, and later imported using Fuel deserialization. No URL available at the moment. Aside - before Fuel/SandstoneDb, I was using Glorp/PostgreSQL. I've gained a lot of simplicity, but lost sequence numbers and object caching (and maybe more things, that have not been a problem so far). I've written a seqno replacement, but the lack of object caching is not a problem at the moment (i.e. small data sets, so far). |
Re: [Pharo-users] Who is using Fuel?
|
|
Am 28.02.2012 um 19:15 schrieb Yanni Chiu: > On 28/02/12 11:03 AM, Norbert Hartl wrote: >> >> I first used image persistence but the image grow to large. Then I added >> fuel as journal to be written before image save in order to be able to >> recover from an emergency. >> Now we redo the persistence part. An account object graph is dissected >> into configuration part and payload part. The payload is written >> directly to mongo db in a Json format. The configuration part will be >> written as fuel blob to mongo. Planned is that at startup a configured >> amount of account fuel blobs are loaded and started. Those >> configurations have their own proces ses that write back payload to >> mongo. Etc. > > That's similar to what I do, except no image save and no mongo db. > Norbert > Here's how it goes: > - on restart the image reconstructs a Pier kernel saved via Fuel to a file > - included in the Pier kernel is what you're calling the "configuration part" - there are custom (non-std Pier) components here. > - the components in the reconstructed image can read/write a "payload" part, which is serialized to files using Fuel and SandstoneDb. > - additionally, a "sub-tree" of the Pier kernel can be exported to a file using Fuel serialization, and later imported using Fuel deserialization. > > No URL available at the moment. > > Aside - before Fuel/SandstoneDb, I was using Glorp/PostgreSQL. I've gained a lot of simplicity, but lost sequence numbers and object caching (and maybe more things, that have not been a problem so far). I've written a seqno replacement, but the lack of object caching is not a problem at the moment (i.e. small data sets, so far). > > |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by NorbertHartl
Norbert,
On 28 Feb 2012, at 17:32, Norbert Hartl wrote: > I experience image corruption with big images. I am assuming you are using a normal image on a 32-bit Cog VM on Linux ? I am very interested in the practical limits on image size (not necessarily saving it). What vm switches did you apply ? I played a bit with some tests, and I could allocate about 1Gb in a 1.5GB heap VM. What is your experience ? Thx, Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Yanni Chiu
On Tue, Feb 28, 2012 at 7:15 PM, Yanni Chiu <[hidden email]> wrote:
WOW. I didn't know there was a real user about that ;) - additionally, a "sub-tree" of the Pier kernel can be exported to a file using Fuel serialization, and later imported using Fuel deserialization. That's interesting! No URL available at the moment. Ok, let me know if you have one at some point in the future and I add it. Aside - before Fuel/SandstoneDb, I was using Glorp/PostgreSQL. I've gained a lot of simplicity, but lost sequence numbers Sequence numbers are the less thing you should care about since you can alwasy do it in image side ;) and object caching (and maybe more things, that have not been a problem so far). I've written a seqno replacement, but the lack of object caching is not a problem at the moment (i.e. small data sets, so far). Yes, object caching is NOT a problem itself, it is a feature of Glorp or whatever. If what you use (like Fuel) is slow, then it may require some tunning. It can be caching object or not. What I suggest is that you stress your system with big data sets and you will see if you really miss object caching or not. -- Mariano http://marianopeck.wordpress.com |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Sven Van Caekenberghe
Am 28.02.2012 um 20:35 schrieb Sven Van Caekenberghe: > Norbert, > > On 28 Feb 2012, at 17:32, Norbert Hartl wrote: > >> I experience image corruption with big images. > > I am assuming you are using a normal image on a 32-bit Cog VM on Linux ? > yes > I am very interested in the practical limits on image size (not necessarily saving it). > Ah, I was just talking. The latest image grew to 280MB on disk. I just experienced a corrupted image after a few days. I did a more or less clean procedure by stopping all threads while saving. But sometime the image seems to be not fully written on disk making it unloadable on next start. The only limit that I tested is a maximum of threads of 1100 or 1400, can't remember right now. The image is stuck then. But that's no problem to me. > What vm switches did you apply ? > -encoding latin1 -nodisplay -nosound -mmap 512m > I played a bit with some tests, and I could allocate about 1Gb in a 1.5GB heap VM. > I had only the save experience but there could be a huge amount of reasons why it failed. My feeling is that my images are getting worse the longer they run. But I have nothing real to base this assumption on. > What is your experience ? I'm sorry that's all. I just started to do deployments with pharo lately. I was doing everything in gemstone until 2 month ago. Norbert |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Mariano Martinez Peck
On 28/02/12 2:48 PM, Mariano Martinez Peck wrote:
> > Yes, object caching is NOT a problem itself, it is a feature of Glorp or > whatever. If what you use (like Fuel) is slow, then it may require some > tunning. It can be caching object or not. What I suggest is that you > stress your system with big data sets and you will see if you really > miss object caching or not. It not a speed issue. It's about how big the image gets if all the on-disk objects get pulled into memory. The Glorp cache had a max cache size and LRU policy, IIRC. |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by NorbertHartl
On 28 Feb 2012, at 21:30, Norbert Hartl wrote: > Am 28.02.2012 um 20:35 schrieb Sven Van Caekenberghe: > >> Norbert, >> >> On 28 Feb 2012, at 17:32, Norbert Hartl wrote: >> >>> I experience image corruption with big images. >> >> I am assuming you are using a normal image on a 32-bit Cog VM on Linux ? >> > yes > >> I am very interested in the practical limits on image size (not necessarily saving it). >> > Ah, I was just talking. The latest image grew to 280MB on disk. I just experienced a corrupted image after a few days. I did a more or less clean procedure by stopping all threads while saving. But sometime the image seems to be not fully written on disk making it unloadable on next start. The only limit that I tested is a maximum of threads of 1100 or 1400, can't remember right now. The image is stuck then. But that's no problem to me. > >> What vm switches did you apply ? >> > -encoding latin1 -nodisplay -nosound -mmap 512m > >> I played a bit with some tests, and I could allocate about 1Gb in a 1.5GB heap VM. >> > I had only the save experience but there could be a huge amount of reasons why it failed. My feeling is that my images are getting worse the longer they run. But I have nothing real to base this assumption on. > >> What is your experience ? > > I'm sorry that's all. I just started to do deployments with pharo lately. I was doing everything in gemstone until 2 month ago. > > Norbert Thx for the reply. Some time ago, I tried this: http://forum.world.st/Big-Image-Tests-td4188045.html#a4188548 Sven |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by NorbertHartl
On 28/02/12 3:30 PM, Norbert Hartl wrote:
> > I'm sorry that's all. I just started to do deployments with pharo > lately. I was doing everything in gemstone until 2 month ago. That's interesting. Have you moved everything to Pharo, or is some part still in GemStone? Can you outline why GemStone was not the better solution for your use case? (I suspect that your dataset was naturally partitioned on user boundaries). |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Yanni Chiu
On Tue, Feb 28, 2012 at 9:46 PM, Yanni Chiu <[hidden email]> wrote:
If I understand you correctly, then what you are talking about is not caching objects but rather lazy loading. You mean when Glorp lets some proxies of the loaded graph so that they can load the rest of the graph later on? -- Mariano http://marianopeck.wordpress.com |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Yanni Chiu
Am 28.02.2012 um 22:09 schrieb Yanni Chiu: > On 28/02/12 3:30 PM, Norbert Hartl wrote: >> >> I'm sorry that's all. I just started to do deployments with pharo >> lately. I was doing everything in gemstone until 2 month ago. > > That's interesting. Have you moved everything to Pharo, or is some part still in GemStone? Can you outline why GemStone was not the better solution for your use case? (I suspect that your dataset was naturally partitioned on user boundaries). Sorry for the confusion. The part about pharo and gemstone was not referring to this project. Most of my projects run in GemStone and I'm quite satisified with it. I started this project in pharo because the main assets are http messaging and multiple long running processes. Basically the zinc components are available in GemStone, too. But they are neither well integrated nor up to date. Could be better but that isn't a show stopper. It's more the long running processes. In GemStone I would need to start a separate vm in the system that cares about the running processes. Then you need to work out a communication scheme between your seaside UI model and the service vm. I would do this just to store a lot of request/response (dumb) data inside GemStone. I was thinking about it and felt that it just does not fit GemStone that well. Last but not least I liked the idea to be forced to play with some of these modern storage systems. To me GemStone and a combination of pharo with mongo db are not competitors. I find them pretty orthogonal in use cases possible. On the one hand there is GemStone where you can have a huge object graph with all the flexibility we like so much. I use this for more complex and highly interlinked models. Externalization of objects always restrict you in this respect. GemStone is a centralized database but one that can be distributed. On the other hand pharo is more flexible and has the more modern tools. The pharo use case here is to start a couple of tens, hundreds,... of images that either collaborate directly or via a distributed storage. And pharo can scale vertical in machine size being used. I would like to use pharo on tiny machines and let them participate in all of this. In between those two there is magma and on another far end there is amber. I think I want all of them so I always have the right tool whatever use case there might appear on the next day. If we are talking about scale than as a conclusion I can say that: I love smalltalk because it scales in "things possible" Uh, sorry for getting _slightly_ verbose and off-topic :) Norbert |
Re: [Pharo-users] Who is using Fuel?
|
|
On 28/02/12 6:20 PM, Norbert Hartl wrote:
> > Uh, sorry for getting _slightly_ verbose and off-topic :) Thanks for the informative answer. It's good to learn how others tackle problems, and why they chose their solution. |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Mariano Martinez Peck
On 28/02/12 4:30 PM, Mariano Martinez Peck wrote:
> > If I understand you correctly, then what you are talking about is not > caching objects but rather lazy loading. You mean when Glorp lets some > proxies of the loaded graph so that they can load the rest of the graph > later on? Okay, the concept I meant is lazy loading using proxies. So what is "object cache" - i.e. what concept did you have in mind, when I used the term "object cache" earlier? |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by NorbertHartl
Norbert,
On 28 Feb 2012, at 21:30, Norbert Hartl wrote: > The only limit that I tested is a maximum of threads of 1100 or 1400, can't remember right now. The image is stuck then. I know we are way off topic, but are you sure you were not running into some OS limit (max open files/sockets which is often 1024 at the user level) ? You can inspect the limits by doing: $ ulimit -Hn $ ulimit -Sn for the hard and soft limits Sven |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Yanni Chiu
On Wed, Feb 29, 2012 at 4:24 AM, Yanni Chiu <[hidden email]> wrote:
heheheh ok, then we agree :) Indeed, it would be nice to have that. What I am not sure is how to decide when and where to cut the graph and let some proxies.
some internal optimization commonly done by ORMs (like Glorp) where they have a cache in primary memory of objects which were recently asked/queriered. The idea is to avoid going to the disk to fetch the object. -- Mariano http://marianopeck.wordpress.com |
Re: [Pharo-users] Who is using Fuel?
|
|
In reply to this post by Sven Van Caekenberghe
Am 29.02.2012 um 09:10 schrieb Sven Van Caekenberghe: > Norbert, > > On 28 Feb 2012, at 21:30, Norbert Hartl wrote: > >> The only limit that I tested is a maximum of threads of 1100 or 1400, can't remember right now. The image is stuck then. > > I know we are way off topic, but are you sure you were not running into some OS limit (max open files/sockets which is often 1024 at the user level) ? > > You can inspect the limits by doing: > > $ ulimit -Hn > $ ulimit -Sn > > for the hard and soft limits I know, thanks. But my test was as simple as 14000 timesRepeat: [ [ (Delay forSeconds: 60) wait ] forkAt: Processor userBackgroundPriority ] I tested it again and it works. If I open the process browser and turn auto update on then it deadlocks. I made this "mistake" the last time, too. Otherwise with the above test I could start more than 25000 threads without problem. Norbert |
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |