Partition/Sync ready databases for Pharo?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Partition/Sync ready databases for Pharo?

Esteban A. Maringolo
I have the requirement that an app that I'm prospecting must be able
to work offline and synchronize changes once the connection is
restored.

I want to avoid having to write "sync" logic manually.

Is anybody using Pharo with a database that supports automatic
synchronization (replicas?) like the one provided by Firebase
(Realtime Sync or Firestore)?

I rather use PostgreSQL/SQLite locally and sync with a "central" DB,
because I'm familiar with SQL/RDBMS, but I'm open to options.

Regards,

Esteban A. Maringolo

Reply | Threaded
Open this post in threaded view
|

Re: Partition/Sync ready databases for Pharo?

Pierce Ng-3
On Mon, Feb 18, 2019 at 02:56:10PM -0300, Esteban Maringolo wrote:
> I have the requirement that an app that I'm prospecting must be able
> to work offline and synchronize changes once the connection is
> restored.
>
> I want to avoid having to write "sync" logic manually.

I've not done this for real before, but here are some possibilities:

- https://github.com/sqlite-sync/SQLite-sync.com
- http://www.symmetricds.org/
- https://github.com/CanonicalLtd/dqlite
- Use Fossil as your database



Reply | Threaded
Open this post in threaded view
|

Re: Partition/Sync ready databases for Pharo?

Tim Mackinnon
When you say Fossil - are you referring to "https://fossil-scm.org/index.html/doc/trunk/www/index.wiki” ?

I’ve seen it come up a few times - is it good? But rather than subvert Esteban’s thread - if it is the above, can you hook into it to save application runtime artefacts such that a distributed application can read them back (I’d just sort of assumed it was for saving code and docs) - but it sounds like it can go a bit deeper than that (which seems quite interesting and perhaps what Esteban is after?)

Tim


> On 19 Feb 2019, at 01:46, Pierce Ng <[hidden email]> wrote:
>
> On Mon, Feb 18, 2019 at 02:56:10PM -0300, Esteban Maringolo wrote:
>> I have the requirement that an app that I'm prospecting must be able
>> to work offline and synchronize changes once the connection is
>> restored.
>>
>> I want to avoid having to write "sync" logic manually.
>
> I've not done this for real before, but here are some possibilities:
>
> - https://github.com/sqlite-sync/SQLite-sync.com
> - http://www.symmetricds.org/
> - https://github.com/CanonicalLtd/dqlite
> - Use Fossil as your database
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Partition/Sync ready databases for Pharo?

Esteban A. Maringolo
Hi Tim, Pierce,

No problem, I was going to ask about Fossil as well, I know it is a
self contained SCM, with issue tracker, and whatnot. People that uses
it don't want to use git except if forced to.
So I also ask how this would work. :)

I saw SQLite-sync before, but it is a commercial product, and its
platform support/setup seems convoluted.

dqlite might be a viable alternative, from more than reliable
provider, it's is not clear whether I have to use a different library
to access SQLite (it seems so) or a special wire protocol/api calls.
The only drawback I see is that, according to the the FAQ [1], from
the CAP theorem the consensus algorithm puts Consistency and Partition
and leaves out Availability.

I'm looking into CouchDB and reading its "Definitive Guide"[2] book.
It seems straightforward to install, distribute and uses a MVCC
approach that puts availability as its top priority, and since my
domain data has a low probability of conflicts, the eventual
consistency will be more than enough. OTOH I'm not used to document
based DBMS, which will require me to rethink a lot of my domain and
application models.

Also, it has a simple HTTP API (JSON/REST) and there is an existing
Smalltalk client that might be ported and adapted to Pharo
https://cwiki.apache.org/confluence/display/COUCHDB/Smalltalk


Background: I have the prospect to develop an application to replace
an existing one that is a desktop app that "syncs" from/to a central
server every 15', most of the data entry happens locally and the
central server is read only for the most part, it is old, ugly, but it
works "reliably", so I must come with an alternative that is as
reliable as that, and then provide a better UI/UX, new features, etc.

Regards!


Esteban A. Maringolo


[1] From https://github.com/CanonicalLtd/go-dqlite/
Q: When not enough nodes are available, are writes hung until consensus?
A: Yes, however there's a (configurable) timeout. This is a
consequence of Raft sitting in the CP spectrum of the CAP theorem: in
case of a network partition it chooses consistency and sacrifices
availability.

[2] http://guide.couchdb.org/draft/index.html

El mar., 19 feb. 2019 a las 10:03, Tim Mackinnon (<[hidden email]>) escribió:

>
> When you say Fossil - are you referring to "https://fossil-scm.org/index.html/doc/trunk/www/index.wiki” ?
>
> I’ve seen it come up a few times - is it good? But rather than subvert Esteban’s thread - if it is the above, can you hook into it to save application runtime artefacts such that a distributed application can read them back (I’d just sort of assumed it was for saving code and docs) - but it sounds like it can go a bit deeper than that (which seems quite interesting and perhaps what Esteban is after?)
>
> Tim
>
>
> > On 19 Feb 2019, at 01:46, Pierce Ng <[hidden email]> wrote:
> >
> > On Mon, Feb 18, 2019 at 02:56:10PM -0300, Esteban Maringolo wrote:
> >> I have the requirement that an app that I'm prospecting must be able
> >> to work offline and synchronize changes once the connection is
> >> restored.
> >>
> >> I want to avoid having to write "sync" logic manually.
> >
> > I've not done this for real before, but here are some possibilities:
> >
> > - https://github.com/sqlite-sync/SQLite-sync.com
> > - http://www.symmetricds.org/
> > - https://github.com/CanonicalLtd/dqlite
> > - Use Fossil as your database
> >
> >
> >
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Partition/Sync ready databases for Pharo?

Offray Vladimir Luna Cárdenas-2
Hi,

I think that using Fossil depends on what things are you synchronizing
and how fast.

In the Grafoscopio community we use Fossil to sync documentation
(Grafoscopio notebooks and Wiki pages) and issues, which means documents
that move slow, and we use CodiMD [1] (Firebase based) when we do
synchronization on documents we write together in real time (like
hackathon/documentathon notes). Usually we publish them in a single
Fossil repository (see [2] for an example of this combination of slow +
quick sync combining Fossil and CodiMD).

For alternatives to Firebase I have been starting to look at [3] and
[4]. I would really love to see a Pharo client to any of such
alternatives as I would like to have real time collaboration on
Grafoscopio based notebooks.

[1] http://demo.codimd.org/
[2] https://mutabit.com/repos.fossil/datafem/
[3]
https://www.quora.com/Whats-the-closest-open-source-alternative-to-Firebase-Im-mainly-interested-in-alternatives-to-the-software-framework-I-prefer-a-more-lightweight-solution-that-integrates-well-with-a-Backbone-js-application-and-acts-more-like-a-data-sync-layer
[4] https://alternativeto.net/software/firebase/?license=opensource

Hope it helps,

Offray

On 19/2/19 8:36, Esteban Maringolo wrote:

> Hi Tim, Pierce,
>
> No problem, I was going to ask about Fossil as well, I know it is a
> self contained SCM, with issue tracker, and whatnot. People that uses
> it don't want to use git except if forced to.
> So I also ask how this would work. :)
>
> I saw SQLite-sync before, but it is a commercial product, and its
> platform support/setup seems convoluted.
>
> dqlite might be a viable alternative, from more than reliable
> provider, it's is not clear whether I have to use a different library
> to access SQLite (it seems so) or a special wire protocol/api calls.
> The only drawback I see is that, according to the the FAQ [1], from
> the CAP theorem the consensus algorithm puts Consistency and Partition
> and leaves out Availability.
>
> I'm looking into CouchDB and reading its "Definitive Guide"[2] book.
> It seems straightforward to install, distribute and uses a MVCC
> approach that puts availability as its top priority, and since my
> domain data has a low probability of conflicts, the eventual
> consistency will be more than enough. OTOH I'm not used to document
> based DBMS, which will require me to rethink a lot of my domain and
> application models.
>
> Also, it has a simple HTTP API (JSON/REST) and there is an existing
> Smalltalk client that might be ported and adapted to Pharo
> https://cwiki.apache.org/confluence/display/COUCHDB/Smalltalk
>
>
> Background: I have the prospect to develop an application to replace
> an existing one that is a desktop app that "syncs" from/to a central
> server every 15', most of the data entry happens locally and the
> central server is read only for the most part, it is old, ugly, but it
> works "reliably", so I must come with an alternative that is as
> reliable as that, and then provide a better UI/UX, new features, etc.
>
> Regards!
>
>
> Esteban A. Maringolo
>
>
> [1] From https://github.com/CanonicalLtd/go-dqlite/
> Q: When not enough nodes are available, are writes hung until consensus?
> A: Yes, however there's a (configurable) timeout. This is a
> consequence of Raft sitting in the CP spectrum of the CAP theorem: in
> case of a network partition it chooses consistency and sacrifices
> availability.
>
> [2] http://guide.couchdb.org/draft/index.html
>
> El mar., 19 feb. 2019 a las 10:03, Tim Mackinnon (<[hidden email]>) escribió:
>> When you say Fossil - are you referring to "https://fossil-scm.org/index.html/doc/trunk/www/index.wiki” ?
>>
>> I’ve seen it come up a few times - is it good? But rather than subvert Esteban’s thread - if it is the above, can you hook into it to save application runtime artefacts such that a distributed application can read them back (I’d just sort of assumed it was for saving code and docs) - but it sounds like it can go a bit deeper than that (which seems quite interesting and perhaps what Esteban is after?)
>>
>> Tim
>>
>>
>>> On 19 Feb 2019, at 01:46, Pierce Ng <[hidden email]> wrote:
>>>
>>> On Mon, Feb 18, 2019 at 02:56:10PM -0300, Esteban Maringolo wrote:
>>>> I have the requirement that an app that I'm prospecting must be able
>>>> to work offline and synchronize changes once the connection is
>>>> restored.
>>>>
>>>> I want to avoid having to write "sync" logic manually.
>>> I've not done this for real before, but here are some possibilities:
>>>
>>> - https://github.com/sqlite-sync/SQLite-sync.com
>>> - http://www.symmetricds.org/
>>> - https://github.com/CanonicalLtd/dqlite
>>> - Use Fossil as your database
>>>
>>>
>>>
>>
>


Reply | Threaded
Open this post in threaded view
|

Re: Partition/Sync ready databases for Pharo?

Esteban A. Maringolo
El mar., 19 feb. 2019 a las 13:40, Offray Vladimir Luna Cárdenas
(<[hidden email]>) escribió:

> For alternatives to Firebase I have been starting to look at [3] and
> [4]. I would really love to see a Pharo client to any of such
> alternatives as I would like to have real time collaboration on
> Grafoscopio based notebooks.
>
> [1] http://demo.codimd.org/
> [2] https://mutabit.com/repos.fossil/datafem/
> [3]

> https://www.quora.com/Whats-the-closest-open-source-alternative-to-Firebase-Im-mainly-interested-in-alternatives-to-the-software-framework-I-prefer-a-more-lightweight-solution-that-integrates-well-with-a-Backbone-js-application-and-acts-more-like-a-data-sync-layer
> [4] https://alternativeto.net/software/firebase/?license=opensource

These takes the "old" Firebase approach of a pure JS client (or
Android/iOS with SDK).
In general I don't like the "Firebase" approach of a huge JSON tree as
your database, nor how it merges changes (PATCH/PUT).
Firestore [1] seems to orient everything to a Collection based NoSQL
approach (ala MongoDB).

I want something I can access from Pharo, at least via a REST API.
Although I'm biased to choose RDBMS (and SQL).

Regards!

Esteban A. Maringolo

Reply | Threaded
Open this post in threaded view
|

Re: Partition/Sync ready databases for Pharo?

Offray Vladimir Luna Cárdenas-2
Maybe something like the SQLite based Core Object [1], from the Etoile
folks, can your for you... at least as inspiration, because the project
looks pretty unmaintained.

[1] http://coreobject.org/

Cheers,

Offray

On 19/2/19 12:45, Esteban Maringolo wrote:

> El mar., 19 feb. 2019 a las 13:40, Offray Vladimir Luna Cárdenas
> (<[hidden email]>) escribió:
>
>> For alternatives to Firebase I have been starting to look at [3] and
>> [4]. I would really love to see a Pharo client to any of such
>> alternatives as I would like to have real time collaboration on
>> Grafoscopio based notebooks.
>>
>> [1] http://demo.codimd.org/
>> [2] https://mutabit.com/repos.fossil/datafem/
>> [3]
>> https://www.quora.com/Whats-the-closest-open-source-alternative-to-Firebase-Im-mainly-interested-in-alternatives-to-the-software-framework-I-prefer-a-more-lightweight-solution-that-integrates-well-with-a-Backbone-js-application-and-acts-more-like-a-data-sync-layer
>> [4] https://alternativeto.net/software/firebase/?license=opensource
> These takes the "old" Firebase approach of a pure JS client (or
> Android/iOS with SDK).
> In general I don't like the "Firebase" approach of a huge JSON tree as
> your database, nor how it merges changes (PATCH/PUT).
> Firestore [1] seems to orient everything to a Collection based NoSQL
> approach (ala MongoDB).
>
> I want something I can access from Pharo, at least via a REST API.
> Although I'm biased to choose RDBMS (and SQL).
>
> Regards!
>
> Esteban A. Maringolo
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Partition/Sync ready databases for Pharo?

Pierce Ng-3
In reply to this post by Esteban A. Maringolo
On Tue, Feb 19, 2019 at 10:36:18AM -0300, Esteban Maringolo wrote:
> No problem, I was going to ask about Fossil as well, I know it is a
> self contained SCM, with issue tracker, and whatnot. People that uses
> it don't want to use git except if forced to.
> So I also ask how this would work. :)

Fossil's built-in sync mechanism is described at
https://fossil-scm.org/index.html/doc/trunk/www/sync.wiki.

Fossil's traffic works through tunneling - this is how I 'fossil push'
new blog posts from my laptop to my server through an SSH tunnel. I had
also tested Fossil behind Internet-reachable Caddy HTTPS reverse proxy,
which works too.

For your use case, I might approach it like this: Write data locally and
commit to Fossil as blobs. To sync, run Fossil under OSSubprocess to
sync with the central server, either directly (if Fossil on central
server is directly reachable) or thru a tunnel created by your
application if your application runs on both ends.

> I saw SQLite-sync before, but it is a commercial product, and its
> platform support/setup seems convoluted.
>
> dqlite might be a viable alternative, from more than reliable
> provider, it's is not clear whether I have to use a different library
> to access SQLite (it seems so) or a special wire protocol/api calls.

I was actually thinking of https://github.com/rqlite/rqlite/ but did not
remember the name when I replied originally.

But it seems that, however you choose to sync, your application still
needs to handle any data conflict that arises.

Pierce

Reply | Threaded
Open this post in threaded view
|

Re: Partition/Sync ready databases for Pharo?

Pierce Ng-3
In reply to this post by Tim Mackinnon
On Tue, Feb 19, 2019 at 01:02:51PM +0000, Tim Mackinnon wrote:
> When you say Fossil - are you referring to
> "https://fossil-scm.org/index.html/doc/trunk/www/index.wiki” ?
> I’ve seen it come up a few times - is it good?

Yes I mean the Fossil SCM. I use it as a personal wiki and file repo. It
comes with built-in ticketing system and forum, all wrapped into one
executable file. Fossil also imports/exports Git repos.

I'd use Fossil for small development teams too. Except for game
development and other cases involving multi-GB repos with lots of binary
blobs.

> can you hook into it to save application runtime artefacts such that a
> distributed application can read them back

Fossil was designed as a standalone executable. It is difficult to
"library-ify" and the developers have no plan to do so. But it is just a
single executable, supports JSON output for some of its functions, and
can be invoked via OSSubprocess and the like, conceptually similar to
how Iceberg uses libgit2 to do Git stuff.

Pierce