An IPFS interface for Pharo

Hi everyone,

Over the last weeks I have started to explore IPFS more seriously.
IPFS, the Inter-Planetary File System, is supposed to be the
next-generation Web: a decentralized content-addressed database.

Since there is nothing better then Pharo for exploring databases,
I have started to write an IPFS interface to Pharo:

   https://github.com/khinsen/ipfs-pharo

It connects to a local IPFS server, so you have to have one
running. It's surprisingly straightforward to install and configure,
unless you have to fight with firewalls that block IPFS traffic.

Feedback of any kind is welcome!

Cheers,
  Konrad.

Am 06.06.19 um 19:37 schrieb Serge Stinckwich:

> Great work ! I think Marcus was interested to have something like that.
> It is a solution to distribute scientific datasets ?

That's what I am trying to explore. But IPFS is a nice basis for many
more applications. I have seen an IPFS-based GitHub (Radicle), various
social networks, personal information management (Textile), and a lot more.

Konrad.

I would like something like that for Pharo images. Can you sync files.pharo.org on ipfs ?

On 6 Jun 2019, at 19:37, Serge Stinckwich <[hidden email]> wrote:

On Thu, Jun 6, 2019 at 6:35 PM Konrad Hinsen <[hidden email]> wrote:

Hi everyone,

Over the last weeks I have started to explore IPFS more seriously.
IPFS, the Inter-Planetary File System, is supposed to be the
next-generation Web: a decentralized content-addressed database.

Since there is nothing better then Pharo for exploring databases,
I have started to write an IPFS interface to Pharo:

   https://github.com/khinsen/ipfs-pharo

It connects to a local IPFS server, so you have to have one
running. It's surprisingly straightforward to install and configure,
unless you have to fight with firewalls that block IPFS traffic.

Great work ! I think Marcus was interested to have something like that.

yes!

IPFS has nice properties for many things… e.g. one simple things to try is to distribute images via IPFS… it has all the

propose of Bittorrent for that.  Imagine your case in Africa… very nice for that already.

Marcus

[hidden email] writes:

> I would like something like that for Pharo images. Can you sync
> files.pharo.org on ipfs ?

That should be straightforward using nothing but the IPFS command-line
client. But it's worth thinking about the goals. What are you trying to
improve exactly? The main advantage that IPFS provides is the
possibility to create full or partial mirrors of the data (and thus
faster access), including on your personal machine. To make this
effective, you have to think a bit about good data management
strategies. Few people will want to mirror all the history of Pharo.
So you have to make interesting subcollections available under
permanent names (via IPNS).

Konrad.

Konrad Hinsen <[hidden email]>
wrote:
:
> . To make this
> effective, you have to think a bit about good data management
> strategies. Few people will want to mirror all the history of Pharo.

Why? Fits on an SD card in a raspberry pi

Stephan

On 11/06/2019 18:07, Stephan Eggermont wrote:
> Konrad Hinsen <[hidden email]>
> wrote:
> :
>> . To make this
>> effective, you have to think a bit about good data management
>> strategies. Few people will want to mirror all the history of Pharo.
> Why? Fits on an SD card in a raspberry pi


All of http://files.pharo.org/ ? So how many GB is that?

Konrad.

Konrad Hinsen <[hidden email]>
wrote:
It is only a few thousand changes per release. There is no reason why that
shouldn’t compress well

Stephan

Stephan Eggermont <[hidden email]> writes:

>> All of http://files.pharo.org/ ? So how many GB is that?
>
> It is only a few thousand changes per release. There is no reason why that
> shouldn’t compress well

Did anybody try?

In IPFS, files are cut into blocks of roughly 256 KB. Blocks shared
between multiple files are stored only once. So if changes in Pharo
images from one version to the next happen mainly in one or two specific
places (such as beginning and end), they would be stored efficiently on
IPFS without any effort.

But again, the best way to know is to try it out.

Konrad.

Indeed. Another thing to try: if someone has a local IPFS running, would it cache
only those images that either this machine or near machines requested?

This way we could have one “full” copy and caching (e.g. in the “African class room”
example) would happen automatically.

Of course the downside is that one needs to speak the IPFS protocol, thus either running
a client (e.g the go client)… so real transparent use would only be possible if Pharo could
implement the protocol…

(all based on a not complete understanding of IPFS)

        Marcus

There are many other ideas to explore (based on my partial knowledge)… more brainstorming
than anything else:

- Could we have an image formate that would be “block friendly”?
- How could one move resources out of the image into IPFS?

- Content addressing (like it is done in Git, too) and Objects, how can it mix?
        — Could one have “content pointers” instead of “memory pointers”?

I am sure there is much more…


        Marcus

Marcus Denker <[hidden email]> writes:

> Indeed. Another thing to try: if someone has a local IPFS running, would it cache
> only those images that either this machine or near machines requested?

Yes. But note that "nearby" is probably not what you expect, since IPFS
connects more or less randomly to other peers in the network. It's
neither geographically nearby nor nearby in the local network. All that
can be configured.

> This way we could have one “full” copy and caching (e.g. in the “African class room”
> example) would happen automatically.

That might work. A safer approach is to have one server machine in the
classroom on which the required files are pinned (IPFS jargon for "not
removed during garbage collection"). Then configure the other machines
nearby to "bootstrap" from that server.

> Of course the downside is that one needs to speak the IPFS protocol,
> thus either running a client (e.g the go client)… so real transparent
> use would only be possible if Pharo could implement the protocol…

The generally recommended way to use IPFS on a desktop machine is what I
do in my Pharo interface: connect via HTTP to go-ipfs running as a
daemon process. Even the IPFS command-line client does that. The
advantage is that you have only one data repository per machine, rather
than one per process.

For getting a Pharo image, you would then use the command-line
interface, or implement IPFS access in Pharo launcher via the daemon
(which should be straightforward, since it's really just an HTTP
request).

> There are many other ideas to explore (based on my partial knowledge)…
> more brainstorming than anything else:
>
> - Could we have an image formate that would be “block friendly”?

Probably. If some stable core is in one part of the image and the stuff
that changes with each release in another part, then many block could be
shared. Which IPFS does automatically.

> - How could one move resources out of the image into IPFS?

This is what I am currently experimenting with. My idea is to use IPFS
as something like an extended image, mainly for data that I would also
like to access from non-Pharo code.

> - Content addressing (like it is done in Git, too) and Objects, how can it mix?
> — Could one have “content pointers” instead of “memory pointers”?

It can certainly be done (just put all objects into a dictionary with
the hash as the key), but memory efficiency and access time could be
serious obstacles. This sounds more like a research project.

Konrad.