[squeak-dev] Beyond email, the Social Semantic Desktop

Ken wrote:
> I agree that for now we should simply move on but perhaps give a little
> thought to what might be a workable alternative if this does turn into a
> real problem in the future.

The recent spam with forged senders to the Squeak list
http://lists.squeakfoundation.org/pipermail/squeak-dev/2009-January/133725.html
suggests public email lists are going to be less and less useful.

I know from one email forged to appear to be from me on one mailing list
related to Doug Engelbart's Unrev-II colloquium (many years ago), how
unsettling that can be for the one whose email address gets forged.
Ultimately such things probably cannot be prevented entirely (security is
never perfect), but such events can certainly be made less annoying and much
rarer.

One might ask how Squeak or similar systems could help with that.

Filtering at one point will just never be perfect. One issue with email
(unlike web forums) is that the community can't easily tag information
*after* it is sent out in a machine-readable format, like as "forged" or
even just "boring" or "interesting". One can even imagine tagging email
information after it is sent with a complex set of information like Slashdot
does, say, related to moderation by various human moderators, then
interpreted locally in your email client using different moderator
reputations you yourself or someone you trust supplies. Perhaps you might
only read emails to the Squeak list that Tim Rowledge marked as VM-related.
:-) Or, one can imagine better ways of splitting up long emails into a
variety of topics after they were sent and creating better links between the
ideas. So, if we have a way to tag information after it is sent out to a
bunch of people, we can work collaboratively and stigmergically to build
information into knowledge (or an approximation thereof). This is in some
sense how the first email-like system by Doug Engelbart worked with Augment,
although done on only one central machine. There are some commercial systems
that do support shared distributed workspaces for communications, but we all
know a dynamic and open source language platform should be able to support
this better. :-) Wikis like Ward Cunningham invented in Smalltalk already
work this way in a sense (continued modifications after the first post), but
they are generally not distributed and wikis are generally restricted to a
textual approach (although "semantic wikis" are emerging, like Semantic
MediaWiki).

Distributed systems like email are really good approaches for a lot of
reasons like by minimizing bottlenecks if everyone used the same server for
reading a web forum or by creating redundancy by everyone having a local
copy of each message after it is sent. So, the basic technical idea of email
is not that bad in terms of being distributed (likewise Usenet was a really
good idea in some ways to move stuff around in a distributed way). But email
is showing its age because email is still perceived mostly as being about
sending free form text, so there is no obvious way to hook tagging into it.
And of course HTML mail was a step in the wrong direction because it didn't
address the general problem of sending complex information people could
easily collaborate to enhance it, while at the same time HTML mail made
plain text email more difficult to use. But it still showed one can build
stuff on top of email, as does the notion of MIME types and attachments. We
still could use something better, and many agree, but it seems like an
impossible task to upgrade email and not worth the bother. Backwards
compatibility is nice, so one can imagine a new system would ideally need to
support conventional email, perhaps adding extra tagging information inline
or in attachments. But that all still seems more complicated than it is
worth. But what if there were other good reasons to switch to a new
communications platform?

I'd suggest the future may not be so much in reinventing mailing lists and
email with authentication, but rather more in moving entirely to a new
multi-purpose distributed paradigm like the "Social Semantic Desktop" (SSD)
idea:
   http://www.semanticdesktop.org
 From there: "The Internet, electronic mail, and the Web have revolutionized
the way we communicate and collaborate - their mass adoption is one of the
major technological success stories of the 20th century. We all are now much
more connected, and in turn face new resulting problems: information
overload caused by insufficient support for information organization and
collaboration. For example, sending a single file to a mailing list
multiplies the cognitive processing effort of filtering and organizing this
file times the number of recipients -- leading to more and more of peoples'
time going into information filtering and information management activities.
There is a need for smarter and more fine-grained computer support for
personal and networked information that has to blend the boundaries between
personal and group data, while simultaneously safeguarding privacy and
establishing and deploying trust among collaborators."

That page goes on to provide more details including: "P2P and Grid
computing, especially in combination with the Semantic Web field, develops
technology to interconnect large communities without centralized
infrastructures for data and computation sharing, which is necessary to
build heterogeneous, multi-organizational collaboration networks."

You could imagine a Social Semantic Desktop from one point of view as being
like a really smart email client, where you could send out emails to mailing
lists saying how to tag previous emails. But, one can imagine supporting all
sorts of backends besides email to let small workgroups communicate in
various ways. And one can imagine having all sorts of applications running
on top of this distributed infrastructure (my wife and I have a couple new
ones in mind myself related to manufacturing and storytelling). Squeak does
have distributed applications like Croquet,
   http://en.wikipedia.org/wiki/Croquet_project
but I'm talking about a more common infrastructure for all these ideas.
Perhaps the TeaTime TObject idea, while great for what it does with Croquet,
is not the only approach to abstraction for the general problem of
collaborative work on information?

I have some notions on this myself which I have been pursuing in Jython for
the JVM:
   http://sourceforge.net/projects/pointrel/
Sorry it's not in Squeak, although earlier versions of that code before the
SSD focus from years back were for Squeak. Essentially, I've been working on
a triple store called Pointrel on-and-off for about a quarter century. That
work even predates WordNet which was in a tiny way perhaps inspired by it.
Why isn't it done yet? Good question. :-)

I'd suggest that using Squeak as the basis for a similar or better Social
Semantic Desktop might be the day that Squeak conquers the world. :-) Or at
least a bigger part of it than it has already with Seaside, Croquet,
Scratch, etc.. :-)

Maybe it could be done using TeaTime, but here is another possibility coming
at this situation from a different perspective.

Here is a rough idea of what I think is a good architecture for a Social
Semantic Desktop and which I am working towards right now. Although I'm
working in Jython to leverage Java, obviously anybody could work on in
Squeak as friendly co-opetition, and so people are welcome to sign up for
the Pointrel SSD mailing list on SourceForge and use it as a place to bounce
around Squeak-related version ideas for now. That list:
http://sourceforge.net/mailarchive/forum.php?forum_name=pointrel-discuss
Obviously, the hope is to replace that list using the system itself. :-)
So, it's just for bootstrapping. :-)

Basic implementation ideas:

* Internally, information is stored in the equivalent of RDF triples.
   http://en.wikipedia.org/wiki/Resource_Description_Framework
I'm using a variant of a triple with a context field as well, like NEPOMUK does:
   http://nepomuk.semanticdesktop.org
(NEPOMUK is a SSD attempt for KDE.) Triples are a general purpose way to
store information, essentially just saying how digital objects link
together. One might expect there would be ways to place Smalltalk objects
into sets of triples (or even just strings) and get them back out again.
Each of the four fields I'm using has both a data field an a namespace
describing how to interpret the data. Using the RDF naming convention, there
are "subject", "predicate", and "object" fields. (I actually prefer
"object", "attribute", "value" which are more OO-like.)  The fourth
"context" field is sort-of equivalent to a file type and file name. These
triples are defined in approach I am currently taking using "transactions"
which are sets of triples to add or remove from the triple store all together.

* Objects are essentially defined by these triples, and their complete
history is implicit in the list of all transactions which is stored on disk
like a Smalltalk changes file (although possibly transactions might be
stored as lots of little files with UUID names until they are periodically
joined together in a larger file). Storing the entire history of the shared
database is wasteful in some ways, but disk space is cheap these days;
likely the whole history will be in memory as well (and that's how I'm doing
the Jython implementation). For any implementation, the RDF database is
either entirely in memory (easy for the first try and coding it yourself) or
on-disk and cached. There are multiple RDF triple-stores that do this
already on disk with a memory cache in efficient ways, including with query
languages etc., though not to my knowledge with these distributed
transaction files like I mention below, though maybe one does?

* Applications like email, shared to-do lists, virtual worlds, shared eToys,
source code control systems, etc. would then all run on top of this
distributed system. Not every client might have all transactions, in which
case their data may have inconsistencies, but applications should be written
to be forgiving of this. :-) Or, application that are less forgiving would
require using more exacting backends than email lists (like a central
database or a IRC-like system or TeaTime for real-time low-latency
applications). Essentially, different distributed databases might have
different preferred coordination backends depending on the need for
reliability or low-latency or public accessibility. This system could
potentially replace the Squeak changes file and sources file and various
Squeak source control systems someday, like say if each change to the system
was written out as a transaction in an XML file. (I've also implemented an
OO-like system purely on triples that supported a crude Smalltalk-like
environment as an experiment a few years back, but that's another issue; for
now I just see this approach as a flexible shared database which transmits
the data usually in objects, but I point that proof-of-concept out to show
how general purpose triples can be.)

* Information could be exchanged as "transactions" that are distributed in
some way like via mailing list email attachments, source code control system
like git, CGI-based system, remote database, WebDAV, TeaTime, or whatever
backends one had access to for transmitting these transaction files (or the
equivalent). I have not implemented this part yet, as the system I have so
far just uses one big file and for shared use can redirect changes to that
file through a CGI script which is somewhat like a mailing list archive that
clients can consult to get all the recent changes they don't have yet. These
transactions could have digital signatures to make forgeries more difficult
(though I have not implemented that either). Each transaction is mainly a
list of triple additions or deletions which modify the triple store (with a
timestamp and author and licenses granted for each action -- in general the
system now goes overboard with explicit license granting). Each transaction
also has a timestamp and the file also specifies which distributed database
it goes with (via a UUID). (It might make sense to have more than one
transaction in a file perhaps.) Probably XML is an OK choice for these
transaction files, which in a sense represent complex messages about how to
change the state of a triple store. I currently use a different plain text
format that is easy to read, but I'm thinking of bowing to the inevitable
standard issue of XML (especially so email attachments could just be
innocent looking XML file). It really does not matter much what the
interchanged data looks like, and the local repository could be in a
different format. Each transaction has a UUID and it is OK to receive the
same transaction twice as long as it is identical.

There are some other details, but that is the basic picture.

Maybe people well-versed in OO design or distributed systems here might have
better ideas, or even point to existing systems which might be better
matches for either Squeak or the JVM than what I propose. If so, I'd
appreciate hearing about them. Here is one somewhat related idea released as
open source by NASA:
    http://infolab.stanford.edu/~maluf/papers/xdb_ipg_ggf03.pdf
"This paper describes XDB-IPG, an open and extensible database architecture
that supports efficient  and flexible integration of heterogeneous and
distributed information resources.  XDB-IPG provides a  novel “schema-less”
database approach using a document-centered object-relational XML database
mapping. This enables structured, unstructured, and semi-structured
information to be integrated  without requiring document schemas or
translation tables. XDB-IPG utilizes existing international  protocol
standards of the World Wide Web Consortium Architecture Domain and the
Internet  Engineering Task Force, primarily HTTP, XML and WebDAV . Through a
combination of these  international protocols, universal database record
identifiers, and physical address data types, XDB-IPG enables an unlimited
number of desktops and distributed information sources to be linked
seamlessly and efficiently into an information grid.  XDB-IPG has been used
to create a powerful set  of novel information management systems for a
variety of scientific and engineering applications."

Mine is mainly just simpler. :-)

I do waffle sometimes myself about going back to Squeak to do this (as I
waffle about spending time on getting a Squeak-like system on the JVM). I am
really impressed by Scratch, for example, as a stand-alone application. If
there was a motivated group of people interested here in such a system I
might be tempted to move back to the Squeak side for a time (maybe for
prototyping and then seeing how it goes after that), especially as the
Squeak license issues are getting cleaned up, although I'd be very rusty in
Squeak at this point so it is not my first choice at this point given my own
expertise in Jython. But the core I describe here is not very hard to build
in any language if you do the brute force approach (everything in memory
when you want to use a distributed database); what is the big time
requirement is writing the GUI applications on top of the core (stuff that
works like an email list, or like Wikipedia or Knol, or like SVN, or like
any of many other systems for stand-alone work or collaboration). Squeak has
the rudiments of many of those things, so there is a good argument one could
build on top of, say, Celeste as an email client, or use Scamper to build a
distributed wiki system. This sort of project might really be facilitated by
all the years of hard work people have put into Squeak applications, but
leveraging them all those somewhat copycat Pink Plane efforts in a radically
new Blue Plane distributed database sense. Even the original Augment code
that has been redone in Squeak could be integrated for fun. In any case, I'd
be happy to discuss these issues with Squeakers who wanted to build their
own system even if I stayed with Jython, with an eye to compatibility for
the files or other protocols used to interchange transactions or handle
other issues. Although the more I think about the possibilities of
leveraging all those previous Squeak efforts to build a self-contained
environment, the more interesting it sounds to do this in Squeak. Still,
there obviously are free and open source clients to do email and web
browsing etc. in a lot of languages, so one would expect that after the
system was defined and usable, that many other people would adopt the
distributed back end for their own systems (like write Thunderbird plugins
or whatever).

I know this all sounds ambitious, but the key idea is that workgroup
software is already being used, since all it takes to make this useful is a
few people who want to work together (like with Croquet), and an email
gateway lets the rest of the world that does not adopt the system still stay
in touch with the workgroup.

Anyway, I know I'm taking advantage of an unfortunate situation to toot my
own project's horn, sorry. And I probably would not have sent this email if
I had not seen that spam, as I'm mostly into Jython right now. Still,
basically, even regular spam is very annoying as I already pretty much will
never see email to me that is not one of:
* On a mailing list I have a filter for (but this spam got around that, and
suggests that approach won't work much longer for public lists),
* Has my name in it (even that is getting iffy since I signed up for some
Google groups that somehow spammers now connect my name and email),
* Has one of a few other common terms I am interested in (actually the spam
to Squeakdev triggered a filter I have or I might not have noticed it since
I don't follow this list closely these days), or
* Is on a whitelist of some senders I know and put on there manually (and
that may fall apart if the spammers improve in terms of their forgeries).
The rest of my recent email sits in a pile of over 40000 unread messages
(all spam, I hope, though sometimes I search on it and notice something that
slipped by).  And that is 40000 spam messages even with SpamAssassin on the
email server filtering out the worst ones. And that's 40000 spams just since
last I cleaned that file out some months ago. Granted, my email address has
been on the web for more than ten years.

But in any case, the best reason to do this is actually not to get rid of
spam. It is to enable people to stigmergically refine knowledge-related
digital artifacts that people like Doug Engelbart, Vannevar Bush, Ted
Nelson, and Theodore Sturgeon envisioned decades ago.
   http://en.wikipedia.org/wiki/Stigmergy

Still, I'd suggest this unfortunate incident might be more motivation for
Squeakers to do something about email in general to the extent technology
can help, and the above Social Semantic Desktop idea is one approach which
some Squeakers might be interested in. No doubt spammers will catch up
eventually, but in the meanwhile things may improve for a time, plus there
will be the new distributed applications. In the long term, social change to
a world of abundance for all may be a better solution, of course, at least
to reduce the motivation for most commercial spam; so in that sense the
abundance being facilitated by the internet will help the internet defend
itself from spammers in one way or another. :-)

--Paul Fernhout
http://www.pdfernhout.net/