Persistent Object Interop with GLASS and Maglev

We're moving towards the development stage of a new project with Gemstone and Seaside using GLASS initially. I think later we will want to move to the commercial version, but there's also been some talk of checking out Maglev later down the line. I'm having some trouble finding good info about the interop behaviors between Smalltalk and Ruby on Maglev.

Can someone identify and outline what are the rules and limitations when dealing with Gemstone, Smalltalk, and Ruby (via Maglev)? It's not quite clear to me how interop between the two works, especially regarding persistent objects.

For example, if we write our application in Smalltalk today, all of our persistent objects will be Smalltalk objects. If we then move all the code and data to Maglev at some point and we want Ruby code to talk to those existing objects, are there any issues? How does it handle persistence if the object was created in Smalltalk instead of Ruby? For instance, if we have some scripts in Ruby that load up a persistent object that is a Smalltalk class, can it transparently write back if we modify it in Ruby? What about the other way around?

I guess what I am asking is if a side-by-side approach of two languages running on Gemstone in the same application, same Stone requires any special code. This came up because there's a wealth of Ruby scripts that do some of the background stuff we might want to do as well as some other logic, but we want to stick to the rest of the GLASS stack for the majority of code.

Thanks.

> I'm having some trouble finding good info about the interop behaviors
> between Smalltalk and Ruby on Maglev.

  That's probably because we haven't published much on that!  I'll try and
  answer some of your questions below, but we currently don't have a lot of
  experience in this area.

> Can someone identify and outline what are the rules and limitations when
> dealing with Gemstone, Smalltalk, and Ruby (via Maglev)? It's not quite
> clear to me how interop between the two works, especially regarding
> persistent objects.

  A couple of guiding principles (not hard and fast rules):

  1. In general, Smalltalk does not know about Ruby, and it takes an
     explicit effort to touch the Ruby side of things from Smalltalk
     (finding objects, invoking methods).

  2. In general, Ruby is built on top of Smalltalk, but does not explicitly
     know about it, and it takes effort to find Smalltalk objects and
     invoke methods on them.

  There are ways to write a mixed Ruby/Smalltalk application, but you will
  be on the vanguard, and you may have to hack your way through a bit of
  underbrush for a while until we discover and address the issues. Today,
  it should be possible to write a mixed Ruby/Smalltalk app, and we hope
  that it will eventually by pleasant to do so.  Please give us feedback!

  Classes and Objects

  From the VM perspective, there really is not a lot of difference between
  a Smalltalk object and a Ruby object (there *are* some differences, but
  nothing relevant to this, already long, discussion). It doesn't make
  sense to talk about "Smalltalk objects" vs "Ruby Objects": to the VM,
  they are both simply "objects".

  Smalltalk code can find and operate on Ruby objects, and vice-versa.
  It's just that you have to be explicit about what you're doing (we don't
  want programmers having to worry about the other language, unless they
  specifically want to take advantage of it; no accidental calls between
  the two languages).

  Each language has its own way of easily finding well known classes in
  that language (standard symbol dictionaries for Smalltalk, the Ruby
  namespace hierarchy for Ruby).  Those mechanisms do not prevent discovery
  from the other language.  In fact, many of the classes are shared.  E.g.,
  Object, Array, Float are the same identical class.  I.e.,

     Smalltalk:  Object asOop    "72193"
     Ruby:       Object.__id__   # => 72193

  Some of the Ruby classes show up in the standard Smalltalk symbol
  dictionaries (try browsing for "Ruby" in a class browser).  Some of the
  classes are not easily accessible from Smalltalk (e.g., library defined
  classes, user defined classes), because they are only registered in the
  Ruby namespace.  But you can still access these from Smalltalk.  E.g.,
  here is how you'd access the ruby constant Maglev::PERSISTENT_ROOT:

    |ns moduleMaglev proot|
    ns := Object persistentNameSpace: 1 .
    moduleMaglev := (ns rubyConstAssociationAt: #Maglev) value.
    proot := moduleMaglev rubyConstAt: #PERSISTENT_ROOT env: 1.

  We first get the RubyNameSpace object for the Object class.  This is
  where the Ruby constants for class Object are stored.  We then get the
  value of the :Maglev constant (a module) and then ask the Maglev module
  for the value of the constant :PERSISTENT_ROOT.  We could also store
  values into these namespaces from Smalltalk, and they would become
  visible to Ruby.

  So, objects are just objects, and can be accessed from either language.

  Persistence:

  Persistence works the same for both languages.  If an object is reachable
  from another persistent object, then that object will be persisted at the
  next commit.  Since there aren't really "smalltalk objects" and "ruby
  objects", the only criterion for persistence is reachability (nothing
  changed here).  For instance, you could put a Smalltalk-ish object, say a
  MetacelloVersionNumber object, into the persistent RubyNameSpace for
  Object, commit, and it would be visible to Ruby code.  The Ruby code will
  have difficulty effectively using that object, since there will be no
  methods visible to the Ruby code for the MetacelloVersionNumber object.
  Which brings us to...

  Methods and Environments:

  The 3.0 GemStone VM has the notion of an Environment.  An environment is
  a namespace for message sends.  Smalltalk is in environment 0, and Ruby
  is in environment 1.  Each environment has its own method dictionaries.
  A message send, by default, stays within its own environment.  This means
  the author of a method only has to be concerned about which methods are
  visible in that environment.  You wont accidentally send a Ruby message
  from a Smalltalk method. Likewise, all of the Ruby methods operate in
  environment 1, and do not see any of the Smalltalk methods.  Each
  language preserves its own method namespace.  Since Ruby depends on
  Smalltalk, there are a lot of new methods in environment 0 (the Smalltalk
  side of things) that support ruby.  They tend to be in either the
  *maglev-runtime* or RubySupport categories, but you can look at them from
  the Smalltalk class/method browser.

  There are mechanisms to call from one environment to the other.  Many
  methods have a variant that take an "env" parameter.  You can then call
  from Smalltalk into the Ruby environment 1 method dictionaries. There is
  a variant for perform: named perform:env: you could use to send arbitrary
  ruby methods to objects.  E.g. Ruby defines the "object_id" method, but
  you can't call it directly from Smalltalk:

     Object object_id   "raises a MessageNotUnderstood"

  But using perform:env:, you can call it from Smalltalk:

     Object perform: #object_id env: 1 .   "72193"

  And if you try to invoke it in env 0, you get the MessageNotUnderstood
  again:

     Object perform: #object_id env: 0 .   "raises a MessageNotUnderstood"

  You can also invoke Smalltalk methods from Ruby, but it is a bit more
  complicated.  Try the following:

    $ cd $MAGLEV_HOME
    $ rake stwrappers

  And then look at the files in
  $MAGLEV_HOME/lib/ruby/site_ruby/1.8/smalltalk.  E.g., here is some of the
  code from one of those files, with some comments I added for this email:

    # Get a reference to the RcCounter class.
    RcCounter = __resolve_smalltalk_global(:RcCounter)

    # Open up (monkey patch) the class
    class RcCounter

      # the class_primitive_nobridge and primitive_nobridge methods takes
      # two parameters, the ruby name and the smalltalk name.  They then
      # make the smalltalk method available to ruby via the ruby name.
      # The class_primitive_nobridge works on the class side, and the
      # primitive_nobridge works for instance methods.
      #
      # E.g., this makes it possible to send the :comment message to the
      # environment 0 method dictionaries from Ruby like this:
      #
      #    RcCounter._st_comment()  # => "blah blah blah"
      #
      class_primitive_nobridge '_st_comment', 'comment'


      # This allows ruby to do:
      #   an_rc_counter_obj._st_decrement()
      primitive_nobridge '_st_decrement', 'decrement'

      # Called like:  an_rc_counter_obj._st_decrementBy_(10)
      primitive_nobridge '_st_decrementBy_', 'decrementBy:'

    end

  There are other mechanisms we developed to implemmet MagLev, but aren't
  really polished for general use, e.g., from Smalltalk -> Ruby:

     anObject @ruby:ruby_method: param

  We obviously need a lot more documentation here, and probably better
  support for the Ruby->Smalltalk direction (e.g., we don't even have the
  Ruby equivalent of the perform:env: method defined!).

> For example, if we write our application in Smalltalk today, all of our
> persistent objects will be Smalltalk objects. If we then move all the
> code and data to Maglev at some point and we want Ruby code to talk to
> those existing objects, are there any issues?

  In general, we expect that you should be able to do this.  There may be
  issues, but those should be mostly ease of use issues.  But again, we
  don't yet have a lot of experience with real applications calling back
  and forth between Smalltalk and Ruby.  We are striving to make it
  seamless, but we still need more work to make that happen.

> How does it handle persistence if the object was created in Smalltalk
> instead of Ruby? For instance, if we have some scripts in Ruby that load
> up a persistent object that is a Smalltalk class, can it transparently
> write back if we modify it in Ruby?  What about the other way around?

  Objects are just objects.  Ditto persistent objects.  The only thing that
  matters for persistence is reachability.  If objectA is reachable
  from a persistent object, then objectA will be persisted at the next
  commit.  So, there is no fundamental issue here.

  There are some extra steps you might have to take to ensure this works.
  E.g., by default, Ruby classes read in from files are not persistent.
  The rules of persistence require that an object's class (and super class,
  mixed in modules, etc.) be persistent as well.  So, you'll need to take
  care that the task yourself, otherwise the commit will fail.  But this is
  a "managing the persistence of the Ruby class hierarchy" issue, and
  not a fundamental VM-level issue.

  There may also be issues arising from the different mechanisms Ruby and
  Smalltalk use to find classes, and the tools/support each language has
  for class versioning and migration. But they are not fundamental VM-level
  issues, rather "ease of use" issues.  We have some ideas on how to
  address these, but we are hoping that real-world experience will bring
  more insight in how to manage Ruby persistence (from the project
  management, operational, design pattern side of things), You might want
  to read $MAGLEV_HOME/examples/persistence/migrations/migrations.org for
  some initial thoughts on Ruby persistence.

> I guess what I am asking is if a side-by-side approach of two languages
> running on Gemstone in the same application, same Stone requires any
> special code.

  It will require special code, because we require a special syntax to
  call from one environment to the other.  But, in theory, that should be
  it (we'll see how well theory holds up in practice....;^).

> This came up because there's a wealth of Ruby scripts that do some of the
> background stuff we might want to do as well as some other logic, but we
> want to stick to the rest of the GLASS stack for the majority of code.

  You should be able to do this.  This is one of the things we're hoping
  for: allow you to use the best tools and libraries from both languages to
  solve problems.

  So, the MagLev team needs to document things better, and we expect that
  we'll have to provide a more polished API for inter-language message
  passing.  But, I think we'll be more successful if we get some real-world
  experience before we set things in concrete.   We have some rough, basic
  mechanism in place today, and look forward to making it better in the
  future.

--  
Peter McLain
[hidden email]

As Peter points out, we'd love to hear _your_ thoughts on what a seamless Smalltalk to Ruby and Ruby to Smalltalk interface would be. We tend to be in one camp or the other, I suspect there are actually more of you out there who use both.

-- Monty

Thanks. This is exactly what we were looking for. I figured persistent objects should be the same. The namespacing and separate dictionaries make total sense as well. I think we're awhile off from using a mixed approach in a real app, but this helps us that we can plan to do so.

Gaving worked in the past with C and various object dbs including Gemstone (also Objectivity, Vesant, and a few others), I realize it can be a pain talking between languages. It usually ends up that one language is king and the others are just in a sense translating through some crude API. This is one issue we had with the later versions of objectivity - Smalltalk and some other supported languages became second-class citizens. I think what you guys have done here looks a lot better than what we encountered in the past.

As long as the rules are consistent and documented, what you describe is not a huge issue. I think mostly we're interested in using objects created by Smalltak with Ruby, and in some cases calling Ruby from Smalltalk.  I do wish we could use Pharo's browsers to work on Ruby too, but it's not easy matching things that don't align perfectly.

As for real-world apps, having gone through many language a talks to language b integrations from java<->smalltalk, c#<->java, lisp<->c++, etc. I find it's always good to also have some application-level glue. I know that the low-level stuff is needed for all the situations we talked about and especially persistence,  but sometimes you really just want to grab a snapshot of data or check a simple value. I think for some of these cases in our applications we might consider building some facade, web services, or RESTful APIs where the web is involved and super speed and persistence are not really as important.

Thanks for the detailed reply. I look forward to when things are further along that you can post some of this info. I've checked your Twitter and I always check the corporate page, but there's not much at the moment. The blogs from Dale and James on GLASS have been a huge help to us and we really appreciate it. I suppose the lack of documentation and less frequent updates to all of the above means you guys are doing actual work. We have similar issues so I empathize. I also know doing documentation early sometimes is like rolling a boulder up a hill or doesn't make sense at all. In any case, keep up the good work, we appreciate it.