JNIPort preview for Squeak and Pharo available on SqueakSource

I have made a preview release of JNIPort for Pharo/Squeak available on SqueakSource:
    http://www.squeaksource.com/JNIPort.html

JNIPort is a Smalltalk library which allows Java code to be invoked from Smalltalk. It acts as a bridge between the world of Smalltalk objects and a Java Virtual Machine (JVM) where Java code is executing.

If you want to know what can be done with JNIPort, see
    http://philemonworks.wordpress.com/2010/04/29/google-api-access-from-smalltalk-using-jniport/
for an example.

Installation instructions are on the Wiki at SqueakSource. See http://jniport.wikispaces.com/ for more information about JNIPort.

The current state of the Pharo/Squeak version of JNIPort (as of June 8, 2010):

• Starting a JVM seems to lead to a deadlock in Squeak 4.1, but not in Pharo. When you start a JVM, nothing happens until you hit command-. (on a Mac), wait until a notifier for the user interrupt appears, and proceed. The problem seems to be somewhere in WeakRegistry; it disappears when you replace Squeak’s WeakRegistry by the WeakRegistry class from a Pharo image.
• The UTF16TextConverter in Squeak 4.1 has a bug. I found the same problem in Pharo and reported it there, a patch is already in the queue for Pharo. I have no idea if this will automatically make it into Squeak. See
  

http://code.google.com/p/pharo/issues/detail?id=2516

This means that java.lang.String objects will probably result in junk when they are converted to Smalltalk Strings in Squeak 4.1.

• Callbacks from Java to Smalltalk are not yet supported.
• In contrast to VisualWorks and Dolphin, JNIPort does not create ghost classes as wrappers for Java classes, but creates “real” classes which can be seen in the system browser. It also does not copy prototypes of CompiledMethods when a wrapper class or a Java class is created, because this causes VM failures in Pharo 1.0. Instead, methods are compiled from source code. This has a performance impact, of course.
• Another difference from the VisualWorks and Dolphin versions is that wrappers for Java methods without arguments don’t have selectors ending with an awkward “_null” suffix.
• JNIPort has been developed and tested with Pharo 1.0 on Mac OS X 10.6.3. If it works on other operating systems has yet to be tested.
• I don’t have a Metacello configuration yet.
  

The next steps will be:

• Adding support for callbacks from Java to Smalltalk.
• Testing it on more platforms.
• Reorganization of the documentation on the JNIPort wiki, and addition of Pharo/Squeak specific documentation similar to the documentation for the VisualWorks version.
• Release of JNIPort 2.0 for VisualWorks and Pharo/Squeak when everything is stable. This will be after the release of Pharo 1.1.
  

If it works in Squeak 4.1 or on Linux or Windows, please let me know, especially if you had to do something which might be worth mentioning in the installation instructions.

Have fun with JNIPort, and let me know if it works!

Joachim Geidel

On Tue, 8 Jun 2010, Joachim Geidel wrote:

That's really interesting. I tried to reproduce it on windows without
success, the jvm just failed to start, then alien crashed the image. Can
you send a stack trace of the deadlock?


Levente

Am 09.06.10 16:11 schrieb Levente Uzonyi: The attachment contains a stack trace of the place where a user interrupt
has always interrupted the stuck process so far. I suspect that the
finalization process somehow interferes with the registration of new objects
in the WeakRegistry. The problem is in a loop which creates wrapper methods
for each of the Java classes found so far. It does not always happen at the
same iteration, I have seen it happen for the first and for the fourth class
in the list (I checked this only twice so far).

What do you mean by "the jvm just failed to start"? Does it mean that the
jvm.dll could not be found, i.e. the JNIPortJNIInterface could not load the
library?

HTH,
Joachim

On Wed, 9 Jun 2010, Joachim Geidel wrote:

Thanks. It would be good to know if another process (for example the
finalization process) is inside the critical section, or the semaphore is
left without being signaled.

>
> What do you mean by "the jvm just failed to start"? Does it mean that the
> jvm.dll could not be found, i.e. the JNIPortJNIInterface could not load the
> library?

Exactly. Do I have to tell the path to the dll somewhere?


Levente

>
> HTH,
> Joachim
>
>

Am 09.06.10 19:32 schrieb Levente Uzonyi:
As far as I can tell, there can only be two processes which access the
WeakRegistry. One of them is the process running the JVM startup code, the
other one would be the finalization process. JNIPort does not fork other
processes. How could this be analyzed? By instrumenting the finalization
process? How? In VisualWorks, there are code probes which can be used to
monitor variables and more - does something similar exist in Squeak?

>> What do you mean by "the jvm just failed to start"? Does it mean that the
>> jvm.dll could not be found, i.e. the JNIPortJNIInterface could not load the
>> library?
>
> Exactly. Do I have to tell the path to the dll somewhere?

As far as I know, Alien does not yet accept paths to a library from
Smalltalk. I have not used it on Windows, so I don't know what has to be
done there. The jvm.dll is typically located at
  C:\Program Files\Java\jre1.6.0_NN\bin\client\jvm.dll
where NN is the patch level of the JRE. You could try adding the "client"
directory to your PATH. If this doesn't help, try copying jvm.dll into the
directory where the Squeak VM is installed, but I think this might not work.
I faintly remember reading somewhere that on Windows, the jvm.dll looks for
other dlls in ".." relative to its own location, and that this seems to be
hardwired. I don't know for sure, however. Someone who knows more about
Alien might know what to do (Eliot Miranda, John McIntosh,...).

Joachim

Hi Joachim,

Am 09.06.10 20:49 schrieb Eliot Miranda:
>> As far as I know, Alien does not yet accept paths to a library from
>> Smalltalk.
>
> I don't understand this. Of course Alien accepts paths to libraries from
> Smalltalk.  Look at Alien class>>primLoadLibrary:, which is wrapped by Alien
> class>>lookup: symbol inLibrary: libraryName.

Apparently I have thoroughly misunderstood something, probably from one of
the threads about having to place a symbolic link to the library in the VM's
resources directory on Mac OS X 10.6. Thanks for the correction!

For JNIPort, the library file name can be configured before loading the
library, so this should actually be easy:

| jvmSettings jvm zfClass zipfile entries |
JNIPortJNILibrary
    libraryFile: 'C:\Program Files\Java\jre1.6.0_16\bin\client\jvm.dll'.
jvmSettings := JVMSettings new.
jvmSettings usesGhosts: true.
jvmSettings jniPortSettings useJNIHelperLibrary: false.
jvmSettings runtimeSettings
    classpath: '.;C:\somepath\JNIPort.jar;C:\somepath\JNIPort-Tests.jar'.
Cursor execute showWhile: [JVM newWithSettings: jvmSettings].

Best regards,
Joachim Geidel

I have some news, and maybe a solution.

I managed to produce a log file for the #wait and #signal messages sent to
the Semaphore of the WeakRegistry (see below). I still can't explain why the
deadlock occurs and why proceeding after a user interrupt helps.

What I found however is that changing the priority of WeakArray's
finalizationProcess from userInterruptPriority to systemBackgroundPriority
solved the problem - no more deadlocks when starting JNIPort's JVM. I have
no idea why this works, and I also have no idea if changing the priority of
the process can have any negative side-effects.

I find WeakKeyDictionary>>finalizeValues suspicious. It does a linear scan
of the hash table, and when it nils a slot, it rehashes objects found above
this slot. I think this can corrupt the hash table when there are colliding
hashes. A simplified example with a hash table with 5 slots which abstracts
from the fact that the elements are actually WeakKeyAssociations:

Initial state:
[nil nil nil nil nil]
Add object A with hash 4:
[nil nil nil A nil]
Add object B with hash 5:
[nil nil nil A B]
Add object C with hash 4:
[C nil nil A B]
Expire object A, finalizeValues:
[C nil nil nil B]

WeakKeyDictionary>>finalizeValues will detect that the object in slot 4 was
garbage collected, and try to rehash objects from there to the end. However,
it will not detect that object C needs to be moved to slot 4. This means
that C will not be found by scanFor: (tests if it is present will give the
wrong answer), and it can be added again to the WeakKeyDictionary. If C
expires after being added a second time, it will be finalized twice, which
can lead to errors.

Or did I get something wrong here?

Best regards,
Joachim Geidel

----

In the log, + stands for #wait, - for #signal. The numbers are the hashes of
the active process. 1033371648 is the Process executing the JVM startup,
560201728 is the hash of the finalization process.

+1033371648
-1033371648
[many repetitions]
+1033371648
-1033371648
+560201728
-560201728
+1033371648
-1033371648
+1033371648
-1033371648
+1033371648
-1033371648
+1033371648
-1033371648
+1033371648
-1033371648
+560201728
-560201728
[more repetitions: 1 access by the finalization process after 6-8 accesses
by the JVM process]
+560201728
-560201728
+1033371648
-1033371648
+560201728
-560201728
+1033371648
-1033371648
+560201728
-560201728
+1033371648
-1033371648
[more repetitions]
+1033371648
-1033371648
+560201728
+1033371648
[Deadlock, User Interrupt & Proceed here]
-1033371648
-560201728
+560201728
-560201728
Etc.

On Sat, 12 Jun 2010, Joachim Geidel wrote:

Very nice find, this is definitely the bug. Here is a snippet to reproduce
it:

| objectWithHashModulo w a b c |
objectWithHashModulo := [ :requestedHash :modulo |
  | object |
  [
  object := Object new.
  object hash \\ modulo = requestedHash ] whileFalse.
  object ].
w := WeakKeyDictionary new.
a := objectWithHashModulo value: 3 value: 5.
w at: a put: 1.
b := objectWithHashModulo value: 4 value: 5.
w at: b put: 2.
c := objectWithHashModulo value: 3 value: 5.
w at: c put: 3.
self assert: w capacity = 5.
self assert: (w array at: 4) key == a.
self assert: (w array at: 5) key == b.
self assert: (w array at: 1) key == c.
a := nil.
Smalltalk garbageCollect.
w finalizeValues.
self assert: (w includesKey: c)

I will fix it soon.


Thanks,
Levente