Changeset: Enhanced integration of drag'n'drop from host

Am 23.10.2020 14:11:28 schrieb Thiede, Christoph <[hidden email]>:

Hi all! :-) I'm glad to officially submit my dropFiles changeset today, finally. It enhances the integration of drag'n'drop (DnD) events delivered by the VM from the host operating system to the image, following one of Marcel's ideas: Instead of generating separate DropFilesEvent instances, every DnD event from the VM is translated into a TransferMorph that can be dragged into every morph or tool like every other Morph of TransferMorph, too.

What is the goal of this changeset?

This design change entails two key advantages: First, every dragged host element is represented by a first-class object (which is a TransferMorph) instance. Second, as a consequence, applications do no longer need to implement a separate protocol for handling host DnDs (which was #dropFiles:/#wantsDropFiles:) but only need to provide one single proper implementation of #acceptDroppingMorph:event:/#wantsDroppedTransferMorph:. In particular, this also enables developers who use the ToolBuilder(Spec) framework to handle contents dropped from the host system since the old dropFiles protocol was never forwarded to the ToolBuilder framework.

How does it work?

There have already been complete implementations of the DropPlugin on the VM side for all important platforms (Windows, Linux, and macOS) for a long time (almost 20 years ...) which record four different types of drag events, namely: #dragEnter, #dragMove, #dragLeave, and #dragDrop. However, until now, only the last of them, #dragDrop, has been handled on the image side (see HandMorph >> #generateDropFilesEvent:). The main work of this changeset can be found in just this method where I added support for the other drag types in order to create, move, and release a TransferMorph accordingly. Other changes include:

• Deprecated, but kept working the old DropFilesEvent protocol (these events are still processed, but a DeprecationWarning is signaled if any morph actually handles them).
• Migrated the only implementation of the old dropFiles protocol, which is PasteUpMorph, to the unified TransferMorph protocol.
• Introduced a new event class + protocol for SystemLaunchEvent (aka #launchDrop) which is generated by a singleton VM when it has been invoked with a new image file (see class comment). That #launchDrop is a curious hack on the VM side to reuse the DnD protocol but has completely different semantics that now are separated on the image side.
• Implemented the SystemLaunchEvent protocol on PasteUpMorph.

Besides the changeset, I have also fixed a few minor inconsistencies with several VM-specific implementations in the OpenSmalltalk-VM repository (see #508, #514, and #518) that have already been merged (thanks to all reviewers!).

What's the current state of the changeset?

I have successfully tested the changeset on Windows (Win 2004) and Ubuntu/X11. Also, I made sure it works with Squeak.js and TruffleSqueak. However, I do not have any possibility to test it on the OSVM implementations for macOS and iOS, so your help and feedback especially for these platforms will be greatly appreciated!

Please see the attached demo video on how to test the changeset: Simply drag some files into an inspector field or a workspace and you should receive a list of FileStreams and/or FileDirectories. Also, aborting a drag operation should not fail, i.e. drag a file over the image but then drop the file in another window.

What's next?

There is much more potential for a comfortable host DnD integration! Concretely, I am having two downstream goals in mind: First, dropping other content types besides files and/or directories in your image, such as texts or images (currently, the DropPlugin implementations that I could try out are restricted to the former two types). Second, dragging objects out of the VM to save them as a file, or even better, to drop them in a second image! Vanessa has told me that this was even possible in the past for certain platforms, but unfortunately, the relevant code appears to be dead nowadays. Nevertheless, I have stumbled upon a lot of stubs for both features in the DropPlugin and HandMorph code, and I'm looking forward to reviving them in some interesting follow-up projects!

Which related contributions/discussions are there else?

• System-ct.1156 (Inbox, fixes double prompt for handling a dropped file when the first dialog was canceled; not included in the changeset)
  
• [squeak-dev] Re: Please try out | Inspector Refactoring =) (the thread contains some interesting brainstorming about MIME type support for dragging inside the image)
  

Please test and review! And many thanks to Marcel for his helpful tips!

Best,

Christoph

Am 26.10.2020 07:47:02 schrieb Marcel Taeumel <[hidden email]>:

Hi Christoph!

This is great news! I will take a look at it and test it for the Windows platform.

Best,

Marcel

Am 23.10.2020 14:11:28 schrieb Thiede, Christoph <[hidden email]>:

Hi all! :-) I'm glad to officially submit my dropFiles changeset today, finally. It enhances the integration of drag'n'drop (DnD) events delivered by the VM from the host operating system to the image, following one of Marcel's ideas: Instead of generating separate DropFilesEvent instances, every DnD event from the VM is translated into a TransferMorph that can be dragged into every morph or tool like every other Morph of TransferMorph, too.

What is the goal of this changeset?

This design change entails two key advantages: First, every dragged host element is represented by a first-class object (which is a TransferMorph) instance. Second, as a consequence, applications do no longer need to implement a separate protocol for handling host DnDs (which was #dropFiles:/#wantsDropFiles:) but only need to provide one single proper implementation of #acceptDroppingMorph:event:/#wantsDroppedTransferMorph:. In particular, this also enables developers who use the ToolBuilder(Spec) framework to handle contents dropped from the host system since the old dropFiles protocol was never forwarded to the ToolBuilder framework.

How does it work?

There have already been complete implementations of the DropPlugin on the VM side for all important platforms (Windows, Linux, and macOS) for a long time (almost 20 years ...) which record four different types of drag events, namely: #dragEnter, #dragMove, #dragLeave, and #dragDrop. However, until now, only the last of them, #dragDrop, has been handled on the image side (see HandMorph >> #generateDropFilesEvent:). The main work of this changeset can be found in just this method where I added support for the other drag types in order to create, move, and release a TransferMorph accordingly. Other changes include:

• Deprecated, but kept working the old DropFilesEvent protocol (these events are still processed, but a DeprecationWarning is signaled if any morph actually handles them).
• Migrated the only implementation of the old dropFiles protocol, which is PasteUpMorph, to the unified TransferMorph protocol.
• Introduced a new event class + protocol for SystemLaunchEvent (aka #launchDrop) which is generated by a singleton VM when it has been invoked with a new image file (see class comment). That #launchDrop is a curious hack on the VM side to reuse the DnD protocol but has completely different semantics that now are separated on the image side.
• Implemented the SystemLaunchEvent protocol on PasteUpMorph.

Besides the changeset, I have also fixed a few minor inconsistencies with several VM-specific implementations in the OpenSmalltalk-VM repository (see #508, #514, and #518) that have already been merged (thanks to all reviewers!).

What's the current state of the changeset?

I have successfully tested the changeset on Windows (Win 2004) and Ubuntu/X11. Also, I made sure it works with Squeak.js and TruffleSqueak. However, I do not have any possibility to test it on the OSVM implementations for macOS and iOS, so your help and feedback especially for these platforms will be greatly appreciated!

Please see the attached demo video on how to test the changeset: Simply drag some files into an inspector field or a workspace and you should receive a list of FileStreams and/or FileDirectories. Also, aborting a drag operation should not fail, i.e. drag a file over the image but then drop the file in another window.

What's next?

There is much more potential for a comfortable host DnD integration! Concretely, I am having two downstream goals in mind: First, dropping other content types besides files and/or directories in your image, such as texts or images (currently, the DropPlugin implementations that I could try out are restricted to the former two types). Second, dragging objects out of the VM to save them as a file, or even better, to drop them in a second image! Vanessa has told me that this was even possible in the past for certain platforms, but unfortunately, the relevant code appears to be dead nowadays. Nevertheless, I have stumbled upon a lot of stubs for both features in the DropPlugin and HandMorph code, and I'm looking forward to reviving them in some interesting follow-up projects!

Which related contributions/discussions are there else?

• System-ct.1156 (Inbox, fixes double prompt for handling a dropped file when the first dialog was canceled; not included in the changeset)
  
• [squeak-dev] Re: Please try out | Inspector Refactoring =) (the thread contains some interesting brainstorming about MIME type support for dragging inside the image)
  

Please test and review! And many thanks to Marcel for his helpful tips!

Best,

Christoph

MorphicEvent subclass: #SystemLaunchEvent
	instanceVariableNames: 'arguments wasHandled'
	classVariableNames: ''
	poolDictionaries: ''
	category: 'Morphic-Events'!

!SystemLaunchEvent commentStamp: 'ct 1/18/586455 02:22' prior: 0!
I am signaled by the VM that is configured as a singleton application when it has been invoked with a new image file.

This feature is also known as #launchDrop as well as under several platform-specific names:

	Unix - runAsSingleInstance
	Win32 - RunSingleApp

Instance Variables
	arguments:		A collection of FileStreams.
	wasHandled:		A Boolean.!

!Morph methodsFor: 'event handling' stamp: 'ct 10/15/2020 19:31'!
handlesSystemLaunch: evt
	"Do I want to receive #systemLaunch: when the VM instance is invoked with a second image file? See #handleSystemLaunch:."

	^ false! !

!Morph methodsFor: 'event handling' stamp: 'ct 10/15/2020 19:36'!
systemLaunch: evt
	"Handle a systemLaunch event, signaled when the VM instance is invoked with a second image file. See #handleSystemLaunch:. Hook for subclasses."! !

!Morph methodsFor: 'events-processing' stamp: 'ct 10/15/2020 19:32'!
handleSystemLaunch: anEvent
	"Handle a system launch event from the OS. Signaled if them VM is configured as a singleton application when it has been invoked again with a second image file. See SystemLaunchEvent comment."

	anEvent wasHandled ifTrue: [^ self]. "not interested"
	(self handlesSystemLaunch: anEvent) ifFalse: [^ self].
	anEvent wasHandled: true.
	self systemLaunch: anEvent.! !

!PasteUpMorph methodsFor: 'event handling' stamp: 'ct 10/15/2020 19:32'!
handlesSystemLaunch: anEvent

	^ self isWorldMorph! !

!PasteUpMorph methodsFor: 'event handling' stamp: 'ct 10/15/2020 19:57'!
systemLaunch: anEvent
	"Handle a number of files the singleton VM was called with again. See SystemLaunchEvent comment."

	self flag: #todo. "Do something more useful with the image here, e. g. tell the VM to load it."
	^ self dropFiles: anEvent arguments event: anEvent! !

!HandMorph methodsFor: 'private events' stamp: 'ct 1/18/586455 10:01'!
generateDropFilesEvent: evtBuf 
	"Generate the appropriate mouse event for the given raw event buffer."

	| position buttons modifiers stamp numFiles dragType |
	stamp := evtBuf second.
	stamp = 0 ifTrue: [stamp := Time eventMillisecondClock].
	dragType := evtBuf third.
	position := evtBuf fourth @ evtBuf fifth.
	buttons := 1. "red" "hacked because necessary for correct mouseMoveDragging handling"
	modifiers := evtBuf sixth.
	buttons := buttons bitOr: (modifiers bitShift: 3).
	numFiles := evtBuf seventh.
	
	dragType caseOf: {
		...
		[4] -> [ "dragDrop"
			| oldButtons |
			externalDropMorph ifNil: [
				"dragDrop has been sent without prior dragging. This happens if the VM is configured as a singleton application and has been called again (aka #launchDrop)."
				^ SystemLaunchEvent new
					setArguments: (self collectDropFilesAndDirectories: numFiles)
					source: self
					stamp: stamp].
			...

Hi Jakob, thanks for your reply! :-)

So you are proposing to build a separate dispatching mechanism for system launch events, if necessary, which sounds very reasonable with regard to possible performance issues. In this case, there would be no need for an extra event class and I could couple #generateDropFilesEvent: just a little bit stronger to the world.

> If the world morph were to dispatch the event, what about non-Morphic applications?