[long] Squeak and High-DPI

Hi all,
[this is a longish post, you might want to use this toc]

[TOC]
- [INTRO]
- [TERMINOLOGY]
  - [DPI]
  - [DEFAULT DPI]
- [STATUS]
  - [WINDOWS DPI]
  - [OSX CARBON DPI]
  - [OSX COCOA DPI]
- [SUGGESTIONS]      <<<< PLEASE COMMENT
  - [OSX COCOA PATCH]
  - [VM/IMAGE INTERFACE]

[INTRO]

For some time now, OSX and also Windows support screens of different pixel per inch (DPI, :P).

The approaches are slightly different; but both support a fallback mode where the pixel content of non-aware applications is automatically scaled to look like on "default dpi"[See DEFAULT DPI]. Effectively, we end up with pixel doubling.

One way or another, this means Squeak looks somewhat pixelated on new hardware.

These are some thoughts of mine on how to address this.


[TERMINOLOGY]
  [DPI] Actually "dots-per-inch" but freely used in multiple senses by different people. For displays it used to mean "how many pixels are there approximately in an inch?", but for a long time now, The [DEFAULT DPI] that OSes report usually was fixed/defaulted (Win: 96, OS X: 72), _regardless_ of the actual screen's dpi.

  [DEFAULT DPI] On OS X, the system reports default DPI as 72 dpi and Retina typically as 144 dpi, _although_ the "actual" dpi is different. For example, My MacBook Pro 15" is reported as  144dpi, but https://www.sven.de/dpi/ calculates that I have 220.53 pixel to the inch. That's why Apple is moving away from DPI and just talking about a "scale factor"[1],[2] and the abstract "point" principle, moving away from actual pixels for "user content"[3]. On Windows, the default dpi is 96 dpi, but Windows always made it a little easier to change that.

  Squeak also has a default dpi, namely 96.0 dpi.


[STATUS]
  [WINDOWS DPI]

  Windows supports different levels of DPI awareness[4], available depending on OS Version[5].

  Long story short, with calling a special function[6] or (As we did/do/can do) setting a key in the executables ".manifest" file to "true" or "true/PM", Windows treats our Application as DPI-Aware[7].

  Currently the effect of enabling this for our VMs means:
   - Looks ok on non-High-DPI screens
   - Looks tiny on High-DPI screens
   - Moving the VM between High-DPI and non-High-DPI screens changes the outer frame size of the application, the pixel size reported _to_ the VM remains the same.

  [OSX CARBON DPI]

  DPI-awareness and High-DPI are not possible with the QuickDraw-API the Carbon VM used to use.

  [OSX COCOA DPI]

  OS X now has a Device-in-pixel mode and a Userspace-in-"point" mode. When opted-in via Info.plist file, on AppKit-Level, one can convert between those with +[NSScreen backingScaleFactor], -convertRectToBacking:, -convertRectFromBacking and similar. On Quartz2D/CoreGraphics-Level, CGContextGetUserSpaceToDeviceSpaceTransform(), CGContextConvertRectToDeviceSpace(), and CGContextConvertRectToUserSpace() can be used. Both approaches are slightly different. OpenGL needs an additional Opt-in.

  In contrast (currently) to Windows, when moving between high- and non-high-DPI, the outer frame size of the application stays the same but the pixels of the backing store change.

  Up until now, the Cocoa branch is not DPI-aware.


[SUGGESTIONS]

  [OSX COCOA PATCH]
  Attached is a patch that makes the VM High-DPI ready. PLEASE COMMENT.
  Summary:
   - Add Info.plist entry to state whether we want High DPI (NSHighResolutionCapable)
     THIS IS DISABLED BY DEFAULT (as with the Windows VM's manifest)
   - All Views:
     -add -sqScreenSize for the Pixel-dimensions to report to Squeak (and use for calculation in Squeak coordinates)
     -add -sqMousePosition: to convert Mouse coordinates from the View's system (maybe high-dpi) to Squeak coordinates
   - OpenGL view:
     - Opt-in for High resolution (-setWantsBestResolutionOpenGLSurface:) (overridden by Info.plist's entry)
     - Make texture size/tracking rects based on -sqScreenSize
   - Quartz2D/CoreGraphics view:
     - Work around deprecation of [[NSGraphicsContext currentContext] graphicsPort] in OS X 10.10
     - Draw image in -sqScreenSize size, and position correctly in the View's coordinate system
     - (remove some duplication from OpenGL view)
   - iPhone SqueakUIView:
     - use Fallbacks for -sqScreenSize/-sqMousePosition:, no High-DPI yet.
  Note: this patch is against the Cog/platform/ios directory, Interpreter needs similar treating.

  [VM/IMAGE INTERFACE]
  With the current Windows VM and the Cocoa VM (+patch as above), users can Opt-in for High-DPI.
 
  However, the Image does not know that its high pixel count is on a small display (hello Etoys!). I _want_ the image to know and to be able to react.

  I think that the image should treat DPI changes similar screen size changes. Here are my ideas:
   - The VM exposes screen size as prim 106, screen depth as 'primitiveScreenDepth'
     * We should expose 'primitivePixelPerInch' with a fallback of '^ 96.0'.
     * 96.0 dpi shall be the [DEFAULT DPI] of Squeak.
       > For OS X that would mean, on non-high-dpi it shall report 96.0 dpi (not 72 dpi), on high-dpi it shall report 192.0 dpi (not 144)
       > Windows already has 96.0 dpi as [DEFAULT DPI], it shall reuse that and adhere to [8]
   - The image regularly checks the display size (DisplayScreen class>>checkForNewScreenSize).
     * We should do the same for DPI (Morphic main loop etc.)
   - There's already a client of DPI changes, TextStyles.
     * The image shall store changed dpi settings to TextStyles>>pixelsPerInch:
   - Projects can react to display size changes via #displaySizeChanged, But only fonts can react to DPI changes via #pixelsPerInchChanged
     * We shall make project able to react to dpi changes, maybe via #pixelsPerInchChanged
   - The standard tools are currently pixel-oriented and use hard-coded pixel values
     * I am unsure.
     * We could introduce conventions to act dpi-dependent (integers: pixels with autoscaling (pixels / default dpi * actual dpi or so), floats or scaled: dpi-based, via TextStyle class>>pointsToPixels:).
     * We could introduce actual units and say "this tool is 3 cm wide" and calculate pixels via #pointsToPixels:.
     * We could eliminate hard-coded values and base tools on font sizes. I tried that with the FontImporterTool (for example: FontImporterTool>>#buttonHeight, #filenameHeight. This may be hard and not very traceable, tho.
    - We could move the dpi api from TextStyle to somewhere more general.


I know there are some rough edges here and there; I'd like to hear your suggestions.

Let's make Squeak look nice and sharp :)

Best
        -Tobias

PS: Thanks to Marcel for some valuable suggestions and help to get Quartz transformations right.



[1]: https://developer.apple.com/library/mac/documentation/Cocoa/Reference/ApplicationKit/Classes/NSScreen_Class/index.html#//apple_ref/occ/instm/NSScreen/backingScaleFactor
[2]: https://developer.apple.com/library/ios/documentation/UIKit/Reference/UIView_Class/index.html#//apple_ref/occ/instp/UIView/contentScaleFactor
[3]: https://developer.apple.com/library/mac/documentation/GraphicsAnimation/Conceptual/HighResolutionOSX/Explained/Explained.html#//apple_ref/doc/uid/TP40012302-CH4-SW2
[4]: https://msdn.microsoft.com/en-us/library/windows/desktop/dn280512(v=vs.85).aspx
[5]: https://msdn.microsoft.com/en-us/library/dn469266(v=vs.85).aspx#high_dpi_features_in_windows
[6]: https://msdn.microsoft.com/en-us/library/dn302122(v=vs.85).aspx
[7]: http://www.squeakvm.org/svn/squeak/branches/Cog/build.win32x86/squeak.cog.spur/Squeak.exe.manifest
[8]: https://msdn.microsoft.com/en-us/library/dn469266(v=vs.85).aspx#dpi_and_the_desktop_scaling_factor

Hi, there!

+1 for updating the OSX VM to support hi-dpi.

+1 for adding "pixelsPerInchChanged" to Squeak consistently with "displaySizeChanged".

---

Considering the DPI-awareness of Morphic, MVC, and the base system (e.g. text/paragraph rendering), this may need further discussion. However, this is can be deferred until the other two above work.

---

We could also provide in-image preference to modify dpi-awareness by editing plist/.manifest and restarting the VM. Is there support for restarting the VM from within Squeak?

Best,
Marcel

Hi Marcel,

On 03.05.2016, at 10:04, marcel.taeumel <[hidden email]> wrote:

> Hi, there!
>
> +1 for updating the OSX VM to support hi-dpi.
>
> +1 for adding "pixelsPerInchChanged" to Squeak consistently with
> "displaySizeChanged".

It's present on TextStyle class. But we might want to move it.

Btw: windows can change this on-the-fly w/o changing the manifest.
and changing the plist on osx does not immediately change this, there are caches that must be invalidated.

Best regards
        -Tobias :)


>
> Best,
> Marcel
>

What happens when the window is on two displays at the same time?

Stephan

Hi Stephan,

when moved from the source screen, it will then look bigger or smaller on the target screen until fully transferred to the target screen, which then applies its DPI. At least in Windows 10.

Best,
Marcel