Generic input

A challenge when making user interfaces is efficient input handling across different devices, such as:

  • Touch devices with narrow screens
  • Touch with wide screen
  • Virtual reality + controllers
  • Gaming controllers (xbox, steam deck etc.)
  • Keyboard + mouse

This describes a new approach to efficient cross-platform input handling. Input consist of two parts:

  • A command grid with 44 labeled buttons. On-screen keyboard/hotkeys with discoverable bindings.
    • or 36 (4x9) or 48
  • 2D/3D graphical area supporting one pointer and one button

Command grid

The command grid consist of 44-button labeled buttons. This allows efficient input on different devices:

  • On touch devices shown as a grid(virtual keyboard). Responsive layouts supports different device widths (from 8 to 15 buttons per row).
  • On keyboard, each key is bound to a button, and an on-screen keyboard with labels is shown for discoverable hotkeys.
  • On controllers and VR a 4x11 grid is shown. "Joystick" position selects column, and the 4 direction-buttons selects row.

This allows for muscle memory as well as discoverability of hotkeys/buttons. And the layout is also familiar for text input.

In the illustrations below, the letters represents the key-bindings. _ is space, A is shift, > is tab, < is backspace, and = is enter. In practise the letters would be replaced by labels for the commands in the grid.

The responsive layouts are:

 1 2 3 4 5 , . ; 
 6 7 8 9 0 < o p
 q w e r t y u i
 a s d f g h j k
 z x c v b n m l
 AAA >>> ___ ===


 1 2 3 4 5 6 7 8 9
 q w e r t y u i o
 a s d f g h j k l
 A z x c v b n m p
 0 < > ___ , . ; =


 1 2 3 4 5 6 7 8 9 0 <
 q w e r t y u i o p >
 a s d f g h j k l ; =
 A z x c v b n m , . _

 q w e r t y u i o p ; 7 8 9 <
 a s d f g h j k l , . 4 5 6 >
 A z x c v b n m ___ 0 1 2 3 =

The controller combinations (action-button + joystick position) is displayed below. The on-screen display is just be highlighted cols of the 4x11 layout above and action-button symbols added.

[◁]           [△]           [▷]           [▽]  

     2 3 4         w e r         s d f         z x c
    1     5       q     t       a     g       A     v
       <             >             =             _   
    0     6       p     y       ;     h       .     b
     9 8 7         o i u         l k j         , m n

Pointer

Single-touch/single-button-mouse on a 2d-graphic view.

If the view is 3d, then there is the following ui for navigating the view

  • wasd_Aq in the command grid is bound to movement (wasd is directions, space/shift is up/down, q is toggle between 2d-pointer and 3d-pan)
  • Mouse: is either 2d-pointer or 3d panning. Left mouse button is touch-down. Right mouse button swaps pointer and pan when pressed.
  • Touch devices: virtual joysticks for moving and panning – beside or above the command grid. Two-finger interactions can also be used for panning.
  • Gaming controller: 2nd joystick is either 2d-pointer or 3d-panning. One of the secondary buttons is touch-down. Another one swaps pointer and pan when pressed.
  • VR with 3d-controllers: Left controller-direction/trigger is used for movement. Right controller-direction/trigger is used as (laser-)pointer.
  • VR with gaming-controller: 2nd joystick is pointer (pointer movement is on virtual sphere around camera). The pointer is fixed on its 3d-position when camera moves. Laser between an origin point relative to camera and the pointed-to-position. Otherwise as gaming controller.

The 2d-graphic view is just a surface in the 3d view, that fits the viewport in default view. The font size and padding is such there is a main column with a width of ~66 characters.

Typography notes

For ~66 readable chars/line:

Mobile-portrait:

  • On 320px screen width, 310px column with 12.5px Roboto Condensed Light with 1.5 line-height. 6x8 input grid
  • On 360px screen width, 350px column with 14px Roboto Condensed Light with 1.5 line-height. 5x9 input grid
  • On 375px screen width, 365px column with 14.5px Roboto Condensed Light with 1.5 line-height. 5x9 input grid
  • On 414px screen width, 400px column with 16px Roboto Condensed Light with 1.5 line-height. 4x11 input grid

Mobile-portrait has black background to make non-screen part of phone function as additional text-margin, to get more screen-estate.

Tablet/desktop/mobile-landscape:

  • On 640px screen width, 600px column with 20px Open Sans Regular with 1.6 line-height. 4x11 input grid
  • On 768px+ screen width, 640px column with 21px Open Sans Regular with 1.6 line-height. 3x15 input grid