A challenge when making user interfaces is efficient input handling across different devices, such as:
- Touch devices with narrow screens
- Touch with wide screen
- Virtual reality + controllers
- Gaming controllers (xbox, steam deck etc.)
- Keyboard + mouse
This describes a new approach to efficient cross-platform input handling. Input consist of two parts:
- A command grid with 44 labeled buttons. On-screen keyboard/hotkeys with discoverable bindings.
- 2D/3D graphical area supporting one pointer and one button
The command grid consist of 44-button labeled buttons. This allows efficient input on different devices:
- On touch devices shown as a grid(virtual keyboard). Responsive layouts supports different device widths (from 8 to 15 buttons per row).
- On keyboard, each key is bound to a button, and an on-screen keyboard with labels is shown for discoverable hotkeys.
- On controllers and VR a 4x11 grid is shown. "Joystick" position selects column, and the 4 direction-buttons selects row.
This allows for muscle memory as well as discoverability of hotkeys/buttons. And the layout is also familiar for text input.
In the illustrations below, the letters represents the key-bindings.
_ is space,
A is shift,
> is tab,
< is backspace, and
= is enter. In practise the letters would be replaced by labels for the commands in the grid.
The responsive layouts are:
1 2 3 4 5 6 7 8 9 0 , . ; < o p q w e r t y u i a s d f g h j k z x c v b n m l AAA >>> ___ === 1 2 3 4 5 6 7 8 9 q w e r t y u i o a s d f g h j k l A z x c v b n m p 0 < > ___ , . ; = 1 2 3 4 5 6 7 8 9 0 < q w e r t y u i o p > a s d f g h j k l ; = A z x c v b n m , . _ q w e r t y u i o p ; 7 8 9 < a s d f g h j k l , . 4 5 6 > A z x c v b n m ___ 0 1 2 3 =
The controller combinations (action-button + joystick position) is displayed below. The on-screen display is just be highlighted cols of the 4x11 layout above and action-button symbols added.
[◁] [△] [▷] [▽] 2 3 4 w e r s d f z x c 1 5 q t a g A v < > = _ 0 6 p y ; h . b 9 8 7 o i u l k j , m n
Single-touch/single-button-mouse on a 2d-graphic view.
If the view is 3d, then there is the following ui for navigating the view
wasd_Aqin the command grid is bound to movement (
wasdis directions, space/shift is up/down,
qis toggle between 2d-pointer and 3d-pan)
- Mouse: is either 2d-pointer or 3d panning. Left mouse button is touch-down. Right mouse button swaps pointer and pan when pressed.
- Touch devices: virtual joysticks for moving and panning – beside or above the command grid. Two-finger interactions can also be used for panning.
- Gaming controller: 2nd joystick is either 2d-pointer or 3d-panning. One of the secondary buttons is touch-down. Another one swaps pointer and pan when pressed.
- VR with 3d-controllers: Left controller-direction/trigger is used for movement. Right controller-direction/trigger is used as (laser-)pointer.
- VR with gaming-controller: 2nd joystick is pointer (pointer movement is on virtual sphere around camera). The pointer is fixed on its 3d-position when camera moves. Laser between an origin point relative to camera and the pointed-to-position. Otherwise as gaming controller.
The 2d-graphic view is just a surface in the 3d view, that fits the viewport in default view. The font size and padding is such there is a main column with a width of ~66 characters.
For ~66 readable chars/line:
- On 320px screen width, 310px column with 12.5px Roboto Condensed Light with 1.5 line-height. 6x8 input grid
- On 360px screen width, 350px column with 14px Roboto Condensed Light with 1.5 line-height. 5x9 input grid
- On 375px screen width, 365px column with 14.5px Roboto Condensed Light with 1.5 line-height. 5x9 input grid
- On 414px screen width, 400px column with 16px Roboto Condensed Light with 1.5 line-height. 4x11 input grid
Mobile-portrait has black background to make non-screen part of phone function as additional text-margin, to get more screen-estate.
- On 640px screen width, 600px column with 20px Open Sans Regular with 1.6 line-height. 4x11 input grid
- On 768px+ screen width, 640px column with 21px Open Sans Regular with 1.6 line-height. 3x15 input grid