Generic input
A challenge when making user interfaces is efficient input handling across different devices, such as:
- Touch devices with narrow screens
- Touch with wide screen
- Virtual reality + controllers
- Gaming controllers (xbox, steam deck etc.)
- Keyboard + mouse
This describes a new approach to efficient cross-platform input handling. Input consist of two parts:
- A command grid with 44 labeled buttons. On-screen keyboard/hotkeys with discoverable bindings.
- or 36 (4x9) or 48
- 2D/3D graphical area supporting one pointer and one button
Command grid
The command grid consist of 44-button labeled buttons. This allows efficient input on different devices:
- On touch devices shown as a grid(virtual keyboard). Responsive layouts supports different device widths (from 8 to 15 buttons per row).
- On keyboard, each key is bound to a button, and an on-screen keyboard with labels is shown for discoverable hotkeys.
- On controllers and VR a 4x11 grid is shown. "Joystick" position selects column, and the 4 direction-buttons selects row.
This allows for muscle memory as well as discoverability of hotkeys/buttons. And the layout is also familiar for text input.
In the illustrations below, the letters represents the key-bindings. _
is space, A
is shift, >
is tab, <
is backspace, and =
is enter. In practise the letters would be replaced by labels for the commands in the grid.
The responsive layouts are:
1 2 3 4 5 , . ;
6 7 8 9 0 < o p
q w e r t y u i
a s d f g h j k
z x c v b n m l
AAA >>> ___ ===
1 2 3 4 5 6 7 8 9
q w e r t y u i o
a s d f g h j k l
A z x c v b n m p
0 < > ___ , . ; =
1 2 3 4 5 6 7 8 9 0 <
q w e r t y u i o p >
a s d f g h j k l ; =
A z x c v b n m , . _
q w e r t y u i o p ; 7 8 9 <
a s d f g h j k l , . 4 5 6 >
A z x c v b n m ___ 0 1 2 3 =
The controller combinations (action-button + joystick position) is displayed below. The on-screen display is just be highlighted cols of the 4x11 layout above and action-button symbols added.
[◁] [△] [▷] [▽]
2 3 4 w e r s d f z x c
1 5 q t a g A v
< > = _
0 6 p y ; h . b
9 8 7 o i u l k j , m n
Pointer
Single-touch/single-button-mouse on a 2d-graphic view.
If the view is 3d, then there is the following ui for navigating the view
wasd_Aq
in the command grid is bound to movement (wasd
is directions, space/shift is up/down,q
is toggle between 2d-pointer and 3d-pan)- Mouse: is either 2d-pointer or 3d panning. Left mouse button is touch-down. Right mouse button swaps pointer and pan when pressed.
- Touch devices: virtual joysticks for moving and panning – beside or above the command grid. Two-finger interactions can also be used for panning.
- Gaming controller: 2nd joystick is either 2d-pointer or 3d-panning. One of the secondary buttons is touch-down. Another one swaps pointer and pan when pressed.
- VR with 3d-controllers: Left controller-direction/trigger is used for movement. Right controller-direction/trigger is used as (laser-)pointer.
- VR with gaming-controller: 2nd joystick is pointer (pointer movement is on virtual sphere around camera). The pointer is fixed on its 3d-position when camera moves. Laser between an origin point relative to camera and the pointed-to-position. Otherwise as gaming controller.
The 2d-graphic view is just a surface in the 3d view, that fits the viewport in default view. The font size and padding is such there is a main column with a width of ~66 characters.
Typography notes
For ~66 readable chars/line:
Mobile-portrait:
- On 320px screen width, 310px column with 12.5px Roboto Condensed Light with 1.5 line-height. 6x8 input grid
- On 360px screen width, 350px column with 14px Roboto Condensed Light with 1.5 line-height. 5x9 input grid
- On 375px screen width, 365px column with 14.5px Roboto Condensed Light with 1.5 line-height. 5x9 input grid
- On 414px screen width, 400px column with 16px Roboto Condensed Light with 1.5 line-height. 4x11 input grid
Mobile-portrait has black background to make non-screen part of phone function as additional text-margin, to get more screen-estate.
Tablet/desktop/mobile-landscape:
- On 640px screen width, 600px column with 20px Open Sans Regular with 1.6 line-height. 4x11 input grid
- On 768px+ screen width, 640px column with 21px Open Sans Regular with 1.6 line-height. 3x15 input grid