Mouse Gestures

Ability to give commands to an application using only the mouse. An arbitrary sequence of mouse motions can be assigned to an application activity. For example, you could go "Back" by moving the mouse to the left, close the application by moving it up and down, etc. Unsupported application activities can be added via customized (usually JavaScript) scripts. Common as an add-on in web browsers like FireFox, Opera, and Maxthon, but I see no reason it couldn't be used in any application, or even as an OS extension A Mouse Gestures tab in the Mouse control panel would be nice. Anybody know of such an application? -- JosephStyons

See also: GestureRecognition MozillaFireFox? FingerBoard

I remember back in college, when I was taking VLSI design, that the damnable MentorGraphics CAD tools had numerous such gestures for numerous operations. Man, it was a pain in the butt to use... perhaps as an expert UI MouseGestures might be a good idea - but it's something that you have to train people to use. -- ScottJohnson

I was thinking more of an optional, user-defined set of mouse gestures. Not required, just available as a convenience. Most everybody agrees that if you can work without moving your hands from the keyboard, then you save time. The same should be true of the mouse. If more actions can be completed between the keyboard<>mouse transition, then efficiency should go up. -- JosephStyons

Complex mouse gestures are just awful, but I can't live without the simple gestures I use every day in Firefox: hold down the right mouse button and flick the mouse left for back, right for forward ... and if I need a "boss key", down for minimize ;) I'm reminded of the game BlackAndWhite ... an otherwise enjoyable game nearly ruined by its complete reliance on a gesture interface.

Fundamentally, mouse gestures are an extension of the "hotkey" concept from keyboards to the mouse. The conceptually tricky mapping of directions to command actions is similar to the conceptually tricky mapping of a key to a command action. However, the problem is that where keyboards have a much larger space of commands (50-ish keys and 3-ish buckybits) mice only really have 4 directions and 3 holdable-buttons... and 2 of the hold-features of the buttons are generally taken (drag and scroll for left and middle). So imagine binding keyboard hotkeys when you can only use the four directional keys + ALT. As a result, mouse gestures rely much more on "chording" - sequences of hotkeys... which is about as pleasant as abandoning the fast "ctrl" hotkeys in favour of the alt-key-key-key approach that key-driven menus provide. Combined with extra limitations - you can't "repeat" motions the way you can "repeat" keys by tapping them repeatedly - every sequential line of a gesture must be in a different direction, leaving only 3 directions for subsequent lines. You can expand the space to 8 by using diagonals, but this becomes frustrating and tricky. That being said, mouse-gestures work really well if you remember them for what they are: a very small set of hotkeys for the mouse. So, use them in cases where you are making a mouse-driven application where you have a very short list of commands the user needs to access with the mouse - preferably commands that can easily be conceptually mapped to directions.

Also, don't neglect that this kind of "hotkey"-like behaviour can be done on any interface device. "Rocker" gestures for FireFox are a wonderful mechanism, for example: hold the right mouse button and then use the mousewheel to cycle through opened tabs - much more natural than ctrl-tab and ctrl-shift-tab.

How about in non-desktop environments? Are there MouseGesture? facilities for PalmPilots?, TabletPCs, etc? Or should we call them PenGestures? in that domain? Maybe they would be more useful in an environment where keyboarding is unintuitive. -- JosephStyons

The AppleNewton incorporated a number of gestures into its core interface:

A scrubbing motion would delete the objects within the bounding rectangle of the scrub
Tap and hold (down up down) would turn on selection. You could then lasso or draw over objects to be selected.
Drawing a caret within text would insert a space.
Drawing a horizontal paren would remove a space. (This is the standard proofreading notation.)
A single line up or down through selected text would capitalize or lowercase the text, respectively.
You move a selected object by dragging it. You copy the object by tap and drag (down up down drag).
Dragging selected objects to the side of the screen would attach an icon there and put the object in the clipboard. You could then switch tasks and those items would remain there. This is the most natural clipboard implementation I have ever encountered.

In many respects, the AppleNewton's interface has improved on most if its successors. -- IanOsgood

You know, it shouldn't be difficult to enable ordinary users to define and bind their own mouse gestures. All you need to do is code them up into a digit sequence. For instance:

<nothing> is 0
north is 1
NE is 2
east is 3
SE is 4

...

E clockwise half-circle is 23
S clockwise half-circle is 24
W clockwise half-circle is 25

...

clockwise full circle is 37

...

anticlockwise ...

and then you just let the user combine them, so that:

0 is a single-click
0-0 is a double-click
51515 is a Fireball

...

Yet somehow, I doubt there's any package for extensible mouse gestures. -- RK

You better get started, then

Not a priority.

There is a Windows program available at http://www.tcbmi.com/strokeit/ that can learn new gestures as you do them, and assign custom actions to them. -- MichaelSparks

Mouse gestures are a one-finger sign language. Everywhere one finds mouse gestures, they're an inferior substitute for voice recognition and spoken commands or video recognition and ASL. Even the simplest and most primitive voice recognition system is more usable and has a lower learning curve than a mouse gesture system. Plus, voice recognition is available off the shelf whereas mouse gestures have to be custom built!

In the case of BlackAndWhite, mouse gestures should have been replaced by Power Words of a spoken spellcasting language. The use of power words can be extended for an ObjectBrowser to trigger WheelMenu commands.

In the case of so-called implicit mouse gestures, they would be replaced by continuous phonemes like:

But such a system would be obviously stupid. Note that the mouse -> voice translation doesn't introduce the stupidity, it just exposes it. Any use of the mouse designed to implement a very small and highly regular language reduces the mouse to the MouseKeys. And any attempt to implement an ubiquitous hardware component using a different (nevermind a marginally less common one!) hardware component is blatantly stupid.

In summary, mouse gestures are a horrific mistake of HCI design and should never ever be used. They're not even a good gimmick since they don't have as much of a Wow factor as the simplest voice recognition. Mouse gestures are the kind of stuff programmers end up producing because they never realize what they're doing.