Advanced gestures

For more complicated gestures, the Gestures C library provides gesture recognizers for touch events that occur when you place one or more fingers on a touch screen. In the Gestures library there are two classes of gestures:

Composite
A composite (also called transform or continuous) gesture is one that may send multiple notifications to the application as you continue to touch or move your fingers on the screen. The composite gestures are:
  • Swipe: One continuous horizontal or vertical motion consisting of a tap, a movement, and a release with one finger
  • Pinch: One continuous motion consisting of a two-finger tap, a movement inward or outward, and a release
  • Rotate: One long press with one finger, followed shortly by a swipe in an arc by your second finger.
  • Two-finger pan: One continuous motion consisting of a long press, a movement, and a release with two fingers
Discrete
Discrete gestures send only a single notification to the application. The discrete gestures are:
  • Tap: One touch with one finger
  • Double-tap: Two touches in quick succession with one finger
  • Triple-tap: Three touches in quick succession with one finger
  • Long press: One touch with a pause before releasing
  • Press and tap: One long press with one finger, followed by a touch with a second finger
  • Two-finger tap: One touch with two fingers

Gesture recognizers

A gesture recognizer is a self-contained state machine. It progresses through its various states in reaction to the touch events that it receives. Depending on the gesture, a gesture recognizer may need to interpret single- or multiple-touch events in order to detect a single gesture. That is, a gesture recognizer could transition through multiple states before being able to determine the gesture that the user intended. After the gesture recognizer detects a gesture, its gesture callback is invoked. It is the responsibility of your application to handle the detected gesture in your gesture callback function. In the context of the Gestures library, you can see that the gesture recognizers are referred to as gestures.

Gesture recognizers for some of the widely used gestures are already provided for you by the Gestures library:

Gesture Recognizer Gesture class Uses timer(s)
Tap Discrete
Double-tap Discrete
Triple-tap Discrete
Long press Discrete
Press and tap Discrete
Two-finger tap Discrete
Swipe Composite
Pinch Composite
Rotate Composite
Two-finger pan Composite

Some gesture recognizers use timers as part of their detection of gestures. If you need to detect a gesture that the Gestures library doesn't support, you can define your own gesture recognizer. For more information about gesture recognizers, see Gesture recognition.

Touch events

Touch events are events generated by the touch controller. In the Gestures library, an mtouch_event_t represents a touch event. mtouch_event_t contains various types of information about the touch event, such as the type of touch event (touch, move, or release), the time stamp of the event, the coordinates of the touch, and so on. These events are also referred to as mtouch events. See the file input/event_types.h for more information.

Gesture sets

A gesture set is a set of gesture recognizers; it can detect multiple types of gestures. Your application defines a gesture set by allocating, to the set, the gesture recognizers that detect the gestures that are of interest to you.

You can think of a gesture set as the interface between gesture recognizers and the application. The application sends mtouch events to a gesture set, not to individual gesture recognizers. Individual gesture recognizers must belong to a gesture set to be able to receive mtouch events and invoke callback functions when the gesture-recognizer state transitions happen.

The gesture set manages all relationships between gesture recognizers and dependencies. Therefore, individual gesture recognizers are simple to implement and allow the application to customize the desired gesture-recognizer relationships for its own needs. Also, apps that are interested only in a small subset of gestures can choose its gesture recognizers of interest.

A gesture set handles all that is required of the state transition. Examples of what the gesture set handles are:

  • invokes the gesture-recognizer callback function when valid
  • resets each gesture recognizer when events in the gesture set have transitioned to either GESTURE_STATE_COMPLETE or GESTURE_STATE_FAILED
  • manages failure dependencies between gesture recognizers
  • timer handling for gesture recognizers that need timer events (as opposed to mtouch events)
  • failure notification when all gestures recognizers in a set have transitioned to GESTURE_STATE_FAILED

Gesture recognition

Applications are responsible for selecting the gestures they're interested in and adding the corresponding gesture recognizers to one or more gesture sets to achieve the desired recognition behavior. When a gesture-recognizer state transition occurs, the gesture set invokes the application callback function for that gesture. Applications can also add their own gesture recognizers when they require specialized or custom gesture recognition.

State transitions

It's important to understand the states and valid state transitions of the gesture recognizers. Understanding is especially important when you're defining your own custom gesture recognizer because you need to provide the function to handle these state transitions:

gesture_state_e (*process_event)(struct contact_id_map* map,
                                 struct gesture_base* gesture,
                                 mtouch_event_t* event,
                                 int* consumed)

Composite gestures and discrete gestures go through different state transitions.

Composite gestures

Diagram showing the gesture-recognizer states for composite gestures.

Discrete gestures

Diagram showing the gesture-recognizer states for discrete gestures.

GESTURE_STATE_RECOGNIZED and GESTURE_STATE_UPDATING aren't valid states for recognizers of discrete gestures; these gesture recognizers transitions directly from GESTURE_STATE_UNRECOGNIZED to either GESTURE_STATE_FAILED or GESTURE_STATE_COMPLETE.

State transitions are specific to the gesture that the recognizer is detecting. Most state transitions are a result of mtouch or timer events that are of interest to the gesture recognizer.

None (GESTURE_STATE_NONE)
The initial state of a gesture recognizer; you can initialize the state to GESTURE_STATE_NONE by calling gesture_base_init().
Unrecognized (GESTURE_STATE_UNRECOGNIZED)
The state of a gesture recognizer after it has been added to a gesture set; it is now ready to receive mtouch and timer events. The gesture recognizer returns to this state after the gesture set calls reset().
Recognized (GESTURE_STATE_RECOGNIZED)
The state of a gesture recognizer after it has received one mtouch or timer event that moves the gesture recognizer to GESTURE_STATE_COMPLETE.
Updating (GESTURE_STATE_UPDATING)
The state of a gesture recognizer while it receives mtouch or timer events that move the gesture recognizer to GESTURE_STATE_COMPLETE.
Complete (GESTURE_STATE_COMPLETE)
The state of a gesture recognizer when it has received all mtouch or timer events that fulfill the requirements of detecting its gesture.
Failed (GESTURE_STATE_FAILED)
The state of a gesture recognizer when requirements of detecting its gesture aren't fulfilled.

Simultaneous recognition

Unless there is a failure dependency indicated, each gesture recognizer in a gesture set receives all mtouch events sent to the gestures set, meaning that your app can simultaneously recognize several gestures.

A single mtouch event can result in the invocation of multiple gesture-recognizer callback functions because several gestures can be recognized simultaneously.

Callback invocation

Your application must provide the gesture-recognizer callback function. This function defines what the app does when a gesture is recognized or updated:

void(*gesture_callback_f)(struct gesture_base* gesture,
                          mtouch_event_t* event,
                          void* param, int async);

The gesture parameter contains information about the gesture, event contains information about the mtouch event that started the gesture, and async identifies whether this callback is invoked from an mtouch event (async = 0) or from a timer event (async = 1).

This callback function is invoked as a result of either an mtouch or a timer event. Although your gesture callback is invoked on other state transitions, your application is interested only when the gesture has been recognized or not. That is, you would usually implement application behavior based on your gesture recognizer in the GESTURE_STATE_COMPLETE or the GESTURE_STATE_FAILED state.

The goal of this callback function is to identify the gesture that is recognized based on mtouch events that are received and to define your application's actions based on the gesture that is recognized. Typically, your application copies information from the incoming mtouch event to a local structure and uses that information accordingly.

Timer support

A gesture recognizer can change states on timer events. Gesture callback functions are invoked after the invocation of the timer callback from the context of the timer thread. The timer thread doesn't detect mtouch events.

Gesture callbacks of gesture recognizers that have timers (for example, double tap, triple tap, or long press), can be invoked from both the application thread or the timer thread. The processing performed during your gesture callback blocks either mtouch or timer events, depending on which thread invoked the callback.

The following diagrams show that the gesture set is shared between two separate threads. Here's an example of how an mtouch event invokes gesture_callback_f() and blocks a timer event from accessing the gesture set.

Diagram showing an example of mtouch event-invoked gesture_callback_f() blocking a timer event.

And here's an example of how a timer event invokes gesture_callback_f() and blocks an mtouch event from accessing the gesture set.

Diagram showing an example of timer event-invoked gesture_callback_f() blocking an mtouch event.

The Gestures library is thread-safe and assures that any data shared between multiple threads are not accessed simultaneously. However, you need to consider the possibility of threads blocking, or being blocked. For example, if your gesture callback function performs rendering operations, you are likely blocking important mtouch or timer events from being processed. As a result, the behavior of your application may become unpredictable.

To help with the synchronization between mtouch-invoked and timer-invoked gesture callback functions, gesture recognizers and gesture sets provide a parameter, async, in the gesture_callback_f() and gestures_set_fail_f() functions. This async parameter is set to 1 when the gesture callback is called from the timer thread.

Gesture recognizers that don't use timers aren't guaranteed to never have their gesture callback function invoked asynchronously. Your callback functions are synchronous to the thread where gestures_set_process_event() is called only when none of the gestures in your gesture set use timers. If you are using gesture recognizers from the Gestures Library, note that the double tap, triple tap, and long press gesture recognizers use timers.

Lists

Timers behave differently, depending on what the app passes:

  • event list (for example, gestures_set_process_event_list()) or,
  • single event (for example, gestures_set_process_event())

It doesn't make sense to use the wall clock for timer events when a gesture recognizer set is passed a list of events that occurred in the past. Gesture recognizers receive events at a much higher rate than when the events come in at real time. For this reason, when processing event lists, the gesture set causes timers to expire and invoke their callback functions using the event time stamps as the timebase.

If unexpired timers are left after the event-list processing finishes, they will be converted to real-time timers based on the differences between the current wall clock time and the time stamp of the last event in the list.

Timer handling for gestures recognizers with failure dependencies behave in the same way.

Failure dependencies

You can define a gesture recognizer to have failure dependencies on other gestures.

A failure dependency is when the detection of one gesture recognizer depends on the failure of another. That is, if one gesture recognizer moves to GESTURE_STATE_FAILED and it has failure dependents, then the gesture recognizers that depend on this failed gesture recognizer are processed.

For example, you have an application that has one gesture set. This gesture set includes both tap and double-tap gesture recognizers. Your app needs only one of the tap or double-tap gesture to be recognized at a time. Your application sets a failure dependency to indicate that in this gesture set, the tap gesture recognition depends on the failure of the double-tap gesture recognition. To set this failure dependency in your application, you must use the gesture_add_mustfail() function. In this particular example, you include the following code in the initialization of your gesture recognizers:

gesture_tap_t* tap;
gesture_double_tap_t double_tap;
gesture_add_mustfail(&tap->base, &double_tap->base);

The code sample above indicates that when a double-tap gesture fails, the gesture recognizer tries to recognize the mtouch event as a tap gesture.

Failure notification and event lists

Apps can register gesture sets for failure notifications. The gesture set delivers these notifications using a failure callback function that is separate from the gesture-recognizer callback function:
void(*gestures_set_fail_f)(struct gestures_set* set, 
                           struct event_list* list, 
                           int async);

This failure callback function is invoked only if all gesture recognizers in the gesture set have transitioned to GESTURE_STATE_FAILED. If at least one gesture recognizer is in the GESTURE_STATE_COMPLETE state, the failure-notification callback function isn't invoked.

The event_list parameter contains the list of events that were delivered to the gesture set that caused all gestures to fail. This list of events is passed to the failure callback function of the gesture set. These events can either be processed individually or delivered to another gesture set for further processing.

Event lists are used to keep copies of events should failure dependencies need to be fulfilled or failure notifications need to be delivered. Event lists can contain up to 1024 events. If more events come in after the list is full, the oldest nonkey (mtouch or release) events are dropped from the list.

Reset

When a gesture recognizer in a gesture set transitions its state to either GESTURE_STATE_COMPLETE or GESTURE_STATE_FAILED, it's reset only when all other gesture recognizers in the same gesture set have also transitioned to either of these states.

For example, if an application defines a gesture set with tap and double-tap gestures, the application would receive two callbacks:
  • single tap after first release
  • double tap after second release
Assuming that no failure dependencies have been configured, the application would not receive two callbacks for two single taps because the completed single tap will already be in the state, GESTURE_STATE_COMPLETE, after the first tap. When the double tap either completes or fails, the gesture set calls
void (*reset)(struct gesture_base* gesture);

on each of its gesture recognizers.

Custom gestures

You can define gesture recognizers to detect gestures that the Gestures library doesn't already support. Gesture recognizers that are used to detect custom gestures can be compiled with the application code and added to a gesture set in the same way that system gesture recognizers are added to a gesture set.

Custom gesture-recognizer data types and functions

If you're defining your own custom gesture recognizers, you must provide the following as part of the definition and implementation of your own gesture recognizer:

Definition of your custom gesture

typedef struct {
   gesture_base_t base; /* The gesture base data structure. */
   // ...               /* The information specific to your custom gesture */
   int timer_id; /* The ID of the timer for this custom gesture (if needed) */
} gesture_custom_t;

This structure represents information for your custom gesture recognizer; this structure must include gesture_base_t followed by additional members to capture your specific information. For example, if your custom gesture recognizer uses a timer, you need to include the ID of the timer as a specific parameter.

Definition and implementation of the alloc() function

gesture_custom_t* custom_gesture_alloc(gesture_custom_params_t* params,
                                       gesture_callback_f callback,
                                       struct gestures_set* set);
where types and parameters are as follows:
gesture_custom_t
The structure of your custom gesture recognizer.
custom_gesture_alloc
The name of your gesture's alloc() function.
gesture_custom_params_t
A structure that represents the parameters of your custom gesture recognizer.
callback
The application gesture callback.
set
A gesture set that your custom gesture recognizer is added to.
Your alloc() function must:
  1. Allocate the memory necessary for your custom gesture recognizer.
  2. Invoke gesture_base_init() to initialize the data structure for the gesture base.
  3. Invoke gesture_set_add() to add your custom gesture to the gesture set.
  4. Set the gesture recognizer type as GESTURE_USER.
  5. Set the process_event(), the reset(), and the free() functions.
  6. Set the gesture callback: callback.
  7. Perform any custom gesture-specific initialization that isn't part of the reset.
  8. Store the custom gesture-specific parameters with the gesture recognizer, and set default parameters. If your custom gesture recognizer uses a timer, use gesture_timer_create() to obtain the ID for your timer.

Definition and implementation of the process_event() function

gesture_state_e (*process_event)(struct contact_id_map* map,
                                 struct gesture_base* gesture,
                                 mtouch_event_t* event, int* consumed)

Your process_event() function must handle state transitions and return the new, or unchanged, gesture recognizer state. If your custom gesture is time-based, you must adjust the timers accordingly. The Gestures library provides API functions for you to set and reset your timers.

Definition and implementation of the free() function

void (*free)(struct gesture_base* gesture)

Your free() function must release all the memory that your alloc() function allocates.

Definition and implementation of the reset() function

void (*reset)(struct gesture_base* gesture);

Your reset() function must reset the gesture-specific data structures to their initial states.

Definition of parameters specific to your custom gesture (optional)

This structure represents the parameters specific to your custom gesture recognizer.

Definition of states specific to your custom gesture (optional)

These constants represent the states specific to your custom gesture recognizer; these states are in addition to the set of states defined in gesture_state_e.

Definition and implementation of the gesture_timer_callback() function (optional)

gesture_state_e(*gesture_timer_callback_t)(struct gesture_base* gesture, 
                                           void* param);

Gesture sets provide time-based notifications to gesture recognizers that use timers. A notification is implemented as a callback function to the gesture recognizer. If your custom gesture recognizer is timer-based, you need to implement this timer callback function.

A gesture recognizer can transition states on timer events. Similar to the process_event() function for mtouch events, the gesture recognizer's time-based state transitions come from its timer callback function. This function returns the new, or unchanged, state based on the timer event received.

Contact ID map

A contact ID is an identifier that is used to identify mtouch events. The mtouch event data structure contains the contact_id element that is assigned a value from a zero-based index and corresponds to the individual fingers that touch the screen. The contact ID doesn't change until that finger is released.

User-defined gestures typically need to associate a specific mtouch event with a contact ID to associate the streams of events with the finger that caused them. The custom gesture can'd use the contact ID from the mtouch directly. Instead, user-defined gestures must invoke map_contact_id() to obtain a zero-based contact ID that is remapped from the gesture set's perspective.

The remapping is necessary because a contact ID of 1 for an mtouch event could actually correspond to a gesture set's contact ID 0. This mapping could be the case if there are multiple gesture sets in play, or if the user's finger is resting on the touch-sensitive bevel.

Helper functions

Helper functions are available if you are defining your own gestures. These functions are:

void save_coords(mtouch_event_t *event,gesture_coords_t *coords)
This function saves the coordinates of a mtouch event in the specified gesture_coords_t data structure. This function is useful if your gesture is sensitive to the placement of the touch event. For example, in a double-tap gesture, the coordinates of the first tap are saved and compared to the coordinates of the second tap. If these coordinates are within an acceptable range, the gesture recognizer can consider the gesture to be a double tap.
int32_t diff_time_ms(gesture_coords_t *coords1, gesture_coords_t *coords2)
This function returns the elapsed time between the two specified gesture events, and is useful if your gesture depends on the receipt of multiple mtouch events. For example, in a double-tap gesture, the time elapsed between the first and second tap cannot exceed an acceptable time. If too much time has elapsed between the two taps, the double-tap gesture is considered to have failed.
uint32_t max_displacement_abs(gesture_coords_t *coords1, gesture_coords_t *coords2)
This function returns the maximum displacement, in pixels, between two gesture events. For example, if the absolute value of the difference between the x coordinates of the two gestures is greater than the absolute value of the difference between the y coordinates, the function returns the former. For example, in a double-tap gesture, this function can be used to help determine whether or not the two taps received are close enough together on the screen to be considered a double-tap gesture.
int map_contact_id(struct contact_id_map *map, unsigned contact_id)
This function remaps contact identifiers from mtouch events to contact identifiers to be used by gestures. This function is typically one of the first calls in your custom gesture recognizer's process_event() function. You need to first map the contact_id from the mtouch event received to a contact_id that your custom gesture recognizer can use.
int gesture_timer_create(struct gesture_base* gesture, gesture_timer_callback_t callback, void* param)
This function creates a timer that invokes your gesture recognizer's timer callback function when it expires. You need to use this function with time-based gesture recognizers.
int gesture_timer_set_now(struct gesture_base* gesture, int timer_id, unsigned ms)
This function sets a timer using the current time as the reference time. You can use this function with time-based gesture recognizers.
int gesture_timer_set_ms(struct gesture_base* gesture, int timer_id, unsigned ms, _Uint64t base_nsec)
This function sets a timer using a specified time stamp as the reference time. You can use this function with time-based gesture recognizers.
int gesture_timer_set_event(struct gesture_base* gesture, int timer_id, unsigned ms, struct mtouch_event* base_event)
This function sets a timer using an mtouch event time stamp as the reference time. You can use this function with time-based gesture recognizers.
void gesture_timer_destroy(struct gesture_base* gesture, int timer_id)
This function resets the specified timer. You can use this function with time-based gesture recognizers.
int gesture_timer_query(struct gesture_base* gesture, int timer_id, int* pending, _Uint64t* expiry)
This function queries the information for the specified timer. You can use this function with time-based gesture recognizers.

Last modified: 2015-07-24



Got questions about leaving a comment? Get answers from our Disqus FAQ.

comments powered by Disqus