Towards context-aware gesture enabled user interfaces

Abstract

Conventional graphical user interface techniques appear to be ill-suited for the kinds of interactive platforms that are required for future generations of computing devices. 3D graphics and immersive virtual reality applications require interactive 3D object manipulation and navigation. Perceptual user interfaces using speech and gestures are in high demand to provide a more natural human-computer interaction modality. The major challenge facing Perceptual user interfaces is the lack of a standard application programming interfaces capable of handling ambiguity and providing the means to include domain-specific knowledge about the context in which the user interface is used. In this dissertation, we study dynamic hand gestures, which are defined as a sequence of hand postures. We emphasize the generality of our dynamic gesture model, which is capable of recognizing essentially any dynamic hand gesture confined in a sequence of postures. Hand postures are static poses and are defined by an array of posture attributes. We use a generic definition hand postures capable of covering the space of hand postures at different levels of granularity and abstraction; and we timely monitor the posture variation as it unfolds within the dynamic gesture. We also study the role of context in gesture interpretation without making assumptions about a specific application. We view the hand-tracking and gesture-recognition subsystems as integral parts of a larger distributed and multi-user multi-service application, where gesture interpretation plays the role of resolving ambiguity of the recognized gesture. We identify the relevant aspects to hand gesture interpretation and we propose agent-based system architecture for gesture interpretation. We finally propose a framework for gesture-enabled system design, where context is placed in a middleware layer that interfaces with all sub modules in the system and plays a dialectic role and keeping the overall system stable.