Gesture recognition system application to early childhood education

Human-Computer Interaction (HCI) is a growing field of research which focuses on the relation between humans and technologies. HCI seeks to understand the methods that humans can use to communicate with computers and defines new interaction paradigms according as technology and the devices are progressing. These new interaction models have a wide range of applications ranging from research to industry, entertainment, or education. To enhance playing and learning for children is one of the most common uses of human-computer interaction. In recent years, the games industry, looking for more interactive experiences, has developed a set of new devices and technologies that provide users with a more natural interaction than a simple gamepad, keyboard, or mouse can offer. These devices, such as Kinect, or Wiimote, allow tracking of the body and hands of the players, recognizing their movements and gestures, and allow a more natural interactive experience. These devices also allow the development of new somatosensory applications which increase user immersion and motivation, providing more entertaining games. Recently, all these ideas, new HCI paradigms, games and somatosensory devices, have been combined to develop a set of applications generically called Gesture Interactive Game- Based Learning (GIGL), which aims to improve learning performance through interactive games. GIGL opens up new opportunities to learn complex content using new paradigms, for example, through movement of the body or hands, which provide a basis for new learning models. These techniques are especially interesting in primary and secondary education for several reasons. Firstly, more natural interfaces can be created. Secondly, learners can use their body as a tool; this will reduce physical passivity and increase their motivation. Thirdly, the child can be supervised by teachers while solving specific tasks. Many studies report that GIGL can increase both the ability to learn and motor skills in different ranges of ages. However, the overall education of children should stimulate other kinds of abilities rather than simply learning and motor competences. In this work, Gesture interactive game-based learning (GIGL) was used to test whether these types of applications were suitable for the stimulation of working memory (WM) and basic mathematical skills (BMS) in early childhood (5-6 years) by developing a set of serious games, based on computer, in which the interaction person-computer has been implemented using a system of recognition of gestures with hands. The user (child) gestures with the XX hand and controls the interaction with the game. Motion capture has been developed using the Kinect device, while for the development of the game interfaces, the Unity 3D graphics engine has been chosen. Executive functions (EF) are brain-based cognitive feature skills that facilitate, essentially, thought and self-regulation. Executive functions are based in the prefrontal cortex of our brains and assist with goal-setting and decision-making. There are three main types of executive functions: working memory, inhibitory control, and cognitive flexibility. As the system has been developed to be used for very young children, 5-6 years old, the type of interaction must to be very simple. We have used only a simple set of gestures: select, drag, drop. These gestures are, in general, standard to every application and the devices should recognize them without problems. The purpose of the activity was to engage the participants in drag and drop actions with the Kinect. The drag and drop actions were implemented as follows: the user could select the desired object by moving the Kinect pointer, which is a hand on screen, over it, and closing their palm after. Then she could move it to the correct place by moving their hand while keeping the palm closed and drop it by opening their palm. The user could use either hand. The system has a backoffice application that allows configuration. It is based on a MySQL database managed with PHP MyAdmin, which is a free open source tool written in PHP that allows you to manage MySQL through a web browser. A web page was created using PHP and HTML to manage the configuration of the applications We have identified two main actors: the students themselves, and the teacher who can configure specific properties in the system that concern the development of the games, in addition to registering their students. Through this system, the teacher can create a profile for each child containing her name, weight, height, and a photo, to be used in the GIGL. The system allows generating reports on the interaction of the different students so that the teacher can analyze their evolution. The research, carried out in collaboration with the GIPDAE group of the University of A Coruña, was developed from a quasi-experimental design with pre-test and post-test, using both an experimental and a control group through three phases: the first was the previous evaluation of the student’s skills; a second phase in which the use of technology was developed; and a final phase of evaluation. In the evaluation phases, the working memory was measured by using the Corsi Test, and the basic mathematical skills by using the test for the diagnosis of basic mathematical competences (TEDI-MATH). This part of the evaluation of the different XXI proposed games, as well as a large part of their design, has been carried out by the GIPDAE group. The main contribution of this study is proving that at a very early stage of childhood, it can be observed that the executive functions, with a focus on working memory, can be improved and positively impact the mathematical skills developed through GIGL. This hypothesis was proven experimentally by a standard test, which shows an increase in children’s cognitive abilities through computer game play. The set of applications developed with the GIGL paradigm overcome the limitations of traditional interfaces used in early childhood education, such as the keyboard, mouse or gamepad. The integration of a device such as Kinect in a graphics engine such as Unity 3D and a lightweight database manager such as MySQL has allowed us to develop a set of interactive educational games, which have been validated by educational experts and have allowed us to work with pre-school children regarding executive functions.

keywords: Basic Mathematical Skills, Early Childhood Education, Gesture Interactive Game-Based Learning, Human-Computer Interaction, Working Memory