Add more features using our RAPP store!

Delivery and Navigation

ladia, an elderly person, is requesting robot assistance by pressing a button on her tablet.

TIAGo, the robot assistant, receives the request and moves to the location of the tablet. Gladia issues the voice command to be brought a cup of tea.

TIAGo confirms the task it is about to execute and moves to the kitchen. In the kitchen, the robot issues the voice command to Adam, the caregiver. Adam prepares a cup of tea and places it on TIAGo’s tray, then issues the voice command that the order can be delivered. TIAGo moves to Gladia’s location and asks for confirmation. Gladia issues the voice command that TIAGo rotates. TIAGo executes the command. Gladia takes the cup of tea and places it on the coffee table in front of her, then confirms. After the confirmation, TIAGo moves to the initial position.

Tiago operates as an Agent in a smart environment organized as a multi-agent system with multiple smart devices that provide continuous information, connected through wireless network. The PAL robotic software is directly available to the Core Agent and is made available to the ecosystem through task requests.

The control system is based on the RAPP architecture extended by a Task Harmoniser Agent and a Task Requester Agent. The main nodes are:

  • The Core Agent – drives the hardware of the robot and provides the services to be called by other Agents, such as navigation, control of torso and head, text-to-speech and the translation of voice commands in intents.
  • The Task Harmoniser Agent – receives task requests from the Task Requester Agents and schedules Dynamic Agents basing on a specified algorithm (e.g. priorities).
  • Task Requester Agents – request the Task Harmoniser Agent to perform tasks based on the data from robot’s or additional sensors.

The multi-agent system is implemented using SMACH, a ROS-compatible agent framework.

 

Simulator Real

Fall Reaction

In this scenario, Tiago is in rest mode in the living room. In the kitchen, Michael falls. The INCARE fall and indoor localization module in its bracelet identifies a hazardous event and sends it to the server. From the server, the signal is received by Tiago, which immediately starts moving towards the signal location. When Tiago arrives at Michael, it detects him in an unusual pose and establishes audio and video connection with a remote caregiver, streaming its camera and microphone input. Tiago checks on Michael, asking him if he is feeling well.

 

Simulator Real

Pick and Place

This scenario exemplifies Tiago’s object manipulation capabilities by grasping objects on a table and lifting them a few centimeters above. Execution of dexterity tasks, combined with scene understanding, extends the usability of the robot in a new range of scenarios, such as cleaning a room by picking out-of-place objects and placing them in a basket or bringing objects to the user at the request.

The core of the mechanism is based on machine learning, the grasping poses being identified using a Generative Adversarial Network (GAN) trained on the Cornell Grasp Dataset in a task of distinction between correct and incorrect grasps for a multitude of household objects in different poses. The input of the GAN are 4-channel images containing colour and depth information, preprocessed to a fixed size of (256, 256, 4), randomly rotated or scaled for data augmentation and pseudo-random alteration of pixel intensities in the colour channel to make the learning process invariant of lighting conditions. The best grasp candidates are identified as centers in the 3D space of the robot, opening of the gripper and rotation angle of the wrist. The learning process is conducted using mini-batches over multiple epochs until convergence, employs CUDA tensors for computational speed and penalizes a channel-wise L1 loss between target and predictions. The optimal motion to reach the target pose is computed using the KDL Inverse Kinematics solver.

Towel Pick Part 1 Pick Part 2

Scene Understanding

This scenario proves Tiago’s ability of understanding the surrounding environment and answering simple questions about it. Tiago navigates through the house and answers several questions about the rooms through which it passes. Tiago can:

  • recognize basic objects from the house;
  • count instances;
  • recognize colors;
  • answer yes/no questions.

As Gladia is in another room, she can send Tiago to check things for her. For example, she sends Tiago to the bathroom to check how many towels are in there and whether the bathtub contains water. Then, she sends Tiago to the bedroom, to check whether her husband, Michael, is sitting on the bed. Finally, she wants to know the color of the bed sheets, to make sure that she hasn’t forgotten to change them.

Tiago’s reasoning mechanism is based on 3 components:

  1. a model for visual question answering that combines:
    1. a VGG-16 architecture with inputs of size 224×224 pre-trained on ImageNet (for understanding images);
    2. time series vectors with each word transformed into a 300 dimensional representation calculated using Global Vectors (for understanding questions);
  2. a color detector that extracts the dominant color from a given region;
  3. an external knowledge base (Visual Genome) with logical correlations between nouns and verbs.

Tiago still has difficulties in answering questions that involve details. For example, even if it recognizes a clock on the wall, it is not yet capable of reading it, to tell the exact hour. It is also hard for Tiago to count partially occluded objects—given a couch with several cushions, it does not count them all.

Cognitive Games

Match Game

Initially, a set of cards with images are shown upside down. Cards contain pairs of images. The game consists in the getting all the pairs of two cards. If both cards have the same image, the score will be increased, otherwise the cards will roll again. The game is won when the all pairs of images are found. There are 3 types of levels depending on the number of cards. It can be 8, 12 or 16 cards. The player can choose to leave the game or if he/she has not returned any cards he can turn over all face cards for a short period of time.

This game is included for free within any purchased INCARE system.

 

Collect & Create

The game “Collect&Create” contains two mini games that are based on collecting different objects that depending on the game will be used to make a pie or to make a puzzle. The player has an avatar that is controlled by touch screen gesture. The avatar is placed in a 3D environment.

This game is included for free within any purchased INCARE system.

 

Collect Objects

The player is placed on a road on which he / she must collect one type of object that is shown at the beginning of the game. The object is randomly generated between a collection of predefined objects.

This game is included for free within any purchased INCARE system.

 

Get the Differences

The game consists in 2 scenes: left and right. The left scene is different from the right one. It may contain:

  • different objects from the right scene.
  • Or same objects (as on the right) but some of them are rotated compared with the objects from the left image.

When the player click on the left side on the object that isn’t on the right side / or has a different orientation, the color will be changed in gray (it wouldn’t be clicked in future).

 

Organize Numbers

The game consists in 15 squares with numbers and an empty square arranged in the form of 4 x4 grid (4 rows and 4 columns). The squares contain numbers from 1 to 15. Initially, all these numbers are shuffled. The player must arrange the numbers in an ascending order.

 

Object Maze

The game consists in moving a red square through a maze.

 

Multiplayer Games

Game for Team (G4T)

Game for Team (G4T) – Team planting version is part of Active Silver Life (ASL) Activation Framework. The goal of this framework is to increase activity level of elderly people, in the social, physical, and cognitive domain. Motivation of elderly people is supported by Game for Team gamification system. Simultaneous cooperative and competitive gameplay energizes social life and strengthens connections. Players receive their game points on daily basis, based on the completion rate of their activity plan in ASL calendar.

 

Coming soon