MANGO – Mobile augmented reality with functional eating guidance and food awareness

The prevention of cardiovascular diseases becomes more and more important, as malnutrition accompanies today’s fast moving society. While most people know the importance of adequate nutrition, information on advantageous food is often not at hand, such as in daily activities. Decision making on individual dietary management is closely linked to the food shopping decision. Since food shopping often requires fast decision making, due to stressful and crowded situations, the user needs a meaningful assistance, with clear and rapidly available associations from food items to dietary recommendations. This paper presents first results of the Austrian project (MANGO) which develops mobile assistance for instant, situated information access via Augmented Reality (AR) functionality to support the user during everyday grocery shopping. Within a modern diet - the functional eating concept - the user is advised which fruits and vegetables to buy according to his individual profile. This specific oxidative stress profile is created through a short in-app survey. Using a built-in image recognition system, the application automatically classifies video captured food using machine learning and computer vision methodology, such as Random Forests classification and multiple color feature spaces. The user can decide to display additional nutrition information along with alternative proposals. We demonstrate, that the application is able to recognize food classes in real-time, under real world shopping conditions, and associates dietary recommendations using situated AR assistance.