During the UbiComp / ISWC 2020 virtual conference the following 49 accepted Posters and Demos originated in five topics will be presented on Monday, September 14, 2020 from 11:30 to 13:00.

Topic Overview

Poster and Demo Details

User Interfaces

This paper presents the designing and evaluation of PizzaBox, a 3D printed, tangible food ordering system that aims to differ from conventional food ordering systems and provide an unique experience when ordering a pizza by incorporating underlying technologies that support ubiquitous computing. The PizzaBox has gone through both low and medium fidelity testing while working collaboratively with participants to co-design and refine a product that is approachable to all age groups while maintaining a simple process for ordering food from start to finish. We utilised this artefact to conduct an user study at an independent pizzeria to uncover potential opportunities. We present two of the main themes identified through the discussions: 1) end user engagement (from entertainment to education), and 2) healthy eating and living. We found that our approach could potentially utilise towards promoting a healthier lifestyle as well as an educational tool.


Research has shown that turn-by-turn navigation guidance has made users overly reliant on such guidance, impairing their independent wayfinding ability. This paper compares the impacts of two new types of navigation guidance – reference-based and orientation-based – on their users’ ability to independently navigate to the same destinations, both as compared to each other, and as compared to two types of traditional turn-by-turn guidance, i.e., map-based and augmented-reality (AR) based. The results of our within-subjects experiment indicate that, while the use of reference-based guidance led to users taking more time to navigate when first receiving it, it boosted their subsequent ability to independently navigate to the same destination in less time, via more efficient routes, and with less assistance-seeking from their phones than either map-based or AR-based turn-by-turn navigation guidance did.


In Human-Computer Interactions (HCI), to reduce the dependency of bulky devices like physical keyboards and joysticks, many gesture-based HCI schemes are adopted. As a typical HCI technology, text input has aroused much concern and many virtual or wearable keyboards have been proposed. To further remove the keyboard and allow people to type in a device-free way, we propose AirTyping, i.e., a mid-air typing scheme based on Leap Motion. During the typing process, the Leap Motion Controller captures the typing gestures with cameras and provides the coordinates of finger joints. Then, AirTyping detects the possible keystrokes, infers the typed words based on Bayesian method, and outputs the inputted word sequence. The experiment results show that our system can detect the keystrokes and infer the typed text efficiently, i.e., the true positive rate of keystroke detection is 92.2\%, while the accuracy that the top-1 inferred word is the typed word achieves 90.2\%.


There is a relative lack of affective technology to support young children in communicating their emotions, particularly in educational contexts. This paper is concerned with discussing prior design challenges and vision for creating an emotion-communication technology for young children. We describe a system – the Catbus – which conceptualizes a digital and interactive storytelling of children’s emotions fed through their wristbands, as well as online resources to support adoption by educators and students.


A parent’s capacity to understand the mental states of both him/herself and the child is regarded as important in the parent-child relationship. We propose Dyadic Mirror, a wearable smart mirror that is designed to foster this parental capacity in everyday parent-child interactions. Its key feature is to provide a parent with a second-person live-view from the child, i.e., the parent’s own face as seen by the child, during their face-to-face interaction. Dyadic Mirror serves as an intuitive cue that helps the parent be aware of (1) his/her emotional state, and (2) the way he/she would be now being seen by the child, thereby facilitating to infer the child’s mental state.


Conversation agents have shifted the way we communicate with ubiquitous services by enabling the use of natural language communication and the analysis of acoustic and linguistic language patterns. Speech skills of children are not yet fully developed; therefore, most conversational agents frequently misunderstand them. In this research, we examined if conversational agents can uncover instances of language discrimination in children. We developed Bolita, a conversational agent using a Google Home and a Sphero Robot to encourage children to practice how to tell a joke. The results of a two week study of the use of Bolita by 37 Mexican children showed a conversational agent is more likely to misunderstand children with speech skills below average. Our results indicate that they speak less, use fewer words, and need more time to answer when interacting with a conversational agent, which may explain the challenges with the conversational agent understanding them. We close discussing the potential of conversational agents to uncover digital markers in children with language differences and suggest ways that conversational agents could be built to be more inclusive.


Receiving feedback based on the combination of self-confidence and correctness of an answer can help learners to improve learning efficiency. In this study, we propose a self-confidence estimation method using a simple touch up/move/down events that can be measured in a classroom environment. We recorded handwriting behavior during the answering vocabulary questions with a tablet and a stylus pen, estimating self-reported confidence. We successfully built a method that can predict the user’s self-confidence with a maximum of 73% accuracy.


Many current human-robot interactive systems tend to use accurate and fast — but also costly — actuators and tracking systems to establish working prototypes that are safe to use and deploy for user studies. This paper presents an embedded framework to build a desktop space for human-robot interaction, using an open-source robot arm, as well as two RGB cameras connected to a Raspberry Pi-based controller that allow a fast yet low-cost object tracking and manipulation in 3D. We show in our evaluations that this facilitates prototyping a number of systems in which user and robot arm can commonly interact with physical objects.


Audio streaming services are used daily by millions worldwide, enabling on-demand listening and the discovery of songs, artists and podcasts that closely align with the listener’s preferences. Meanwhile, traditional terrestrial radio persists as another ubiquitous and still viable mode of accessing more pre-programmed music and news content, including traffic reports and weather information. While both media services offer listeners a distinct set of value propositions, efforts to combine the ‘best of both worlds’ have been few and far between. Towards this objective, we describe our preliminary efforts to understand audio media consumers’ music streaming and traditional radio listening habits and preferences as part of a project aimed at creating an integrated experience for individual listeners and their close networks of family and friends. Through rapid prototyping, and the speed dating method, we explore the design implications for creating and validating radio-like experiences that are at once personal, customizable and shareable.


Several privacy sensitive apps, e.g., dating apps, and medical counselling apps are equipped with Instant Messaging (IM) features. Messenger features in such apps, let users to chat romantically or as a part of getting personal counselling. However, frequently interacting with the app’s messenger service privately, is difficult in public spaces. We term such public spaces as \emph{Casual Acquaintance-prone Spaces} (CAS). To overcome the limitations associated with interacting with sensitive apps, we propose IMception, a design solution to camouflage sensitive apps’ messenger-feature within another app’s UI. To conceptualize the design of IMception, we conducted a two-week long survey. Our key findings include that participants felt concerned not only with the content (text messages) from their sensitive apps but also with the appearance of the app from its UI. At last, we explore and discuss the design of IMception and highlight its important design considerations.


Mobile User Interfaces

The homework for low-grade pupils often contains simple arithmetic problems, i.e., four arithmetic operations. To evaluate the learning quality of pupils, teachers and parents often need to check the homework manually, which is time and labor consuming. In this paper, we propose a homework auto-checking system HmwkCheck, which checks the four arithmetic operations automatically. Specifically, HmwkCheck utilizes the embedded camera of a smartphone to capture the homework as an image, and then processes the image in the smartphone to detect, segment and recognize both printed characters and handwritten characters. We implement HmwkCheck in an Android smartphone. The experiment results show that HmwkCheck can check homework efficiently, i.e., the average precision, recall and F1-score of character recognition achieve 94.03\%, 93.41\% and 93.72\%, respectively.


The concept of “Industrial Internet” was first proposed by General Electric in 2012. It aims to promote the intellectualization of the whole service system. However, with the development of the Industrial Internet, some criminals launch attacks on industrial control terminals (such as computers and mobile devices), causing the failure of industrial control terminals or wrong instructions, which resulting in factory losses. Therefore, there is an immediate need to extract valuable information from mobile network streaming, accurately detect abnormal behaviors and timely raise the alarm.

In this paper, we propose a method of anomaly detection for mobile devices in Industrial Internet based on knowledge graph and demonstrate the results by using visualization technology. First, we use the optimized data mining algorithm based on frequent item sets to analyse the data, so that our method can accurately detect different kinds of concurrent attacks. Second, this method is able to locate the IP addresses of the attacker and the victim accurately. Third, we design an anomaly alarm module, which can visualize the results in multiple dimensions and assist security administrators to understand complex network situation in real time and take corresponding measures according to the network anomaly.


Prior interruptibility research has focused on identifying interrupt-ible or opportune moments for users to handle notifications. Yet, users may not want to attend to all notifications even at these moments. Research has shown that users’ current practices for selective attendance are through speculating about notification sources. Yet, sometimes the above information is insufficient, mak-ing speculations difficult. This paper describes the first research attempt to examine how well a machine learning model can predict the moments when users would incorrectly speculate the sender of a notification. We built a machine learning model that can achieve an recall: 84.39%, precision: 56.78%, and F1 score of 0.68. We also show that important features for predicting these moments.


With the growth of interactive text or voice-enabled systems, such as intelligent personal assistants and chatbots, it is now possible to easily measure a user’s mood using a conversation-based interaction instead of traditional questionnaires. However, it is still unclear if such mood measurements would be valid, akin to traditional measures, and user-engaging. Using smartphones, we compare in this paper two of the most popular traditional measures of mood: International PANAS-Short Form (I-PANAS-SF) and Affect Grid. For each of these measures, we then investigate the validity of mood measurement with a modified, chatbot-based user interface design. Our preliminary results suggest that some mood measures may not be resilient to modifications and that their alteration could lead to invalid, if not meaningless results. This exploratory paper then presents and discusses four voice-based mood tracker designs and summarizes user perception of and satisfaction with these tools.


Existing mobile-phone notification systems sort and display notifications in notification ‘drawers’ according to predetermined categories, as well as their arrival times and other factors. However, the order in which users attend to notifications often fails to match their display order, because the former is situated. Yet, there is little understanding of the order in which users would prefer their notifications to be sorted to meet their in-situ needs across various situations. This paper therefore presents the preliminary results of a mixed-methods study of the difference between smartphone users’ attendance order and their desired display order of smartphone notifications. Our preliminary results show that a mismatch between attendance order and desired display order existed in nearly half of cases. Specifically, many users desired certain categories of notifications to be placed higher in their notification drawers than their actual notification-attendance behaviors would tend to suggest. Additionally, while our participants felt that some notifications have low-attractiveness senders or content, such as shopping-related ones, they would want the system to give them a higher priority.


Vocabulary acquisition is the basis of learning a language, and using flashcards applications is a popular method for learners to memorize the meaning of unknown words. Unfortunately, this method alone is not effective for learners to remember the meaning of words when they appear in sentences. To solve this, we developed the Mobile Vocabulometer which allows users to acquire new vocabulary with context-based learning. Based on the correlation between comprehension and interests, we use the learning materials that adapt to users’ interests and language skills. This system harnesses the power of the original Vocabulometer, and modifies it to be effective for mobile learning. An experiment on Japanese university students showed that, overall, learners achieved better results compared to using a simple flashcard application. This result indicates that this system provides a significant advantage over context-free learning systems.


It would be hard to overstate the importance of Computer Vision (CV), applications of which can be found from self-driving cars, through facial recognition to augmented reality and the healthcare industry. Recent years have witnessed dramatic progress in visual-object recognition, partially ascribable to the availability of labeled data. Unfortunately, recognition of obscure, unclear, and ambiguous photos that are taken from unusual angles or distances remains a major challenge, as recently shown by the creation of the ObjectNet [1]. This paper complements that work via a game in which obscure, unclear, and ambiguous photos are collaboratively created and labeled by the players, who adopt the role of detectives collecting evidence against in-game criminals. The game rules enforce the creation of images that are challenging to identify for CV and people alike, as a means of ensuring the high quality of players’ input.


Mobile phones have become a new means of accessing and executing crowdsourcing tasks in a variety of situations. Yet, while it is commonly assumed that people are likely to perform these tasks during activity breakpoints, it remains unclear whether different types of such breakpoints affect the likelihood that crowdsourcing tasks will be performed. To explore this question, we classified breakpoints into five types, according to phone users’ preceding, current, and upcoming activities, and conducted a six-week experience sampling method study of 30 users’ breakpoint-type-specific crowdsourcing-task performance behavior. We found that these participants tended to engage in crowdsourcing tasks when they were at breakpoints between two different activities, rather than within an activity, and also when breakpoints were long. Additionally, the higher the complexity of their previous activity, the lower the crowdsourcing-task execution rate. However, high complexity of the post- crowdsourcing task activity had no obvious impact on execution rate.


A memorable city exploration experience requires some unexpected surprises. For pedestrians exploring in city blocks, ordinary route planning and navigation system cannot meet the need of interesting exploration and would even miss the possible surprises on the way. In order to enhance resident user experience in city exploration, we designed a gamified exploratory navigation system. Our system would engage the user when they are close to a point of interest (POI) by proposing interactive activities and “conversing” with them. We conducted preliminary field experiment with 5 participants to evaluate our system and observe how mobile technology and navigation system are practical used in city exploration. We hope our study could provide some reflecting for the further design of these kinds of services and systems which would engage residents in exploring the city and strengthen the connection with the city.


We present our ongoing effort to capture, represent, and interact with the sounds of our loved ones’ laughter in order to offer unique opportunities for us to celebrate the positive affect in our shared lived experiences. We present our informal evaluation of laughter visualizations and argue for applications in ubiquitous computing scenarios including Mobile Augmented Reality (MAR).



This work explores the potential of a set comprised of wearable sensors, a performative lighting installation, and a public museum space, to inspire performative and collaborative social behavior among members of the public. Our installation, The Light, was first exhibited as part of the Late at Tate Britain event in 2019. In this paper we discuss the concept and technological implementation behind the work, and present an initial qualitative study of observations made of the people who interacted with it. The study provides a subjective evaluation based on people’s facial expressions and body language as they improvise and coordinate their movements with one another and with the installation.


Respiratory related events (RE) during nocturnal sleep disturb the natural physiological pattern of sleep. This events may include all types of apnea and hypopnea, respiratory-event-related arousals and snoring. The particular importance of breath analysis is currently associated with the COVID-19 pandemic. This paper proposes a new accessible and convenient approach to RE detection, apnea-hypopnea index (AHI) estimation and screening of respiratory threat severity using photoplethysmography and accelerometer sensors data taken from available on market consumer (Samsung) smartwatches. The proposed algorithm is a deep learning model with long short-term memory cells for RE detection for each 1 minute epoch during nocturnal sleep. Our approach provides the basis for a smartwatch based respiratory-related sleep pattern analysis (accuracy of epoch-by-epoch classification is greater than 80 %), can be applied for a potential risk of respiratory-related diseases screening (mean absolute error of AHI estimation is about 6.5 events/h on the test set, which includes participants with all types of apnea severity; two class screening accuracy (AHI threshold is 15 events/h) is greater than 90 %). Our experimental results demonstrate that proposed model constitutes a noninvasive and inexpensive screening system for the RE detection and analysis.


Commercially available flexible printed circuit boards (FPCBs) have the potential to embed electronics, connectivity, and interactivity into the same surface. This makes them an ideal platform for untethered and interactive wearable devices. However, we lack an understanding how well FPCB-based antennas and sensors perform when worn directly on the body. This work contributes an understanding by studying body-worn FPCBs in three technical evaluations: First, we study the integration of Bluetooth Low Energy and compare the signal strength of our body-worn FPCB with a rigid BLE developer board. Second, we study the accuracy of embedded capacitive touch sensing with two electrode sizes. Finally, we develop a resistive flex sensor based on commercially available FPCB materials and compare its accuracy with a state-of-the-art flex sensor. Taken together, our results demonstrate a high usability of FPCB-based wearable devices.


The development of technology has enabled the construction of various expert systems that can now assist experts in disease recognition. Nowadays, most of the expert systems are based on decision trees. The parameters inside a decision tree are based on experts’ knowledge and scientific data which vary slightly depending on the expert and source. To deal with this problem, we developed a complete medical system in which electrical impedance tomography and body surface potential mapping are used to measure the patient biosignals. The system focuses on the cardiorespiratory bio-signals. With the use of machine learning, we analyze each measured channel and show which channel should be taken into account in the detection of pathologies. We show that the channels that are important for machine learning are different from those used by experts in 12-channel ECG.


Leg bouncing is assumed to be related to anxiety, engrossment, boredom, excitement, fatigue, impatience, and disinterest. Objective detection of this behaviour would enable researching its relation to different mental and emotional states. However, differentiating this behaviour from other movements is less studied. Also, it is less known which sensor placements are best for such detection. We collected recordings of everyday movements, including leg bouncing, from six leg bouncers using tri-axial accelerometers at three leg positions. Using a Random Forest Classifier and data collected at the ankle, we could obtain a 90\% accuracy in the classification of the recorded everyday movements. Further, we obtained a 94\% accuracy in classifying four types of leg bouncing. Based on the subjects’ opinion on leg bouncing patterns and experience with wearables, we discuss future research opportunities in this domain.


Wearable health devices have the potential to incentivize individuals in health-promoting behaviors and to assist in the monitoring of health conditions. Wearable epilepsy seizure monitoring devices are now evolving that can support individuals and their caregivers via the automated sensing, reporting and logging of epileptic seizures. This work contributes a novel reflection on the interface requirements of wearer users and non-wearer stakeholder users. We evaluate the “guessability” of the light pattern interface of the Empatica Embrace wrist-worn epileptic seizure monitor and provide box plot results for eight interface indications. We also report summarised feedback from a heuristic analysis with fourteen participant evaluators. The results indicate some satisfaction with the minimal aesthetic of a simple light pattern interface as well as some concerns about confusion between different indications, accessibility and reliance on recall.


Doze is an on-skin, hydrogel-based sleep mask which seeks to improve, enhance, and augment sleep through the use of programmed scent diffusion in tune with the user’s cortical rhythms. Taking advantage of hydrogels’ unique properties, the Doze mask encapsulates and emits therapeutic scents at a regulated pace. The release of scent is controlled by an embedded heater within the layers of the mask and communicates remotely to a smart device. This communication allows for a personalized dosage release based on the user’s biometric or contextual data. Investigating both the pervasive power of smell in enhancing sleep as well as natural topical remedies, this personalized mask explores the potential for unintrusive solutions to the ever-growing rarity of a good night’s sleep.


Individuals with mobility impairments often discuss the challenges associated with donning and doffing shirts (i.e. putting them on and taking them off). Limited previous work has tackled this issue, but the comfort and aesthetic integrity of the shirt is often forgotten. In this paper, we co-designed an adaptive shirt with individuals with mobility impairments and personal support workers. With the insights from these discussions, we developed an augmented top that transforms wide sizes (for the easy donning and doffing) into their preferred fit. The study resulted in the design of SMAller Aid, which uses Shape Memory Alloy (SMA) springs to retract to a smaller size. The shirt adapts to their needs while retaining its aesthetic integrity to empower them with independence and no required assistance.


This work demonstrates a connected smart helmet platform, HeadgearX, aimed at improving personnel safety and real-time monitoring of construction sites. The smart helmet hardware design is driven by flexible and expandable sensing and actuating capabilities to adapt to various workplace requirements and functionalities. In our demonstrator, the system consists of ten different sensors, visual and haptic feedback mechanism, and Bluetooth connectivity. A companion Android application is also developed to add further functionalities including those configurable over-the-air. The construction project supervisors can monitor all on-site personnel’s real-time statuses from a central web server which communicates to individual HeadgearX helmets via the companion app. Several use case scenarios are demonstrated as examples, while further specific functionalities can be added into HeadgearX by either software reconfigurations with the existing system or hardware modifications.


Daytime sleepiness, the difficulty to maintain an alert waking state during the day, is a serious problem causing vehicle accidents and adverse effects on well-being, health, and productivity. Our research aims at predicting daytime sleepiness using wearable sensing in everyday life to raise awareness and help people to manage their energy better. This study presents a first exploration of comparing body temperature (wrist, forehead, in-ear) with users alertness, measured over a reaction test: Psychomotor vigilance task (PVT) in 7 participants over 2 days in real-life conditions (168 hours in total). The results indicate a weak correlation between some body temperature measures and the PVT scores for certain subjects. This underlines that unobtrusive on-body temperature sensing can be an interesting modality to understand and explore daytime sleepiness.



Technology abuse refers to the excessive use of personal technology devices, which can have a negative impact on adolescent patients’ lifestyles and might lead to negative physical and mental health outcomes. This study conducted a needs assessment study to gain guidelines for the development of assistive systems to help adolescents deal with technology abuse issues. Our results identify current difficulties to depict screen use on multiple devices for the recording of device usage data as well as behavioral data related to lifestyles (e.g., sleep conditions). We also proposed a preliminary design of technology solutions to make the information sharing among patients and parents possible for constructive communication between them and provide treatment teams with the data necessary for diagnosis and the formulation of treatment plans.


For avoiding excessive congestion of tourists that causes overtourism, we propose a Generic Point of Interest (POI), which is an alternative sightseeing spot potentially attractive enough for tourists to replace a well-known sightseeing spot. We also propose a method to discover generic POIs and evaluate it. While the rapid spread of social networking services (SNSs) and social media makes tourism more familiar to people, it is further aggravating overtoursim around the world due to the nature of SNSs and social media, where users simultaneously find the same posts or articles recommending specific tourist spots and are attracted to the same destinations at the same time. As overtourism has severe influences on both visitors and local residents, it is essential to solve this problem. Although there are many studies providing ways of recommending less crowded tourist spots or mining less-known spots in a famous sightseeing area, we cannot apply those methods as a fundamental solution for overtourism for two reasons: 1) in many cases, the number of tourists already exceeds the touring area’s total capacity; and 2) many approaches relying on a number of user-generated data points cannot discover unbusy sightseeing spots since users hardly post reviews nor images. To address these challenges, we propose a novel concept of generic POIs, alternative sightseeing spots to famous spots, and we propose a method to discover generic POIs, whose images are similar to those of existing famous sightseeing spots. We also evaluate our method with collected examples of generic POIs. We hope that the proposed method will help alleviate the overtourism problem in the real world as a first step.


This paper presents the visualization of lifelog based on the amount of social and physical activities for well-being. The motivation is that enables users to aware their social, physical, and moderate activities for behavioral change aiming a comfortable how to spend life for individuals. In this paper, three experiments were conducted to examine the feasibility of measuring and visualizing daily activities. We classified the one student’s various daily activities to see the tendency of activity levels and classes. Also, we examined individual differences of three people in the same spatiotemporal space. Finally, we examined how the one student’s activity changes of half-day can be visualized.


Adhering to the prescribed medication schedule is one of the crucial steps that lead to successful recovery or treatment for chronic diseases. However, more than 50% of prescription medicine is not taken as instructed. Existing interventions that focus on reminders often lack detailed insights into people’s daily intake routines. Ubiquitous sensor systems in combination with qualitative data can facilitate detailed insights into medication routines. We draw on the Data-Enabled Design framework to gain a better understanding of behaviors around medicine intake. This study implements the contextual step by collecting data with a sensor module as an attachment to an existing pillbox. The resulting data is then discussed with the participant to reveal novel insights into medication non-adherence. We show the first promising results from a one-week user-test with one participant and discuss the next steps in the Data-Enabled Design process.


We designed and developed DOOM (Adversarial-DRL based Op-code level Obfuscator to generate Metamorphic malware), a novel system that uses adversarial deep reinforcement learning to obfuscate malware at the op-code level for the enhancement of IDS. The ultimate goal of DOOM is not to give a potent weapon in the hands of cyber-attackers, but to create defensive-mechanisms against advanced zero-day attacks.
Experimental results indicate that the obfuscated malware created by DOOM could effectively mimic multiple-simultaneous zero-day attacks. To the best of our knowledge, DOOM is the first system that could generate obfuscated malware detailed to individual op-code level. DOOM is also the first-ever system to use efficient continuous action control based deep reinforcement learning in the area of malware generation and defense. Experimental results indicate that over 67% of the metamorphic malware generated by DOOM could easily evade detection from even the most potent IDS. This achievement gains significance, as with this, even IDS augment with advanced routing sub-system can be easily evaded by the malware generated by DOOM.


Recalling medical instructions provided during a doctor’s visit can be difficult due to access barriers, primarily for older adults who visit doctors multiple times per year and rely on their memory to act on doctor’s recommendations. There are several interventions that aid patients in recalling information after doctors’ visits; however, some have been proven ineffective, and those that are effective can present additional challenges for older adults. In this paper, we explore the challenges that older adults with chronic illnesses face when collecting and recalling medical instructions from multiple doctors’ visits and discuss implications for AI-assisted tools to enable older adults better access medical instructions. We interviewed 12 older adults to understand their strategies for gathering and recalling information, the challenges they face, and their opinions about automatic transcription of their conversations with doctors to help them recall information after a visit. We found that participants face accessibility challenges such as hearing information and recalling medical instructions that require additional time or follow-up with the doctor. Therefore, patients saw potential value for a tool that automatically transcribes and helps with recall of medical instructions, but desired additional features to summarize, categorize, and highlight critical information from the conversations with their doctors.


We present DeviceAR, a set of techniques to facilitate AR-based interactions with connected devices. These techniques allow HoloLens to discover and establish communication with devices, control them, and seamlessly exchange digital information using holographic interfaces.


Long-term biosignal monitoring should adopt continual learning models considering past and current temporal traits. Continual monitoring requires both high quality data that capture each user’s temporal traits and high performance models. Thanks to the latest sensing and device technology, user-local continual data acquisition has become easier.
However, the data of many users distributed remotely need to be gathered to label data, train models, and analyze them in a real system. The biggest problems are the communication volume and the cloud capacity with uploading and storing raw biosignal data of all users. We, therefore, propose a distributed active sampling method for continual learning. Our method adaptively enables both a model to be efficiently trained and temporal traits to be extracted for each user at each time adaptively. We also verify our method in a use case of arrhythmia observation with ECG. The experimental results indicate its effectiveness in terms of efficiency for model training, graspability for temporal traits, and adaptability.


In this paper, we propose a dance pose evaluation method for a dance learning application using a smartphone. In the past, methods for classifying and comparing dance gestures through 3-D joint information obtained through a 3-D camera have been proposed, but there is a problem in using them for accurate dance pose evaluation. That is, these methods simply compare the similarity between the dance gestures without evaluation of the exact dance pose. To solve this problem, we propose a new method that can be operated on a smartphone for exact dance pose evaluation that simultaneously performs an affine transformation and an evaluation method to compare the joint position and joint angle information. In addition, we prove that the proposed method is suitable for dance learning applications through comparative experiments on a smartphone with real-world datasets.


The use of text-to-speech (TTS) technology to generate radio content is largely unexplored, despite the importance of radio, in particular in remote parts of the world where TTS offers a robust means of transforming existing data into media for low-literate audiences and those without regular internet access. Is synthetic speech able to meet the expectations of radio listeners and add value to community radio stations in remote areas? We present a preliminary analysis of the design and use of TTS applications in the context of two emerging community radio stations in rural Romania. We find that while the applications developed so far are generally perceived as useful for the running of the station, future work should focus on identifying additional use cases that add value beyond that of ‘filling time’ or simply replacing the need for a human voice.



In distance learning contexts, drowsiness is a major factor which disturbs learning. However, it is not easy for instructors to monitor students’ wakefulness.
In order to improve learning efficacy, accurate estimation of wakefulness is needed.
In this study, we propose a multimodal wakefulness estimation method based on face and body movement information.
We utilize web-cameras to obtain facial and head (face-head) movements and pressure mats for body movements, the latter of which can record the distribution of upper body pressure while watching video lectures.
To confirm the effectiveness of multimodal data for wakefulness estimation, we conducted an experiment to collect data from students as they engaged in e-learning and their level of wakefulness was annotated in one-second windows. We extracted 45 features from face-head movements, and 80 features from seat pressure data.
Two types of fusion methods, early and decision level fusion were applied, and the late fusion approach achieved an average F1-macro score of 0.70 in three levels of wakefulness estimation, which is higher than the unimodal approach.
This result indicates that fusion of facial images and seat pressure features can be effective for learner wakefulness estimation.


Smartphones, with their ubiquity and plethora of embedded sensors enable on-the-go measurement. In this poster, we describe one novel measurement potential – weight measurement – by turning an everyday smartphone into a weighing scale. We describe VibroScale, our vibration-based approach to measuring weights of objects, that are small in size. Being able to objectively measure the weight of objects in free-living settings, without the burden of carrying a weighing scale has several possible use cases, particularly in weighing of small food items. We designed a smartphone app and regression algorithm that estimates the relative induced intensity of an object placed on the smartphone. We tested our proposed method on more than 50 fruits and other everyday objects of different sizes and weights. The results demonstrate that our smartphone-based method can measure the weight of fruits without relying on an actual weighing scale. Overall, we observed that VibroScale can measure one type of object with a mean absolute error of 12.4 grams and a mean absolute percentage error of 7.7%. We believe that in future this approach can be generalized to estimate calories and measure weight of various types of objects.


Blood pressure (BP), as a crucial vital sign of human beings, reflects the physical state of the cardiovascular system. Currently, blood pressure is mainly measured by collecting the changes in pressure in the vessel using cuff-sensors. It is a manual operation and cannot achieve continuous BP monitoring. In this work, we developed OfficeBP, a novel non-intrusive BP monitoring system for a typical office environment. OfficeBP relies on measuring the pulse transit time (PTT) between the pulse propagate from arterial proximal to the distal site on once heartbeat. For calculating the PTT, the user’s face and thumb fingertip are regarded as the start and end points respectively. A twin-channel PPG sensing system is presented, that is, the fingertip pulse recording photoplethysmography (PPG) is obtained by a low-cost photoelectric sensor integrated with a mouse. Using image processing the face pulse is acquired by remote-PPG (rPPG) that based on a commercial off-the-shelf camera collecting facial video frames. OfficeBP was evaluated on 11 participants in different working conditions including the external illumination factor and personal internal factors, and achieved RMSE result of diastolic blood pressure 4.81 mmHg, systolic blood pressure 5.35 mmHg, demonstrate the feasibility of the system in an office environment.


91\% of the world’s population lives in areas where air pollution exceeds safety limits\footnote{\url{https://www.who.int/health-topics/air-pollution}}. Research has focused on monitoring ambient air pollution, but individual exposure to air pollution is not equal to ambient and is thus important to measure. Our work (in progress) measures individual exposures of different categories of people on an academic campus. We highlight some anecdotal findings and surprising insights from monitoring, such as \textbf{a)} Indoor CO$_2$ concentration of 1.8 times higher than the permissible limit. Over 10 times the WHO limit of PM$_{2.5}$ exposure during \textbf{b)} construction-related activities, and \textbf{c)} cooking (despite the use of exhaust). We also found that during transit, the PM$_{2.5}$ exposure is at least two times higher than indoor. Our current work though in progress, already shows important findings affecting different people associated with an academic campus. In the future, we plan to do a more exhaustive study and reduce the form factor and energy needs for our sensors to scale the study.


The article presents a cyber-physical system for acquiring, processing and reconstructing images from measurement data. The technology is based on process tomography, intelligent sensors, machine learning, Big Data, Cloud Computing, as well as Internet of Things as a solution for industry 4.0. Industrial tomography allows observation of physical and chemical phenomena without the need for internal penetration, in a non-destructive way and allows monitoring of manufacturing processes in real time. The application contains a dedicated algorithm based on discrete cosine transformation to solve the inverse problem and a specialized intelligent system for tomographic measurements.


In recent years, we have seen efforts made to simultaneously monitor the respiration of multiple persons based on the channel state information (CSI) retrieved from commodity WiFi devices. However, existing approaches only work when multiple persons exhibit dramatically different respiration rates and the performance degrades significantly when the targeted subjects have similar rates. What’s more, they can only obtain the average respiration rate over a period of time and fail to capture the detailed rate change over time. These two constraints greatly limit the application of the proposed approaches in real life. Different from the existing approaches that apply spectral analysis to the CSI amplitude (or phase difference) to obtain respiration rate information, we leverage the multiple antennas provided by the commodity WiFi hardware and model the multi-person respiration sensing as a blind source separation (BSS) problem. Then, we solve it using independent component analysis (ICA) to obtain the reparation information of each person. In this demo, we will demonstrate MultiSense — a multi-person respiration monitoring system using COTS WiFi devices.


Given the pandemic and the high air pollution in large parts of the world, masks have become ubiquitous. In this poster, we present our vision and work-in-progress (WIP) towards leveraging the ubiquity of masks for health sensing and persuasion. We envision masks to monitor health-related parameters such as i) temperature; ii) lung activity, among others. We also envision that retrofitting masks with sensors and display to show localized pollution can create awareness about air pollution. In this WIP, we present a smart mask, \emph{Naqaab}\footnote{Naqaab means mask in Hindi}, that measures forced vital capacity (FVC) of the lung using a retrofitted microphone. We evaluated the measured lung parameter on eight persons using an Incentive Spirometer\footnote{https://en.wikipedia.org/wiki/Incentive\_spirometer} and found that our smart mask accurately measures incentive lung capacity. \emph{Naqaab} also measures pollution exposure and indicates via different LED colours. We envision using such a system for eco feedback.


In this work we present an indoor emergency context monitoring system based on ground vibration caused by persons in the target area. The system is designed for production plants and large buildings to perceive the safety status of this area. Our approach is privacy-protecting, because it requires neither video nor sound. Instead, piezo sensors on the floor measure vibrations, which are analyzed with machine learning to compute the safety status of the covered area. This way our system can determine whether an emergency occurred, but it is not straight forward possible to attach names to the detected persons. We compare the impact of different feature extraction methods and different types of classifiers on the classification results. Our experiments show that we can determine an emergency event with an average F1 score of 0.97.


Recognizing the working appliances is of great importance for smart environment to provide services including energy conservation, user activity recognition, fire hazard prevention, etc. There have been many methods proposed to recognize appliances by analyzing the power voltage, current, electromagnetic emissions, vibration, light, and sound from appliances. Among these methods, measuring the power voltage and current requires installing intrusive sensors to each appliance, or granting the access to the smart meter and applying power decomposition schemes to the total power consumption to infer the working states of appliances. Measuring the electromagnetic emissions and vibration requires sensors to be attached or close (e.g., $<15cm$) to the appliances. Methods relying on light are not universally applicable since only part of appliances generate light. Similarly, methods using sound relying on the sound from motor vibration or mechanical collision so are not applicable for many appliances. As a result, existing methods for appliance fingerprinting are intrusive, have high deployment cost, or only work for part of appliances. In this work, we proposed to use the inaudible high-frequency sound generated by the switching-mode power supply (SMPS) of the appliances as fingerprints to recognize appliances. Since SMPS is widely adopted in home appliances, the proposed method can work for most appliances. Our preliminary experiments on $18$ household appliances (where $10$ are of the same models) showed that the recognition accuracy achieves $97.6\%$.



Virtual Conference:
September 12-17, 2020

Posters & Demos Show:
September 14, 2020
11:30 – 13:00 (EDT)


Past Conferences

The ACM international joint conference on pervasive and ubiquitous computing (ubicomp) is the result of a merger of the two most renowned conferences in the field: pervasive and ubicomp. while it retains the name of the latter in recognition of the visionary work of mark weiser, its long name reflects the dual history of the new event. a complete list of both ubicomp and pervasive past conferences is provided below.

UbiComp 2019, London, England

UbiComp 2018, Singapore

UbiComp 2017, Maui, USA

UbiComp 2016, Heidelberg, Germany

UbiComp 2015, Osaka, Japan

UbiComp 2014, Seattle, USA

UbiComp 2013, Zurich, Switzerland

UbiComp 2012, Pittsburgh (PA), USA

Pervasive 2012, Newcastle, England

UbiComp 2011, Beijing, China

View All