UbiComp/ISWC '23 Adjunct: Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing

Characterizing Information Processing Activities (IPAs) such as reading, listening, speaking, and writing, with physiological signals captured by wearable sensors can broaden the understanding of how people produce and consume information. However, sensors are highly sensitive to external conditions that are not trivial to control – not even in lab user studies. We conducted a pilot study (N = 7) to assess the robustness and sensitivity of physiological signals across four IPAs (READ, LISTEN, SPEAK, and WRITE) using multiple sensors. The collected signals include Electrodermal Activities, Blood Volume Pulse, gaze, and head motion. We observed consistent trends across participants, and ten features with statistically significant differences across the four IPAs. Our results provide preliminary quantitative evidence of differences in physiological responses when users encounter IPAs, revealing the necessity to inspect the signals separately according to the IPAs. The next step of this study moves into a specific context, information retrieval, and the IPAs are considered as the interaction modalities with the search system, for instance, submitting the search query by speaking or typing.

Current commercial Augmented Reality (AR) headsets are expensive and limited in availability, posing challenges for researchers and designers in resource-constrained settings. To address this, we introduce Attina, an open-source, low-cost optical see-through AR headset using a smartphone as its display. Overcoming limitations of previous work, the system features stereoscopic rendering for enhanced depth perception and a 3-DoF controller for virtual interactions. We discuss Attina’s design and outline opportunities for a usability study to evaluate its user experience.

This paper explores the challenges and opportunities in achieving a high level of realism and presence in Virtual Reality (VR) experiences. Realism and presence, which refer to the level of authenticity of the virtual environment and the user’s subjective perception of being in it, respectively, are fundamental components of engaging and satisfactory VR experiences. Despite advancements in VR technology, users can often distinguish between real and virtual experiences, thereby limiting the potential of VR in various fields. This paper reviews various challenges inherent in achieving indistinguishable realism in VR, encompassing visual, auditory, and tactile components. These insights are critical in guiding the continued development of more immersive and realistic VR experiences.

Empathy is the ability to perceive and understand other people's emotions and thoughts. Virtual reality (VR), as a new technology, offers a unique opportunity to induce empathy in individuals by allowing them to immerse themselves in realistic situations from another person's perspective. In this study, we have developed a virtual reality system that incorporates situational learning and immersive interaction. Through first-person perspective simulations, participants can experience the challenges and situations encountered by individuals with positive symptoms of schizophrenia. The aim is to investigate the potential for altering public empathy and attitudes through this simulated experiential system. We hope our study could provide a new approach to health education strategies to enhance public empathy and attitudes toward people with schizophrenia.

With the widespread use of wearable devices, it is becoming easier to predict the future of an individual’s behavior regarding health. However, little is known about how presenting future predictions about an individual’s behavior affects awareness and behavior. The “StepUp Forecast” project is investigating walking behavior to clarify the impact of presenting predictions of the number of steps an individual will take on human behavior. In a previous study, it was found that self-efficacy and the number of steps taken increased significantly when the predicted number of steps was presented, compared with when only a record of the number of steps taken was shown. To advance this research, we investigated the effects of intentionally and secretly increasing or decreasing the predicted value rather than simply presenting it. Specifically, we observed changes in awareness and behavior when users were presented with a predicted value with 1,000 steps added or subtracted. In our 5-week experiment, 40 participants used StepUp Forecast, an application that presents the predicted number of steps on the basis of past life logs. The results indicate a non-significant but increasing trend in self-efficacy and number of steps added or subtracted compared with when only the step-count record was presented.

Within the HCI community, the impact of social dynamics on behavior change has gained considerable interest. This paper introduces the Binary Social Game (BSG), a principle that utilizes asynchronous social interactions to motivate desired behaviors. Initially, we developed GrowFlower based on the BSG principle, and its effectiveness was evaluated through a two-week wizard-of-oz study. After gaining insights from the preliminary study, we refined our approach and created GENGO, a bingo-style game that is specifically designed to encourage exercise as a targeted behavior. A subsequent two-month user study in real-world conditions, involving 82 participants, validated GENGO’s effectiveness in fostering consistent exercise habits. These findings highlight the potential of BSG as a useful tool for promoting desired behaviors within a group context, especially through asynchronous social interactions. This study contributes to the understanding of how social dynamics can be leveraged for behavior change.

Ensuring proper developmental and behavioral assessments for children is crucial, particularly in contexts where resources are scarce, developmental delays carry a stigma, and specialists are lacking. However, adequate and culturally appropriate methods for screening and assessing children in the Majority World still need to be improved. In this context, mobile technology can be crucial in providing accessible assessments and screenings that better fulfill the needs. In this work, we present the design and development of Mazi Umntanakho "Know Your Child," a conversational agent on WhatsApp that supports South African home visitors in assessing and tracking children’s socio-emotional skills. The agent is based on the Strengths and Difficulties Questionnaire (SDQ) and International Development and Early Learning Assessment (IDELA) to provide culturally appropriate assessments. The agent is personalized to user preferences and language and provides feedback and advice in various multimedia formats, including infographics, video, audio, and photos.

With many contemporary video conferencing platforms available, there is still a need for platforms that afford a researcher workflow to conduct controlled online experiments. We have developed an open source experimental video conferencing platform that enables researchers to design and conduct remote experiments. Our platform provides a high level of control over the user interface and video streams, which is essential for studying the differences between remote and in-person social interactions. We give an overview of our platform’s usage and architecture and conduct a take-home study (N=9) to evaluate how accessible our system is to potential new contributors. We also follow up with an initial evaluation of technical performance bottlenecks for when our experimental platform is deployed, and show that the computational resources increases per each video stream as well as the type of filters applied to each participant. We end with a short discussion on next steps and the experimental hub’s potential to be extended as a sandbox for testing browser based augmented reality (WebAR) filters to be adopted in interdisciplinary experimental procedures.

Understanding drivers’ states is essential for providing personalized interventions and adaptive feedback in vehicles, thereby ensuring safer driving and a more comfortable driver experience. As driving tasks necessitate understanding and reaction to the rapidly changing road environment and successful management of working memory to prevent dual-task interference, driver situation awareness (SA) should be primarily considered for such interventions and feedback. However, due to its complex nature, most of the current methods for measuring SA rely on questionnaires. In this work, we aim to develop sensor-based assessments of driver SA, especially towards road signs, which play an important role in immediate decision-making during driving. We collected eye-tracking data and physiological responses from 32 participants during simulated driving and annotated this data according to the levels of SA that drivers achieved. Our ensemble-based machine learning model that uses physiological measures and gaze-related features demonstrated an accuracy of 78.02% in a three-class driver SA prediction. Since SA is a key component in evaluating vehicular interfaces, our VR-based approach holds the potential for the iterative design of in-car infotainment applications and road environments. The method also lays a foundation for SA-adaptive cooperation between human drivers and AI in the context of advanced driver assistance.

Battery degradation, a gradual loss of capacity and performance due to frequent charging and discharging cycles, is a significant challenge to the widespread adoption of electric vehicles (EVs). This study proposes a BMS algorithm that delays full charging under selective conditions and completes charging immediately just before use to reduce battery degradation rate caused by fully charged state time. Our goal is to predict the charging end time based on an individual’s departure time by capturing digital behavioral markers extracted from smartphone data, while minimizing reduction in driving range due to undesired predictions. Preliminary experiment was conducted with 41 subjects to assess the feasibility of the proposed approach. Our results demonstrate that the mobile passive features are capable of learning the departure behavior pattern, achieving an average mean absolute error (MAE) of 0.2336.

This work presents ‘Lightron’, a wearable sensor system designed for boxing training assistance, improving punching accuracy for novices. This system combines accelerometers, stretch sensors, and flex sensors to detect the user’s movements, providing LED light feedback to the user. This adjustable design ensures a tight fit of the device to the body, allowing the sensor to collect accurate arm movement data without impeding training movements. A simple neural network is used to enable real-time motion detection and analysis, which can run on low-cost embedded devices. Contrary to merely using accelerometers on the wrist, Lightron collects motion data from the elbow and shoulder, enhancing the precision of punch accuracy assessment. Primary user studies conducted among boxing amateurs have shown that using Lightron in boxing training increases the performance of amateur players both in single and periodic training sessions, demonstrating its potential utility in the sports training domain.

Underwater activities are enjoyed around the world. Divers wear dive computers that allow visualization of diving information including depth and dive time, to avoid the risk of decompression sickness. However, dive computers are so expensive that they burden novice divers. A method of estimating water depth with barometric pressure sensors in smartphones, instead of dive computers, was studied [3]. Unfortunately, the conventional method [3] for estimating water depth using a barometric pressure sensor in a smartphone is limited to estimating shallow depths of about  1 m to 2 m due to the sensor only being able to detect up to about 1100 hPa to 1200 hPa. We propose a method of estimating water depths of several tens of meters with a barometric pressure sensor in a smartphone by adopting a waterproof hard case with a decompression adjustment function and a polynomial regression model. We implemented a prototype of the proposed method and conducted experiments to evaluate the precision of estimating water depths down to  20 m in the sea. The experiments revealed that the proposed method could cover water from 0 to  20 m depths and estimate water depths with a root mean squared error (RMSE) of less than 1 m.

Stress plays a significant role in fatal accidents, highlighting the importance of timely monitoring of driver stress to facilitate effective interventions and reduce road accidents. However, monitoring driver stress presents numerous challenges in the context of driving. First, state-of-the-art techniques such as self-stress evaluation and periodic cortisol level checks are not suitable for the driving scenario. Second, existing unimodal solutions does not provide a comprehensive and holistic assessment of the driver’s stress. Although some research utilizes multimodal features, the use of wearables attached to the driver’s body in real-life situations is impractical and highly discomforting. Our proposed solution tackles these challenges by offering a contactless and non-intrusive approach that prioritizes the driver’s comfort during the collection of multimodal data, which includes capturing heart rate variability (HRV), respiration rate, and microfacial expressions. Through feature-level data fusion, we combine and integrate these diverse modalities to generate comprehensive insights. These insights are then utilized by the multimodal learning pipeline to predict the driver’s stress levels in real driving scenarios.

Smart textile sensors have attracted increasing interest in the domain of wearable computing for human motion monitoring. Previous studies have shown that textile sensor layout has a major impact on the effectiveness and performance of wearable prototypes. However, it is still a trick and time-consuming issue to determine textile sensor layout in a quantitative approach as it involves figuring out the number, placement, and even orientations of sensors, yet there is no streamlined digital platform or tool specifically addressing this issue. In this paper, we introduce LayTex, a digital tool capable of generating layout proposals for personalized scenarios, which aims at facilitating designers and researchers to construct prototypes efficiently. The preliminary evaluation with designers on smart garments for scoliosis indicates that LayTex has great potential to lower the barriers and simplify the process of textile prototype construction.

This research deals with how to build reliable AI models using shared sensitive data. Confidential computing is gaining attention in AI services for a ubiquitous computing field. It makes legal and social regulations strict, including protecting privacy and ensuring security for sensitive data and AI models. As is also the case in ubiquitous computing research. The research phase also requires a manual trial-and-error process to create value through new combinations of shared data and AI models. Conventional confidential computing cannot realize the human workflow. This paper proposes and verifies “Sandbox AI” that can build precise AI models through efficient manual processes without revealing the shared data or models to one another. Sandbox AI utilizes privacy-preserving synthetic data generation and active learning on a confidential computing architecture. Specifically, Sandbox AI works in a sandbox container and generates synthetic data on the basis of real data, interactively extracts synthetic data that are useful for efficient model training while considering the real data distribution, discloses only the extracted synthetic data for annotation, and re-trains the model on the labeled synthetic data. Experimental results show that the achieved model accuracy is as good as that of conventional learning using real data with privacy violations.

With the widespread adoption of smartphones, users now have easy access to similar services, leading to increased churn. As a result, it has become essential for service providers to prevent churn caused by customers’ switch to competing services. The most common approach for service providers to prevent their customers’ churn is to make churn predictions by monitoring customers’ usage patterns of their own services. However, despite the importance of insights concerning customers’ usage of competing services for the retention of customers, such information are yet to be integrated into churn prediction models due to the lack of suitable monitoring methods. Here, we propose an approach to predict user churn leveraging the event logs from smartphones and tablets. Instead of conventional churn prediction methods that solely rely on the users’ usage patterns of their own service, our approach predicts churn by utilizing users’ usage patterns of competing services, including their trial use of service before switch to competitor’s. We evaluated the prototyped prediction model using smartphone logs collected from NTT DOCOMO smartphone and tablet users who consented to data collection between April 2020 and March 2021. The results demonstrated that the proposed method achieved AUC values ranging from 0.844 to 0.923. Moreover, our approach improved the performance of the conventional method that predicts churn without leveraging the features of the competitor’s app by 1.8% to 7.5%.

This research delves into the intersection of illustration art and artificial intelligence (AI), focusing on how illustrators engage with AI agents that embody their original characters (OCs). We introduce ’ORIBA’, a customizable AI chatbot that enables illustrators to converse with their OCs. This approach allows artists to not only receive responses from their OCs but also to observe their inner monologues and behavior. Despite the existing tension between artists and AI, our study explores innovative collaboration methods that are inspiring to illustrators. By examining the impact of AI on the creative process and the boundaries of authorship, we aim to enhance human-AI interactions in creative fields, with potential applications extending beyond illustration to interactive storytelling and more.

Smartwatches are used by millions of people for applications in health, finance, and communication. As the computational power and range of applications supported by these devices expand, it is becoming more and more important to secure access to them. While various user authentication technologies have been extensively explored in smartphone use scenarios (e.g., FaceID, fingerprint, PIN, or pattern) the applicability of these approaches to smartwatches is typically limited due to the small watch form factor. To improve authentication on smartwatches, we propose SonarAuth, a novel user authentication system for unmodified commercial smartwatches using behavioral biometrics derived from motion, touch, and around-device motions. To capture in-air hand motions, we adapted an existing sonar system for smartwatches. We collected data from 24 participants from single touch to the watch screen with the thumb, index, and middle fingers. Using a multimodal deep learning classifier, we achieved a promising mean Equal Error Rate(EER) of 6.41% for user authentication based on a single thumb tap. We note that our system is usable and has good potential to be combined with other authentication modalities.

Blood pressure (BP) is one of the vital signssignals used to evaluate the health status of the human body, a. And continuous ambulatory blood pressure (ABP) monitoring can accurately and comprehensively reflect the physiological status of the cardiovascular system. ABP is mainly measured using ambulatory sphygmomanometer, by regularly collecting the pressure changes in blood vessels. However, this method is usually intrusive, inconvenient, and expensive. In this paper, we develop SleepABP, a novel natural, continuous and accurate ABP monitoring system for daily sleep scenarios. SleepABP is based on the Ballistocardiogram (BCG) signal, which relies on measuring the tiny fluctuations (impulses) in the body caused by the continuous beating of the heart. Specifically, the proposed BCG sensing system monitors the pulses from the user's head applying the piezoelectric ceramic sensor. The BCG signals are recorded simultaneously, and the BCG cycle restoration algorithm is used to extract the continuous and complete BCG cycles for constructing the multi-dimensional features of the blood pressure prediction. SleepABP was evaluated on 16 participants under various sleeping conditions, positions, and personal internal factors. The RMSE results of diastolic blood pressure 3.22 mmHg and systolic blood pressure 3.5 mmHg demonstrates the system's effectiveness and feasibility in a daily sleeping environment.

Virtual Reality (VR) makes use of psychological stressors to enable users to feel immersed in different domains. Although stressor tests, like the Color-Word Stroop Test (CWST), have been shown to be valid and reliable, there are limited deployments of such stressors that exhibit appropriate immersion and user experience in VR. In this paper, we present the development and evaluation of two VR simulators of the CWST. We conducted a between-subjects study with 27 participants who completed the conventional CWST, a VR-only simulator of the CWST, and a gamified VR simulator of the CWST. We measured and conducted a statistical analysis of participants’ CWST scores, perceived stress, immersion, user experience, playability, and heart rate. Our results show that there are no significant differences in using the two CWST simulators of the VR. We found that users’ heart rate is significantly higher when using the VR simulators than the conventional CWST and that the VR game simulator significantly shows a higher score in immersion than the VR-only simulator. However, users’ perceived stress was significantly less when using the VR game simulator. Finally, we reflect on design insights for stressors tests in VR and discuss directions for future work.

Mobile notifications, crucial to our daily activities and informational needs, are often undermined by insufficient management features. Previous research has suggested enhancements like automatic sorting, filtering, and categorization, but empirical evidence supporting these strategies is yet to be seen. This study bridges the gap, developing an Android application to assess these proposed features’ efficacy in improving notification management efficiency and user experience. We utilized the Experience Sampling Method (ESM) for in-depth user insights, and our preliminary findings indicate a perceived superiority for a hybrid system combining automatic and manual functionalities, over systems solely dependent on either approach. This research paves the way for an optimized notification system, better equipped to assist users in managing mobile notifications effectively.

Ads are ubiquitous on online video platforms, providing significant commercial value but also often interrupting viewers at inopportune moments. This can be particularly disruptive for mobile users, whose diverse contexts of use can mean time for video watching is limited. Prior research indicates that allowing users to negotiate or defer interruptions can reduce disruption. Inspired by this, we developed AdJustMoment, a mobile video player application that imports content from online platforms and offers users the option to defer ads through "Snooze" or "Ads-debt" - user-determined ads-viewing times. Our preliminary qualitative feedback indicates a preference for "Ads-debt", as users felt it provided greater control over when ads appear. Additionally, users also desire to receive positive feedback upon completing an ad, as it serves as a motivational factor.

Engagement is a human experience relevant in multiple contexts, including classrooms, presentations and workplaces. Stemming from flow theory, engagement in these contexts has been studied using wearable devices, which can unobtrusively measure physiological changes, specifically Electrodermal Activity (EDA). However, researchers have not explored how EDA markers might be similar or different between various engagement scenarios, namely student, audience and workplace engagement. In this study, we investigated possible similarities through the use of three datasets containing EDA data and engagement self-report labels, collected in the wild in different settings using research-grade wrist-worn wearable devices. We analysed the correlation between hand-crafted EDA features and the engagement level and we leveraged a machine learning framework for engagement prediction. We found that similar features are correlated with the engagement level across the various settings. We also found that our machine learning model identified related markers as important across the three engagement contexts. Our results highlight that similarities are present in the EDA features between different engagement contexts, while also identifying possible dataset specific differences.

As smartphone notifications proliferate, it becomes increasingly challenging for users to review them all. In response, we’ve developed NotiSummary, an application leveraging ChatGPT to present a concise summary of notifications, thereby reducing the time and effort required to review them. The application also integrates customization capabilities to further enhance the user-centric experience. Our study investigates the potential applicability of AI-based summarization techniques in notifications and explores user interaction patterns with these summaries. Preliminary results show that users find these summaries helpful in quickly accessing specific information, with preferences emerging for receiving summaries early in the morning and late at night. These findings underscore the potential value of notification summarization while highlighting the need to further investigate user preferences for summary appearance and the impact of summary generation on notification management behavior.

We identify features of mood-tracking apps for managing mental health that foster engagement and sustained use by adolescents—a population that expresses a preference for digital apps over face-to-face support, yet demonstrates low levels of engagement with such apps. We developed a prototype of an adolescent-focused mood-tracking app, informed by literature about existing apps’ approaches to recording patients’ symptoms, the role of data representations in long-term mental health management, and the potential benefits of peer support tools. We then conducted a survey (n = 88) to assess adolescents’ preferences for various aspects of this prototype. We found that participants prefer tools for self-reflection and self-awareness over those for gamification or social support, and that they value function over entertainment when choosing wellness apps, especially among participants who disclosed a history of managing mental health. Qualitative analysis of open-ended responses revealed that customization and self-reflection are important design themes. Our findings have implications for the design of mental health apps that cater to the specific needs and preferences of adolescent users.

For the application of telepresence robots, the on-site person usually has been regarded as a main element in remote interaction. However, in the context of a remote tour, the visual presence of an on-site person unexpectedly interrupting a robot’s view may worsen the remote visitor’s viewing experience. Moreover, the presence of a telepresence robot with its camera on may arise the privacy concerns of an on-site person. In this sense, we developed TeleVIP, a platform that provides telepresence applications with easy-to-use APIs to remove the figures of on-site people and selectively distill the useful information from the presence or context of on-site people, according to the application’s demand. We implemented a working prototype of TeleVIP and performed pilot experiments in terms of both quantitative metrics and qualitative results, indicating the preliminary usability and feasibility of TeleVIP.

People with motor and speech impairments has limited communication abilities, resulting in a heavy reliance on Augmentative and Alternative Communication (AAC) tools. Existing commercial AAC tools provide simple combination of fixed words, satisfying basic physiological needs while encounter huge challenges in social communication. This social communication is significant for users’ mental health, especially for those with limited motor abilities. This article thus developed SocializeChat, an AAC tool that employs LLM technology to boost social chat with gaze inputs. Specifically, SocializeChat generates multiple sentences of conversation in real time, offers suggestions tailoring to users' preferences of topics, and phrases sentences in a style in accord with the relationship of people in conversation. This is achieved through a user dataset containing content preferences and social closeness, and through dedicated design of prompts and procedures. In a brief testing, SocializeChat was rated as effectively embodying subjects' own content preferences and communication style.

Detecting whether a target crosses the given zone (e.g., a door) can enable various practical applications in smart homes, including intelligent security and people counting. The traditional infrared-based approach only covers a line and can be easily cracked. In contrast, reusing the ubiquitous WiFi devices deployed in homes has the potential to cover a larger area of interest as WiFi signals are scattered throughout the entire space. By detecting the walking direction (i.e., approaching and moving away) with WiFi signal strength change, existing work can identify the behavior of crossing between WiFi transceiver pair. However, this method mistakenly classifies the turn-back behavior as crossing behavior, resulting in a high false alarm rate. In this paper, we propose WiCross, which can accurately distinguish the turn-back behavior with the phase statistics pattern of WiFi signals and thus robustly identify whether the target crosses the area between the WiFi transceiver pair. We implement WiCross with commercial WiFi devices and extensive experiments demonstrate that WiCross can achieve an accuracy higher than 95% with a false alarm rate of less than 5%.

The shoulder joint plays a crucial role in the recovery of upper limb function. However, conventional wearable technologies employed for monitoring shoulder joint movements predominantly rely on inertial sensing units (IMUs), which may suffer alignment errors and compromise the freedom and wearability experienced by patients during their daily activities. This paper contributes in two facets, first, it presents the design, implementation, and technical evaluation of a new wearable system, a customized unilateral shoulder wrap that utilizes flexible and breathable textile sensors. Diverging from earlier studies, our system not only facilitates the monitoring of glenohumeral joint angles but also concurrently tracks the movement angles of the scapula. Secondly, to estimate joint angles, we propose a specific model called the Channel-Temporal Encoding Network (CTEN), which leverages Transformer and Long Short-Term Memory (LSTM) architectures. In a preliminary technical evaluation, the results demonstrate root mean square errors (RMSEs) of 2.24°and 1.13°for the glenohumeral joint and scapula, respectively. This study is intended to contribute to the development of more advanced wearables tailored for shoulder joint rehabilitation training.

Traditional in-person counseling encounters limitations in terms of accessibility, flexibility, and social stigma. Additionally, low mental health literacy and embarrassment among individuals hinder help-seeking behavior. Meanwhile, the introduction of more sophisticated sensors embedded in ubiquitous devices such as smartphones and smartwatches, and the release of a powerful large language model, i.e., chatGPT, create new opportunities to address the existing limitations of traditional counseling services. In that regard, we propose WMGPT, a system that offers round-the-clock mental health counseling services. By leveraging continuous analysis of user context and digital phenotype, WMGPT delivers personalized counseling support. Through 24/7 passive monitoring, it continuously assesses individuals’ mental state, initiates conversations on their behalf, and potentially triggers counseling services. These specialized counseling services are facilitated by a fine-tuned chatGPT model. WMGPT presents a promising solution to overcome the limitations of traditional counseling by providing accessible, personalized, and timely mental health support, paving the way for a convenient and effective service for improving well-being.

Spectroscopic analysis of physiological phenomena has remained an important yet underutilized application in wearable technology today. Lumos has recently been introduced as an open-source wearable device capable of on-body spectroscopic research across the visible spectrum, enabling scientists and researchers to study the optical properties of various clinical biomarkers in real-time. However, a key limitation in the data output of this device is the lengthy process required to visualize and plot the spectral responses of observed mediums. In this paper, we present SpectraVue, an interactive web application that allows for visualization of Lumos spectral data. Utilizing a user-friendly interface, SpectraVue enables researchers to quickly generate three-dimensional plots from Lumos data stored in csv or text files, providing a comprehensive view of the spectral response of the medium under investigation. Additionally, SpectraVue offers features such as comparison of spectral data with a clinical biomarker, various data export options, and interactive plotting, further enhancing the user experience and researcher efficiency. The output graphs can be used to provide a standardization of spectral responses across a wide range of mediums, including characterization of these responses in clinical biomarkers such as glucose and alcohol. SpectraVue aims to facilitate these investigations by streamlining the data processing and visualization workflow, thereby accelerating clinical diagnostic research.

Although many works bring exercise monitoring to smartphones and smartwatches, inertial sensors used in such systems require the device to be in motion to detect exercises. We demonstrate our full paper ProxiFit, a practical on-device exercise monitoring system capable of classifying and counting exercises despite the device being still. ProxiFit remotely detects adjacent exercises with magnetic field fluctuations induced by the motions of ferrous exercise equipment. Novel proximal sensing nature of ProxiFit (1) extends coverage of wearable exercise monitoring to exercises that do not involve device motion such as lower-body machine exercise, and (2) brings a new off-body exercise monitoring mode with line-of-sight screen visibility, namely signage mode, to a smartphone mounted in front of the user.

This paper proposes a novel demo application named CityScouter that utilizes multimodal data to analyze various aspects of urban characteristics quantitatively. Existing studies have proposed systems to examine either the physical characteristics of cities or the nature of people residing there. However, there is a lack of systems that analyze the characteristics of cities from both the physical and the residents’ aspects. CityScouter addresses this challenge by leveraging computer vision technologies to quantify the quality of the urban landscape atmosphere and combining it with location information and user search history to reveal the desires of people visiting the area. The application is user-friendly and compatible with mobile devices, enabling users to conveniently enhance their understanding of cities while exploring them. Additionally, we provide reviews from urban development experts, offering insights into the applicability of our application. Furthermore, we showcase the usefulness and user experience of CityScouter through live demonstrations at the conference venue.

The paper demonstrates that the introduction of heterogeneous crowd robots improves the accuracy and completeness of indoor 3D mapping. First, three ground robots (sweeping robot, inspection robot, and guidance robot) are used to perform active exploration and mapping tasks in the iGibson indoor environment. However, the mapping results of the ground robot reveal that there are obvious blind spots in key areas (such as tables, stoves, and beds), resulting in the lack of point cloud data. To overcome this challenge, we introduce a drone for active 3D mapping using its bird’s eye view and powerful perception capabilities. Experimental results show that by introducing drones, we have successfully eliminated the blind areas of vision existing in-ground robot mapping and achieved more comprehensive and accurate mapping results. This demonstration fully demonstrates the advantages of introducing heterogeneous crowd robots, and how the complementary capabilities of different types of robots can work together to improve the indoor 3D mapping process.

We present CaptAinGlove, a textile-based, low-power (≤ 1.15Watts), privacy-conscious, real-time on-the-edge (RTE) glove-based solution with a tiny memory footprint (≤ 2MB), designed to recognize hand gestures used for drone control. We employ lightweight convolutional neural networks as the backbone models and a hierarchical multimodal fusion to reduce power consumption and improve accuracy. The system yields an F1-score of 80% for the offline evaluation of nine classes; eight hand gesture commands and null activity. For the RTE, we obtained an F1-score of 67% (one user).

Perceived real-time interactions between humans and a metaverse often require synchronizing actions in the virtual and real world. Latency is a main blocking factor due to processing and transmitting data over long distances because cloud data centres that deploy metaverse locate far away. We present an anticipatory computing solution to predict precisely human actions grasping objects based on hand kinesthetics leveraging smart gloves with IMU and flexion sensors. We demonstrated that the gain from the anticipation of up to several hundred milliseconds could compensate for computing and transmission latency, enabling immersive interactions over extremely long distances. Our solution is flexible and is free from locking into particular smart glove vendors, making it easy for future extensions.

The advent of Automatic Speech Recognition (ASR) has made real-time captioning for the Deaf and Hard-of-Hearing (DHH) community possible, and integration of ASR into Head-worn Displays (HWD) is gaining momentum. We propose a demonstration of an open source, Android-based, captioning toolkit intended to help researchers and early adopters more easily develop interfaces and test usability. Attendees will briefly learn about the the technical architecture, use-cases and features of the toolkit as well as have the opportunity to experience using the captioning glasses on the tooz HWD while engaging in conversation with the demonstrators.

Ensuring universal accessibility to information cannot be overstated. Unlike visual readers, however, screen reader users are given inefficient and restricted channels to acquire the given information. In particular, we focus on the initial step of information acquisition – quickly scanning the overall structure of a textual document so that the reader makes an informed decision about where to jump and read the details. While this step is inherently quick for visual users, screen reader users passively listen to the slow, sequential list of items read aloud. To close this gap, we call for a technique that accelerates screen reader users’ scanning process. Our system, VoiceCogs, takes multi-itemed text sources and synthesizes audio that concurrently plays multiple text-to-speech from a respective text source while facilitating the discernibility of individual sources. To this end, we devise and implement two interlocking techniques to minimize phonetic interferences between concurrent speeches.

A common method of generating a garment pattern is by draping fabric on a 3D form. Many apparel designers prefer this tangible process rather than the more abstract process of drafting patterns in 2D using CAD software, but the visualization benefits of 3D rendering remain important to product development. Currently, draped patterns must be manually digitized to transfer the shape to the CAD environment, a cumbersome process. Here, we explore augmenting the fabric used to drape patterns with e-textile components, such that pinning the fabric to the form directly digitizes the pattern shape. The e-textile interface uses a keypad matrix approach with rows and columns on opposite sides of the fabric. Passing a metallic pin through a row/column intersection creates a circuit connection that a microcontroller reads and maps to a 2D spatial layout on the screen. The proof-of-concept device developed here is a first step toward an automatic digitizing system that allows designers to preserve manual skills and processes while integrating more efficiently with digital systems.

Imagine a near-future smart home. Home-embedded visual AI sensors continuously monitor the resident, inferring her activities and internal states that enable higher-level services. Here, as home-embedded sensors passively monitor a free person, good inferences happen inconsistently. The inferences’ confidence highly depends on how congruent her momentary conditions are to the conditions favored by the AI models, e.g., front-facing or unobstructed.

We envision new strategies of AI-to-Human Actuation (AHA) that boost the sensory AI’s robustness by inducing favorable conditions from the person with proactive actuations. To demonstrate our concept, in this demo, we show how the inference quality of the AI model changes relative to the person’s conditions and introduce possible actuations, used in our full paper experiments, that could drive more favorable conditions for visual AIs.

Ubiquitous computing refers to the concept of integrating computers and technology into everyday objects and environments, making them constantly available and seamlessly interconnected. In recent times, there has been a growing adoption of the concept in industrial settings as well. This contribution targets the replacement of an industrial manual repair process through a robotic solution. The focus of this work is on the novel interface utilizing a digital pen and spatial augmented reality, which mimics the original job preparation process but ubiquitously enables automated robot programming. This shifts the work of the human from a dull, dirty and dangerous process to the cognitively more demanding part of inspection and process strategy definition, whereas the robot is used as a dexterous and tenacious intelligent process tool to perform the surface processing operation itself.

Recently self-supervised learning (SSL) has been proposed in the field of human activity recognition (HAR) as a solution to the labelled data availability problem. The idea being that by using pretext tasks such as reconstruction or contrastive predictive coding, useful representations can be learned that then can be used for classification. Those approaches follow the pretrain, freeze and fine-tune procedure. In this work we investigate how a simple change - not freezing the representation - leads to substantial performance gains across pretext tasks. The improvement was found in all four investigated datasets and across all four pretext tasks and is inversely proportional to amount of labelled data. Moreover the effect is present whether the pretext task is carried on the Capture24 dataset or directly in unlabelled data of the target dataset.

Folding is a unique structural technique to equip planar materials with motion or 3D mechanical properties. Textile-based capacitive sensing has shown to be sensitive to the geometry deformation and relative motion of conductive textiles. In this work, we propose a novel self-tracking foldable smart textile by combining folded fabric structures and capacitive sensing to detect the structural motions using state-of-the-art sensing circuits and deep learning technologies. We created two folding patterns, Accordion and Chevron, each with two layouts of capacitive sensors in the form of thermobonded conductive textile patches. In an experiment of manually moving patches of the folding patterns, we developed deep neural network to learn and reconstruct the vision-tracked shape of the patches. Through our approach, the geometry primitives defining the patch shape can be reconstructed from the capacitive signals with R-squared value of up to 95% and tracking error of 1cm for 22.5cm long patches. With mechanical, electrical and sensing properties, Capafoldable could enable a new range of smart textile applications.

We propose a method to estimate the temperature of an object being grasped using a PPG sensor. Evaluation experiments showed a 94% accuracy for three subjects in two temperature ranges.

Facial expressions recognition and eating monitoring technologies can detect stress levels and emotional triggers that lead to unhealthy eating behaviors. Wearables offer a ubiquitous solution to help the individual develop coping mechanisms to manage stress and maintain a healthy lifestyle. Introducing FaceEat, a privacy-focused, real-time, and on-the-edge (RTE) wearable solution with minimal power consumption (≤ 0.55 Watts) and utilizing a tiny memory space (11 − 19KB). Its purpose is to recognize facial expressions and eating/drinking activities. At the heart of FaceEat are lightweight convolutional neural networks, serving as the backbone models for both facial and eating scenarios. During the RTE evaluation, the system achieved an F1-score of over 86% in facial expression recognition. Additionally, we achieved an F1-score of 90% for monitoring eating and drinking activities for the user-independent case with an unseen volunteer for the RTE.

We propose a method to estimate the sampling frequency from only the 3-axis acceleration data. The proposed method calculates the absolute difference between two consecutive samples, creates a histogram, and constructs a Transformer-based regression model.

A venous insufficiency, to which the deep veins of the lower human extremities are particularly susceptible, can lead to serious diseases, such as a deep vein thrombosis (DVT) with subsequent risks of severe implications, e.g. pulmonary embolism or a post-thrombotic syndrome (PTS) [6]. The current standard procedure to diagnose venous insufficiency is performed exclusively in medical offices and hospitals in the form of in-patient treatments with special medical equipment. This hurdle for the patient, combined with the often diffuse symptoms of venous insufficiency [7], may lead to a late discovery of diseases such as DVTs and increases the risk of secondary diseases as well as treatment costs [3]. To address these issues, we propose a novel method for continuous monitoring of the current venous function by adapting the Light Reflection Rheography (LLR) and using low-cost wearable sensor technology and a smartphone app, aiming to deliver critical early stage information about pathological changes of the blood flow in the lower limbs.

AirSpec is an extensible, environment-focused, research and development smart glasses platform that evolved from an existing open-source, psychophysiological monitoring system. We created custom supporting software toolkits that allow users to interact with the device, easily view real-time data, and perform remote data collection. In its base configuration, the system includes a variety of sensors that sample physiological and environmental signals and stream that data to a Bluetooth-connected client, either a phone running the AirSpec App or a Bluetooth-equipped computer via our website or a python script. In addition, AirSpecs have been made more extensible with multiple external electrical connections to support more applications and future sensor subsystems.

Recognizing human activities from sensor data is a vital task in various domains, but obtaining diverse and labeled sensor data remains challenging and costly. In this paper, we propose an unsupervised statistical feature-guided diffusion model for sensor-based human activity recognition. The proposed method aims to generate synthetic time-series sensor data without relying on labeled data, addressing the scarcity and annotation difficulties associated with real-world sensor data. By conditioning the diffusion model on statistical information such as mean, standard deviation, Z-score, and skewness, we generate diverse and representative synthetic sensor data. We conducted experiments on public human activity recognition datasets and compared the proposed method to conventional oversampling methods and state-of-the-art generative adversarial network methods. The experimental results demonstrate that the proposed method can improve the performance of human activity recognition and outperform existing techniques.

Conducting user studies on social perception of wearable technology requires a framework with sufficient sensitivity to emulate perception in real-life. Using videos can efficiently reach a large number of participants, but what level of video fidelity is necessary to achieve results similar to live interaction? In a study of 48 participants, we examine four different viewing conditions: phone, laptop, big screen, and live demonstration, while investigating the social weight of wearing different glasses (Vuzix Blade, Tooz Devkit, North Focals, or prescription glasses). Contrary to our hypothesis, the live condition was not the most sensitive.

We explore an atypical use of bio-impedance by leveraging the unique temporal impedance patterns caused by the dynamic circuit changes between a pair of electrodes due to the body motions, and interactions with metal utensils and food during dining activities. Specifically, we present iEat, a wearable impedance-sensing device for automatic food intake monitoring without using external devices such as smart utensils. Using only one impedance channel with one electrode on each wrist, iEat detects food intake activities (e.g. cutting, putting food in the mouth with or without utensils, drinking, etc.) and food types from a defined category. At idle, iEat measures the normal body impedance between the wrists; while eating, new parallel circuits will be formed between the hands through the utensils and food. To quantitatively evaluate iEat in real-world settings, a food intake experiment was conducted including 40 meals performed by ten volunteers in a realistic table-dining environment. With a light-weight convolutional neural network and leaving one subject out cross-validation, iEat could detect five food intake-related activities with 86.27 % average accuracy, and classify eight types of foods with 77.73 % average accuracy.

We present WatchPPG, an open-source toolkit that enables raw photoplethysmography (PPG) data collection and stress detection using off-the-shelf smartwatches.

Smartphones have become an integral part of our daily lives, with an ever-increasing number of users relying on mobile applications to meet their needs. This widespread usage necessitates a deep understanding of how people utilize their devices to develop more effective features. Leveraging context-based information from these devices offers insights into user needs and behaviours. This PhD study focuses on three primary aspects: deciphering context signals to determine user needs, modelling mobile user behaviour based on these signals, and generating synthetic smartphone user patterns. Our goal is to provide invaluable data that can drive the evolution of future mobile applications, further enhancing the user experience.

In our everyday lives, we often follow standardized norms based on the “standard person”. However, these existing norms, created by averaging data from large populations, often fail to fit an individual. It is also challenging to implement guidelines based on these norms due to various real-life constraints in our life. We propose a hyper-personalization approach, leveraging principled generation to overcome these limitations. We explore a method that deconstructs the standard norms into principles and embodiments, and then generates contextualized guidelines by producing embodiments based on these principles. We initially identified the efficacy of our research by designing a real-life actionable sleep recommendation system. Our next step is the personalization of vocabulary in children’s language learning. In the end, we aim to design and implement a framework that can be applicable to potential cases in the future.

Electronic textiles (e-textiles) development and manufacturing processes have largely been based on individual (Do it yourself) DIY manufacturing procedures without rigorous production process standardization. Because of this, textile circuits have mostly been subjected to a trial & error development approach. In line with this motivation, this dissertation will investigate, identify, and synthesize e-textile fabrication parameters to develop a design guideline for woven stitched textile circuits. The first phase investigates and synthesizes expert knowledge that will allow for the formulation of design rules regarding the manufacturability of an e-textile. The second phase involves the development of design rules for e-textiles manufacturability to standardize the development process of e-textiles in a more reliable and repeatable way, especially because repeatability and reliability are the most important factors in manufacturability [1]. The third phase involves the deployment of these rules in a solar e-textile circuit application.

Mental, emotional, and behavioral disorders are highly prevalent in preschool-aged children and can significantly affect their social-emotional development and adaptive functioning. However, identifying signs of problematic behavior at this age is extremely challenging due to several structural and phenomenological barriers. This work leverages mobile and wearable devices to build accurate, usable, and scalable assessment tools that can be deployed in home settings to screen for common disorders in young children. It describes the development of novel screening algorithms that utilize behavioral and neurophysiological signals recorded during brief, naturalistic tasks, and presents stakeholder perspectives toward the usability and clinical utility of such screening tools.

Population aging is a global phenomenon. It is expected that by 2050, the older adult population will be 21.4% of the total population in US alone, according to Pew Research. WHO recognizes social engagement as an important factor for reducing isolation. Researchers have historically used digital technologies to build connections, but they offer disadvantages like accessibility issues and have not been a good match for the aged population. My doctoral research aims to (1) understand older adults’ needs, preferences, and challenges in maintaining social connections (2) investigate the benefits of tangible technologies to help older adults stay connected (3) create a design framework for tangible technologies to improve social connection for older adults. This doctoral consortium paper discusses my research on exploring the potential of tangible technologies, as opposed to digital, to reduce social isolation among older adults.

It is well-known that cognitive bias influences information retrievers to receive information fairly. Among all the biases, we focus on confirmation bias, which is the most impactful and sometimes leads to polarization. Numerous projects have been working on improving search systems to provide fair results, while little work has been done on enhancing fairness from the user’s side. Thus, this project aims to investigate a quantifiable approach to make the systems aware that retrievers carry confirmation bias and perform biased behaviors. Besides, some works have attempted to detect confirmation bias using web-logging or eye-tracking but failed to find differences between the search behaviors. In this regard, other quantifiable and objective measures should be applied. This project aims to collect multi-modal behavioral and physiological data using wearable sensors, and used as the input for machine learning techniques and build a bias-aware model.

Wearables have become ubiquitous technologies that allow people to capture rich data to track their health information, immerse themselves in virtual reality, and stay connected. However, there is a consensus that wearables are only available to a select few due to cost, reducing the potential benefits they can provide to underserved populations. To ensure that wearables are designed more equitably and include the voices of underserved communities in the design process, I conducted a qualitative study with 19 participants from low-income communities and found that safety was a critical issue and that wearables can potentially be used to address their daily safety concerns. In this thesis position paper, I motivate the importance of codesigning wearables with and for underserved communities and propose a novel approach at the hardware and software levels for the development of safety-based wearable devices based on the needs of these communities.

Wearable technologies have gained significant traction in various domains with healthcare being a particularly prominent area for their use, often in conjunction with mobile (health) applications. The increasing demand and versatility of these technologies pose challenges in meeting the specific needs of patients, healthy subjects, as well as healthcare professionals. Despite their growing adoption, there is a notable lack of empirical research focusing on user behavior, adaptation, acceptance, and usability evaluation for wearable technologies and companion apps. Research focusing on human factors and usability is currently inadequate in this specific area. Therefore, the objective of my research is to develop a structured guideline to address these limitations. The primary goal is to create a practical and flexible framework for assessing the usability of wearables and their associated mobile apps. This framework will be designed to be accessible and adaptable, allowing both usability experts and non-experts such as clinical researchers to evaluate wearable technology together with its companion mobile app. Through the development, validation, and implementation of this framework, this investigation aims to support the evaluation in a structured manner, and thus enhance the long-term adoption and improve the acceptance of wearable technologies and mobile (health) apps.

Hardware and software advances have paved the way for large-scale environmental urban sensor networks, but there is no guidance on where to place a finite number of nodes and how to assess the network design. In this work, I develop and test two quality metrics for urban sensor networks—one that focuses on the impacts of urban form and one that acknowledges cities as social spaces. I then propose a data-driven algorithm grounded in urban planning theory for a new urban sensor network design. Through real-world deployments and simulations using open data, I will compare my proposed design and quality metrics to those used in prior sensor networks. This work will enable the deployment of environmental sensor networks that produce useful citywide data for numerous stakeholders despite the complexities of urban environments.

Knowledge Transfer is one of the essential principles in education. The teacher’s knowledge is encoded through speech and writing and transmitted to the student. The student then decodes the transmitted information according to individual capabilities and absorbs it as knowledge. This paper presents an approach to accelerate knowledge transfer using sensor technology and social-cognitive states. So far, we have worked on quantifying meeting discussions, analyzing lecture studies, and estimating domain knowledge from web browsing. The contribution of this study is to estimate the degree of achievement of knowledge transfer and to accelerate it based on the estimated results.

People with Vision Impairments face various barriers in physical activities that are highly visual (e.g., cooking, visiting art museums), which strongly impact their agency, autonomy, and quality of life. Cyber-Physical Systems (CPS) provide opportunities for people with vision impairments to better interact with physical space. In this research, we first explore current practices and challenges of the adaptation of CPS by people with vision impairments in physical activities. We then establish a comprehensive design space and design guidelines for CPS to satisfy the needs of blind individuals. Finally, we integrate the user-centered design guidelines, implement CPS for people with vision impairments to reduce the barrier of interacting with physical space, and conduct deployment studies to validate our design. Overall, our research provide a roadmap of augmenting accessible CPS for people with disabilities in the physical space.

Skin-strain – the act of stretching the skin – is an interesting (but generally understudied) mode of haptic stimulation. Producing artificial skin strain on the body using a wearable device requires a clear understanding of the body product relationship between the device and the user. We present a wearable device capable of creating dozens of unique skin strain experiences on the wearer through a novel shape memory alloy (SMA) actuator + reconfigurable hook-and-eye attachment architecture. Not only can this architecture create spatially- and temporally-customizable skin strain experiences, it does so without the use of temporary / permanent adhesives (a typical limitation of other skin strain device designs). We present the iterative design process from early working prototypes to the most recently developed devices, including the underlying design criteria and decisions

Anger dysregulation can lead to aggression and peer rejection in early childhood. Anger episodes and temper tantrums frequently escalate, leaving parents and young children needing effective emotion regulation interventions. This research investigates how designing a playful interaction with ubiquitous technology can help parents teach young children emotion regulation strategies such as deep breathing. Breezy is a soft toy with an app and a storybook that helps parents and engages them in teaching deep breathing as an anger regulation strategy. The soft toy has sensory and physical features and can be held as a puppet, which helps young children to connect with the toy. The digital app addresses facial expressions and physiological components of emotions through sensory and interactive features that encourage identifying emotions. The storybook facilitates a familiar context for parents' engagement through playful learning. The interactive system provides parents with opportunities to teach their young children to identify and label anger and practice and model age-appropriate breathing exercises. This research aims to inform the design of future educational and playful interactions for emotional competence skills, as well as to broaden their application and promote their use in homes.

In this work, we promote capacitive sensing as a versatile smart textile modality through a collection of functional wearable designs. Considering the large variety of possible garment design concepts, we outline an approach to implement smart sensing technology into garments while maintaining these diverse design possibilities. After introducing the basic functionalities of capacitive sensing and the process of designing and building a smart garment, we present an assortment of garments enabled by this technology within the MoCa’Collection. Each of the projects serves a different purpose, built by people representing different backgrounds from electrical engineers, computer scientists, digital artists to smart fashion designers, starting from technical design over digital art to our latest design of a strongly design-oriented full-body capturing suit implementing the proposed technology.

With the progression of technology as integrated into daily life, physical tech has become increasingly embedded or hidden from the user’s view. Because of this design change, many of the aesthetics that previously defined everyday technology have disappeared from the public eye. Our ability to connect the capability of technology with the spacial world it utilizes has disappeared with it. This notion is especially true for antenna design. An object that was once visible on cars, houses, and phones is now so embedded in the devices that use it that its technology is essentially formless. Kirigami Antennas is a research exploration centering on e-textile meta-materials, designing antennas that are cognizant of their use and relation to space. This collection of antennas are not embedded nor hidden. They instead borrow from the art of Kirigami to re-insert themselves into the 3D space from which they receive their signals. Through experimentation with Kirigami antenna shapes, we are able to design freestanding lace antennas that effectively received electromagnetic signals at a wide range of frequencies, picking up AM, FM, and HAM radio, along with other data transmissions. Kirigami Antennas provides a space for experimentation with antennas as objects that help us reach and search through space.

Bio-based materials facilitate the development of more sustainable devices and wearables, expanding the range of design possibilities beyond conventional materials. Our work with biofoam explores one such quality, dissolving, as a unique affordance for designing and interacting with wearables. We developed techniques to make biofoam yarns, and used them to craft three wearables: “Seasonal Footwear", a “Reveal Bralette", and an “Unfolding Lace Top". These wearables incorporate sections that dissolve in water, allowing customization to suit the user’s needs. These wearables illustrate short-term use cases, such as a one-time reveal or shape change. We explore this novel design space as sustainable ephemeral fashion, where bio-based dissolving materials enable revealing, transformative, and interactive functionalities.

Urinary incontinence (UI) is a prevalent condition affecting millions of individuals worldwide, leading to various physical, social, and psychological challenges that diminish their quality of life. Current management approaches primarily focus on containment rather than proactive monitoring and warning systems. This paper presents the development and evaluation of a novel wearable technology called Privee, designed as an unobtrusive undergarment to monitor bladder fullness in real-time. Privee utilizes e-textile-based bioimpedance spectroscopy technology, which noninvasively assesses bladder fullness by analyzing the electrical properties of body tissues and fluids. The undergarment incorporates eight embroidered electrodes and textile transmission lines seamlessly integrated into the fabric. By continuously monitoring the bioimpedance signals from the bladder, Privee provides real-time information about the bladder’s fullness level. This data is processed using a specialized algorithm to estimate the need for urination. The noninvasive nature of Privee eliminates the discomfort and risks associated with invasive monitoring methods, offering a user-friendly and convenient solution for individuals with UI, overactive bladder, or post-operative care needs. This innovative technology has the potential to improve patients’ quality of life and optimize healthcare costs associated with UI management.

Prosthetic makeup is the use of prosthetic materials for cosmetic or makeup effects to extend the skin and features. Commonly used to simulate wounds or exaggerate physical characteristics, prosthetic makeup is usually created for film or theatrical purposes, rather than for everyday fashion or social wearability. Social Prosthesis is a design project which aims to introduce interactivity, movement, and aesthetic within silicone prosthetics by providing design considerations and fabrication techniques unique to on-face wearables. Through opening up opportunities for cosmetic expression and storytelling through dynamic makeup, Social Prosthesis invokes the sociality of beauty—the change and movement that happens when we alter our appearances in contact with others.

Combined with the unprecedented stress of the COVID-19 crisis and the increase in social unrest, human-caused environmental disasters are having a profound impact on well-being, resulting in a dramatic spike in mental health issues. Studies are emerging daily around concepts of ecological grieving stress, depression, anxiety, and a host of emotions that are surfacing and increasing in our modern times. From eco-nostalgia to eco-anxiety and eco-grief, our responses to climate change, environmental devastation, and social unrest can prevent us from taking positive action, often leading to existential crises. Our proposed project, Haikeus: Transmuting Ecological Grieving into Action, works directly at the interface of some of humanity's wicked problems, which are complex, challenging to solve, and hard to fully understand. The aim of this project is to bring awareness and motivation for transmuting such an emotion into an action through the power of creativity. We further argue that the established methods could facilitate a more nuanced understanding of organizational barriers to communicate its potential value to proceed with the change.

Fit of an e-textile garment has become increasingly important as more advanced wearable technology applications demand precise placement of sensors and actuators. Human anthropometry is complex and varied, and for many applications, a single e-textile garment cannot accurately fit a variety of users. One such example is lower-limb rehabilitation applications, which rely on precise placement of sensors and actuators with respect to joints and muscles for each individual user. Here, we present the development of a multi-layer, stretchable, flexible e-textile system, which affords quick and easy repositioning of components on the garment surface. Beyond custom fitting, this infrastructure also affords dynamic functionality of the garment, by adding and replacing sensors and actuators to enable a wide variety of applications.

This paper presents the use of weaving as a technique to create functional augmented reality (AR) markers using different textile structures and colors. We conducted experiments with plain, twill, and satin weaves, as well as varying colors in the warp, to test the effectiveness of the markers. Our findings show that weaving is a viable method for creating AR markers, and the software can detect markers even with varying colors and slightly misaligned quadrants. This work opens up new possibilities for weaving and textile structures in AR design.

This paper presents BioSparks, a wearable device that detects glucose levels in sweat through electrochemical biosensors crafted with traditional jewelry techniques. Unlike conventional biosensors that are disposed of after use, BioSparks employs a repurposing method, allowing for the reuse of discarded electrodes within the jewelry’s chain, as pendants or earrings. It incorporates interchangeable electrodes that facilitates their replacement after timelife. The modular design enables the wearable to be placed on various body parts, including the neck, wrist and waist. The paper outlines our design considerations for Wearability Factors for Jewelry Biosensors, and the fabrication process combining traditional jewelry techniques and electromistry. Our technical evaluation shows the performance of our biosensor under ten different glucose concentrations.

Users listening to music, making phone calls or listening to the environment they are in (i.e. situational awareness) are having a sub-optimal user experience just because the in-ear headsets they are using are not optimised for each user but rather optimised for a general population (i.e. population average). The proposed method is one way of alleviating this problem by compensating for the users’ ear-canal shapes and size difference and including a correction EQqualiser (EQ) into the signal processing chain, for best and most compelling user experience.

In recent years, more and more consumer earphones come equipped with inertial measurement units (IMUs). Using such sensors, past research has broadly explored earables for fitness applications, ranging from tracking higher-level workout activities to extracting precise gait-related parameters. Tying in with this research, we initially evaluated how the accelerometer inside earables may be leveraged to conduct vertical jump testing, which is a common standard measure to assess fitness. In a small study, four participants performed seven jump trials each. Using a simple analytical approach, we find that jump height can be calculated at 0.04 m mean absolute error with a median error of 0.02 m.

In the rapidly expanding universe of smart IoT, earable devices, such as smart headphones and hearing aids, are gaining remarkable popularity. As we anticipate a future where a myriad of sophisticated applications—interaction, communication, health monitoring, and fitness guidance—migrate to earable devices handling sensitive and private information, the need for a robust, continuous authentication system for these devices becomes more critical than ever. Yet, current earable-based solutions, which rely predominantly on audio signals, are marred by inherent drawbacks such as privacy concerns, high costs, and noise interference. In light of these challenges, we investigate the potential of leveraging photoplethysmogram (PPG) sensors, which monitor key cardiac activities and reflect the uniqueness of an individual’s cardiac system, for earable authentication. Our study presents EarPass, an innovative ear-worn system that introduces a novel pipeline for the extraction and classification of in-ear PPG features to enable continuous user authentication. Initially, we preprocess the input in-ear PPG signals to facilitate this feature extraction and classification. Additionally, we present a method for detecting and eliminating motion artifacts (MAs) caused by head motions. Through extensive experiments, we not only demonstrate the effectiveness of our proposed design, but also establish the feasibility of using in-ear PPG for continuous user authentication—a significant stride towards more secure and efficient earable technologies.

Earables have been gaining popularity over the past few years for their ease of use and convenience over wired earphones. However, modern-day earables usually have a limited interface, inhibiting their potential as an accessible medium of input. To this end, we present EarBender: an ear-based real-time system that bridges the gap between earables and on-body interaction, providing a more diverse and natural form of interaction with devices. EarBender enables touch-based hand-to-ear gestures on mobile devices by leveraging inertial sensors in commercially available earable devices. Our proposed system detects the slight deformation in a user’s ear resulting from different ear-based actions including swiping and tapping and classifies the action performed. EarBender is designed to be energy-efficient, easy to deploy and robust to different users, requiring little to no calibration. We implement a prototype of EarBender using eSense, a multi-sensory earable platform, and evaluate it in different scenarios and parameter settings. Results show that the system can detect the occurrence of gestures with a 96.8% accuracy and classify seven different hand-to-ear gestures with an accuracy up to 97.4% maintained across four subjects.

Earables computing is an emerging research community as the industry witnesses the soaring of True Wireless Stereo (TWS) Active Noise Canceling (ANC) earbuds in the past ten years. There is an increasing trend of newly initiated earable research spanning across mobile health, user-interfaces, speech processing, and context-awareness. Head-worn devices are anticipated to be the next generation Mobile Computing and Human-Computer Interaction (HCI) platform. In this paper, we share our design experiences and lessons learned in building hearable sensing systems from the industry perspective. We also give our takes on future directions of the earable research.

The significant increase in data production resulting from the widespread adoption of mobile and IoT technologies has revolutionized healthcare but also presents significant privacy and ethical challenges. The field of medical data collection is no exception and has limitations in terms of the source, variety and quantity of records from studies on healthcare and wellness. One way to address this dilemma is the use of the Blockchain for patient data collection and use. The anonymity of a centralized network allows the patient’s identity to be protected. The structure formed by nodes allows the information to be always available and does not depend on a main server. The immutability of records in the chain ensures unambiguous traceability of information flow by the healthcare provider. Finally, the network’s consensus and reward mechanisms could motivate new users to participate in active sensing. In this article we will expose the architecture of an application that relies on the Blockchain to meet the above information needs by leveraging the potential of the Ethereum network. In addition, we present a use case where consciously collected data from our platform is used to train a machine learning model automatically, using a P2P Browser-Based Computational Notebook as execution and distribution environment.

Federated learning (FL), a decentralized machine learning technique, enhances privacy by enabling multiple devices to collaboratively train a model without transferring data to a central server. FL is used in Human Activity Recognition (HAR) problems, where multiple users generating private wearable data share models with a server to learn a useful global model. However, FL may compromise data privacy through model information sharing during training. Moreover, it adheres to a one-size-fits-all approach toward data privacy, potentially neglecting varied user preferences in collaborative scenarios such as HAR. In response to these challenges, this paper presents a collaborative learning framework integrating differential privacy (DP) and FL, thus providing a tailored approach to privacy protection. While some existing works integrate DP and FL, they do not allow clients to have different privacy preferences. In this work, we introduce a framework that allows different clients to have different privacy preferences and hence more flexibility in terms of privacy. In our framework, DP adds individualized noise to individual clients’ gradient updates for privacy. However, such noised updates can also be interpreted as an attack on the FL system. Defending against these attacks might result in excluding honest private clients altogether from training, posing a fairness concern. On the other hand, not having any defensive measures might allow malicious users to attack the system, posing a security issue. Thus, to address security and fairness, our framework incorporates a client selection strategy that protects the global model from malicious clients and provides fair model access to honest private clients. We have demonstrated the effectiveness of our system on a HAR dataset and provided insights into our framework’s privacy, utility, and fairness.

Preserving the individuals’ privacy in sharing spatial-temporal datasets is critical to prevent re-identification attacks based on unique trajectories. Existing privacy techniques tend to propose ideal privacy-utility tradeoffs (PUT), however, largely ignore the fairness implications of mobility models and whether such techniques perform equally for different groups of users. The quantification between fairness and privacy of PUT models is still unclear and there exists limited metrics for measuring fairness in the spatial-temporal context. In this work, we define a set of fairness metrics designed explicitly for human mobility, based on structural similarity and entropy of the trajectories. Under these definitions, we examine the fairness of two state-of-the-art privacy-preserving models that rely on GAN and representation learning to reduce the re-identification rate of users. Our results show that these models violate individual fairness criteria, indicating that users with highly similar trajectories receive disparate privacy gain.

Over the past few decades, ubiquitous sensors and systems have been an integral part of humans’ everyday life. They augment human capabilities and provide personalized experiences across diverse contexts such as healthcare, education, and transportation. However, the widespread adoption of ubiquitous computing has also brought forth concerns regarding fairness and equitable treatment. As these systems can make automated decisions that impact individuals, it is essential to ensure that they do not perpetuate biases or discriminate against specific groups. While fairness in ubiquitous computing has been an acknowledged concern since the 1990s, it remains understudied within the field. To bridge this gap, we propose a framework that incorporates fairness considerations into system design, including prioritizing stakeholder perspectives, inclusive data collection, fairness-aware algorithms, appropriate evaluation criteria, enhancing human engagement while addressing privacy concerns, and interactive improvement and regular monitoring. Our framework aims to guide the development of fair and unbiased ubiquitous computing systems, ensuring equal treatment and positive societal impact.

Measuring and ensuring machine learning model fairness is a challenging task that is especially difficult in federated learning (FL) settings where the model developer is not privy to clients’ local data. We propose Inflorescence, a framework that explores how the application of clustered FL strategies, which are designed to handle data distribution skew across federated clients, affects fairness. We share empirical simulation results quantifying the extent to which clustered FL impacts various group and individual fairness metrics, finding that it tends to improve fairness in terms of accuracy, precision, and error entropy, but not in terms of disparate impact or equal odds. We open-source the proposed framework as a Python package to facilitate research on fairness in distributed systems.

This research presents an exploratory study focusing on developing the immersive mindfulness meditation group experience. The primary goal is to observe the impact of Extended Reality (XR) on enhancing people’s awareness and cultivating inner tranquility. Previous works have explored the benefit of group meditation and the potential of utilizing multisensory feedback in XR. Our study designs an activity that employs interactive digital art to help the therapist verbally convey the concept of mindfulness meditation through participants’ senses, including visual, olfactory, gustatory, auditory, haptic, interoceptive, and proprioceptive perceptions. Unlike the traditional mindfulness meditation experience, the XR experience provides multiple channels for participants to engage in mindfulness, extending beyond mere imagination. By surrounding the therapist’s healing script in immersive projection, the experience can stimulate participants in the space in which they exist physically. To collect participant feedback, we did a short questionnaire with 68 responses. The result shows that few attendees would connect the activity with other traditional psychological therapy activities they attended, but most participants indicated that they had a different experience. Like the conventional activity, it brings them a calm and peaceful mind. The study gathered considerations for designing group mindfulness meditation, including participant preparation and selection, the provision of multisensory stimulation, and essential factors when collecting feedback.

In the realm of virtual reality (VR), haptic feedback plays a pivotal role in enhancing user immersion. However, the implementation of non-contact thermal haptics presents numerous challenges. In this study, we introduce a novel approach for creating thermal haptic feedback in a VR environment, combining an ultrasonic phased array with a heating circuit. Our design generates multiple beams that deliver heated air to the user, thus simulating the sensation of temperature change. This approach enables an immersive, responsive VR experience that dynamically adjusts the temperature of the surrounding air based on the VR content. The innovation resides in our method’s ability to overcome traditional phased array limitations, such as grating lobes and high hardware costs, by leveraging the directionality of ultrasonic transducers within the array. This results in a more cost-effective, compact, and efficient thermal haptic feedback system. Our work thus offers a significant contribution to the field of VR, presenting a new way of enhancing user immersion through non-contact thermal haptics.

This paper presents the experience in the implementation and application of a large-scale Indoor Location-Based Service (LBS) in an academic library. We deployed an indoor positioning system within the school library that leverages 550 Bluetooth beacons, covering an area of approximately 12,000 square meters. This system allows users to engage with location-aware book navigation services via a web interface on their mobile devices. Upon locating a book of interest, the system enters navigation mode, updating and guiding users based on their current location within the library. Utilizing the users’ positional data and borrowing history, the system is capable of recommending other potentially interesting books, exhibits, and library events. The system went live in February 2023, with a recorded usage sessions of 21,540 instances which includes 17,271 uses of the retrieval services and 2,065 indoor navigation services, as of May 2023. This paper outlines the system architecture of this large-scale indoor LBS, shares user engagement data, and after anonymization, makes this data publicly available for academic analysis and research. We are hoping that our experience can shed light on the understanding and future development of large-scale indoor LBS technologies.

In this short paper we provide some observations on the (im)possibility of distinguishing between real and virtual in augmented reality worlds. Distinguishing may be possible because of imperfect technology, impossibilities, and manipulating perceptual capabilities. The question must be asked, if this distinction cannot always be made, does this place responsibility on the owner or administrator of the Extended Reality (XR) environments? And who might be the owner or administrator in the case of ever-present extended reality?

In this paper, we present the design and development of HarmonicThreads, a smart, cost-effective fabric augmented by generative machine learning algorithms to create music in real time according to the user's interaction. In this manner, we hypothesize that individuals with sensory differences could take advantage of the fabric's flexibility, the music will adapt according to users' interaction, and the affordable hardware we propose will make it more accessible. We follow a design thinking methodology using data from a multidisciplinary team in Mexico and the United States. Then we will close this paper by discussing challenges in developing accessible smart fabrics in different contexts.

Since 2018, Venezuelans have contributed to 75% of leading AI crowd work platforms’ total workforce [15], and it is very likely other Latin American and Caribbean (LAC) countries will follow in the context of the post covid-19 economic recovery [7]. While crowd work presents new opportunities for employment in regions of the world where local economies have stagnated[11], few initiatives have investigated the impact of such work in the Global South through the lens of feminist theory. To address this knowledge gap, we surveyed 55 LAC women on the crowd work platform Toloka to understand their personal goals, professional values, and hardships faced in their work. Our results revealed that most participants shared a desire to hear the experiences of other women crowdworkers, mainly to help them navigate tasks, develop technical and soft skills, and manage their finances more efficiently. Additionally, 75% of the women reported that they completed crowd work tasks on top of caring for their families, while over 50% confirmed they needed to negotiate their family responsibilities to pursue crowd work in the first place. These findings demonstrated a vital component lacking from the experiences of these women was a sense of connection with one another. Based on these observations, we propose a system designed to foster community between LAC women in crowd work to improve their personal and professional advancement.

Research involving participatory design with children necessarily implies a careful review of the research methods to ensure participants’ safety; However, in Latin America may be different than in the Global North countries. Over the past 13 years, we have partnered with a school-clinic in Mexico that specializes in assisting children with autism. During this time, we designed, developed, and assessed UbiComp technology to assist in various stages of a child's therapy cycle. We have been collaborating with the school-stakeholders such as children and young adults with autism, parents, teachers, psychologists, and therapists, including them in all the phases of the projects. In this paper, we reflect on our experiences at this school-clinic, discussing the formality and ethics when conducting research. We close by presenting the implications we have learned while conducting research with this population.

Built environments (BE) are increasingly integrating sensing and interactivity capabilities, changing how we co-exist and live. In recent years, HCI, ubiquitous computing, and architecture have contributed to the interdisciplinary field of Human-Building Interaction (HBI). HBI represents the growing complexities of human experience within BE, which includes utilizing sensing capabilities to gracefully support people’s needs. We present an initial taxonomy classifying interactions with HBI devices, the HBI Interactivity Taxonomy, a novel contribution to this field. We employed an integrative research strategy, sampling device descriptions from commercial and academic sources. Taxonomy features were extracted through thematic analysis. The resulting list of characteristics describes how users interact with HBI devices. We offer an initial version of this taxonomy as a tool to facilitate communication and enhance the design and evaluation of future HBI devices in both academia and industry.

Human physiology and reactions to external stimuli differ between individuals. Researchers have developed strategies to adapt to these differences but the adaptations generally require data from each individual beforehand. A cold-start occurs when there is no data from a new individual. To address this, current study proposes user calibration which uses short segments of easily obtainable baseline data to adapt to new individuals. Experiments were conducted on two public stress and affect detection datasets, WESAD and SWELL-KW, to assess the effectiveness of the proposed calibration method and to determine suitable duration of the baseline measurement. Results showed that user calibration always beat the non-personalized model and segments of 3-8 minutes seemed to be most promising to consider for future use.

This study explores the potential of conversational agents to improve well-being during food preparation and consumption. We describe the process of designing and evaluating a conversational agent that promotes mindful eating and cooking. We first discuss the findings of an exploratory study conducted to understand changes in food behaviors of young adults in Mexico during the COVID-19 pandemic and their impact on well-being. Subsequently, an iterative methodology is used to design and evaluate two interactive systems with 2 different studies. The results of these interventions provide evidence that the systems improve the eating/cooking experience and promote mindfulness when performing these activities.

Coughs and sneezes are important symptoms of respiratory, allergic and Influenza-Like Illnesses. Monitoring the number of occurrences of these symptoms can not only track patients’ health but also help with understanding the dynamics of outbreaks. While there has been extensive research around detecting coughs, creating reliable methods to identify sneezes are still unexplored, partially owing to scarce availability of data. Most respiratory symptom detection algorithms operate at high sampling rates and thereby are privacy-invasive. By downsampling the data extensively, we can ensure that the speech cannot be reconstructed, yet retain enough signal characteristics to identify sneeze and cough sounds. Using a lightweight machine learning based approach, we propose a privacy-aware and energy-efficient pipeline that can detect coughs and sneezes. Such a system can lead to higher adoption and thus make tracking diseases in individuals and communities easier.

Stress detection is important for ensuring overall mental well-being of an individual. Literature suggests several approaches for prediction or classification of stress. However, the performance of these approaches varies a lot across subjects and tasks. Moreover, perception of stress is highly subjective and hence it is difficult to create a generic model/devices for prediction of stress. In this study, we have proposed an approach for creating a generic stress prediction model by combining the knowledge and variety from multiple public datasets containing galvanic skin response (GSR) data recorded during different context and activities. Most significant features are selected from these recorded signals and a voting based approach was finally adopted to develop a model for predicting mental stress. Proposed model has been validated using test data as well as a set of completely unseen data collected in our lab. We achieved an average classification accuracy of 89% (F-score 0.87) for test data and similar performance for completely unseen data as well. Results show that the proposed model outperforms the training models created using individual datasets. In addition, our model is created using skin response data recorded using off-the-shelf devices. Thus, our proposed model with selected feature set can be used for monitoring stress in real life scenarios and to create mass-market stress prediction products.

Human activity recognition has several applications in healthcare, sports, and industrial settings. In the latter, it can monitor industry workers and evaluate if the required activities are appropriately performed. In this paper, we employ state-of-the-art Deep Learning techniques to recognize ten distinct packing activities performed by sixteen participants from the Openpack dataset. Our proposed architecture combines data from various sensors and leverages Convolutional Neural Networks and Long Short-Term Memory networks to process spatial and temporal data. Additionally, we incorporate Transformers into our network, resulting in an improved F1-Score performance of 98.21.

Many IMU-data-based HAR datasets have been proposed. However, integrating the use of those is difficult because each dataset is measured by its unique sensor positions. Thus, we have developed a garment device with 396 IMU sensors placed in a grid with approximately 7 cm spacing to enable the matching of IMU data from various positions and the integrated use of multiple databases. In this paper, the best and most suitable sensor positions for classification were explored with this device for six basic activities and eight daily activities with the expectation of determining the key IMU positions for the integrated dataset and creating standard sensor positions when building a new dataset. The single sensor positions with the highest classification performance in all activities, basic activities, and daily activities were the upper-middle part of the dominant lower arm, the front of the thigh, and the inner wrist of the dominant arm. The position with the highest classification score for daily activities except Writing and Typing was the inner wrist of the dominant arm. The sensor position with the highest classification score for Writing was the upper part of the dominant upper arm, and Typing was the upper-middle part of the non-dominant lower arm. Furthermore, the combination of the four sensors showed the best macro-F1 scores for all activity sets: 0.90 with all activities, 1.00 with basic activities, and 0.83 with daily activities.

In the event of sudden cardiac arrest in places where medical care is not available, such as in a house or in the city, primary life-saving treatment by a person present at the scene is extremely important. However, only about half of the patients receive primary life-saving treatment, and the implementation rate of primary life-saving treatment is low [5]. One of the reasons for the low implementation rate is the high psychological hurdle caused by the lack of knowledge and experience in primary life support. Although many studies have been conducted to promote the implementation of primary life-saving procedures, CPR (cardiopulmonary resuscitation) practice mannequins are still generally required to practice primary life-saving procedures, which increases the practice cost. In this research, we propose an easier cardiac massage practice method using only a smartphone equipped with a barometer, which is widely used in the world, and a transparent sealed bag used to store food. In this research, we developed a system that estimates the pressing strength, and estimates the pressing interval of cardiac massage based on the information obtained from the barometer of the smartphone, and provides real-time feedback to the smartphone screen. The system also has a test mode to check the user’s cardiac massage skills. We conducted an experiment to evaluate the screen design and practicality of the developed application. The interval of cardiac massage was evaluated as good for practice, but the strength of the cardiac massage was evaluated as just enough to get the gist.

In this work, we aim to collect data on the emotion of the elderly using the care application FonLog and try to develop a predictive model and analyze the daily emotion data of the elderly in a geriatric care facility. We used weather information and the level of care needed by the elderly as input features and the random forest as a prediction model. As a result, the method performs differently in different facilities. Feature correlations of weather data differed among each elderly in each facility. In the textual analysis based on emotion change data, we found a strong repetition in the recording of textual data. This implies that we need to provide caregivers with a more detailed rationale for how to record changes in emotion.

Owing to the widespread utilization of activity recognition, numerous deep learning models have been proposed to facilitate recognition in diverse contexts. Federated learning (FL) is an emerging learning paradigm that enables the collaborative learning of a shared model without compromising the data privacy of users. However, the assumption of FL is to rely on the annotated data on clients, which is difficult to acquire the annotations for human activity recognition (HAR) on all clients due to the lack of expertise or resource. Moreover, a general model is not suitable for each person due to data heterogeneity, resulting from the different physical characteristics and various contextual information. To this end, we propose a semi-supervised learning method for personalized federated HAR, in which clients have completely unlabeled data, while the server has a small amount of labeled data contributed by volunteers. Clients conduct unsupervised learning on autoencoders with locally unlabeled data to collaboratively learn a general representation model. The server conducts supervised learning on an activity classifier with labeled data stored on the server. After that, the shared model is personalized using individually pseudo-labeled data on each client side, wherein both confidence and uncertainty are taken into account concurrently, with the aim of achieving a balanced selection for assigning pseudo-labels to samples. We conduct extensive experiments with two different real-world HAR datasets, demonstrating the effectiveness of the proposed methods.

We analyzed the eye movements of cognitively healthy older adults while reading Japanese text and compared them with those of younger adults. We found that it is essential to match the reading speeds of older and younger adults to accurately compare eye movement parameters during their reading. Cognitively healthy older adults had longer fixation durations, fewer fixations, and fewer extra fixations than younger adults. Meanwhile, cognitively healthy older adults had a length of forward saccades comparable to that of younger adults. These results suggest that the reduced efficiency due to a longer fixation duration is compensated for by fewer fixations in cognitively healthy older adults who read at the same speed as younger adults.

In this paper, we propose a feature pioneering method using Large Language Models (LLMs). In the proposed method, we use ChatGPT 1 to find new sensor locations and new features. Then we evaluate the machine learning model which uses the found features using an open dataset. In current machine learning, humans make features, for this engineers visit real sites and have discussions with experts and veteran workers. However, this method has the problem that the quality of the features depends on the engineer. In order to solve this problem, we propose a way to make new features using LLMs. As a result, we obtain almost the same level of accuracy as the proposed model which used fewer sensors and the model uses all sensors in the dataset. This indicates that the proposed method is able to extract important features efficiently.

Deep Learning (DL) has enabled considerable increases in the accuracy of classification tasks in several domains, including Human Activity Recognition (HAR). It is well-known that when data distribution changes between the training and test datasets, the accuracy can drop, sometimes significantly. However, some variability sources in HAR, such as sensor orientation, are only sometimes considered when evaluating these models. Therefore, we must understand how much such changes could impact current DL architectures. In this paper, under an orientation variability scenario, we evaluate three common DL architectures, DeepConvLSTM, TinyHAR, and Attend-and-Discriminate, to quantify the performance drop attributed to this shift. Our results show that all architectures show performance drops on average, as expected, but participants are affected differently from them, so they would fall short for some in classification accuracy in real-life settings where orientation can change across the wearing sessions of one participant or across participants. The performance change is related to the difference in distribution distance.

Known as the Hawthorne Effect, studies have shown that participants alter their behavior and execution of activities in response to being observed. With researchers from a multitude of human-centered studies knowing of the existence of the said effect, quantitative studies investigating the neutrality and quality of data gathered in monitored versus unmonitored setups, particularly in the context of Human Activity Recognition (HAR), remain largely under-explored. With the development of tracking devices providing the possibility of carrying out less invasive observation of participants’ conduct, this study provides a data-driven approach to measure the effects of observation on participants’ execution of five workout-based activities. Using both classical feature analysis and deep learning-based methods we analyze the accelerometer data of 10 participants, showing that a different degree of observation only marginally influences captured patterns and predictive performance of classification algorithms. Although our findings do not dismiss the existence of the Hawthorne Effect, it does challenge the prevailing notion of the applicability of laboratory compared to in-the-wild recorded data. The dataset and code to reproduce our experiments are available via https://github.com/mariusbock/hawthorne_har.

With the increasing number of IoT devices, there is a growing demand for energy-free sensors. Human activity recognition holds immense value in numerous daily healthcare applications. However, the majority of current sensing modalities consume energy, thus limiting their sustainable adoption. In this paper, we present a novel activity recognition system that not only operates without requiring energy for sensing but also harvests energy. Our proposed system utilizes photovoltaic cells, attached to the wrist and shoes, as eco-friendly sensing devices for activity recognition. By capturing photovoltaic readings and employing a deep transformer model with powerful learning capabilities, the system effectively recognizes user activities. To ensure robust performance across various subjects, time periods, and lighting conditions, the system incorporates feature extraction and different processing modules. The evaluation of the proposed system on realistic indoor and outdoor environments demonstrated its ability to recognize activities with an accuracy of 91.7%.

LLMs for vision and NLP domain has been popular by the widespread use of ChatGPT and GPT-4. This paper tackles to build LLMs for sensor domain of one-dimensional signals whose downstream task is activity recognition and emotion detection. We propose a new architecture of Transformer-based self-supervised learner which we name SENvT. This SENvT builds the LLMs for sensor data using 7 pretext objectives in multi-task learning together with contrastive learning. Experimental results show these three. First, we obtained better results for contrastive learning and the masked token task but not for other pretext tasks. Second, the masked token task was better in 60% rather than in 10%. Third, the RGW worked best in accuracy while the masked token task worked best in F1.

The Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge collects sensor data for activity recognition, garnering significant interest among researchers. Our team, named "Juliet", extracts features from multi-source data. It is worth noting that we incorporate OpenStreetMap (OSM) data as supplementary information, which significantly improves the prediction performance. We employ an ensemble model that combines both machine learning and deep learning techniques. For machine learning, we utilize XGBoost, LightGBM, and CatBoost models that take hand-crafted features. For deep learning, we adopt a CNN-RNN-Transformer framework that accepts both raw features and hand-crafted features as input. By combining the ensemble model with post-smoothing, our approach enhances the accuracy of SHL recognition.

The fifth edition of Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge aims at determining the mode of locomotion and transportation of users using sensor-equipped devices. This recognition task relies on measurements captured from Inertial Measurement Unit and radio sensors placed on the user’s hand, bag, hips and torso. The provided dataset presents several challenges, including its size, asynchronicity between the data types, lack of time continuity, and imbalanced distribution of the locomotion and transportation modes. To address these issues and enhance the recognition performance, our team, KDDI Research, performed data pre-processing and hand-crafted additional features. We explored different classifiers based on both Machine Learning or Deep Learning methods. Finally, the XGBoost Classifier model achieved the highest accuracy and f1-score across different validation datasets (bag, hands, hips, and torso). This model, used for our final submission on the testing dataset, achieved an average accuracy of 0.75 on these datasets.

Transportation mode detection (TMD) is a context-aware computing technology with significant potential in several applications. However, the development of TMD technologies for real-world scenarios remains challenging, including user-independent evaluations and multimodal analyses. In this study, our team (HYU-CSE) suggested a TMD model as part of the Sussex-Huawei Locomotion (SHL) recognition challenge, and we used the SHL motion and location data. The proposed TMD model was based on the DenseNet architecture, and post-processing using voting schemes was applied to refine the detection performance. The results suggested that the proposed method achieved 94.13% of an F1 score with user-independent analysis. We hope that our study will ultimately help in the design of better TMD applications.

This paper presents the ideas utilized by Team Ds at the 2023 Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge. The 2023 SHL Challenge involves classifying eight modes of locomotion and transportation using data acquired by smartphone motion and GPS sensors. The data were obtained from three users, User1 provided the training set, and parts of the data acquired by User2 and User3 were used as the validation and test sets. Long-period data (up to 110 s) were used to estimate the modes of locomotion and transportation for a sample in the main processing. We have created a model to estimate the modes of locomotion and transportation using User1’s accelerometer, gyro sensor, magnetic sensor, and GPS location data. The neural network model comprised four sections, i.e., two types of a one-dimensional convolutional neural networks (1D-CNNs) with different filter size using raw waveform data for nine sensors, one 2D-CNN using the spectrograms of accelerometer and gyro sensor, and two fully-connected layers of 22 statistical data computed from the sensors. In a post-processing step, a model was constructed to output the labels from the SoftMax output obtained from the previous model, and this output was used as the final estimation. The developed model was evaluated experimentally, and the F1-score obtained on the validation data was 0.952.

Participants of the fifth edition of SHL recognition challenge 2023 aim to recognize eight locomotion or transportation modes in a user-independent manner based on motion and GPS sensor data. The "Fighting_zsn" team proposes an ensemble machine learning model based on road network which is experimentally shown to significantly improve model performance. First, both time-domain and frequency-domain features are extracted from the provided data, and public spatial-domain information is incorporated to get road network features. Besides, contextual information is captured by changing and window-based features derived from features mentioned above. With comprehensive experiments on the validation data, the ensemble model based on XGBoost, LightGBM, and Random Forest is chosen for the solution. Finally, results show that the model performs well when recognizing activity modes, achieving a weighted F1 score of 0.80 and an averaged precision score of 0.82. The source codes and experimental results are available from: https://github.com/zhaoyaya1234/SHL2023.

Mobile sensors have improved traffic prediction through transportation mode recognition. Researchers are interested in exploring mobile sensor-based recognition methods for transportation modes. The SHL Recognition Challenge is a prominent competition in this field. SHL Challenge 2023 introduced a diverse dataset with GNSS-based and motion sensor data for transportation mode recognition. Our team, "we-can-fly," presents a fine-grained method using a multi-scale fusion approach that incorporates motion sensor and GNSS information. With temporal and topological attention mechanisms, we capture scene characteristics and enhance contextual understanding for transportation mode recognition. On the validation set, our method achieves an impressive 72.39% accuracy and 71.25% F1 score, confirming the effectiveness of our multimodal fusion transportation mode recognition algorithm.

We address the problem of the automatic recognition of transportation mode from sensor data collected using personal devices. We leverage a dataset that includes data from motion and location sensors embedded in smartphones carried by three different users at different body positions – hand, hips, torso and bag. The data was labelled by the users using one of eight activities: still, walking, run, bike, car, bus, train, subway. This dataset was made available in the context of the SHL recognition challenge 2023. We, team MUSIC, propose DecayXGBoost, an enhanced version of the classic XGBoost classifier. DecayXGBoost leverages statistical and frequency-domain features to discriminate among the eight activities and adds a post-inference smoothing stage that encodes several heuristics into the classification. Our results show that DecayXGBoost achieves a F1 score of 84.5%, demonstrating its effectiveness in accurately predicting transportation modes from motion and location data.

The primary research objective of this study is to develop an algorithm pipeline for recognizing human locomotion activities using multimodal sensor data from smartphones, while minimizing prediction errors due to data differences between individuals. The multimodal sensor data provided for the 2023 SHL recognition challenge comprises three types of motion data and two types of radio sensor data. Our team, ‘HELP,’ presents an approach that aligns all the multimodal data to derive a form of vector composed of 106 features, and then blends predictions from multiple learning models which are trained using different number of feature vectors. The proposed neural network models, trained solely on data from a specific individual, yield F1 scores of up to 0.8 in recognizing the locomotion activities of other users. Through post-processing operations, including the ensemble of multiple learning models, it is expected to achieve a performance improvement of 10% or greater in terms of F1 score.

The goal of Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge in 2023 is to recognize 8 modes of locomotion and transportation (activities) in a user-independent manner based on motion and GPS sensor data. The main challenges of this competition are sensor diversity, timestamp asynchrony, and the unknown positions of sensors in the test set. We, team "WinGPT", construct special features like velocity from the raw dataset, and extract various features from both time domain and frequency domain. Additionally, this article calculates the distance between users and the nearest places or roads as a feature using map information obtained from OpenStreetMap. We use a dataset with a total of 202 features to train classical machine learning models such as decision tree, random forest, LightGBM, and XGBoost, among which the XGBoost model performs the best, achieving a macro F1 score of 78.95% on the validation set. Moreover, based on our predictions, we determine that the sensor location in the test set is positioned on the hand. Through a post-processing procedure applied to the model, we ultimately achieve a final macro F1 score of 90.86% on the validation set from the hand. In addition, we open the source code of feature extraction and model training and publish it on GitHub: https://github.com/Therebe123/SHL2023.

This paper aims to address the Sussex-Huawei Locomotion - Transportation (SHL) recognition challenge organized at the HASCA Workshop of UbiComp 2023. The challenge focuses on achieving user-independent recognition of eight different modes of activities using motion and GPS sensor data[3, 9]. Our team, named DataScience SHL Team, proposes a pipeline, which involves extracting features including time domain, motion position, road map, and differential features, utilizing the OpenStreetMap platform as a additional resource. We carefully select the Random Forest model as our classification model. Additionally, a post-processing approach is introduced to modify labels. Since the test data partition lacks identification, we have aggregated models trained at four sites, enhancing the overall performance and robustness. The proposed pipeline has achieved an accuracy of 84.38% and an f1-score of 57.49% for hand data during the validation phase, demonstrating a significant improvement in motion state recognition.

This paper presents our solution to the SHL recognition challenge 2023 which focuses on recognizing 8 transportation modes in a user-independent manner based on motion and GPS sensor data. Our team ZZL propose an ensemble framework based on fine multi-window feature engineering and overfitting prevention. Firstly, we extracted a large and diverse set of features in the feature engineering process, including incorporating OpenStreetMap data to better leverage location data, and introducing multiple time windows to extract long, medium, and short term aggregated features, providing rich feature inputs. Secondly, we proposed an ensemble framework that comprehensively utilizes different techniques to prevent overfitting, including data downsampling, fine-tuning data distribution, designed train-test splitting, and model integration. Moreover, we applied post-processing on the model predictions to smooth the predicted results. Finally, we achieve F1-score of 0.868 on validation dataset.

The Suessex-Huawei Locomotion-Transportation (SHL) Recognition Challenge presents a large real-world dataset derived from multimodal smartphone sensors, with the aim of accurately distinguishing between eight different states of locomotion and transportation. Based on the end-to-end principle, our team (Yummy MacMuffin) proposed AttenDenseNet, which integrates channel and spatial attention as well as convolutional layers for feature extraction and implements DenseNet as our classification model. In our experiments, we viewed the data on different axes as multi-channel input, splitting them into 5-second windows and then performing the classification task with each window as a unit. Data leakage was avoided by dividing the training data by timestamps and the generalization ability was enhanced by increasing the sample size. In the end, we achieved an average F1 score of 0.5442 on the validation set.

In this paper we summarize the contributions of participants to the fifth Sussex-Huawei Locomotion-Transportation (SHL) Recognition Challenge organized at the HASCA Workshop of UbiComp/ISWC 2023. The goal of this machine learning/data science challenge is to recognize eight locomotion and transportation activities (Still, Walk, Run, Bike, Bus, Car, Train, Subway) from the motion (accelerometer, gyroscope, magnetometer) and GPS (GPS location, GPS reception) sensor data of a smartphone in a user-independent manner. The training data of a “train” user is available from smartphones placed at four body positions (Hand, Torso, Bag and Hips). The testing data originates from “test” users with a smartphone placed at one, but unknown, body position. We introduce the dataset used in the challenge and the protocol of the competition. We present a meta-analysis of the contributions from 15 submissions, their approaches, the software tools used, computational cost and the achieved results. The challenge evaluates the recognition performance by comparing predicted to ground-truth labels at every 10 milliseconds, but puts no constraints on the maximum decision window length. Overall, five submissions achieved F1 scores above 90%, three between 80% and 90%, two between 70% and 80%, three between 50% and 70%, and two below 50%. While the task this year is facing the technical challenges of sensor unavailability, irregular sampling, and sensor diversity, the overall performance based on GPS and motion sensors is better than previous years (e.g. the best performance reported in SHL 2020, 2021 and 2023 are 88.5%, 75.4% and 96.0%, respectively). This is possibly due to the complementary between the GPS and motion sensors and also the removal of constraints on the decision window length. Finally, we present a baseline implementation to help understand the contribution of each sensor modality to the recognition task.

This study describes the human activity recognition method for the Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge by team TDU_BSA. The user and phone location of the test data were estimated; the phone location of the test data was expected to be in the hand of the user. For each segment of the test data, users and phone location were then estimated. By combining motion sensors (accelerometer, gyroscope, and magnetometer) with location, we achieved a higher accuracy than in the case of previous challenges. The classification accuracy was improved by ensemble learning of LSTM(Long Short-Term Memory)-based deep learning and XGBoost model. Finally, by applying a mode filter to the estimation results, the F-measure of the SHL validation set was 98.5% at the submission stage.

Human Activity Recognition (HAR) holds valuable potential as a digital biomarker, offering insights into individuals' physical activities and well-being. Leveraging wearable devices to monitor human activities can serve as an integral component of digital therapeutics, offering valuable insights into an individual's health and behavior. However, privacy concerns in HAR systems pose a significant challenge as users' data can be at risk of exposure and necessitates robust protection measures. To mitigate these concerns, the adoption of privacy-preserving techniques, such as federated learning, becomes necessary. This paper addresses the topic of Privacy-aware Human Activity Recognition (HAR) with Smart Glasses and explores a federated learning approach to achieve accurate recognition while preserving data privacy. The study highlights the trade-off between model complexity, recognition performance, and data privacy. Initially, a comparison was conducted between a complex deep learning network (STResNet) and a simpler deep learning network (DANA). The complex network exhibited an F1-Macro score of 0.87, while the simpler network achieved 0.82. Subsequently, the simpler network was further evaluated by comparing it to its federated learning implementation. The results revealed that the simpler network attained an F1-Macro score of 0.85, whereas federated learning achieved a slightly lower score of 0.80. These findings shed light on the performance disparities between the different approaches.

86% of children with Autism Spectrum Disorders (ASD) exhibit tactile impairments. Recent research has shown Mobile Tactile Stimulation is effective as a sensory processing therapy for children with ASD. The amount of time, duration, and intensity of such tactile stimulation depends on the appreciation of therapists and their interpretability based on experience and judgment. So, there is an untapped potential to propose an approach that can objectively quantify progress by leveraging the capabilities of Mobile Tactile Stimulation to gather continuous streams of data from touch interactions. In this position paper, we present a proposal to use Digital Phenotypes as an approach to adjust the “digital active ingredients” of tactile stimulation. We discuss our vision towards using Digital Phenotypes as a tool to follow up patient trajectories and create adaptive digital therapies. We close, discussing directions for future work.

This paper contributes a methodological framework for the design of digital therapeutics for mental health which leverages on state-of-the-art knowledge on digital twins and virtual coaching solutions to realize more effective and AI-powered digital health interventions. The paper discusses how the deployment of digital twins as computational models for predicting patient's health condition can be combined with human-centered distributed conversational modeling techniques in designing AI powered mental health interventions with key stakeholders. This approach helps to address main challenges in the design of AI-powered digital therapeutics, such as optimizing design efforts and resources, ensuring ethics soundness and transparency of decisions related to the digital treatment, as well as informing the pre-clinical validation of future digital therapeutics.

Stress, particularly occupational, is a major burden for individuals and the society, affecting the health of one worker in four. Digital interventions are a scalable and moderately effective way to tackle it. Just-in-time adaptive interventions (JITAIs), which use sensors and artificial intelligence to adapt to the user’s needs and context, appear to be a promising way to increase the effectiveness. In this paper we first describe a wearable and mobile data collection as well as stress detection experiments intended to serve as the basis for such an intervention. We then present our design for a JITAI based on cognitive behavioural therapy, and highlight a number of open questions applicable to this and similar JITAIs.

Digital Therapeutics (DTx) is an emerging field within Digital Health that is rapidly growing and holds promise for addressing anxiety-related disorders. Moreover, the combination of virtual reality (VR) and wearable sensors holds great potential for developing affordable, accessible, and effective DTx solutions. In this paper, we present our conception of a fully customised DTx for managing and supporting anxiety-related disorders therapy. We highlight five key assumptions that should be considered when developing such a DTx. These assumptions include the customisation of virtual environments (VEs); integration of physiological data and patient feedback; the use of dashboards for data visualisation; session replay and identification of peak response moments; and patient empowerment throughout the therapeutic process. By embracing these assumptions, we describe the main components constituting this DTx and emphasise fundamental features to fulfil the identified assumptions. Furthermore, we explore some of the potential benefits and challenges of DTx applied to mental health.

Recent ubiquitous sensing technologies make it possible to capture streaming digital data that reports aspects of a patient’s physiology, behavior, and symptoms both quantitatively and in real time. As a result, it may be possible to develop streaming disease readouts that are more accurate and less obtrusive than relying on patient and caregiver reports alone. This study investigates the feasibility of leveraging physiological and behavioral signals extracted from a radio frequency sensing device to characterize metrics indicative of breathing, mobility, and sleep patterns. We investigate the variations in these signals between individuals with Systemic Lupus Erythematosus (SLE) and healthy participants in a 6-months longitudinal, exploratory, in-home study involving 19 SLE and 28 healthy participants. Results show that many signals (e.g., breathing rate, sleep efficiency, and gait speed) significantly distinguish SLE and healthy participants and demonstrate the potential of using remote sensing as an unobtrusive low-burden tool to assess disease symptoms continuously and in real time.

Smartwatches are an increasingly popular technology that employs advanced sensors (e.g., location, motion, and microphone) comparable to those used by smartphones. Passive mobile sensing, a method of acquiring human behavior data from mobile and wearable devices inconspicuously, is widely used in research fields related to behavior analysis. In combination with machine learning, passive mobile sensing can be used to interpret various human and environmental contexts without requiring user intervention. Because smartwatches are always worn on the wrist, they have the potential to collect data that cannot be collected by smartphones. However, the effective use of smartwatches as platforms for passive mobile sensing poses challenges in terms of battery life, storage, and communication. To address these challenges, we designed and implemented a tailored framework for off-the-shelf smartwatches. We evaluated power consumption under eight different sensing conditions using three smartwatches. The results demonstrate that the framework can collect sensor data with a battery life of 16-31 h depending on the settings. Finally, we considered potential future solutions for optimizing power consumption in passive sensing with off-the-shelf smartwatches.

Various mobile apps have been released to track and promote mental health and well-being. Despite the high interest in developing these apps, they suffer from high attrition rates. These apps have limited utility if they are delivered in a manner that does not maintain individuals’ engagement. Engagement features are therefore a critical factor to consider for fostering intended benefits. While there is considerable research on analysing the engagement features of these apps available in English, our understanding of engagement features in such Arabic apps is limited. Moreover, much less is known about mobile sensing in Arabic apps. To address this gap, we systematically searched app stores, identified 110 apps available in Arabic, and analyzed their features based on existing mHealth assessment frameworks. Our analysis found that available Arabic apps poorly implemented engagement features, apart from basic features such as sharing and reminders. Surprisingly, Arabic apps missed mobile sensing capabilities and AI applications. This paper highlights the importance of employing mobile sensing and persuasive design principles in the future design of Arabic apps.

Mobile sensing data, approximating human behavior and physiology, can be processed by machine learning models to predict mental health symptoms. While these models are accurate in smaller samples, their generalization accuracy decreases in larger samples, potentially because it is difficult to collect enough mobile sensing and mental health outcomes data at scale to enable generalization. In this study, we hypothesized that augmenting training data with synthetic data samples could improve the generalizability of these machine learning models. We created a data augmentation system that generated synthetic mobile sensing and mental health outcomes data, and evaluated the utility of this system via the downstream machine learning task of predicting daily mood from wearable sensing data. We experimented with both simple (e.g. noise addition) and novel generative data augmentation methods, based upon conditional generative adversarial networks and multi-task learning. Our initial findings suggest that the data augmentation system generated realistic synthetic data, but did not improve mood prediction. We propose future work to validate our findings and test other methods to improve the generalizability of mental health symptom prediction models.

Digital phenotyping has been used in substance use disorder (SUD) research to predict substance consumption and monitor relevant symptoms. While various digital sensors have been utilized in SUD research, there is a lack of consideration for digital phenotypes that reflect cognitive functions. However, previous research has consistently shown the association of cognitive impairments with SUD and the positive effects of cognitive remediation in improving treatment outcomes. Given the role of cognitive functions in SUD, measuring and tracking the cognitive functions of patients can contribute to enhancing the process of intervention and treatment. Thus, this paper aims to facilitate the discussion of identifying and validating cognition-aware digital phenotyping in SUD. As a step toward this goal, this paper suggests the potential of a specific type of digital feature: keystroke dynamics. Keystroke dynamics have been found to be effective in estimating cognitive functions in various clinical domains. Future research needs to investigate if keystroke dynamics can be applied in SUD research and find other digital phenotypes that can measure the cognitive functions of patients with SUD.

This paper presents a novel approach of an ECG-based mental health biometric system that relies on ultra-short duration (2 seconds) of one-channel ECG signal segments from acute stress data for accurate user identification and authentication. The proposed method uses a simple framework for contrastive learning (SimCLR) to train the user identification and authentication models. The performance of the proposed ECG-based biometric system was evaluated for a single-session use case using an in-house dataset. The dataset consisted of ECG signals acquired during a study protocol designed to induce physical and mental stress. The proposed biometric system was able to achieve an accuracy of 98% for user identification and an equal error rate (EER) of 0.02 when trained and tested with a balanced condition with stress and baseline/recovery. Our proposed system was able to retain its accuracy to 95% and the EER to 0.05 even when the training size was significantly reduced.

Mental health conditions have become a global public health issue, especially in the context of the COVID-19 pandemic. To cope with the increasing demand for mental health services, many people have turned to smartphone applications that offer various mental health solutions, such as therapy, counseling, and self-help. However, these applications also pose significant privacy risks for their users, as they collect and share sensitive personal and health information with third parties, often without adequate consent or transparency. In this study, we examine the privacy policies of popular mental health smartphone applications using the Fair Information Practice Principles (FIPPs), a widely recognized privacy framework. Our objective is to assess the extent to which these applications adhere to the FIPPs guidelines and to identify the gaps and challenges in their privacy practices. We hope that our findings can inform and guide policy makers and application developers to design more user-centric and robust privacy policies that ensure the safety and security of users’ information.

Mental health from the pandemic generated in 2019 with SARSCOV19, has increased, likewise contributed to relapse and exacerbation of mental health symptoms in diagnosed patients. In addition, individuals with a recent diagnosis of a mental disorder were found to have a higher risk of COVID-19 infection and also a higher frequency of adverse outcomes, representing an additional risk factor for worsening mental health. The aim of this research is to develop a technological solution is the development and implementation of a web platform and a mobile application to assist and support the therapeutic work related to schizophrenia, both in its aspect of continuous assessment, as well as intervention in different areas. A mobile application is developed to be used to send the tests to the patients in order to evaluate their mental state and, in this way, to foresee possible relapses. A web application was developed for doctors to administer users and ask questions together with the consultation of test results. As result, an average of 87.11 was obtained in the SUS test of the application and 78.54 in the web test. This test evaluates usability and a score higher than 68 is considered good.

The excessive and problematic consumption of alcohol characterizes Alcohol Use Disorder (AUD). In Mexico, the 12-month prevalence of any substance is 2.5%, and the lifetime prevalence of alcohol dependence is 5.9%. Ecological Momentary Interventions (EMIs) are treatments that are provided to patients during their daily lives and in natural settings. EMIs can monitor alcohol consumption, provide personalized feedback, and deliver coping strategies to help individuals manage their cravings and avoid relapse. This work discusses challenges and opportunities associated with technology-based Ecological Momentary Interventions (EMIs) for Alcohol Use Disorder (AUD) in Mexico. We aim to contribute valuable insights into improving AUD interventions and strategies for the Mexican population, which could also provide insights into LATAM countries with similar contexts.

SolareSkin is a self-powered and ubiquitous electronic skin equipped with ultraflexible organic solar cells for visible light sensing and energy harvesting. This dual-functional system captures light signals, transforms them into electrical impulses and enables multi-class gesture and activity recognition. Its design employs a photocurrent model that allows solar cells to serve as energy harvesters and visible light sensors simultaneously. The solar cells demonstrate a decent conversion rate of incident light into electricity, supporting an efficient, sustainable operation. Additionally, the system incorporates advanced system integration with a low-powered data collection board embedded with wireless transmission modules and an intuitive user interface. An algorithm is employed for signal analysis with data pre-processing methods and several machine learning models. The data pre-processing methods comprise filtering, scaling, normalization, segmentation, and downsampling of raw sensor data to reduce noise and increase prediction accuracy. The machine learning model evaluation focuses on three algorithms: Support Vector Machines (SVM), K-Nearest Neighbors (KNN), and Random Forest, due to their efficiency with high-dimensional, nonlinear data. The experimental results suggest the excellent performance of SVM in recognizing 7-class finger gestures and 7-class body activities, with accuracies of 97.3% and 96.7%, respectively. This advancement in electronic skin technology is promising for ubiquitous human-centric sensing, enabling various applications such as healthcare monitoring, human-computer interaction, and smart homes.

In disaster scenarios, unmanned aerial vehicles (UAVs) can serve as mobile base stations because of their maneuverability and synergy. However, due to constrained UAV communication capabilities and limited battery life, UAV base stations resource allocation for mobile sensors in a data-heterogeneous environment is a significant challenge when optimizing communication quality. To address this, we propose AGUZero, an attention-based graph reinforcement learning (RL) framework. Inspired by MuZero [27], AGUZero is designed to handle dynamic and uncontrollable environments based on Monte Carlo Tree Search (MCTS). Additionally, to tackle data heterogeneity, AGUZero represents the states using heterogeneous sub-graphs and employs an attention-based model to capture relationships among UAVs and sensors. The experimental results show that AGUZero outperforms other baseline models consistently when either the number of UAVs or the number of sensors is varying. AGUZero improves the data transmission ratio by 11.03% and 10.35% in the two cases respectively.

The intricate flight patterns exhibited by birds have sparked considerable interest in many fields. However, traditional external visual observation has significant limitations, including susceptibility to line-of-sight occlusion, uneven lighting effects, and lack of continuous detection. In this paper, we design a hybrid embedded system that can perform bird flight postures recognition by integrating inertial measurement unit (IMU) with flexible strain sensor. The proposed system leverages a strain sensor based on laser-induced graphene (LIG), strategically positioned on the bird’s wing joints, in conjunction with a high-precision IMU deployed on the bird’s torso. Through a learning architecture, we achieve an impressive accuracy of 99.48% in identifying eight commonly observed bird flight postures. The integration of the proposed system offers exciting possibilities and serves as a powerful tool for advancing our understanding of the mechanisms underlying creature flight.

The landing process of the quadrotors can be affected by the disturbance from the ground effect when approaching the landing surface. Such a disturbance significantly increases the chances of collision and jittering of the quadrotors, thereby posing threats to the safety of both the quadrotors and the mounted equipment. In light of this, we propose SmoothLander, an aerodynamics and reinforcement learning-based control system to stabilize the quadrotors under the influence of the ground effect and control noise. First, we design a landing trajectory for the quadrotor in accordance with aerodynamics. Then we design a reinforcement learning-based command generator to effectively optimize the quadrotor’s landing behavior. We evaluate our control system through physical feature-based simulation and in-field experiments. The results show that our method can enable the quadrotor to land more smoothly and stably against control noise than the baseline.

Low-cost air pollution sensors (LCS) deployed on urban vehicles (e.g., taxis, buses) have emerged as a cost-effective solution for fine-grained air pollution monitoring. However, these mobile LCSs suffer from measurement drifting in real-world scenarios, necessitating a post-deployment real-time calibration. Unfortunately, the limited availability of urban real reference stations (RRS) restricts the calibration opportunities for LCSs. This paper proposes a non-rendezvous method that addresses this challenge by establishing virtual reference stations (VRS), which offer additional calibration opportunities for LCSs. Through the air pollution field reconstruction, the readings of VRSs are inferred from RRSs’ data. Furthermore, a confidence assessment mechanism is developed to quantify the uncertainty of established VRSs. Finally, a field experiment is conducted to demonstrate the effectiveness of the proposed method, showcasing a 25% improvement over the advanced baseline.

Smart grid communication networks are facing the increasing challenge of heterogeneous facilities and diverse communication requirements. Traditional communication technologies without the ability to adapt cannot meet these requirements. A data-driven and machine learning-based multi-class traffic management scheme is proposed in this study, which classifies network traffic into different service levels for better transmission provision. Numerical experiments with a real-world dataset are conducted to validate the effectiveness of the multi-class traffic management scheme, in which XGBoost achieves an accuracy of 0.9842 and an F1 score of 0.9914.

Accurate peak electrical demand forecasting plays a pivotal role in managing energy consumption in Internet Data Centers (IDCs), where electricity expense forms a major part of operational costs. To intelligently schedule energy storage for shaving the peak load and reducing both energy expense during peak hours as well as the demand charge, IDC operators need precise predictions of the magnitude and timing of daily peak electrical demand. This paper introduces a novel method for peak load forecasting that combines the strengths of both the Seasonal Autoregressive Integrated Moving Average (SARIMA) and Long Short-Term Memory (LSTM) models. Our approach is rigorously validated with the field dataset from three Tencent Inc.’s data center in North China region. The successful application of this method underscores its robustness and potential for broader application within the IDC sector and the wider power industry.

Ear-electroencephalography (EEG) holds significant promise as a practical tool in brain-computer interfaces (BCIs) due to its enhanced unobtrusiveness, comfort, and mobility in comparison to traditional steady-state visual evoked potential (SSVEP)-based BCI systems. However, achieving accurate SSVEP classification in ear-EEG faces a major challenge due to the significant attenuation and distorted amplitude of the signal. To address this challenge, this paper focuses on enhancing ear-EEG feature representations by training the model to learn feature representations similar to those of scalp-EEG. We propose a cross-domain feature distillation (CD-FD) framework, which facilitates the extraction of shared features between the two domains. This framework facilitates the identification of crucial features concealed within ear-EEG signals, leading to more effective SSVEP classification. We evaluate the proposed CD-FD framework through single-session decoding and session-to-session transfer decoding, comparing it with EEGNet and canonical correlation analysis (CCA). The results demonstrate that the proposed framework achieves the best classification results in all experiments.

The Space-Air-Ground Integrated Network (SAGIN), integrating heterogeneous devices including low earth orbit (LEO) satellites, unmanned aerial vehicles (UAVs), and ground users (GUs), holds significant promise for the advancing smart city applications. However, resource management of the SAGIN is a challenge requiring urgent study in that inappropriate resource management will cause poor data transmission, and hence affect the services in smart cities. In this paper, we develop a comprehensive SAGIN system that encompasses five distinct communication links and propose an efficient cooperative multi-type multi-agent deep reinforcement learning (CMT-MARL) method to address the resource management issue. The experimental results highlight the efficacy of proposed CMT-MARL, as evidenced by key performance indicators such as the overall transmission rate and transmission success rate. These results underscore the potential value and feasibility of future implementation of the SAGIN.

Clustered Federated Learning (FL) is a novel approach for handling data heterogeneity in FL and training personalized ML models for clients. However, existing research overlooks network constraints and objectives when implementing clustered FL in a wireless network. Since clients experience varying energy costs when connected to different servers, the cluster formation greatly impacts system energy efficiency. To address this, we present an energy-efficient client clustering problem that optimizes FL performance while minimizing energy costs. We introduce a new metric, the effective number of clients, to predict cluster FL performance based on size and data heterogeneity, guiding the clustering optimization. To find optimal client clusters, we propose an energy-efficient clustering algorithm using parallel Gibbs Sampling. Simulations and data experiments demonstrate that our algorithm achieves tunable trade-offs between data similarity (thus FL performance) and energy cost.

Federated Learning (FL) is often impeded by communication overhead issues. Prompt tuning, as a potential solution, has been introduced to only adjust a few trainable parameters rather than the whole model. However, current single-modality prompt tuning approaches fail to comprehensively portray local clients’ data. To overcome this limitation, we present Twin Prompt Federated learning (TPFL), a pioneering solution that integrates both visual and textual modalities, ensuring a more holistic representation of local clients’ data characteristics. Furthermore, in order to tackle the data heterogeneity issues, we introduce the Augmented TPFL (ATPFL) employing the contrastive learning to TPFL, which not only enhances the global knowledge acquisition of client models but also fosters the development of robust, compact models. The effectiveness of TPFL and ATPFL is substantiated by our extensive evaluations, consistently showing superior performance compared to all baselines.

Data center operators, with the aid of advanced energy storage devices, can reduce the operating cost by maintaining a long-term stable electrical consumption curve. Therefore, accurate electrical consumption forecasting is crucial to enable effective data center electricity systems. In this paper, we develop a data-driven forecasting model for short-term data center electrical consumption. Through examining the results of the traditional statistical model, Seasonal Auto-Regressive Integrated Moving Average (SARIMA), and the deep learning model, Long Short-Term Memory Neural Network (LSTM), we combine them to construct a hybrid model which outperforms each single one of the two models and shows good mobility. The overall performance of our models has been validated on data center electrical consumption data provided by Tencent Inc.

Modern Visual-Based Tactile Sensors (VBTSs) use cost-effective cameras to track elastomer deformation, but struggle with ambient light interference. Solutions typically involve using internal LEDs and blocking external light, thus adding complexity. Creating a VBTS resistant to ambient light with just a camera and an elastomer remains a challenge. In this work, we introduce WStac, a self-illuminating VBTS comprising a mechanoluminescence (ML) whisker elastomer, camera, and 3D printed parts. The ML whisker elastomer, inspired by the touch sensitivity of vibrissae, offers both light isolation and high ML intensity under stress, thereby removing the necessity for additional LED modules. With the incorporation of machine learning, the sensor effectively utilizes the dynamic contact variations of 25 whiskers to successfully perform tasks like speed regression, directional identification, and texture classification. Videos are available at: https://sites.google.com/view/wstac/.

Accurate calibration of low-cost sensors is critical for improving their potential in environmental monitoring. State-of-the-art (SOTA) methods based on supervised learning commonly calibrate sensor measurements with ground truth from the immediate past or future. However, these techniques rely heavily on labeled data which is challenging to obtain in real-world scenarios. Thus, this paper introduces CaliFormer, a novel representation learning model using self-supervised learning to extract time- and spatial-invariant knowledge from unlabeled measurements. Moreover, we propose pre-training enhancements and model architecture modifications to help train CaliFormer. We then fine-tune the calibration model with the learned representations, which is supervised by limited labeled data. Finally, we comprehensively evaluate our calibration model with a dataset collected by low-cost sensors. Results show that our model outperforms other SOTA calibration methods significantly.

UbiComp/ISWC 2023 is featuring a special program called ASSET (Americas Student Symposium on Emerging Technologies) as part of SIGMOBILE’s student outreach program. Student participating to ASSET will be divided into small groups and each group will be provided with an excellent and dynamic research mentor who will help them hone their research skills.

We propose a symposium to foster research on inclusive and equitable ubiquitous computing in Latin America (LATAM). The meeting is inspired by the ACM SIGCHI Across Borders Initiative, and the 1st CSCW@LatAm Research Catalyst Workshop. We aim to: 1) discuss opportunities and challenges of defining and accomplishing inclusive and equitable ubiquitous computing centered on LATAM; 2) collaboratively mentor emerging related projects focused on LATAM; 3) create a space that encourages cross-country collaborations to address together common research challenges, whilst also strengthening the links among all the members, regardless of their level of expertise in the field; and, 4) provide a forum for dialogue between local communities in Latin America (e.g., educational and health workers, NGOs) and UbiComp researchers, with the aim of identifying pathways for codesign-focused collaborations between diverse groups of stakeholders on problems of public interest.

Data annotation is key to a large number of fields, including ubiquitous computing. Documenting the quality and extent of annotation is increasingly recognised as an important aspect of understanding the validity, biases and limitations of systems built using this data: hence, it is also relevant to regulatory and compliance needs and outcomes. However, the process of annotation often receives little attention, and is characterised in the literature as “under-described” and “invisible work”. In this tutorial, we bring together existing resources and methods to present a framework for the iterative development and evaluation of an annotation protocol, from requirements gathering, setting scope, development, documentation, piloting and evaluation, through to scaling-up annotation processes for a production annotation process. We also explore the potential of semi-supervised approaches and state-of-the-art methods such as the use of generative AI in supporting annotation workflows, and how such approaches are validated and their strengths and weaknesses characterised. This tutorial is designed to be suitable for people from a wide range of backgrounds, as annotation can be understood as a highly interdisciplinary task and often requires collaboration with subject matter experts from relevant fields. Participants will trial and evaluate a selection of annotation interfaces and walk through the process of evaluating the outcomes. By the end of the workshop, participants will develop a deeper understanding of the task of developing an annotation protocol and aspects of the requirements and context which should be taken into account.

Presentations and code from this event will be shared openly on a Github repository.

Feature extraction lies at the core of Human Activity Recognition (HAR): the automated inference of what activity is being performed. Traditionally, the HAR community used statistical metrics and distribution-based representations to summarize the movement present in windows of sensor data into feature vectors. More recently, learned representations have been used successfully in lieu of such handcrafted and manually engineered features. In particular, the community has shown substantial interest in self-supervised methods, which leverage large-scale unlabeled data to first learn useful representations that are subsequently fine-tuned to the target applications. In this tutorial, we focus on representations for single-sensor and multi-modal setups, and go beyond the current de facto of learning representations. We also discuss the economic use of existing representations, specifically via transfer learning and domain adaptation. The proposed tutorial will introduce state-of-the-art methods for representation learning in HAR, and provide a forum for researchers from mobile and ubiquitous computing to not only discuss the current state of the field but to also chart future directions for the field itself, including answering what it would take to finally solve the activity recognition problem.

A central research goal of Ubicomp has always been the development of systems and methods that seamlessly support humans in accomplishing complex tasks in everyday life. In the wake of rapid advances in artificial intelligence (AI), topics such as "Human-Centered AI" and "Hybrid Human AI" are showing a growing interest in this very research that puts us humans and our needs at the center of artificial intelligence. While methods for augmenting the human body and the impact of these augmentations on human physical life are being extensively researched, there has been very limited progress in evaluating the impact on human cognitive perception and its impact on the overall outcome of augmentations to the human body. In this tutorial, we will address the question of how to evaluate the cognitive impact of human augmentation. We will address the different levels of cognitive effects, how to measure which methods of augmentation have the best effect, and which cognitive measures have the greatest impact on augmentation, and we will give the audience the opportunity to test and evaluate cognitive human augmentation systems themselves.

Over the last couple of years, there has been a big push toward making immersive and mixed technologies available to the general public. Yet, designing for these new technologies is challenging, as users need to position virtual objects in 3D space. The current state-of-the art technologies used to access these virtual environments (e.g., Head-mount displays (HMD)s also presents additional challenges for designers when considering depth perception issues that affect user precision. Moreover, these challenges are exacerbated when designers consider accessibility needs of special populations. To make new immersive and mixed technologies more accessible, we propose a tutorial at UbiComp / ISWC 2023 to discuss design strategies, research methodologies, and implementation practices with special populations using technologies across the physical-virtual reality spectrum. In this tutorial participants will learn how to make these technologies more accessible by (1) teaching students of the tutorial how to design, prototype, and evaluate these technologies using empirical research. We aim to (2) bring together researchers, practitioners, and students who are interested in making immersive and mixed technologies more accessible and (3) identify common problems when designing new user interfaces.

Rapid technological advances are expanding the practical applicability of virtual reality and/or augmented reality (VR/AR); however, the degree to which new users are able to interact with these technologies is limited by current input modalities. Gaining an intuitive grasp of VR/AR applications requires that users be immersed in the virtual environment, which in turn depends on the integration of multiple realistic sensory feedback mechanisms. This workshop will bring together researchers from the fields of UbiComp and VR/AR to explore alternative input modalities and sensory feedback systems to facilitate the design of coherent and engaging VR/AR experiences comparable to those in the real world.

The recognition of complex and subtle human behaviors from wearable sensors will enable next-generation human-oriented computing in scenarios of high societal value (e.g., dementia care). This will require a large-scale human activity corpus and much-improved methods to recognize activities and the context in which they occur. This workshop deals with the challenges of designing reproducible experimental setups, running large-scale dataset collection campaigns, designing activity and context recognition methods that are robust and adaptive, and evaluating systems in the real world. We wish to reflect on future methods, such as lifelong learning approaches that allow open-ended activity recognition. This year HASCA will welcome papers from participants to the Fifth Sussex-Huawei Locomotion and Transportation Recognition Challenge in a special session.

This paper explores the intersection of Artificial Intelligence and Machine Learning (AI/ML) fairness and mobile human-computer interaction (MobileHCI). Through a comprehensive analysis of MobileHCI proceedings published between 2017 and 2022, we first aim to understand the current state of algorithmic fairness in the community. By manually analyzing 90 papers, we found that only a small portion (5%) thereof adheres to modern fairness reporting, such as analyses conditioned on demographic breakdowns. At the same time, the overwhelming majority draws its findings from highly-educated, employed, and Western populations. We situate these findings within recent efforts to capture the current state of algorithmic fairness in mobile and wearable computing, and envision that our results will serve as an open invitation to the design and development of fairer ubiquitous technologies.

Mental health and well-being are critical components of overall health: suffering from a mental illness can be both debilitating and life-threatening for individuals experiencing symptoms. Detecting symptoms of mental illness early-on and delivering interventions to prevent and/or manage symptoms can improve health and well-being outcomes. Ubiquitous systems are increasingly playing a central role in uncovering clinically relevant contextual information on mental health. Research shows that these systems can passively measure symptoms and enable opportunities to deliver intervention. However, despite this potential, the uptake of ubiquitous technologies into mental healthcare has been slow, and a number of challenges need to be addressed towards the effective implementation of these tools. The goal of this workshop is to bring together researchers, practitioners, and industry professionals interested in identifying, articulating, and addressing such issues and opportunities. Following the success of this workshop for the last seven years, we aim to continue facilitating the UbiComp community in both the conceptualization, translation, and implementation of novel approaches for sensing and intervention in the context of mental health.

As in prescribable medical drug-based therapies, Digital Therapeutics (DTx) solutions introduce the use of software as 1) an active ingredient implemented as digital interventions to improve patients’ condition; and 2) as the excipient through which the intervention is conveyed to the patient. The most common DTx solutions implemented until today deal with the use of software-based solutions for dealing with mental health conditions. However, the potential of DTx is envisioned to grow to treat other health conditions with a number of technologies that still have not been duly explored. The objective of this workshop is to discuss the opportunities, barriers, and challenges of future DTx, the envisioned future use-cases, and the technologies with high potential to drive the uptake of these technologies, and this way to position this field as a solid complement of today's therapeutic approaches for dealing with different health conditions.

With the advancements in ubiquitous computing, ubicomp technology has deeply spread into our daily lives, including office work, home and house-keeping, health management, transportation, or even urban living environments. Furthermore, beyond the initial metrics commonly applied in computing, such as “efficiency” and “productivity”, the benefits that people (users) get from well-being-aware ubiquitous technology have been greatly emphasized in the recent years. Through the sixth “WellComp” (Computing for Well-being) workshop, we will discuss and debate the contribution of ubiquitous computing towards users’ well-being covering physical, mental, and social wellness (and the combinations thereof), from the viewpoints of various different layers of computing. Organized by a diverse international team of ubicomp researchers, WellComp 2023 will bring together researchers and practitioners from both academia and industry to explore versatile topics related to well-being and ubiquitous computing.

With the proliferation of connected devices with advanced sensing, computing, and communication capabilities, ubiquitous computing systems have become prevalent nowadays. They have the potential to revolutionize various industries by enabling new applications and services (e.g., patient monitoring, personalized recommendations, traffic control, home energy management). However, in real-world ubiquitous computing systems, data collection can be expensive or impossible. Due to the limited quantity and quality of data available, pure data-driven methods may not perform well. A promising approach to overcome these limitations is to utilize physical knowledge, including domain knowledge from experts, heuristics based on experience, and analytic models of physical phenomena.

The theme of this workshop is to advance the theoretical understanding, algorithmic development and system implementations of ubiquitous computing systems that integrate physical knowledge with data-driven methods. The workshop will provide an inclusive gathering for researchers and practitioners from various fields and facilitate future collaborations. The workshop welcomes research papers as well as position papers. The overall goal is to grow the community who are dedicated to improving ubiquitous computing systems by fusing physical knowledge into data-driven methods.

The objective of the 4th ACM International Workshop on Earable Computing (EarComp 2023) is to provide an academic forum and bring together researchers, practitioners, and design experts to discuss how sensory earables technologies have and can complement human sensing research. It also aims to provide a launchpad for bold and visionary ideas and serve as a catalyst for advancements in this emerging new Earable Computing research space.

IoT products are increasingly becoming the default, with non-IoT versions of common hardware (e.g., TVs and printers) harder to find. Alongside this adoption surge, lack of support, outdated security, and planned obsolescence present concerning sustainability issues, contribute to eWaste growth and widen digital divides globally. This workshop aims to present and discuss legal, social, technical, and design aspects of repair practices, engaging the Ubicomp community in exploring challenges and opportunities for more repairable IoT devices. Focusing on diverse repair scenarios, the workshop seeks to establish a concise, holistic, and inclusive agenda for this research domain's future. Participants will map key research questions to support the movement towards more repairable technology.