Shardul Sapkota

Photo Credit: Angad Srivastava

I am a second year Computer Science PhD student at Stanford University, where I am advised by Professors James Landay and Scott Delp.

My research focuses on designing sensing systems and interfaces for health and wellbeing. I am currently working on developing new approaches to unobtrusively monitor health by understanding how people interact with their everyday environment (e.g., using computer vision to perform fall risk assessment).

Before Stanford, I was a Senior Machine Learning Engineer at Shopee, where I worked with the team responsible for training neural networks for its advertisement and recommender systems. Prior to joining Shopee, I was fortunate to have been able to study and conduct research on Human-Computer Interaction at the NUS-HCI Lab with Professor Shengdong Zhao, the Augmented Human Lab (Auckland Bioengineering Institute) with Professor Suranga Nanayakkara, and the Fluid Interfaces Group (MIT Media Lab) with Tomás Vega. I received my undergraduate degree from Yale-NUS College, where my thesis focused on using physiological signals to detect changes in people’s attentional states in real-time.

publications

Supporting Physical Activity Behavior Change with LLM-Based Conversational Agents

Matthew Jörke, Shardul Sapkota, Lyndsea Warkenthien, Niklas Vainio, Paul Schmiedmayer, Emma Brunskill, and James Landay

arXiv preprint arXiv:2405.06061 2024

Abstract DOI PDF

Physical activity has significant benefits to health, yet large portions of the population remain physically inactive. Mobile health applications show promising potential for low-cost, scalable physical activity promotion, but existing approaches are often insufficiently personalized to a user’s context and life circumstances. In this work, we explore the potential for large language model (LLM) based conversational agents to motivate physical activity behavior change. Through formative interviews with 12 health professionals and 10 non-experts, we identify design considerations and opportunities for LLM health coaching. We present GPTCoach, a chatbot that implements an evidence-based health coaching program, uses counseling strategies from motivational interviewing, and can query and visualize health data from a wearable through tool use. We evaluate GPTCoach as a technology probe in a user study with 16 participants. Through quantitive and qualitative analyses, we find promising evidence that GPTCoach can adhere to a health coaching program while adopting a facilitative, supportive, and non-judgmental tone. We find more variable support for GPTCoach’s ability to proactively make use of data in ways that foster motivation and empowerment. We conclude with a discussion of our findings, implications for future research, as well as risks and limitations.

Supporting Physical Activity Behavior Change with LLM-Based Conversational Agents

Matthew Jörke, Shardul Sapkota, Lyndsea Warkenthien, Niklas Vainio, Paul Schmiedmayer, Emma Brunskill, and James Landay

arXiv preprint arXiv:2405.06061 2024

Abstract DOI PDF

Physical activity has significant benefits to health, yet large portions of the population remain physically inactive. Mobile health applications show promising potential for low-cost, scalable physical activity promotion, but existing approaches are often insufficiently personalized to a user’s context and life circumstances. In this work, we explore the potential for large language model (LLM) based conversational agents to motivate physical activity behavior change. Through formative interviews with 12 health professionals and 10 non-experts, we identify design considerations and opportunities for LLM health coaching. We present GPTCoach, a chatbot that implements an evidence-based health coaching program, uses counseling strategies from motivational interviewing, and can query and visualize health data from a wearable through tool use. We evaluate GPTCoach as a technology probe in a user study with 16 participants. Through quantitive and qualitative analyses, we find promising evidence that GPTCoach can adhere to a health coaching program while adopting a facilitative, supportive, and non-judgmental tone. We find more variable support for GPTCoach’s ability to proactively make use of data in ways that foster motivation and empowerment. We conclude with a discussion of our findings, implications for future research, as well as risks and limitations.
OpenCap Monocular: Human Movement Dynamics from a Single Smartphone Video

Scott D. Uhlrich, Shardul Sapkota, Antoine Falisse, and Scott L. Delp

American Society of Biomechanics Oral Abstract 2024

Abstract DOI PDF

The ability to easily measure the kinematics and kinetics of human movement could improve the prevention and treatment of neurological and musculoskeletal diseases. The time and expertise required to measure movement with marker-based motion capture has limited its adoption in the clinic and in large-scale research studies. We recently developed OpenCap, open-source software that estimates kinematics and kinetics using two smartphone videos [1]. OpenCap and other markerless motion capture technologies [2] lower the time and cost barriers to movement analysis by orders of magnitude [1]; however, they require multiple cameras. If we can achieve similar accuracy from a single video, the >4 billion smartphone owners around the world would have access to 3D motion capture. The computer vision field has made advances in estimating the global pose of a human mesh from a single video [3], but these algorithms are not designed for or evaluated on their biomedical utility. This study aims to develop and evaluate a pipeline for estimating kinematics and kinetics of common human movements (walking and squatting) from a single video.

OpenCap Monocular: Human Movement Dynamics from a Single Smartphone Video

Scott D. Uhlrich, Shardul Sapkota, Antoine Falisse, and Scott L. Delp

American Society of Biomechanics Oral Abstract 2024

Abstract DOI PDF

The ability to easily measure the kinematics and kinetics of human movement could improve the prevention and treatment of neurological and musculoskeletal diseases. The time and expertise required to measure movement with marker-based motion capture has limited its adoption in the clinic and in large-scale research studies. We recently developed OpenCap, open-source software that estimates kinematics and kinetics using two smartphone videos [1]. OpenCap and other markerless motion capture technologies [2] lower the time and cost barriers to movement analysis by orders of magnitude [1]; however, they require multiple cameras. If we can achieve similar accuracy from a single video, the >4 billion smartphone owners around the world would have access to 3D motion capture. The computer vision field has made advances in estimating the global pose of a human mesh from a single video [3], but these algorithms are not designed for or evaluated on their biomedical utility. This study aims to develop and evaluate a pipeline for estimating kinematics and kinetics of common human movements (walking and squatting) from a single video.
AddBiomechanics Dataset: Capturing the Physics of Human Motion at Scale

Keenon Werling, Janelle Kaneda, Alan Tan, Rishi Agarwal, Six Skov, Tom Van Wouwe, Scott Uhlrich, Nicholas Bianco, Carmichael Ong, Antoine Falisse, Shardul Sapkota, Aidan Chandra, Joshua Carter, Ezio Preatoni, Benjamin Fregly, Jennifer Hicks, Scott L. Delp, and C. Karen Liu

European Conference on Computer Vision 2024

Abstract DOI PDF

While reconstructing human poses in 3D from inexpensive sensors has advanced significantly in recent years, quantifying the dynamics of human motion, including the muscle-generated joint torques and external forces, remains a challenge. Prior attempts to estimate physics from reconstructed human poses have been hampered by a lack of datasets with high-quality pose and force data for a variety of movements. We present the AddBiomechanics Dataset 1.0, which includes physically accurate human dynamics of 273 human subjects, over 70 hours of motion and force plate data, totaling more than 24 million frames. To construct this dataset, novel analytical methods were required, which are also reported here. We propose a benchmark for estimating human dynamics from motion using this dataset, and present several baseline results. The AddBiomechanics Dataset is publicly available at addbiomechanics.org/download data.html.

AddBiomechanics Dataset: Capturing the Physics of Human Motion at Scale

Keenon Werling, Janelle Kaneda, Alan Tan, Rishi Agarwal, Six Skov, Tom Van Wouwe, Scott Uhlrich, Nicholas Bianco, Carmichael Ong, Antoine Falisse, Shardul Sapkota, Aidan Chandra, Joshua Carter, Ezio Preatoni, Benjamin Fregly, Jennifer Hicks, Scott L. Delp, and C. Karen Liu

European Conference on Computer Vision 2024

Abstract DOI PDF

While reconstructing human poses in 3D from inexpensive sensors has advanced significantly in recent years, quantifying the dynamics of human motion, including the muscle-generated joint torques and external forces, remains a challenge. Prior attempts to estimate physics from reconstructed human poses have been hampered by a lack of datasets with high-quality pose and force data for a variety of movements. We present the AddBiomechanics Dataset 1.0, which includes physically accurate human dynamics of 273 human subjects, over 70 hours of motion and force plate data, totaling more than 24 million frames. To construct this dataset, novel analytical methods were required, which are also reported here. We propose a benchmark for estimating human dynamics from motion using this dataset, and present several baseline results. The AddBiomechanics Dataset is publicly available at addbiomechanics.org/download data.html.
Can Icons Outperform Text? Understanding the Role of Pictograms in OHMD Notifications

Nuwan Nanayakkarawasam Peru Kandage Janaka, Shengdong Zhao, and Shardul Sapkota

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems 2023

Abstract DOI PDF

Optical see-through head-mounted displays (OHMDs) can provide just-in-time digital assistance to users while they are engaged in ongoing tasks. However, given users’ limited attentional resources when multitasking, there is a need to concisely and accurately present information in OHMDs. Existing approaches for digital information presentation involve using either text or pictograms. While pictograms have enabled rapid recognition and easier use in warning messages and trafc signs, most studies using pictograms for digital notifcations have exhibited unfavorable results. We thus conducted a series of four iterative studies to understand how we can support efective notifcation presentation on OHMDs during multitasking scenarios. We fnd that while icon-augmented notifcations can outperform text-only notifcations, their efectiveness depends on icon familiarity, encoding density, and environmental brightness. We reveal design implications when using iconaugmented notifcations in OHMDs and present plausible reasons for the observed disparity in literature

Can Icons Outperform Text? Understanding the Role of Pictograms in OHMD Notifications

Nuwan Nanayakkarawasam Peru Kandage Janaka, Shengdong Zhao, and Shardul Sapkota,

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems 2023

Abstract DOI PDF

Optical see-through head-mounted displays (OHMDs) can provide just-in-time digital assistance to users while they are engaged in ongoing tasks. However, given users’ limited attentional resources when multitasking, there is a need to concisely and accurately present information in OHMDs. Existing approaches for digital information presentation involve using either text or pictograms. While pictograms have enabled rapid recognition and easier use in warning messages and trafc signs, most studies using pictograms for digital notifcations have exhibited unfavorable results. We thus conducted a series of four iterative studies to understand how we can support efective notifcation presentation on OHMDs during multitasking scenarios. We fnd that while icon-augmented notifcations can outperform text-only notifcations, their efectiveness depends on icon familiarity, encoding density, and environmental brightness. We reveal design implications when using iconaugmented notifcations in OHMDs and present plausible reasons for the observed disparity in literature
Ubiquitous Interactions for Heads-Up Computing: Understanding Users’ Preferences for Subtle Interaction Techniques in Everyday Settings

Shardul Sapkota*, Ashwin Ram*, and Shengdong Zhao

23rd International Conference on Mobile Human-Computer Interaction (MobileHCI’21) 2021

*Denotes equal contribution

Abstract PDF

In order to satisfy users’ information needs while incurring minimum interference to their ongoing activities, previous studies have proposed using Optical Head-mounted Displays (OHMDs) with different input techniques. However, it is unclear how these techniques compare against one another in terms of being comfortable and non-intrusive to a user’s everyday tasks. Through a wizard-of-oz study, we thus compared four subtle interaction techniques (feet, arms, thumb-index-fingers, and teeth) in three daily hands-busy tasks under different settings (giving a presentation–sitting, carrying bags–walking, and folding clothes–standing). We found that while each interaction technique has its niche, thumb-index-finger interaction has the best overall balance and is most preferred as a cross-scenario subtle interaction technique for smart glasses. We provide further evaluation of thumb-index-finger interaction with an in-the-wild study with 8 users. Our results contribute to an enhanced understanding of user preferences for subtle interaction techniques with smart glasses for everyday use.

Ubiquitous Interactions for Heads-Up Computing: Understanding Users’ Preferences for Subtle Interaction Techniques in Everyday Settings

Shardul Sapkota*, Ashwin Ram*, and Shengdong Zhao

23rd International Conference on Mobile Human-Computer Interaction (MobileHCI’21) 2021

*Denotes equal contribution

Abstract PDF

In order to satisfy users’ information needs while incurring minimum interference to their ongoing activities, previous studies have proposed using Optical Head-mounted Displays (OHMDs) with different input techniques. However, it is unclear how these techniques compare against one another in terms of being comfortable and non-intrusive to a user’s everyday tasks. Through a wizard-of-oz study, we thus compared four subtle interaction techniques (feet, arms, thumb-index-fingers, and teeth) in three daily hands-busy tasks under different settings (giving a presentation–sitting, carrying bags–walking, and folding clothes–standing). We found that while each interaction technique has its niche, thumb-index-finger interaction has the best overall balance and is most preferred as a cross-scenario subtle interaction technique for smart glasses. We provide further evaluation of thumb-index-finger interaction with an in-the-wild study with 8 users. Our results contribute to an enhanced understanding of user preferences for subtle interaction techniques with smart glasses for everyday use.
EyeKnowYou: A DIY Toolkit to Support Monitoring Cognitive Load and Actual Screen Time using a Head-Mounted Webcam

Tharindu Kaluarachchi, Shardul Sapkota, Jules Taradel, Aristée Thevenon, Denys J.C. Matthies, and Suranga Nanayakkara

Extended Abstracts of the 23rd International Conference on Mobile Human-Computer Interaction (MobileHCI’21) 2021

Abstract PDF

Studies show that frequent screen exposure and increased cognitive load can cause mental-health issues. Although expensive systems capable of detecting cognitive load and timers counting on-screen time exist, literature has yet to demonstrate measuring both fac- tors across devices. To address this, we propose an inexpensive DIY-approach using a single head-mounted webcam capturing the user’s eye. By classifying camera feed using a 3D Convolutional Neural Network, we can determine increased cognitive load and actual screen time. This works because the camera feed contains corneal surface reflection, as well as physiological parameters that contain information on cognitive load. Even with a small data set, we were able to develop generalised models showing 70% accuracy. To increase the models’ accuracy, we seek the community’s help by contributing more raw data. Therefore, we provide an opensource software and a DIY-guide to make our toolkit accessible to human factors researchers without an engineering background.

EyeKnowYou: A DIY Toolkit to Support Monitoring Cognitive Load and Actual Screen Time using a Head-Mounted Webcam

Tharindu Kaluarachchi, Shardul Sapkota, Jules Taradel, Aristée Thevenon, Denys J.C. Matthies, and Suranga Nanayakkara

Extended Abstracts of the 23rd International Conference on Mobile Human-Computer Interaction (MobileHCI’21) 2021

Abstract PDF

Studies show that frequent screen exposure and increased cognitive load can cause mental-health issues. Although expensive systems capable of detecting cognitive load and timers counting on-screen time exist, literature has yet to demonstrate measuring both fac- tors across devices. To address this, we propose an inexpensive DIY-approach using a single head-mounted webcam capturing the user’s eye. By classifying camera feed using a 3D Convolutional Neural Network, we can determine increased cognitive load and actual screen time. This works because the camera feed contains corneal surface reflection, as well as physiological parameters that contain information on cognitive load. Even with a small data set, we were able to develop generalised models showing 70% accuracy. To increase the models’ accuracy, we seek the community’s help by contributing more raw data. Therefore, we provide an opensource software and a DIY-guide to make our toolkit accessible to human factors researchers without an engineering background.
Moment-to-Moment Continuous Attention Fluctuation Monitoring through Consumer-Grade EEG Device

Shan Zhang*, Zihan Yan*, Shardul Sapkota, Shengdong Zhao, and Wei Tsang Ooi

Sensors 2021

*Denotes equal contribution

Abstract DOI PDF

While numerous studies have explored using various sensing techniques to measure attention states, moment-to-moment attention fluctuation measurement is unavailable. To bridge this gap, we applied a novel paradigm in psychology, the gradual-onset continuous performance task (gradCPT), to collect the ground truth of attention states. GradCPT allows for the precise labeling of attention fluctuation on an 800 ms time scale. We then developed a new technique for measuring continuous attention fluctuation, based on a machine learning approach that uses the spectral properties of EEG signals as the main features. We demonstrated that, even using a consumer grade EEG device, the detection accuracy of moment-to-moment attention fluctuations was 73.49%. Next, we empirically validated our technique in a video learning scenario and found that our technique match with the classification obtained through thought probes, with an average F1 score of 0.77. Our results suggest the effectiveness of using gradCPT as a ground truth labeling method and the feasibility of using consumer-grade EEG devices for continuous attention fluctuation detection.

Moment-to-Moment Continuous Attention Fluctuation Monitoring through Consumer-Grade EEG Device

Shan Zhang*, Zihan Yan*, Shardul Sapkota, Shengdong Zhao, and Wei Tsang Ooi

Sensors 2021

*Denotes equal contribution

Abstract DOI PDF

While numerous studies have explored using various sensing techniques to measure attention states, moment-to-moment attention fluctuation measurement is unavailable. To bridge this gap, we applied a novel paradigm in psychology, the gradual-onset continuous performance task (gradCPT), to collect the ground truth of attention states. GradCPT allows for the precise labeling of attention fluctuation on an 800 ms time scale. We then developed a new technique for measuring continuous attention fluctuation, based on a machine learning approach that uses the spectral properties of EEG signals as the main features. We demonstrated that, even using a consumer grade EEG device, the detection accuracy of moment-to-moment attention fluctuations was 73.49%. Next, we empirically validated our technique in a video learning scenario and found that our technique match with the classification obtained through thought probes, with an average F1 score of 0.77. Our results suggest the effectiveness of using gradCPT as a ground truth labeling method and the feasibility of using consumer-grade EEG devices for continuous attention fluctuation detection.
Prompto: Investigating Receptivity to Prompts Based on Cognitive Load from Memory Training Conversational Agent

Samantha WT Chan, Shardul Sapkota, Rebecca Mathews, Haimo Zhang, and Suranga Nanayakkara

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2020

Abstract DOI PDF

Prospective memory lapses, which involve forgetting to perform intended actions, affect independent living in older adults. Although memory training using smartphone applications could address them, users are sometimes unaware of available times for training or forget about it, presenting a need for proactive prompts. Existing applications mostly provide time-based prompts and prompts based on users’ cognitive contexts remain an under-explored area. We developed Prompto, a conversational memory coach that detects physiological signals to suggest training sessions when users are relaxed and potentially more receptive. Our study with 21 older adults showed that users were more receptive to prompts and memory training under low cognitive load than under high cognitive load. Interviews and an in-the-wild deployment of Prompto indicated that majority of users appreciated the concept, found it helpful and were likely to respond to its prompts. We contribute towards developing technologies with cognitive context-aware prompting based on users’ physiological readings.

Prompto: Investigating Receptivity to Prompts Based on Cognitive Load from Memory Training Conversational Agent

Samantha WT Chan, Shardul Sapkota, Rebecca Mathews, Haimo Zhang, and Suranga Nanayakkara

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2020

Abstract DOI PDF

Prospective memory lapses, which involve forgetting to perform intended actions, affect independent living in older adults. Although memory training using smartphone applications could address them, users are sometimes unaware of available times for training or forget about it, presenting a need for proactive prompts. Existing applications mostly provide time-based prompts and prompts based on users’ cognitive contexts remain an under-explored area. We developed Prompto, a conversational memory coach that detects physiological signals to suggest training sessions when users are relaxed and potentially more receptive. Our study with 21 older adults showed that users were more receptive to prompts and memory training under low cognitive load than under high cognitive load. Interviews and an in-the-wild deployment of Prompto indicated that majority of users appreciated the concept, found it helpful and were likely to respond to its prompts. We contribute towards developing technologies with cognitive context-aware prompting based on users’ physiological readings.
Byte. it: Discreet Teeth Gestures for Mobile Device Interaction

Tomás Vega Gálvez, Shardul Sapkota, Alexandru Dancu, and Pattie Maes

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems 2019

Abstract DOI PDF

Byte.it is an exploration of the feasibility of using miniaturized, discreet hardware for teeth-clicking as hands-free input for wearable computing. Prior work has been able to identify teeth-clicking of different teeth groups. Byte.it expands on this work by exploring the use of a smaller and more discreetly positioned sensor suite (accelerometer and gyroscope) for detecting four different teeth-clicks for everyday human-computer interaction. Initial results show that an unobtrusive position on the lower mastoid and mandibular condyle can be used to classify teeth-clicking of four different teeth groups with an accuracy of 89%.

Byte. it: Discreet Teeth Gestures for Mobile Device Interaction

Tomás Vega Gálvez, Shardul Sapkota, Alexandru Dancu, and Pattie Maes

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems 2019

Abstract DOI PDF

Byte.it is an exploration of the feasibility of using miniaturized, discreet hardware for teeth-clicking as hands-free input for wearable computing. Prior work has been able to identify teeth-clicking of different teeth groups. Byte.it expands on this work by exploring the use of a smaller and more discreetly positioned sensor suite (accelerometer and gyroscope) for detecting four different teeth-clicks for everyday human-computer interaction. Initial results show that an unobtrusive position on the lower mastoid and mandibular condyle can be used to classify teeth-clicking of four different teeth groups with an accuracy of 89%.