Shardul Sapkota

I am a first year Computer Science PhD student at Stanford University, currently working with Professor James Landay. I spent the previous quarter doing a rotation with Professor Scott Delp.

Before Stanford, I was a Senior Machine Learning Engineer at Shopee, where I worked with the team responsible for training neural networks for its advertisement and recommender systems. Prior to joining Shopee, I was fortunate to have been able to study and conduct research on Human-Computer Interaction at the NUS-HCI Lab, the Augmented Human Lab (Auckland Bioengineering Institute), and the Fluid Interfaces Group (MIT Media Lab). I received my undergraduate degree from Yale-NUS College, where my thesis focused on using physiological signals to detect changes in people’s attentional states in real-time.

I am passionate about building large-scale and accessible tools that detect and respond to subtle changes in cognitive state and social context. These technologies often enable novel applications in health, learning, and productivity.

publications

Ubiquitous Interactions for Heads-Up Computing: Understanding Users’ Preferences for Subtle Interaction Techniques in Everyday Settings

Shardul Sapkota*, Ashwin Ram*, and Shengdong Zhao

23rd International Conference on Mobile Human-Computer Interaction (MobileHCI’21) 2021

*Denotes equal contribution

Abstract PDF

In order to satisfy users’ information needs while incurring minimum interference to their ongoing activities, previous studies have proposed using Optical Head-mounted Displays (OHMDs) with different input techniques. However, it is unclear how these techniques compare against one another in terms of being comfortable and non-intrusive to a user’s everyday tasks. Through a wizard-of-oz study, we thus compared four subtle interaction techniques (feet, arms, thumb-index-fingers, and teeth) in three daily hands-busy tasks under different settings (giving a presentation–sitting, carrying bags–walking, and folding clothes–standing). We found that while each interaction technique has its niche, thumb-index-finger interaction has the best overall balance and is most preferred as a cross-scenario subtle interaction technique for smart glasses. We provide further evaluation of thumb-index-finger interaction with an in-the-wild study with 8 users. Our results contribute to an enhanced understanding of user preferences for subtle interaction techniques with smart glasses for everyday use.

Ubiquitous Interactions for Heads-Up Computing: Understanding Users’ Preferences for Subtle Interaction Techniques in Everyday Settings

Shardul Sapkota*, Ashwin Ram*, and Shengdong Zhao

23rd International Conference on Mobile Human-Computer Interaction (MobileHCI’21) 2021

*Denotes equal contribution

Abstract PDF

In order to satisfy users’ information needs while incurring minimum interference to their ongoing activities, previous studies have proposed using Optical Head-mounted Displays (OHMDs) with different input techniques. However, it is unclear how these techniques compare against one another in terms of being comfortable and non-intrusive to a user’s everyday tasks. Through a wizard-of-oz study, we thus compared four subtle interaction techniques (feet, arms, thumb-index-fingers, and teeth) in three daily hands-busy tasks under different settings (giving a presentation–sitting, carrying bags–walking, and folding clothes–standing). We found that while each interaction technique has its niche, thumb-index-finger interaction has the best overall balance and is most preferred as a cross-scenario subtle interaction technique for smart glasses. We provide further evaluation of thumb-index-finger interaction with an in-the-wild study with 8 users. Our results contribute to an enhanced understanding of user preferences for subtle interaction techniques with smart glasses for everyday use.
EyeKnowYou: A DIY Toolkit to Support Monitoring Cognitive Load and Actual Screen Time using a Head-Mounted Webcam

Tharindu Kaluarachchi, Shardul Sapkota, Jules Taradel, Aristée Thevenon, Denys J.C. Matthies, and Suranga Nanayakkara

Extended Abstracts of the 23rd International Conference on Mobile Human-Computer Interaction (MobileHCI’21) 2021

Abstract PDF

Studies show that frequent screen exposure and increased cognitive load can cause mental-health issues. Although expensive systems capable of detecting cognitive load and timers counting on-screen time exist, literature has yet to demonstrate measuring both fac- tors across devices. To address this, we propose an inexpensive DIY-approach using a single head-mounted webcam capturing the user’s eye. By classifying camera feed using a 3D Convolutional Neural Network, we can determine increased cognitive load and actual screen time. This works because the camera feed contains corneal surface reflection, as well as physiological parameters that contain information on cognitive load. Even with a small data set, we were able to develop generalised models showing 70% accuracy. To increase the models’ accuracy, we seek the community’s help by contributing more raw data. Therefore, we provide an opensource software and a DIY-guide to make our toolkit accessible to human factors researchers without an engineering background.

EyeKnowYou: A DIY Toolkit to Support Monitoring Cognitive Load and Actual Screen Time using a Head-Mounted Webcam

Tharindu Kaluarachchi, Shardul Sapkota, Jules Taradel, Aristée Thevenon, Denys J.C. Matthies, and Suranga Nanayakkara

Extended Abstracts of the 23rd International Conference on Mobile Human-Computer Interaction (MobileHCI’21) 2021

Abstract PDF

Studies show that frequent screen exposure and increased cognitive load can cause mental-health issues. Although expensive systems capable of detecting cognitive load and timers counting on-screen time exist, literature has yet to demonstrate measuring both fac- tors across devices. To address this, we propose an inexpensive DIY-approach using a single head-mounted webcam capturing the user’s eye. By classifying camera feed using a 3D Convolutional Neural Network, we can determine increased cognitive load and actual screen time. This works because the camera feed contains corneal surface reflection, as well as physiological parameters that contain information on cognitive load. Even with a small data set, we were able to develop generalised models showing 70% accuracy. To increase the models’ accuracy, we seek the community’s help by contributing more raw data. Therefore, we provide an opensource software and a DIY-guide to make our toolkit accessible to human factors researchers without an engineering background.
Moment-to-Moment Continuous Attention Fluctuation Monitoring through Consumer-Grade EEG Device

Shan Zhang*, Zihan Yan*, Shardul Sapkota, Shengdong Zhao, and Wei Tsang Ooi

Sensors 2021

*Denotes equal contribution

Abstract DOI PDF

While numerous studies have explored using various sensing techniques to measure attention states, moment-to-moment attention fluctuation measurement is unavailable. To bridge this gap, we applied a novel paradigm in psychology, the gradual-onset continuous performance task (gradCPT), to collect the ground truth of attention states. GradCPT allows for the precise labeling of attention fluctuation on an 800 ms time scale. We then developed a new technique for measuring continuous attention fluctuation, based on a machine learning approach that uses the spectral properties of EEG signals as the main features. We demonstrated that, even using a consumer grade EEG device, the detection accuracy of moment-to-moment attention fluctuations was 73.49%. Next, we empirically validated our technique in a video learning scenario and found that our technique match with the classification obtained through thought probes, with an average F1 score of 0.77. Our results suggest the effectiveness of using gradCPT as a ground truth labeling method and the feasibility of using consumer-grade EEG devices for continuous attention fluctuation detection.

Moment-to-Moment Continuous Attention Fluctuation Monitoring through Consumer-Grade EEG Device

Shan Zhang*, Zihan Yan*, Shardul Sapkota, Shengdong Zhao, and Wei Tsang Ooi

Sensors 2021

*Denotes equal contribution

Abstract DOI PDF

While numerous studies have explored using various sensing techniques to measure attention states, moment-to-moment attention fluctuation measurement is unavailable. To bridge this gap, we applied a novel paradigm in psychology, the gradual-onset continuous performance task (gradCPT), to collect the ground truth of attention states. GradCPT allows for the precise labeling of attention fluctuation on an 800 ms time scale. We then developed a new technique for measuring continuous attention fluctuation, based on a machine learning approach that uses the spectral properties of EEG signals as the main features. We demonstrated that, even using a consumer grade EEG device, the detection accuracy of moment-to-moment attention fluctuations was 73.49%. Next, we empirically validated our technique in a video learning scenario and found that our technique match with the classification obtained through thought probes, with an average F1 score of 0.77. Our results suggest the effectiveness of using gradCPT as a ground truth labeling method and the feasibility of using consumer-grade EEG devices for continuous attention fluctuation detection.
Prompto: Investigating Receptivity to Prompts Based on Cognitive Load from Memory Training Conversational Agent

Samantha WT Chan, Shardul Sapkota, Rebecca Mathews, Haimo Zhang, and Suranga Nanayakkara

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2020

Abstract DOI PDF

Prospective memory lapses, which involve forgetting to perform intended actions, affect independent living in older adults. Although memory training using smartphone applications could address them, users are sometimes unaware of available times for training or forget about it, presenting a need for proactive prompts. Existing applications mostly provide time-based prompts and prompts based on users’ cognitive contexts remain an under-explored area. We developed Prompto, a conversational memory coach that detects physiological signals to suggest training sessions when users are relaxed and potentially more receptive. Our study with 21 older adults showed that users were more receptive to prompts and memory training under low cognitive load than under high cognitive load. Interviews and an in-the-wild deployment of Prompto indicated that majority of users appreciated the concept, found it helpful and were likely to respond to its prompts. We contribute towards developing technologies with cognitive context-aware prompting based on users’ physiological readings.

Prompto: Investigating Receptivity to Prompts Based on Cognitive Load from Memory Training Conversational Agent

Samantha WT Chan, Shardul Sapkota, Rebecca Mathews, Haimo Zhang, and Suranga Nanayakkara

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2020

Abstract DOI PDF

Prospective memory lapses, which involve forgetting to perform intended actions, affect independent living in older adults. Although memory training using smartphone applications could address them, users are sometimes unaware of available times for training or forget about it, presenting a need for proactive prompts. Existing applications mostly provide time-based prompts and prompts based on users’ cognitive contexts remain an under-explored area. We developed Prompto, a conversational memory coach that detects physiological signals to suggest training sessions when users are relaxed and potentially more receptive. Our study with 21 older adults showed that users were more receptive to prompts and memory training under low cognitive load than under high cognitive load. Interviews and an in-the-wild deployment of Prompto indicated that majority of users appreciated the concept, found it helpful and were likely to respond to its prompts. We contribute towards developing technologies with cognitive context-aware prompting based on users’ physiological readings.
Byte. it: discreet teeth gestures for mobile device interaction

Tomás Vega Gálvez, Shardul Sapkota, Alexandru Dancu, and Pattie Maes

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems 2019

Abstract DOI PDF

Byte.it is an exploration of the feasibility of using miniaturized, discreet hardware for teeth-clicking as hands-free input for wearable computing. Prior work has been able to identify teeth-clicking of different teeth groups. Byte.it expands on this work by exploring the use of a smaller and more discreetly positioned sensor suite (accelerometer and gyroscope) for detecting four different teeth-clicks for everyday human-computer interaction. Initial results show that an unobtrusive position on the lower mastoid and mandibular condyle can be used to classify teeth-clicking of four different teeth groups with an accuracy of 89%.

Byte. it: discreet teeth gestures for mobile device interaction

Tomás Vega Gálvez, Shardul Sapkota, Alexandru Dancu, and Pattie Maes

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems 2019

Abstract DOI PDF

Byte.it is an exploration of the feasibility of using miniaturized, discreet hardware for teeth-clicking as hands-free input for wearable computing. Prior work has been able to identify teeth-clicking of different teeth groups. Byte.it expands on this work by exploring the use of a smaller and more discreetly positioned sensor suite (accelerometer and gyroscope) for detecting four different teeth-clicks for everyday human-computer interaction. Initial results show that an unobtrusive position on the lower mastoid and mandibular condyle can be used to classify teeth-clicking of four different teeth groups with an accuracy of 89%.