Summer School on Sensor-Based Behavioral Machine Learning

© Pedro Szekely (Flickr, 2010)

What: Analyzing Bio-behavioral Data to Model Human Behavior (Course code: S3B2-ML)

When: August 3rd-4th, 2020

This year's session has concluded. Our team would like to thank the attendees and speakers for making the event a success. Presentations and slide links will appear in the schedule section as soon as they are available.

Talks will be available on YouTube/SigPort

Various human conditions, such as stress and depression, are both common and are detrimental to our psyche and body. Finding those most at risk, however, can be exceedingly costly without methods to automatically detect high-risk individuals. We are inviting students to a summer school, where they will analyze multimodal bio-behavioral data, such FitBit sensors, to infer and better understand human behaviors. This critical task blends several fields from psychology to signal processing to AI, which are not often studied together. We have invited several experts from diverse fields to how discuss how sensors, data extraction, and machine learning can combine to create critical behavior sensing tools. Students will take knowledge gleaned from these experts to create and tackle projects with real human sensor data. We hope for graduate students and postdocs to see the webinar.

Important Dates

Date Description
May 1st Application deadline [extended] (due by 23:59 PDT)
All students welcome
June 1st Notification of acceptance
July 15th Registration deadline
August 3rd School starts


Theodora Chaspari

Assistant Professor, Department of Computer Science & Engineering
Texas A&M University

Theodora Chaspari is an Assistant Professor at the Computer Science & Engineering Department in Texas A&M University. She has received a Bachelor of Science (2010) in Electrical and Computer Engineering from the National Technical University of Athens, Greece and the Master of Science (2012) and Ph.D. (2017) in Electrical Engineering from the University of Southern California. Between 2010-2017 she was working as a Research Assistant at the Signal Analysis and Interpretation Laboratory at USC. She has also been a Lab Associate Intern at Disney Research (summer 2015). Theodora’s research interests lie in the areas of affective computing, signal processing, data science, and machine learning. She is a recipient of the USC Annenberg Graduate Fellowship 2010, USC Women in Science and Engineering Merit Fellowship 2015, and the TAMU CSE Graduate Faculty Teaching Excellence Award 2019. Papers co-authored with her students have been nominated and won awards at the ACM BuildSys 2019, IEEE ACII 2019, ASCE i3CE 2019, and IEEE BSN 2018 conferences. She has served in various conference organization committees (ACM ACII 2017, IEEE BSN 2018, ACM ICMI 2018, ACM ACII 2019, ACM ACII 2020) and her work is supported by federal and private funding sources (NSF, IARPA, EiF, TAMU DoR, AFRL).

Anind K.Dey

Dean and Professor
University of Washington

Anind K. Dey is a Professor and Dean of the Information School at the University of Washington. Anind is renowned for his early work in context-aware computing, an important theme in modern computing, where computational processes are aware of the context in which they operate and can adapt appropriately to that context. His research is at the intersection of human-computer interaction, machine learning, and ubiquitous computing. For the past few years, Anind has focused on passively collecting large amounts of data about how people interact with their phones and the objects around them, to use for producing detection and classification models for human behaviors of interest. He applies a human-centered and problem-based approach through a collaboration with an amazing collection of domain experts in areas of substance abuse (alcohol, marijuana, opioids), mental health, driving and transportation needs, smart spaces, sustainability, and education. Anind was inducted into the ACM SIGCHI Academy for his significant contributions to the field of human-computer interaction in 2015. Before starting at the University of Washington in 2018, Anind was the Charles M. Geschke Professor and Director of the Human-Computer Interaction Institute at Carnegie Mellon University for 4 years, and was a member of the faculty for 13 years. Previously, he was a Senior Researcher at Intel Research and an Adjunct Assistant Professor of Computer Science at UC Berkeley. Anind received his PhD and MS in computer science, and an MS in aerospace engineering from Georgia Tech, and a Bachelors in Computer Engineering from Simon Fraser University.

Tiago H. Falk

Associate Professor
Institut National de la Recherche Scientifique (INRS)

Tiago Falk received the BSc degree from the Federal University of Pernambuco, Brazil, in 2002, and the MSc and PhD degrees from Queen’s University, Canada, in 2005 and 2008, respectively, all in electrical engineering. In 2007, he was a visiting Research Fellow at the Sound and Image Processing Lab, Royal Institute of Technology (KTH), Sweden, and in 2008 at the Quality and Usability Lab, Deutsche Telekom/TU Berlin, Germany.From 2009-2010 he was an NSERC Postdoctoral Fellow at Holland-Bloorview Kids Rehabilitation Hospital, affiliated with the University of Toronto. He joined the Institut National de la Recherche Scientifique (INRS) in Montreal, Canada in Dec. 2010 as a tenure-track Assistant Professor. In 2015, he was promoted to tenured Associate Professor. He is also an Adjunct Professor at McGill University (ECE Dept.) and an Affiliate Researcher at Concordia University (PERFORM Centre). At INRS, he heads the Multimedia/Multimodal Signal Analysis and Enhancement (MuSAE) Laboratory.

Luca Foschini

Co-founder & Chief Data Scientist
Evidation Health, Inc.

Luca is the Co-founder and Chief Data Scientist at Evidation Health, responsible for data analytics and research and development. At Evidation he has driven research collaborations resulting in numerous publications in the fields of machine learning, behavioral economics, and medical informatics. Previously, Luca held research positions in industry and academic institutions, including Ask.com, Google, ETH Zurich, and UC Santa Barbara. He has co-authored several papers and patents on efficient algorithms for partitioning and detecting anomalies in massive networks. Luca holds MS and PhD degrees in Computer Science from UC Santa Barbara, and ME and BE degrees from the SantAnna School of Pisa, Italy.

Kristina Lerman

Project Leader & Research Associate Professor
Information Sciences Institute, USC

Kristina Lerman is a Principal Scientist at the University of Southern California Information Sciences Institute and holds a joint appointment as a Research Professor in the USC Computer Science Department. Trained as a physicist, she now applies network analysis and machine learning to problems in computational social science, including crowdsourcing, social network and social media analysis. Her recent work onmodeling and understanding cognitive biases in social networks has been covered by the Washington Post, Wall Street Journal, and MIT Tech Review.

Stephen M. Mattingly

Postdoctoral Researcher, Department of Computer Science and Engineering
University of Notre Dame

Stephen Mattingly has a Ph.D in Cognition, Brain, and Behavior from the University of Notre Dame, and works as a postdoctoral researcher in computer science under Aaron Striegel. He is an expert in the cognitive neuroscience of sleep, stress, and memory, and has worked on two large scale longitudinal sensor based studies, Nethealth and Tesserae. He is interested in improving how wearables and other passive sensors measure sleep and stress. In addition, he is interested in applying insights from large scale studies to generate real world improvements in how individuals track and manage sleep and stress.

Jyoti Mishra

Assistant Professor, Psychiatry
UC San Diego

Dr. Mishra is trained in the computational, cognitive and translational neurosciences. She has expertise in studies of attention, learning and brain plasticity, and has contributed to innovations in closed-loop technologies, with over 40 peer-reviewed articles and 7 patents/copyrights.
She is the founder and director of the Neural Engineering and Translational Labs at UC San Diego. NEATLabs innovates neurotechnologies for scalable cognitive brain mapping, and combines these with longitudinal monitoring using wearables. Our analytics are geared towards precision therapeutics in psychiatry across the spectrum of mental health throughout the lifespan and in diverse local and global communities. NEATLabs research has won several grant awards most recently from the Kavli foundation, Sanford Institute for Empathy & Compassion, the Tang Prize foundation and the Stronger Brains foundation. Dr. Mishra has also received the Hellman Award for Early Career Faculty.

Akane Sano

Assistant Professor, Electrical and Computer Engineering
Rice University

Akane Sano is an Assistant Professor at Rice University, Department of Electrical Computer Engineering, and Computer Science. She directs Computational Wellbeing Group. She is a also member of Rice Scalable Health Labs.
Her research focuses on affective, ubiquitous and wearable computing, and biobehavioral sensing and analysis/modeling. Her research targets (1) the analysis and modeling of human ambulatory multimodal time series data including physiological, biological and behavioral data for measuring, predicting, improving, and understanding human physiology and behavior and human factors such as health, wellbeing, and performance and (2) development of human centered computing technologies to support health and wellbeing.
She obtained her PhD at MIT. Before she came to the US, she was a researcher/engineer at Sony Corporation and worked on affective/wearable computing, intelligent systems, and human computer interaction. Her recent awards include Microsoft Productivity Research Award in 2019, the Best Paper Award at IEEE BHI 2019 conference, the Best Paper Award at the NIPS 2016 Workshop on Machine Learning for Health, and the 2014 AAAI Spring Symposium Best Presentation Award.

Björn W. Schuller

Full Professor & Head of the Chair of Embedded Intelligence for Health Care and Wellbeing
University of Augsburg

Björn W. Schuller received his diploma, doctoral degree, habilitation, and Adjunct Teaching Professor in Machine Intelligence and Signal Processing all in EE/IT from TUM in Munich/Germany. He is Full Professor of Artificial Intelligence and the Head of GLAM at Imperial College London/UK, Full Professor and Chair of Embedded Intelligence for Health Careand Wellbeing at the University of Augsburg/Germany, co-founding CEO and current CSO of audEERING – an Audio Intelligence company based near Munich and in Berlin/Germany, and permanent Visiting Professor at HIT/China amongst other Professorships and Affiliations. Previous stays include Full Professor at the University of Passau/Germany, and Researcher at Joanneum Research in Graz/Austria, and the CNRS-LIMSI in Orsay/France. He is a Fellow of the IEEE and Golden Core Awardee of the IEEE Computer Society, Fellow ofthe ISCA, Fellow of the BCS, President-Emeritus of the AAAC, and Senior Member of the ACM. He (co-)authored 900+ publications (30k+ citations, h-index=83), is Field Chief Editor of Frontiers in Digital Health and was Editor in Chief of the IEEE Transactions on Affective Computing amongst manifold further commitments and service to the community. His 30+ awards include having been honoured as one of 40 extraordinary scientists under the age of 40 by the WEF in 2015. He served as Coordinator/PI in 15+ European Projects, is an ERC Starting Grantee, and consultant of companies such as Barclays, GN, Huawei, or Samsung.

Donna Spruijt-Metz

Director USC-CESR Mobile and Connected Health Program
University of Southern California, Dornsife

Donna Spruijt-Metz is Research Professor in Psychology and Professor in Preventive Medicine, housed in the USC Dornsife Center for Economic and Social Research. Her work meshes 21st century technologies with transdisciplinary metabolic, behavioral and environmental research to facilitate dynamic, personalized, contextualized behavioral interventions thatcan be adapted on the fly. She has a deep interest in harnessing mobile health and new media modalities to bring researchers and researched systems into interaction, engage people in their own data, and bring about lasting change in public health. One primary focus is combining sensor and self-report data that is continuous, temporally rich, contextualized. Using data and innovative modeling techniques, Spruijt-Metz collaborates with engineers, health professionals, and data modelers to create new mathematical models of human health-related behavior in real time. She is one of the first to undertake a just-in-time, adaptive intervention (JITAI) in youth, and envisions most or all future interventions being JITAI. Examples of current projects include: Monitoring and Modeling Family Eating Dynamics (M2FED, funded by NSF) and Operationalizing Behavioral Theory for mHealth:Dynamics, Context, and Personalization (funded by NIH). She has led two NSF/EU/NIH-funded international workshops on building new computationally-enabled theoretical models to support health behavior change and maintenance in real- or near-time.

Clemens Stachl

Postdoctoral Researcher, Media and Personality Lab
Stanford University

Clemens Stachl is a postdoctoral scholar at the Communication Department at Stanford University. His work is focused at the description, prediction and explanation of individual differences (i.e., personality traits) from digital behavioral footprints and at the collection of these data with mobile sensing techniques.
Additionally, he is interested in the challenges and opportunities that arise from the widespread use of automated decision making and predictive modeling for science and society alike. Another aspect of his work deals with the personalization of user interfaces and digital systems with regard to individual differences (i.e., personality traits).


Keith Burghardt

Computer Scientist
USC Information Sciences Institute

Benjamin Girault

Postdoctoral Scholar
Unversity of Southern California

Kristina Lerman

Project Leader and Research Associate Professor
Information Sciences Institute, Unversity of Southern California

Emilio Ferrara

Research Team Leader and Assistant Research Professor
Information Sciences Institute, Unversity of Southern California

Shrikanth Narayanan

Niki and C. L. Max Nikias Chair in Engineering
Unversity of Southern California


Students will be learning about the various areas of AI applicable to converting sensor data into viable predictions of human stress, performance, personality, and other psychological factors. Moreover, they will learn about signal processing, embedding, and other ways in which raw data can be converted into useful features.

Time of Day August 3rd August 4rd
08:50 PST [ Keith Burghardt (Opening remarks) ]
09:00 PST Akane Sano Clemens Stachl
09:45 PST Luca Foschini Stephen Mattingly
10:30 PST Jyoti Mishra Donna Spruijt-Metz
11:15 PST Björn Schuller Anind Dey
12:00 PST Theodora Chaspari Kristina Lerman
12:45 PST Tiago Falk (Closing remarks)
13:30 PST (Closing remarks)
(Scroll down to find archived presentations and slides)

Speaker: Akane Sano
Title: Physiological and behavioral data analysis and modeling for health and wellbeing
Abstract: This talk highlights lessons learned from a series of ambulatory studies, developed to measure, forecast and support mood, stress, sleep, and performance, which were run in cohorts of college students, office workers, and shift workers using continuous wearable and mobile phone data. This talk overviews the tools and machine learning algorithms developed in the studies, challenges faced, and some key findings for measuring, forecasting, and supporting mood changes,stress, sleep, and performance.
Presentation: mp4

Speaker: Luca Foschini
Title: Translating Digital Medicine into Practice: Beyond Model Performance
Abstract: Person-Generated Health Data (PGHD) from smartphones, wearables and other sensors have the potential to transform the way health is measured, with broad-ranging applications from clinical research to public health and health care at large. In this talk I will give examples of applications of PGHD across therapeutic areas, including post-op monitoring, screening for cognitive impairment, and COVID-19 detection and quantification. Finally, I will discuss lessons learned in translating PGHD research into benefit for the individuals, and how good analytic performance is a necessary but not at all sufficient condition to engender the trust that clinical investigators, doctors, regulators, and ultimately individuals seek.
Presentation: mp4 Slides: pdf

Speaker: Jyoti Mishra
Title: Personalizing Mental Wellness leveraging Wireless Wearables & ML
Abstract: Mental wellness is sought after by all. For 20% of the American population and 25% of the global population, mental ill-being in the form of depression is a leading cause of disability. Clinical assessment and treatment for depression, however, is subpar and there is much inter-individual variability in treatment outcomes. Current treatments do not rely on important cognitive and brain function states of the individual or vital lifestyle factors such as sleep, physical activity and stress. In this talk, I will describe our progress towards comprehensive and quantified assessments of mental wellbeing in patients that leverage smartphones and smartwatches, and use of N-of-1 machine learning analytics to tailor the personalized wellbeing plan for each individual. This project uses scalable and cost-effective digital and sensor technologies for making precision psychiatry a reality.
Presentation: mp4 Slides: pdf

Speaker: Björn W. Schuller
Title: Wear for Care: Deep Mobile Health Sensing
Abstract: Intelligent mobile sensing has recently repeatedly been praised for its greatest potential to enable Digital Health use cases “on the go” anytime, anywhere, and in real-time. Examples reach from early warnings of heart attacks to depression monitoring and more recently COVID-19 symptoms’ recognition. The range of signals used is thereby rich reaching from heart rate to movement data as well as audio and video, but also UV or NFC, and other sensor data. A crucial factor in this equation is the machine learning back-end that has to be robust and reliable, yet, ideally also privacy-preserving, usable, explainable, efficient, and trustworthy to name but a few of the features needed before according applications can make a real difference. Here, we consider deep learning as the tool of choice to help realise these abilities. We shall discuss the latest and greatest in deep learning with respect to these and further factors including convolutional, recurrent, and generative adversarial architectures with attention mechanisms. Approaches to learning will include efficient automatic, federated, cooperative, self-supervised, reinforced, and life-long learning. Further, we shall look into robustness against attacks and protection of users. In this light, it seems earlier diagnoses and enhanced interventions could be around the corner with the potential to benefit millions.
Presentation: mp4 Slides: pdf

Speaker: Theodora Chaspari
Title: Personalized, low-resource, and privacy-aware machine learning for augmenting human well-being
Abstract: Recent converging advances in sensing and computing, including ambulatory technologies, allow the unobtrusive long-term tracking of individuals yielding a rich set of real-life multimodal bio-behavioral measurements, such as speech, physiology, and facial expressions. While bio-behavioral measurements can afford us useful insights into human behavior empowering physical and mental health, the available data in such applications involve various challenges related to the scarce amount of labels, the high variability across individuals, and the strong presence of privacy-sensitive information. This prevents machine learning systems from making reliable predictions degrading their performance and compromising user trust. This talk will present approaches to address these challenges by: (1) incorporating the inherent inter-individual variability through subject- and group-specific models of human behavior; (2) designing generalizable models of human-related outcomes through novel weakly supervised algorithms; and (3) learning bio-behavioral signal representations that preserve facets of information related to the human state (e.g., emotion), while eliminating information related to a person's identity. We will demonstrate the effectiveness of the proposed approaches through examples in public speaking training, family well-being, and work performance.
Presentation: mp4 Slides: pptx

Speaker: Tiago Falk
Title: Making sense of the noise: the case of wearable data for in-the-wild human performance monitoring
Abstract: Wearable devices are increasingly being used to monitor human performance in highly ecological settings. Wearable devices, however, are very sensitive to movement artifacts that hamper analyses. To this end, signal enhancement methods and noise-robust feature extraction methods are needed. In this presentation, I’ll overview innovations developed in the Multimodal Signal Analysis and Enhancement (MuSAE) Lab (INRS-EMT, University of Quebec, Canada) for biosignal quality measurement and enhancement, as well as for extraction of noise-robust features. Examples will be shown of projects aimed at monitoring workload, stress, anxiety, and fatigue levels of first responders, drivers, and nurses in real-world settings. I will conclude the presentation with a description of a newly developed body-computer-interface-equipped virtual reality headset for user-aware serious gaming, and on new techniques to infer contextual information from the noise patterns in wearable data.
Presentation: mp4 Pre-recorded Presentation: mp4

Speaker: Clemens Stachl
Title: Behavioral Patterns from Smartphones Predict Personality Trait Levels
Abstract: The increasing digitization of our society radically changes how scientific studies are being conducted in the field. In the social sciences, the analysis of online data repositories, digital footprints, and novel possibilities of in-vivo high-frequency data collection now allows for the investigation of formerly intangible psychological constructs, in the wild, objectively, fine-grained, and at large scale. Many recent studies have demonstrated the enormous potential of online digital footprint data for personality research (i.e., research on social media platforms). Even greater potential hold the rich behavioral and contextual data that can be collected as part of everydaysmartphone use. In this talk, we will present data from a 30-days in-vivo smartphone sensing study (N = 624) and show how these data can be used to extract novel and meaningful variables about daily behavior (P = 15,692). Additionally, we will show how Big Five personality trait scores can be predicted from these variables. Using machine learning methods we investigate the accuracy of these predictions and explore the most predictivebehavioral patterns for single personality dimensions. Finally, we will conclude with a privacy focused outlook on future sensing studies.
Presentation: mp4 Slides: pdf

Speaker: Stephen Mattingly
Title: Design of large scale, in-situ, longitudinal studies and what we can discover
Abstract: The increased affordability, variety, and dependability of unobtrusive, passive sensors has allowed researchers to deploy them in larger groups for long periods of time. However, this increased data collection comes with significant difficulties from increased complexity between interacting systems, and requires significant effort to reduce missing data, ensure participant compliance, and finally, to understand what the theoretical underpinnings of what these sensors can, and cannot tell us, both independentlyncreased affordability, variety, and dependability of unobtrusive, passive sensors has allowed researchers to deploy them in larger groups for long periods of time. However, this increased data collection comes with significant difficulties from increased complexity between interacting systems, and requires significant effort to reduce missing data, ensure participant compliance, and finally, to understand what the theoretical underpinnings of what these sensors can, and cannot tell us, both independently and in aggregate. and in aggregate.
Presentation: mp4

Speaker: Donna Spruijt-Metz
Title: Using multilevel, streaming data to intervene on Health Behavior: What do we mean by “Just-In-Time”?
Abstract: One promising current approach to achieving sustained behavior change in the face of highly dynamic, context-dependent, and idiosyncratic behaviors is the just-in-time adaptive intervention (JITAI) framework. This framework leverages recent advances in mobile technologies including smartphones and wearable sensors to continuously monitor physical parameters (eg: heart rate, physical activity levels, stress). These data can be augmented with other context variables (e.g., weather, location), as well as other individual-level variables assessed using momentary self-report. Theaim of the JITAI framework is to use the available data to infer when a specific intervention is likely to be effective for an individual, and to deliver the right intervention in the right time and context so that the individual will be most likely to be receptive. However, existing behavioral theories and computational models lack the ability to represent and efficiently reason about dynamic, multiscale, and idiographic (individualized) behavioral processes and interventions. Where do we go from here? I will discuss some fundamentals and some of the latest advances in technology-facilitated behavioral measurement methods including the meshing sensor data with Ecological Momentary Assessment (EMA),some of the recent mHealth-basedmethods for behavioral interventions, and next steps for modeling and utilizing this complex data in “real” time.

Speaker: Anind Dey
Title: Using Everyday Routines for Understanding Health Behaviors
Abstract: We live in a world where the promise of ubiquitous computing and the Internet of Things is coming true. We have smart devices that pervade our lives, and that are constantly collecting data about us and mostly discarded as irrelevant. I will demonstrate how researchers can extract relevance from this passively collected data and apply it to understanding people's behaviors. I will describe approaches for extracting routines from smart devices, and then how these routines can help us better understand human behavior and anomalies, in the context of studies of depression and substance abuse.
Presentation: mp4

Speaker: Kristina Lerman
Title: Biased Data & Other Threats to Validity of Models
Abstract: Data is often heterogeneous, generated by subgroups with differenttraits and behaviors. The correlations between the traits, behaviors, time, and how the data is collected, create dependencies that bias analysis.Models trained on biased data will make invalid inferences about individuals– what’s known as ecological fallacy. The inferences can also be unfair anddiscriminate against individuals based on their membership in protectedgroups. I describe common sources of bias in heterogeneous data, includingSimpson’s paradox, survivor bias, and longitudinal data fallacy, showing thatignoring these sources of bias can dramatically alter conclusions and lead towrong policy recommendations. I highlight with an example of COVID-19pandemic to show that spatial aggregation of disease statistics exaggeratesestimated growth rates. Finally, I describe a mathematical framework for de-biasing data that addresses these threats to validity of predictive models.The framework creates covariates that do not depend on sensitive features,such as gender or race, and can be used with any model to create fairer,unbiased predictions. The framework promises to learn unbiased modelseven in analytically challenging data environments.
Presentation: mp4 Slides: pptx


Deadline: July 15th (23:59 PDT)

Click here to register

The school is open to everyone interested who interested in how bio-behavioral data, whether they want to analyze data from wearable sensors like FitBit, or model how sensor data correlates with personality traits, stress, and sleep. Students with machine learning background are encouraged to apply, as well as those who have worked on modeling human behavior. The aim of this summer school and workshop is to bring a broad group of researchers interested in how health can be passively tracked to improve livelihoods.


Registration will be free.

Get in touch

4676 Admiralty Way
Suite #1001
Marina del Rey, California USA 90045


(310) 448-9433