Note: This work thread is a part of the PECSS Project, advised by Dr. Rosa I. Arriaga, and funded by the National Science Foundation (Award Number: 1915504)

In my earlier article regarding Digiscape, I explained how the phone’s microphone can be programmed to listen to the environment sound, and the types of computations that we can perform to extract valuable audio features out of it. With this module now integrated into the PECSS app, one natural question that arises is: “What sort of insights can we draw about the patient’s mental state from their audio data?”

In order to answer this research question, I worked on a set of analysis that involved analyzing patient audio. I explain the data, methodology, and results from my experiments in this page.

The Research Question

Are there any correlations between a patient’s speech markers (spectral and prosodic level features) and their levels of distress? How do speech markers compare for individuals when experiencing varying degrees of distress?

Data

Our work primarily constituted of work across two datasets, procured from Emory University:

1. Mock Prolonged Exposure Session Audio Clips (Dataset-1): This dataset consisted of 3 mock prolonged exposure sessions with the patients’ Subjective Unit of Distress (SUD) levels at different timestamps.
2. Trauma Narratives from the ER and follow-up sessions (Dataset-2): This is the dataset from “Narratives in the Immediate Aftermath of Traumatic Injury: Markers of Ongoing Depressive and Posttraumatic Stress Disorder Symptoms”, which contains patient audio files, and their respective personal and clinician-recorded details.

Methodology

We employed a 3-step approach towards answering the research question at hand:

1. Literature Survey: We first conducted a thorough literature survey to identify relevant work in this domain. We identified the most relevant speech markers (such as MFCCs, Zero Crossings, Spectral centroids, phonation rate, etc.) from these two papers primarily:

   1. Speech‐based markers for posttraumatic stress disorder in US veterans. (Marmar Et al.)
   2. Recognizing hotspots in Brief Eclectic Psychotherapy for PTSD by text and audio mining. (Wiegersma Et al.)

2. Investigation Thread-1: We extracted and analyzed the identified features (from the literature survey), on Dataset-1. Our experiments consisted of speech marker comparisons for the patient’s when they exhibit low, medium, and high SUDs.

   Checkout Audio-SUD Analysis Notebook-1, Audio-SUD Analysis Notebook-2, and Audio-SUD Analysis Notebook-3 for the analysis code.

3. Investigation Thread-2: We extracted and analyzed the identified features on Dataset-2, and tried correlating these features with various indices such as the Beck’s Depression Inventory (BDI) and PTSD Symptom Scale (PSS) scores. There were multiple sub-questions that we got to explore in this thread, such as:

   1. How do speech markers vary with increasing BDI? Check out the Jupyter Notebook.
   2. How do speech markers vary with decreasing BDI? Check out the Jupyter Notebook.
   3. Can we classify BDI and PSS values using patient audio? Check out the Jupyter Notebook.
   4. Can we measure clinician effectiveness? Check out the Jupyter Notebook.
   5. Does social support has some correlation with BDI and PSS? Check out the Jupyter Notebook.
   6. Visualizing BDI and PSS scores of patients for visual comparison and analysis. Check out the Jupyter Notebook.

The Github repository containing all notebooks and helper functions can be found here: https://github.com/DiptarkBose/Audio-Analysis

Results

1. We observed that the prosodic features DO show correlations with the patient’s level of distress. For example, we observed that with increasing BDI and PSS, phonation and speech rates decreased, indicating that patients usually speak less when more distressed.
2. We didn’t observe coherent trends for spectral level features. This is most likely due to absence of enough clean patient audio data.