The other major category of settings is related to recording quality. These settings need to be initially confirmed and may need to be occasionally adjusted on the interviewer account, but they also need to be walked through with the participant before beginning recording any interview. Even for repeat subjects, these quality checks should be performed every time, as there is no way of knowing if they have changed their account settings or logged in from a different device since the last interview. A few of these settings additionally may require tweaking from default based on the specifics of the current interview. Remember to revert them to default on your own account after the interview recording is complete if they were altered. For these reasons, it is also worth confirming your own settings before each interview as part of the quality assurance workflow.

The settings that should be adjusted under the "audio" tab of the Zoom client on both computers are:

* If using an external microphone rather than the built-in one, ensure that it is selected.

* Uncheck "automatically adjust microphone volume", then set input volume using Zoom's "test mic" function to ensure that the input audio is not too quiet nor too loud.

* Put "suppress background noise" on the low setting. The interviewer should of course always be in a quiet setting and hopefully not need to tweak this. If the participant is also in a quiet space, they should do the same, but a higher background suppression setting may be needed in some circumstances if you notice a high level of noise coming from their environment.

* Check the box that says “show in-meeting option to ‘turn on original sound’ from microphone.” Then uncheck the boxes underneath that say “high-fidelity music mode,” “echo cancellation”, and “stereo audio”.

* This will cause a "turn original sound on” button to appear in the top left of every video call. It is important to make sure this is clicked before starting recording, for every interview.

* Once you have turned original sound on, please confirm that the participant's audio quality sounds acceptable, and ask the participant to confirm that your audio quality sounds acceptable, before beginning the recording.

* One thing in particular that may need to be adjusted is that depending on participant setup, it may be necessary to turn back on the "echo cancellation" option mentioned.

It is important that settings are confirmed and that audio quality is sanity checked on both ends before beginning recording. Note that not all of these options are available for audio input on the phone Zoom app, which some participants may be using for some interviews, though this should be advised against. If a participant needs to use their phone, please encourage them to use a headset with the phone if possible.

For video recording, there are not many account or app settings to adjust, however there are quality checks that need to be made on the level of the individual interview. On the participant's end, it is important to ensure that their entire face is within the frame, and that the camera is stationary. The latter problem is mostly an issue with phones; the participant can be advised to lean their phone against something to stabilize the camera, but in this case it is important to ensure that the audio input on the phone is not obstructed, unless they are using a headset. Also, for those using a headset, it should be positioned in a way that prioritizes audio quality first, but ideally in a way that does not obstruct the view of the face. Similarly, participants should be asked not to wear a mask if possible, nor any hat that might obstruct view of their eyes.

On the interviewer's end, it is important that the Zoom client is in gallery view. Unfortunately there is not an account wide setting to force all meetings to default to gallery view, so this has to be confirmed manually before beginning recording of any interview. Gallery view can be selected via the "view" panel in the top right of the meeting window. For open interviews in particular, if there are people sitting in on the call besides the subject and the interviewer, such as a subject's parents, please ask them to keep their cameras off. You should then be sure to select "hide non-video participants" in the same "view" panel. Before beginning an open recording, you should confirm that there are exactly two video feeds showing on the call with faces visible: yourself and the participant. Note that "hide non-video participants" is available as an account-wide setting under the "video" panel of the settings menu in the Zoom client, so this could be set as the default for your account to save time during interview set up.

For psychs interviews there are no specific requirements for video, but if anyone will have the camera on, please ensure you have the meeting in gallery view. Regardless, please ensure that no one with their camera off (but a box visible in the call) has a profile picture icon that features a face, as this could cause our initial QC metrics to overestimate the number of psychs interviews that we have available with face data.

§.§.§ On pauses in Zoom recordings

Note that if the recording needs to be paused for a time in the middle of the interview, we recommend using the pause functionality rather than stopping and starting a new recording, as will be discussed in the AMPSCZ project report to come. Stopping recording and then starting again within the same Zoom meeting will create split files within one Zoom interview folder, which can be difficult to systematically handle appropriately with software (and is not supported by the code described here). Pausing on the other hand will cause produced content to have a gap with silence/a black screen during the pause times, so that the final recording accurately reflects the entire duration of the recorded call. If it is desired to have separate sessions within one interview, it is much preferable from a code perspective to end the first Zoom meeting entirely, save that recording, and then start a new Zoom meeting where recording can begin again when ready. This will produce two separate interview folders with different start timestamps, which can be processed normally by my pipeline.

§.§ REDCap/RPMS forms

For clinical interviews in particular, scale ratings for each time point will be recorded via the REDCap/RPMS platform. Regardless of interview format, information on the recording should be carefully entered into the interview runsheet for each session, similar to how a lab notebook would be used for wet lab data collection. This includes standard metadata as well as any notes of potential interest on recording quality, abnormal settings used for interview collection, etc. Details of the REDCap/RPMS instruments within these interview recording protocols are beyond the scope of the current chapter, as handling of REDCap/RPMS data is not performed by my software. However, info found (or unfortunately not found) on REDCap/RPMS has been utilized in conjunction with the pipeline outputs to interpret certain data monitoring anomalies, in addition to their obvious importance in eventual downstream research questions.

§.§ Conducting open interviews

Open interviews are intended to occur at baseline and at a 2 month follow-up point, for each participant enrolled in the study. The goal is to elicit interesting free speech from the participant on any topic possible. Some participants may not want to talk much, and this can of course be clinically relevant, but it is important to make a good attempt to get them to open up during the allotted time.

The AMPSCZ SOP recommends beginning the open interview recording with the following script, or something similar to it:

First, I’d like to thank you for taking the time to talk with me. Like I mentioned, our conversation will be recorded for analysis. This interview is different from the other interviews we do. I would really like to get to know you and learn what your life is like. So, how have things been going for you lately?

The SOP goes on to provide the following tips on conducting a quality open interview:

* You should continue with elaboration questions:

* Restate or rephrase part of what they say as a question to prompt elaboration.

* Prompt them with a follow up question such as, “What is/was it like to [...]?"

* You are trying to elicit stories that are meaningful in the subject’s life. The depths of these stories are more important, rather than the number of stories you can elicit.

* Ideally, the interviewer disappears into the story; it may help to avoid phrases like “tell me”, which draw attention to the interviewer.

* However, sometimes it can help to put yourself into the conversation to express understanding and empathy.

* Make an effort not to derail or redirect the flow of the conversation:

* If the subject is bringing a theme to the conversation, it is best to follow it.

* Leading questions are best and yes/no questions should be avoided.

* If you don't know what to say you can always reiterate what the subject says.

* Try to make the subject know they are being heard. You can do this by acknowledging the emotion behind the story, mirroring what they have said, and expressing gratitude for their openness.

* Sometimes subjects may speak in a way that the interviewer finds confusing. If you lose track of their train of thought, you can request help by saying, “I want to make sure I follow you, can you help me understand...”

* If someone stops speaking altogether, you can repeat what they last said as a question, or even look at them expectantly, but being comfortable with silence is important too. Often people are formulating their thoughts and will continue after a pause.

The SOP also provides the following suggested ending to the open interview:

Thank you so much for taking the time to talk with me [or thank you for sharing with me]. I’m going to stop the recording now.

Remember throughout this process that the main goal is to have subjects speak freely without interruption and to show that you are listening and are interested in what they are saying. This involves not only the prompting used while conducting the interview, but also body language and tone throughout. It is important to let the participant guide the conversation, but at the same time not to let the participant lose interest in the conversation.

By contrast, psychs interviews are much more structured in the prompts required and the responses expected of participants. The goal of the psychs interview is to be providing specific clinical ratings, and these interviews may run for a much longer time but include much briefer (and less rich) individual answers as a result. Many prompts are in fact questions directly about experienced symptom severity and ask for a yes/no style response, which is discouraged in the open context. I will now review the plan for conducting of psychs interviews in the AMPSCZ project.

§.§ Conducting psychs interviews

The psychs interview is a clinical interview based around a new rating scale for clinical high risk (CHR) of Schizophrenia. Per [Brady et al., 2023], AMPSCZ aims to recruit 1,977 individual between 12 and 30 years old who meet the CHR criteria defined by the new "Positive SYmptoms for CAARMS Harmonized with SIPS" (PSYCHS) psychometric instrument. The enrolled CHR subjects will complete screening and baselines psychs interviews, and then will participate in 7 follow-up psychs interviews over the course of 2 years. In parallel, the project aims to recruit 640 healthy controls to complete screening and baseline psychs interviews. $\sim 200$ of the control subjects will also participate in follow-up psychs interviews at 2 month, 12 month, and 24 month study time points [Brady et al., 2023]. Psychs interviews entail both the interview recording protocol detailed, as well as PSYCHS scale ratings based on this interview. All enrolled subjects will also participate in the described open interviews, as well as other project data collection modalities (though subjects may participate in a subset of modalities without consenting to all of them).

Note the PSYCHS scale is based on two existing scales for rating CHR: the Comprehensive Assessment of At Risk Mental State (CAARMS) and the Structured Interview of Prodromal Symptoms (SIPS). The CAARMS and SIPS are the two primary scales used in current practice for clinical high risk of psychosis. These scales are reliable and well-validated, but have only moderate success in actually predicting future development of psychosis, particularly as a large number of people that meet CHR criteria - even with quite high scores - do not end up developing a psychotic disorder [Malda et al., 2019]. Thus one goal of AMPSCZ is to improve on the false positive rate of CHR assessment.

The SIPS identifies three CHR syndromes: Attenuated Psychotic Symptoms Syndrome (APSS), Brief Intermittent Psychosis Syndrome (BIPS), and Genetic Risk and Deterioration (GRD) [Woods et al., 2014]. Per the Orygen Youth Health Research Center's manual for conducting the CAARMS, the full scale involves assessing the following domains.

* Positive Symptoms. Requires rating of:

* Unusual Thought Content

* Non-bizarre Ideas

* Perceptual Abnormalities

* Disorganised Speech

* Cognitive Change – Attention/Concentration. Requires rating of:

* The subjective experience

* Any observed cognitive change

* Emotional Disturbance. Requires rating of:

* Subjective emotional experience

* Observed blunted affect

* Observed inappropriate affect

* Negative Symptoms. Requires rating of:

* Alogia

* Avolition/apathy

* Anhedonia

* Behavioural Change. Requires rating of:

* Social isolation

* Impaired role function

* Disorganising/odd/stigmatising behaviours

* Aggression/dangerous behaviour

* Motor/Physical Changes. Requires rating of:

* Subjective complaints of impaired motor functioning

* Informant reported or observed changes in motor functioning

* Subjective complaints of impaired bodily sensation

* Subjective complaints of impaired autonomic functioning

* General Psychopathology. Requires rating of:

* Mania

* Depression

* Suicidality and self-harm

* Mood swings/lability

* Anxiety

* Obsessive-compulsive disorder symptoms

* Dissociative symptoms

* Impaired tolerance to normal stress

The PSYCHS, which is a new scale building upon these two rating systems, is therefore a quite structured interview where the goal is to assess symptom domains and possible syndromes such as those topics covered in the core SIPS and CAARMS scales. This means that many of the interview questions are probing about specific experiences and asking for subjective symptom ratings that do not necessarily elicit more elaborate patient responses like the open interviews do. On the other hand, the content of the recorded interview can be directly mapped to the resulting PSYCHS scale scores that are reported, in that they are rated based on this clinical interview.

§.§ Details of TranscribeMe settings used for AMPSCZ

As mentioned, the AMPSCZ project uses similar transcription settings with TranscribeMe as our lab does (described in the methods of chapter <ref>). However, there are a few key differences that affect pipeline operations and/or the immediate next steps required for extracting features from the returned transcripts. In this supplemental section, I go through these small but important distinctions.

The requested transcription quality level and verbatim notation - including the conventions for marking of disfluencies and of inaudible words/other noises - are identical, and speaker identification is still performed, with IDs and timestamps formatted in the same way. A key difference for the AMPSCZ interviews is a reduction in the number of timestamps provided, to cut back on cost. These transcripts have lines split by conversational turn rather than sentence, so that a long interviewee response will be contained all on the same line with a single timestamp provided at the start of the response. Additionally, the timestamps report only second-level resolution, rather than providing millisecond digits with accuracy intended up to 100 ms like for BLS. For most analyses, the changes will not be a problem; however, it does alter how one might interpret some of the secondary QC features in the AMPSCZ interview pipeline, which were ported from the work with BLS reported in chapter <ref>.

Another difference between the AMPSCZ transcription protocol and the BLS one is the notation for marking redactions. This has been handled by my interview pipeline and is not something necessary to consider for those monitoring outputs or planning to do downstream analyses on the deidentified transcript data. However, it is worth noting because it was an intentional study design decision made to hopefully generate a good training dataset for future development of automated redaction systems by those with access to the sensitive raw data. In BLS, any redactions of personally identifiable information (PII) are made directly in the returned transcript, so that we only receive a copy with "[redacted]" in place of whatever words were PII. For AMPSCZ, the same guidelines are used by transcribers to identify PII – but instead of removing it, they are instructed to include the verbatim words in the transcription, simply enclosing it in curly braces, which are not otherwise used by TranscribeMe. Thus we can automatically create a version of each transcript that looks just like the BLS redaction style, but we also now have a version that reveals what the PII was for research applications where that is appropriate.

Finally, there is a change in the AMPSCZ transcription protocol that does not affect the form of the outputs in any way, but can have a major impact on what is available for analysis, particularly if sites do not remain aware of it as they are conducting interviews. To prevent running up a large bill from extremely long interviews (as will be discussed it is not uncommon to see psychs recordings exceeding 90 minutes, despite the recommendation being about half that length), TranscribeMe has been instructed not to exceed 35 minutes of transcribed content for a given interview audio. The transcriber will thus intelligently cut off each transcription of a long interview between $30$ and $\sim 32$ minutes based on the content, to prevent a transcript from ending in the middle of a participant response. This cutoff point will begin at the 30 minute mark, but such transcripts will always be charged as a 32 minute submission, so that the vast majority of psychs transcripts will cost $\$58.56$ in English and Spanish, and over $\$100$ in other languages. The resulting transcript can be analyzed like any other, but it is important to keep in mind when comparing across transcripts that the interviewer content might vary more than expected because of the duration cutoff. Note that this issue is functionally exclusive to the psychs interview type.

Once transcripts are returned by TranscribeMe, there are manual review processes in place to ensure PII is correctly redacted. This is a particularly important thing to check, through both human and automated methods, because the redacted transcripts will eventually be shared as part of a broader database that any researchers can apply to access. The pipeline includes some redaction-related sanity checks that will be described; however, the major protocol component of redaction review is an expectation of the sites.

Specifically, sites are expected to carefully review a subset of the transcripts returned from the interviews they conducted, and mark any PII for redaction that was missed by TranscribeMe. They should use the same curly brace convention as TranscribeMe if a redaction needs to be added, and whenever that does occur they should also log the event so there is a clear awareness of the frequency of missed redaction issues. Sites are only expected to review for redactions and not other potential quality issues, and they in fact should not be making any changes to files outside of the scope of redactions. If an egregious quality issue unrelated to redactions is found by a site during the review process, they should contact their point person at the central server to communicate the issue. Central monitors can handle the situation from there, and will likely request a re-transcription of that audio file by TranscribeMe.

Each site will perform a manual redaction review on the first 5 interview transcripts that are produced from interviews submitted by that site, and from that point forward they will review a randomly selected $10\%$ of TranscribeMe transcripts from their interviews. Transcripts for review are returned to sites using the same cloud-based file system as they use for uploading interviews to the central server, and once a manually reviewed transcription is finalized, it will be returned to the central server through a similar mechanism, as sites move the file from the "for review" side of the folder structure to the appropriate spot on the "approved" side. Note that the pipeline (section <ref>) will not process a transcript that was sent for review at all until the final version is returned by the site – though in practice the final version is the same file, as redaction changes are rarely needed.

§ AMPSCZ INTERVIEW MONITORING SUPPLEMENT

§.§ Early results from accounting of open interview availability by subject ID and timepoint across sites

As described in the main text of chapter <ref>, the interview pipeline has enabled detailed accounting of site progress, to identify and address a number of issues – including discrepancies in open interview availability and expected timepoints, documented in detail here for all 14 active Pronet sites as of 1/31/2023.

These stats focus in particular on available open interview recordings and the associated study days tallied for each participant ID with some documented interview involvement (related to the DPChart monitoring view presented in the main text of chapter <ref>, in Figure <ref>):

* There are 102 subjects with at least one recognized interview (i.e. an interview of either type with at least one modality successfully processed by QC).

* Of those, 79 subjects have at least one open interview processed (i.e. at least one modality successfully processed by QC).

* The 23 subject IDs with some interview but no open interview is likely explained in part by site delays in conducting and uploading open interviews versus psychs interviews, though it also may reflect on issues with certain sites incorrectly submitting interviews at a relatively high rate. It is possible as well that a small portion of subjects will be screened out of study participation after a single psychs interview, but the site will still upload the psychs screening, resulting in an ID that will therefore never partake in an open interview.

* Of those, 61 subjects have at least one open interview with a redacted transcript available.

* The 18 subjects with a correctly submitted open interview but no transcription yet are primarily accounted for by the sites that have not been returning the subset of transcripts they were assigned for manual redaction review. There are also a handful of new subjects that are awaiting an output for their first open interview from TranscribeMe, but this process should generally not introduce more than a 1 week delay, especially for English language sites, which represent the vast majority of active Pronet sites to date.

* From those 61 subjects, 14 have transcripts at both expected open interview time points (baseline and 2 month follow-up), and another 27 have a single open transcript from the baseline time point.

* Of course, subjects with only baseline transcripts are a natural occurrence as the study progresses, and we expect to see follow-up transcripts eventually populate for each of these 27.

* There are thus 20 subjects with open transcripts available but that are labeled with potentially questionable time points.

* 4 of these subjects were identified above as clearly problematic submissions from site LA.

* 7 of these subjects have two open interview transcripts with adequate separation (at least 48 days apart), so they are likely intended to represent the two open time points, and are largely acceptable transcripts. However it remains to be confirmed how long after consent it is acceptable for the baseline open interview to occur. All 7 of the mentioned subjects had their first interview after day 30 of study enrollment, including as late as day 65. If this is a concern for the project, sites should be warned even though the data from both open sessions will of course be included in downstream analyses.

* The final 9 subjects of these 20 have a single open interview transcript from an interview conducted past day 30 of study enrollment. It is unclear how many of these subjects will eventually have another interview conducted to join the aforementioned group of 7 subjects (i.e. the current transcript is intended to be a baseline interview even if late), versus how many represent only a follow-up open transcript with unexpectedly missing baseline transcript. The latter could occur due to various issues during the interview upload process or could simply be explained by a baseline transcript that has been produced by TranscribeMe but happened to be assigned for manual review which was subsequently ignored by the site.

As such, a number of potential problems for further follow-up have been identified in this way (to join the many other issues we have identified with various monitoring tools, reported on throughout the main chapter <ref>).

§.§ Constructing the weekly server-wide interview quality accounting tables

The values reported in the weekly server summary table described in the main text of chapter <ref> (with methods details here) are as follows, in order of columns presented (Tables <ref> and <ref>). They are reported separately for each site and interview type:

* The total number of interviews that have QC metrics available for any modality. Note that many of the file management mistakes to be discussed in more depth later (section <ref>) will prevent an interview or a particular interview modality from being processed, so that this number is not necessarily equivalent to the number of interviews recorded.

* The number of interviews with video QC metrics available. This might not match the preceding count due to a file upload mistake that impacts video but not audio, though recall that psychs interviews need not include video and will not include video by design if the hardware recorder is used.

* The number of processed videos with high quality QC results, as defined by an average of between $1.8$ and $2.3$ faces detected per QC-extracted frame.

* The number of processed videos with acceptable QC results, but outside the high quality range. This includes an average of $1.1-1.8$ faces per frame or an average of $> 2.3$ faces per frame. Videos with $1$ or fewer faces per frame have a high chance of being recorded in speaker view rather than gallery view, and regardless are an SOP violation for the open interview format, as the interviewer should have their camera on in gallery view.

* The number of interviews with audio QC metrics available. If there are no errors, this should always match the total number of processed interviews.

* The number of interviews with transcript QC metrics available. This may lag the processed audio count due to the time required for TranscribeMe transcription and in some cases manual site redaction review. It is also possible for this number to be less than the audio count if a recording was rejected because of QC (i.e. volume $< 40$ db), however that has not occurred for any files processed in the AMPSCZ project to date. Otherwise, if there are no mistakes in the TranscribeMe and site review transcript upload processes, the numbers should eventually match.

* The number of returned transcripts with high quality QC results, as defined by fewer than $0.01$ inaudible occurrences per word (i.e. $< 1$ inaudible label per 100 words).

* The number of returned transcripts with acceptable QC results, but outside the high quality range. Here, this is defined as between $0.01$ and $0.05$ inaudible occurrences per word. So far, all finalized transcriptions have had QC of at least acceptable quality.

* Finally, the mean duration (in minutes) is reported from across all of the processed audio files. Recall that open interviews should be at least 10 minutes, and ideally closer to 20. Psychs interviews on the other hand are expected to be $\sim 40$ minutes, and this is also the approximate limit of what will be manually transcribed per the study budget. However, sites are free to upload much longer psychs recordings, with the understanding only the first $\sim 30$ minutes (cropped intelligently by TranscribeMe to avoid cutting off a turn) will be transcribed.

This is of course not an exhaustive consideration of the quality issues that could occur, nor is it even an exhaustive consideration of the quality categories we could define using the pipeline's QC features. The table is meant to be a bird's eye account of dataset collection status, and other tools like the site-specific emails or the DPDash heatmaps described above are necessary for a more nuanced understanding of quality control outputs.

§ CODE ARCHITECTURE (AMPSCZ INTERVIEWS DATAFLOW/QC PIPELINE DOCUMENTATION)

In this section, I provide implementation details and other technical documentation, targeted at both those in the AMPSCZ project that may need to troubleshoot the code in the future and those who may be considering using a version of this code to manage data collection for a different interview recording study. As such, references to inputs and outputs herein will focus on the inputs and outputs of this specific pipeline on the data aggregation servers, with discussion of the larger AMPSCZ infrastructure setup kept as minimal as possible.

The documentation will begin with information on installing (subsection <ref>) and running (<ref>) the code as is. It will then dive into implementation details for each module within the audio (<ref>), transcript (<ref>), and video (<ref>) paths of the pipeline. Unique to this section, I will report the security information presented for the necessary security reviews that were done before installation on the Pronet and Prescient central servers (<ref>). Finally, the section will close with an account of the most likely updates that might be needed to adapt the code to run in somewhat different study contexts, as well as quality of life improvements suggested for the future of AMPSCZ (<ref>).

§.§ Installation

It is highly recommended that this code is run on a Linux machine, but there are minimal hardware requirements as the pipeline was designed with low compute resources in mind. Given the naturally large file sizes of recorded video interviews however, a decent amount of system storage space is expected (more detail on required space can be found in the main chapter section <ref>).

The code requires FFmpeg (tested version 4.0.2) and Python3 (tested version 3.9.6) to be installed, as well as use of standard bash commands, including realpath and dirname. For the email alerting to work, the mailx command must be configured. The python package dependencies can be found in the setup/audio_process.yml file, with the exception of SoundFile and librosa on the audio side and PyFeat on the video side. If Anaconda3 is installed this YML file can be used directly to generate a usable python environment.

For the AMPSCZ project, the above dependencies were already installed and the sendmail command configured on the relevant servers by IT. Thus I installed my code with the following steps (once in the directory location intended for software installs):

* git clone https://github.com/dptools/process_offsite_audio.git

* cd process_offsite_audio/setup

* conda env create -f audio_process.yml

* conda activate audio_process

* pip install soundfile

* pip install librosa

* pip install opencv-python

* pip install torch --no-cache-dir

* pip install py-feat==0.3.7

* sudo yum -y install libsndfile

* This last step was done on Prescient data aggregation server only, to get SoundFile and librosa to work there. It may or may not be necessary to do this then, depending on what libraries are already installed by IT on your machine.

* If it is missing, it might require contacting IT to do the libsndfile installation, if you do not have sudo access (which ideally the account officially running daily AV processing should not, to protect raw file integrity).

Note I installed a few PyFeat dependencies (OpenCV and PyTorch) separately before the PyFeat installation, because we are using low resource servers for data aggregation and they were unable to install PyTorch without the --no-cache-dir flag. On a different machine those two steps before the PyFeat install may not be necessary.

Similarly, version 0.3.7 of PyFeat is the one used across AMPSCZ, and it is known that the latest version is not compatible with the pipeline, because it is no longer possible to specify None for the pose, FAU, and emotion models when initializing a PyFeat detector. On the Pronet data aggregation server there is not enough RAM however to even load those models, and so it is important for the pipeline to be able to load only the lightest weight face detection model when running video QC. For other projects this may not be a concern, in which case it would be fine to install the latest version of PyFeat (as of spring 2023), but would require changing the line in the video QC python function that specifies the models to be loaded. However, for an isolated conda environment to be used for the sole purpose of running this pipeline, there would be minimal benefit to upgrading the PyFeat version at present.

As time goes on it will of course be important to be aware of updates to any of the pipeline dependencies, and maintain the repository accordingly. In addition to the direct dependencies, there are also indirect dependencies on assumptions about Zoom recording settings, and for AMPSCZ there are interactions with other tools written for the purposes of digital psychiatry study management as well – mainly the Lochness [HarvardNRG, 2019] and DPDash [HarvardNRG, 2019] codebases described in section <ref>, which were further adapted specifically for the use of the AMPSCZ project. Lochness is used for pulling the data submitted by sites and enforcing the expected folder structure, but it would be possible to recreate the folder structure manually or with different software if desired. DPDash is used for visually monitoring updates to the dataset via a regularly refreshing web portal, but is not strictly necessary. These are all helpful tools for managing AMPSCZ and would likely be helpful for other future projects too; for the most part, I will assume throughout this section that this pipeline will be integrating with Lochness and DPDash equivalents wherever it is used, and will thus make clear what design decisions are influenced by those tools throughout.

Finally, note that for the transcript portion of the pipeline it will be necessary to have a TranscribeMe SFTP account setup with expected transcription settings. This is largely similar to what was already described for the phone diary code in section <ref>, but a few small (yet important) distinctions were made for AMPSCZ - these are detailed in section <ref> above. Relatedly, the folder structure enforced by the AMPSCZ implementation of Lochness and therefore expected by this pipeline is somewhat different than that expected for the audio journal code. The input expectations for the interview code are provided below (<ref>).

§.§ Running the code

The pipeline will generally be run using the four wrapping bash scripts provided in the top level of the repository [Ennis, 2023], with the path to a settings file as the single argument (shared across branches of the pipeline). The four scripts cover audio preprocessing, transcript preprocessing, video preprocessing, and finally subject-level summarization for monitoring. Each such overarching pipeline branch script will save detailed logging information from each run to a logs subfolder, which is made inside the installation folder of the repository. The audio_process conda environment should be activated prior to launching these scripts. In AMPSCZ, each site is treated by the code as an individual study, and so has its own settings file and will have each part of the pipeline run independently on it. For a template that can be edited into a settings file for a particular site in a project like AMPSCZ (or a particular study on a lab server), see setup/example_config.sh. The available settings in this file will be outlined shortly.

Each major step in a given branch of the pipeline is initiated using a script in the individual_modules folder, see this folder if specific steps need to be run independently. More details on the separate steps will be provided in subsequent sections. First, basic instructions on running the full pipeline to completion will be provided though, for each branch separately (after more details on the config to be used across branches). Note that in addition to the main pipeline, a few utility scripts are provided for combining info across studies. To see how these fit together, refer to the wrapping scripts found within the repo, currently running on a daily cronjob for each of the two AMPSCZ central servers: amp_scz_launch/production_pronet_all_sites_cron_script.sh and amp_scz_launch/production_prescient_all_sites_cron_script.sh respectively.

Specifically for AMPSCZ, the repository folder itself can be found at the path: /opt/software/process_offsite_audio on the Pronet data aggregation server, and at the path: /home/cho/soft/process_offsite_audio on the Prescient data aggregation server.

§.§.§ Available settings

The settings that can be specified for the pipeline (shown in the example_config.sh file) currently are as follows:

* data_root, which is the root path to the main data structure folder. For a given AMPSCZ server this is the same across sites, and within the Baker Lab this is the same across most studies (but not all, as data from some different collaborations are kept on different briefcases). In our terminology, this would be the path on the server to the PHOENIX folder.

* study, which is the name of the study (e.g. BLS) or site (e.g. PronetLA) on the server. This must match the folder name corresponding to the study/site.

* auto_send_on, which for AMPSCZ is expected to always be "Y". If "N", then QC will be computed for the audio without sending anything to TranscribeMe.

* transcribeme_username, which should be the username used to login to the TranscribeMe SFTP server where data transfer for the project is intended to take place. For AMPSCZ this is one email for Pronet and one email for Prescient. For a lab group this may always be the same username across studies, or it may vary depending on payment allocation details.

* length_cutoff, which is the minimum duration for an individual interview recording to be sent to TranscribeMe, in minutes. For example, one might consider an interview of less than 10 minutes to likely be a fragment or incorrectly conducted and not worth sending for transcription. However for AMPSCZ this is always expected to be 0.

* Note that a maximum length per transcript of $\sim 30$ minutes is enforced on TranscribeMe's end for the project, as the science committee wants these audios to be transcribed but does not want to spend too much on them. For a similar setup please discuss with TranscribeMe (though I do not necessarily endorse this approach personally, see Appendix <ref>).

* db_cutoff, which is the minimum volume for a mono interview audio recording to be considered acceptable for transcription. In our hands on prior projects we decided on a 40 db cutoff, so this is also what is used for AMPSCZ. To date it has only been triggered once while $\sim 500$ interviews have now been successfully transcribed, and inaudible rates in the transcribed interviews have largely been very good. Future projects might consider further tweaking this number, though it may be largely a non-issue for both Zoom audios and handheld EVISTR recordings in the context of transcribability. For lab onsite interview recordings using other audio collection methodologies, we have seen more variance in quality and more frequently unusable audio though.

* auto_send_limit_bool, and if this is "Y", auto_send_limit. For AMPSCZ and for internal projects we have kept this setting to be "N", but it is an option provided for safety with research expenditures, especially if the code is run less frequently or left to run only at the end of a study. When the setting is turned on, auto_send_limit should be an integer specifying the maximum number of total minutes that it is acceptable to upload to TranscribeMe in a given run of the code with these settings. If the sum of audio lengths selected for upload exceeds this value, the files will be left set aside in the respective to_send folders (described more in implementation details below), but will not yet be uploaded to TranscribeMe - so the decision can be sorted out manually instead.

* lab_email_list, a comma-separated string of email addresses for the detailed monitoring updates to go to. This was originally intended to contain site-specific addresses for AMPSCZ, but to date monitoring has been largely centralized, and so the addresses here are similar across configs. All of the daily (as needed) site-specific emails will go to the addresses listed under the current site config. For central monitoring across many sites/studies, I would highly recommend setting up targeted email filters, but if monitoring is well-distributed than the rate of messages should be extremely manageable.

* site_email_list, another comma-separated string of email addresses specifying those that should receive emails when there are outstanding transcripts awaiting manual redaction review for that site (first 5 transcripts from each site and a random 10% thereafter). These are the only emails that actually go to site contacts for AMPSCZ at this time. Obviously this is only relevant if manual redaction review will be a part of your project.

* server_version, which is a label that will be applied to email headers to help email filters differentiate between different projects. For AMPSCZ, we primarily used this to easily distinguish between development testing alerts and alerts from real production data when those servers were running simultaneously for Pronet (and later for Prescient) near the start of data collection ramp up, as central server name (e.g. Pronet) is implicit in site name (e.g. PronetLA). In other large scale projects, this string could be used to make a variety of other designations as needed.

* transcription_language, which is a string (currently in all caps) that specifies the language the site will be using to conduct most of their interviews. The pipeline uses this string in the filename it uploads to TranscribeMe, to assist their process in assigning the correct transcription team to a given upload. When transcripts are pulled back this part of the name is removed, so it is used only in the TranscribeMe upload process and not on actual pipeline outputs (or intermediates). This could theoretically be used to convey other messages to TranscribeMe such as transcription settings, in future adaptation of the code. The information here will also be compiled as part of metadata that goes to the NIH data repository for AMPSCZ.

The configs being actively used (or for some sites prepared to be used) for AMPSCZ can be found in the amp_scz_launch/production_pronet_site_configs and amp_scz_launch/production_prescient_site_configs folders for Pronet and Prescient respectively.

To add new sites to the existing processing for this project, one can simply add another config in the appropriate folder for that central server, working from the existing configs for that server as a template (and referring to the above where additional reference is needed). One would then just need to make sure that there are not issues with folder permissions when Lochness initializes the structure for that site. Obviously for setting up a brand new process, one would aim to replicate the current setup for Pronet and Prescient as far as settings files and cron job go, after following the installation instructions in the preceding section to get the code and its dependencies onto the new central server. In the case of a new central server, it would obviously be necessary to change some further settings within each new config – such as the TranscribeMe username, which would require creating a separate SFTP account with TranscribeMe reps first.

On that note, an additional component built into the config .sh file is the management of the password for the specified TranscribeMe SFTP account, which I will next describe how to set up for new servers/projects.

§.§.§ Providing TranscribeMe password to the pipeline

In order for the code to run automatically on a cronjob, the password for the TranscribeMe SFTP server needs to be stored somewhere on the source server. By default, this is assumed to be in a file named .passwords.sh directly under the repo folder where the code is being run. The .gitignore included with the code will ensure that the .passwords.sh file is never pushed back to the central repository (as well as any logs generated by the code more broadly). On both Pronet and Prescient data aggregation servers, this passwords file has 700 permissions and is owned by the account that is set to run the pipeline code. For more on privacy/security, see section <ref> below.

If one will use an identical setup to the AMPSCZ central Lochness servers, then it is straightforward to just create the process_offsite_audio/.passwords.sh file directly and then paste in the following two lines:

export transcribeme_password

Where of course [password] is replaced with the TranscribeMe SFTP account password corresponding to that server's account. Note that TranscribeMe will also periodically require this password to be updated, so for all projects (including both accounts for AMPSCZ) it will be necessary to occasionally update the password contained in the .passwords.sh file directly on the applicable server.

If one wants to store the password file elsewhere or needs to use different password files for different studies/sites operating out of the same server (because different TranscribeMe SFTP accounts are being used), the path to the passwords file can be specified using the passwords_path variable found at the bottom of each config file (like example_config.sh). The passwords file itself needs to be an sh file type with content matching the above to work out of the box, but there is no other restriction on the naming/folder placement (as long as the running account can access) when its path is appropriately specified within the corresponding config.sh settings file.

§.§.§ Using the audio portion of the pipeline

As the particular study (or in the case of the AMPSCZ project, site) of interest is specified as part of the settings files along with all other relevant parameters, the audio branch of the pipeline in Figure <ref> can be easily run with the following command:

bash interview_audio_process.sh example_config.sh

Of course the user should substitute the path to the config file with the settings they actually want to use in the place of "example_config.sh" here.

§.§.§ Using the transcript portion of the pipeline

Similarly, to run the transcript branch of the pipeline in Figure <ref>, the following analogous command can be used:

bash interview_transcript_process.sh example_config.sh

Note that the config file here is identical to the config file used for the audio portion of the pipeline, as the settings were designed to work across all phases of the code.

§.§.§ Using the video portion of the pipeline

Finally, to run the video branch of the pipeline in Figure <ref>, the following analogous command can be used:

bash interview_video_process.sh example_config.sh

Note that the video processing can be run in parallel with the audio in theory, as they do not rely on each other - but the transcript code of course will only work after the audio part of the pipeline has been run and TranscribeMe has returned some transcripts.

§.§.§ Generating additional summary stats for a single site

Recently, a smaller fourth branch was added to the pipeline to facilitate generation of the site-level summary view in DPDash (Figure <ref>) and ultimately to facilitate the site-wide summary supplemental utility scripts mentioned in the main chapter <ref>. However, even when applying this code to a single study scenario, the summary stat portion of the pipeline provides information helpful for monitoring data collection more regularly via quick scans of the summary outputs, so this branch of the code is likely worth running regardless.

To run these summaries on a study of interest, the following command analogous to the above can be used:

bash interview_summary_checks.sh example_config.sh

This script enables the monitoring that is done for each individual site creating subject-wide summaries. The other pipeline components focus on participant-level processing, looping across all participants for a given study/site.

§.§.§ Input expectations/folder structure

For the AMPSCZ project, we expect two different types of interviews (open and psychs), which are both handled by the full pipeline and will both be included in the same email alerts, but are processed independently by each module with outputs saved separately. These expectations are built into the wrapping pipeline scripts, and so at present the names of these two interview types are somewhat hard-coded.

The code expects to work within a PHOENIX data structure matching the conventions described in the AMPSCZ standard operating procedure (SOP). The pulling of data submitted by sites into this folder structure is handled by Lochness in our case, though it is still possible for sites to violate some interview naming conventions within this framework, as Lochness does not enforce e.g. the presence of a date in the folder name as a requirement for upload to the central server. Regardless, I will detail exactly what inputs are expected by the code from the perspective of already being on the central server in this section.

Note in addition to the upcoming expectations for interview naming conventions, the code also requires a metadata file directly within the site/study level folder contained on the GENERAL side, and will only process a particular participant ID if that ID appears as a row in said metadata file with a corresponding valid consent date (per Lochness guidelines, though again this could be replicated independently). If there are issues encountered with the basic study setup appropriate error messages will be logged.

Note the pipeline only ever reads from raw datatypes. All pipeline outputs are saved/modified as processed datatypes. The deidentified data that make it under the GENERAL side of processed will subsequently be pushed by Lochness to the predict server for downstream analyses and eventual upload to the NIH data repository.

More specifically, the raw audio and video data available for an interview are expected to meet the following criteria on the central server, first described for Zoom interviews and then for EVISTR (psychs-only) recordings. For more on the actual conduct of these interviews, see section <ref>.

For Zoom interviews, each session should have all files under a single folder, [Zoom_folder_name]. It should contain the interview components like so, matching automatic naming conventions currently used by Zoom:

* Single video should then be found directly under [Zoom_folder_name]/video*.mp4

* Single collapsed all-speakers (mono) audio file should be directly under [Zoom_folder_name]/audio*.m4a

* Diarized speaker-specific audios should be under a subfolder, [Zoom_folder_name]/Audio Record/*.m4a

* Be aware there is a chance these filenames contain participant names

* represents a wildcard that could match any additional text in the filename string.

We require [Zoom_folder_name] to begin with YYYY-MM-DD hh.mm.ss . Again, this is default Zoom formatting, and the code utilizes the date and time info provided, so it is important to not change it. Anything can come after the date/time metadata (but note the second space at the end to split out the time).

For psychs interviews only, the SOP also allows for interview sessions to be recorded by a single device, so single audio files (each representing one session) of the following form are also allowed directly as a raw interview submission. Like the Zoom interviews, we also require timestamp metadata in the top level name. The SOP requirement here is YYYYMMDDhhmmss.WAV.

In this case, the code is currently strict in requiring exactly this naming format with nothing appended. Note the capitalization of WAV, which is also required at this time (per the default of the approved recording device).

The transcript side of the pipeline primarily relies on outputs from the audio side. It also expects a box_transfer/transcripts folder on the top level of the PHOENIX structure's PROTECTED side, to facilitate transferring completed transcripts to the corresponding sites for correctness review. In finalizing the transcripts that are subsequently returned by sites, it looks for the interviews/transcripts datatype on the PROTECTED side of raw, and it expects these transcripts to have the same name as those originally returned by the pipeline via TranscribeMe.

On that note, the code also expects TranscribeMe to return transcripts with a name identical to the uploaded audio filename, just with the extension switched to .txt. So it is possible, though not an issue we've encountered much, that TranscribeMe could cause a stall in processing by violating naming conventions.

§.§.§ Using cross-site utility functions

In cases where multiple different studies - or for projects like AMPSCZ, sites - are being tracked simultaneously, it will often be desirable to have some monitoring at the server-wide level, summarizing progress per study/site in one place. For the present codebase, I have included supplemental utility scripts for AMPSCZ (but easily adaptable to other contexts) to generate such Pronet- and Prescient- wide summaries. They continue to run automatically for this project and have had minimal issues since implementation, in addition to helping us catch a number of quality issues as a complement to the views in DPDash.

These utility scripts create the weekly histograms, tables, and combined QC + accounting CSVs that are sent out by the code, as described in the primary monitoring report of the main chapter <ref>. They also integrate with DPDash tools to facilitate central monitoring. Note while these functionalities are particularly helpful for monitoring quality of successfully processed interviews (and noticing site-specific trends), data upload issues and other more fundamental SOP violations may not be reflected in these summaries, which focus on those interviews with QC information available. Please continue to refer to core pipeline features on the site-specific level for tracking of logistical problems with interviews.

Recall that the primary resources attached to the weekly monitoring emails are:

* A PDF containing histograms of select QC feature distributions across the server, with open and psychs interview types represented separately via a stacked histogram approach.

* A PDF containing histograms of select QC features distributions across the server for open interview only, but with different sites now represented separately.

* An analogous PDF for psychs interview distribution visualization across sites.

* An HTML table containing counts of good, acceptable, and bad quality interviews by site and type.

* The abridged version of that table focuses only on transcript quality, but there is another table attached with similar counts giving more detailed interview counts across different modalities.

* A CSV containing information on how many subject IDs for each site have transcripts available for both open interview timepoints (baseline and 2 month follow-up) versus just one or the other (or neither, for any IDs that are known to have some interview upload processed across types).

* The combined QC CSV with key QC metrics from across modalities provided for all interviews processed on the server to date – this is similar to what is imported in the combined DPDash view.

A concatenated SOP violations list CSV and overall processed accounting CSV (more detailed contents of which will be described subsequently) are then attached to a second weekly monitoring email for those looking into possible upload/file organization mistakes or who are otherwise doing more detailed data evaluation.

The script coordinating the cross-site monitoring utilities for AMPSCZ can be found at amp_scz_launch/final_all_sites_utility.sh. The arguments it takes are specified within the amp_scz_launch/production_pronet_all_sites_cron_script.sh and amp_scz_launch/production_prescient_all_sites_cron_script.sh scripts that control the running of the entire pipeline for each respective AMPSCZ central server. The final_all_site_utility.sh script organizes the summary CSVs on a daily basis - optionally emailed to a specified list, which is currently just "mennis2<EMAIL_ADDRESS>Those CSVs are used on a weekly basis by another script that is called by the top level cron scripts, again with settings specified therein. Histograms and formatted tables are restricted to this weekly update. That script is amp_scz_launch/weekly_logging_utility.sh, which manages the generation and emailing to a broader list of project staff of the weekly figures and tables every Monday. Both the described utility scripts draw on python functions found under the amp_scz_launch/helper_functions repo folder.

In the main chapter <ref>, I also investigated relationships between some of the QC features in the early AMPSCZ interview dataset. Although scatter plots are not part of automated monitoring updates, the notebooks used to generate them can be found in the amp_scz_launch/preliminary_stats repo folder. By downloading the combined-QC.csv sent by the weekly logging utility, it should be possible to rerun the notebook at any time to get the latest scatter plots, though one may want to tweak some of the visual settings as the number of sites and the number of data points increase over time.

As far as the automatic daily and weekly summarization scripts though, little should need to be updated by AMPSCZ, and the main update for future projects looking to use the code would be the broader issue of handling cases where the interview types are not specifically "open" and "psychs". The primary change that may need to be made periodically for AMPSCZ will simply be to change the list of addresses receiving emails for each central server. The settings for that, particularly relevant for the more widely dispersed weekly figures, can be found towards the bottom of each of production_pronet_all_sites_cron_script.sh and production_prescient_all_sites_cron_script.sh central bash scripts. There the pii_email_list specifies the addresses to receive combined CSVs with SOP violation and processed accounting logs to date – CSVs that contain dates and could potentially contain real names in certain violation scenarios, but are important for logistics tracking. Then the deid_email_list specifies the much longer list of addresses to receive the QC figures primarily described here.

For AMPSCZ members making changes to these sorts of settings, please be sure to update the code on the repository for any longer term updates and also ensure that the new code is correctly pulled to the relevant data aggregation servers. A small additional note to be aware of is that the Pronet data aggregation server uses -A flag for mail command attachments currently whereas Prescient uses the more standard -a flag.

For changes that might need to be made on the DPDash displays of combined QC, note that the utility scripts directly in this repo run daily on the data aggregation servers and only function to inform the email alerting system that is the major component of monitoring implemented directly by this pipeline. Saved combined CSVs and records of sent emails can be found within the pipeline's logging outputs. But because Lochness pushes only outputs stored directly within the GENERAL side of the PHOENIX data aggregation folder structure to predict, which is defined very clearly on the level of the individual subject ID (and interview type), combined CSVs are not pushed to predict. The combined CSVs generated for DPDash then are actually created directly on predict. This is done using very similar code to portions of the cross-site utility scripts that were described here, adapted directly from those scripts. However they are run independently by DPACC, and stored in a different repository related to AMPSCZ's DPDash [AMPSCZ-DPACC, 2021] utilities. That repository additionally includes some other utility functions originally written by me to format other summary information for display on DPDash, in particular the code defining CSVs in the needed form for our DPChart views. In sum, for changes that may need to be made to DPDash steps of the interview monitoring workflow, please contact DPACC instead of working directly in my repository.

Now that the pieces of the pipeline have been overviewed with the most key instructions provided, I will provide more detailed information on the steps contained within each branch and their respective implementations. After that I will discuss security considerations, and finally wrap with a detailed account of various future updates suggested for this codebase.

§.§ Audio processing components

The major steps of the audio side of the pipeline are as follows, presented alongside their corresponding wrapping bash modules (as subbullets), modules which should be reviewed in greater detail if one plans to run only pieces of the pipeline (Figure <ref>):

* Identify new interview audio files in raw by checking against existing QC outputs and interview protocol upload expectations, first for open interviews and then for psychs interviews. This involves converting any newly found top level interview audio files to WAV format (if not already), thus saving WAV versions (a copy if EVISTR WAV format on raw) of all new mono audio files within the corresponding folder on the processed side of PROTECTED.

* These tasks are handled by the run_new_audio_conversion.sh module, which relies on ffmpeg but does not utilize python.

* Run quality control functions on the new audio files, which at the start of this step are found with default date/time naming under a temporary folder on processed. A sliding window audio QC is run to create a single output for each file that will also allow for easy checking of whether a raw interview has already been processed. Then the new audio files are renamed to match SOP convention for processed datatypes, including removing date/time info and instead naming in terms of the study day that the interview occurred (where the corresponding participant's consent date is defined as day 1). Finally, interview-level QC is calculated on the renamed files, to produce final shareable QC outputs that can assist with downstream steps.

* These tasks are handled by the run_audio_qc.sh module, which calls the following python functions to initiate each of the 3 described steps for a given participant ID and interview type:

* sliding_audio_qc_func.py (run per file by the module instead of per participant/type)

* interview_audio_rename.py

* interview_audio_qc.py

* Details on the exact outputs produced by these steps will be provided subsequently.

* Check for sufficient volume levels to determine which mono interview audio files should be uploaded to TranscribeMe (40 db for AMPSCZ). This will also enforce other per interview upload requirements if they are specified in the current settings file (e.g. minimum individual interview duration), and it will ensure that uploaded files indeed met the basic SOP metadata requirements as well, denoting reason for rejection for any file that is not approved to be sent. For psychs interview type specifically, it will prevent the sending of any interview that is not the first from its day number for that subject ID (so that sites cannot get around the hard cap on transcription duration for individual interview sessions that TranscribeMe implements for this project).

* These tasks are handled by the run_audio_selection.sh module, which calls the interview_audio_send_prep.py python script for each subject ID/interview type to actually identify and set aside approved files for the next part of the audio pipeline branch.

* Upload all approved new audio files to TranscribeMe SFTP server (with temporarily appended site-based language marker to the end of the processed file name convention, to assist TranscribeMe in assigning transcribers). Any audio that fails to upload will be kept in the folder with approved audios so that a future run of the code can hopefully successfully upload it. All properly uploaded audio (which in practice is basically all selected audio) are moved to a different subfolder to be used by other parts of the pipeline in tracking pending transcriptions.

* These tasks are handled by the run_transcription_push.sh module, which primarily uses the interview_transcribeme_sftp_push.py script (called for each subject ID and interview type with files awaiting upload) to handle the SFTP push. More details on the upload implementation can be found in security/privacy section <ref> below.

* This step can be turned off via settings file, where it is also possible to add a "stop loss" of a certain max duration sum for uploads from the current run (the latter done where applicable by the same module calling the overall interview_audio_length_check.py python script). However AMPSCZ does not restrict interview uploads to TranscribeMe in any such way.

* Compile and then send an email listing all files that were newly processed, indicating which were successfully uploaded to TranscribeMe and otherwise documenting possible quality issues. Note that this step requires "build up" of content from all previous steps, and should really only be run in the context of full usage of the audio branch of the pipeline at present. Additionally, it only runs in cases where upload to TranscribeMe is turned on (though one could hack around this by setting the mentioned "stop loss" maximum total sending limit to 0 sum minutes).

* These tasks are completed by the run_email_writer.sh module, which calls the study-level interview_audio_email_write.py python helper to add further information to the drafted email body.

* Update the file accounting log to specify all of the raw audio files that were newly recognized and (hopefully) processed on the current run, taking note not only of the present date but also metadata that could be (incorrectly) changed in the future, such as the consent date that is listed for each relevant subject ID at the time of file processing. Further, this step maps the identified day and session numbers to the raw real date/time, it denotes whether a file was rejected for upload to TranscribeMe or not, and it collects extended information on the contents of the corresponding raw folder – such as the number of speaker-specific audio files present in any Zoom interview folder.

* These tasks are completed by the run_final_audio_accounting.sh module, which calls the python script interview_audio_process_account.py on each subject ID and interview type. Note this step, unlike the daily email alerting, occurs even if automatic TranscribeMe upload is turned off.

* The primary use case for the outputs of this step within the broader pipeline will be downstream site-wide monitoring/alerting on possible QC and SOP concerns, processes to be described soon as part of the details on the final branch of the pipeline.

For implementation details, one should refer to the python functions (found under individual_modules/functions_called within the repo folder) that are used by the wrapping bash modules (found directly under individual_modules), as described in this list.

Additionally, there are a number of similarities between the present pipeline and the preprocess portions of the audio diary pipeline detailed in section <ref>, at least with regards to the main audio QC functionality and the interfacing with TranscribeMe. One might refer to that section (as well as other components of chapter <ref>) to complement the knowledge found here and in the main text of chapter <ref>. However for now, I will focus on describing the important outputs produced by the audio branch of the AMPSCZ interview pipeline.

§.§.§ Key direct outputs of audio branch of pipeline

Aside from the renamed and converted audio files that can be used directly for future processing by AMPSCZ code, the primary immediate outputs of this part of the pipeline are the audio QC features. Of course the audio branch also uploads audio to TranscribeMe and does other file accounting to facilitate downstream processes, but those steps (and their subsequent outputs) will be focused on in the upcoming sections on other branches of the pipeline. As such, I will close this section with more detailed information on the audio QC outputs produced by my code.

The primary audio QC output is the interview-level quality summary stats for each mono interview audio, mapped to the final date-removed filenames produced by the renaming step, with one row per interview session. The interview_audio_qc function creates (and subsequently updates) one such QC CSV per subject ID and interview type, saved on the GENERAL side of processed. I will first explain the filenaming conventions and use cases for these CSVs, and then will go into the specific QC metrics contained within.

Under GENERAL, the main interview-level CSV for a given participant and interview type can be found at: [siteFull]/processed/[subject]/interviews/[type]/[site2Digit]-[subject]-interviewMonoAudioQC_[type]-day[start]to[end].csv. To illustrate, an example such path would be: PronetOR/processed/OR03988/interviews/open/OR-OR03988-interviewMonoAudioQC_open-day78to78.csv. Of course as new interviews are processed, the day numbers will expand to reflect the latest range: so the example file mentioned might one day have suffix "day78to138.csv" for instance, if OR03988 has a second open interview conducted 60 days after their initial open interview.

The pipeline will automatically only retain the latest audio QC summary CSV for a given ID and interview type, ensuring that prior records are properly concatenated to the current CSV before deleting CSVs with outdated day numbers in the filename. It is important that outside processes never delete these QC CSVs however, as the persistence of prior QC records is vital to the operation of the pipeline over the course of a project. For example, these CSVs are loaded by the audio selection step of the code, to determine which audios successfully passed the metadata needed for renaming as well as the simple quality control threshold(s).

Because the mentioned CSVs are stored on the GENERAL side of processed, they will additionally be pulled by Lochness to the fully centralized predict server, where DPDash can be used to visually monitor progress updates and where data will get staged for later sharing with the NIH data repository. These QC results that are shared there are indeed used as part of the high level interview metadata mapped to redacted interview transcripts that are planned to be released (prepped by the transcript branch of the pipeline to be described below).

It is also on predict where further merge operations and QC summary stat filtering/binning occur, in order to construct the CSVs that are imported into DPDash to create the central server-wide views shown in the main text of chapter <ref> – including both the combined longitudinal heatmap of all sessions and the per subject counting stats used in the open interview bar charts. Because Lochness does not handle renaming of files (for safety against data deletion), predict will by default store all of the outdated DPDash CSVs with different day numbers in filenames. Future projects using DPDash might consider reevaluating the CSV naming convention that it requires, to avoid this problem with constantly changing filenames; first and last available day numbers are directly contained in the CSV anyway.

Recall that my code does not handle any processing or maintenance on predict, and is not influenced by any operations on predict (though these could hinder the usage of integrated tools like DPDash with my outputs).

Critically, the columns found in each interviewMonoAudioQC file (with values mapped per interview) are as follows:

* Metadata required by DPDash [HarvardNRG, 2019] or to otherwise assist in downstream merges.

* Note that only a necessary subset of DPDash columns are actually filled with values however.

* Length of audio (in minutes).

* Overall volume (in dB).

* Standard deviation of amplitude (though as mean amplitude of audio files is generally 0, this is often a direct transformation from the log scaled dB).

* Mean of spectral flatness (computed by librosa).

The specific column names used for these features are:

With reftime always containing np.nan as the value but present as the first column for DPDash compatibility. The rest of the columns are appropriately filled out based on available interview information. Recall that this QC output focuses on mono interview audio, which will be the top level audio file containing all speakers in the case of Zoom.

It is worth highlighting that an interview might be rejected by the audio selection based on the content of the CSV (too low volume for the corresponding row, multiple psychs interview sessions uploaded on the same day from one ID, etc.), reasons that will be logged in the email alerts. However a file also may be rejected because the queried filename could not be found within this CSV, or because the corresponding CSV was not found in the first place. The former will log as a metadata issue and the latter as a permissions issue, as these are possible reasons for a detected interview audio that passed initial SOP screening to later fail. In practice, especially for long interviews on Pronet, the same error messages might also trigger though because the audio QC function crashed before the information about that interview could be saved. More on troubleshooting this issue (among others) can be found within the future directions of the upcoming supplement section <ref>.

As mentioned, the audio QC processing includes a sliding window QC operation as well, which is performed on the identified new (WAV-converted) mono audio files as a first step under the PROTECTED side of processed before they are renamed according to typical output conventions. One such sliding window QC CSV is saved for each detected mono interview audio file, and the CSV filename matches the date and time given for the corresponding interview in raw – which is then used for filtering out previously processed interview audios on subsequent pipeline runs.

These sliding window QC outputs are not currently used for any other purposes besides processing documentation (and thus should not be deleted - they take up minimal storage space anyway). The filename map txt files giving a very direct command line accessible link between raw interview name and processed interview renaming are similarly important to retain through the duration of a project. Because they stay on the PROTECTED side of processed, these outputs within AMPSCZ are never shared with predict either, and in some sense are thus only intermediates. Still, they could assist in the future for quickly identifying certain finer timescale issues, like psychs interview audios that contain a notable break for a time in the middle of the recording. It is particularly plausible that such outputs might be used to facilitate certain steps of the upcoming feature extraction pipeline, which is to be written for obtaining shareable acoustic measures of possible clinical relevance from interview audio files.

Indeed, a given sliding window output CSV will contain decibel level and mean flatness values as the primary metric columns, with each row corresponding to a particular slice of the interview. By default (and therefore for all AMPSCZ interviews), the sliding window QC computes metrics in 3 second increments with no overlap between these bins. However the default could easily be changed for a future project by supplying additional arguments when the sliding QC python helper is called by the wrapping audio QC module.

§.§ Transcript processing components

The transcript processing branch of the pipeline of course generally relies on successful completion of the audio processing branch of the pipeline, and subsequent return of TranscribeMe transcriptions as specified. Most of the steps thus rely on assumptions about TranscribeMe, though one might still utilize the latter modules in a case where obtained transcripts are analogous but the project uses a different upload/download process, or conversely the earliest modules in a case where TranscribeMe is being used but the specified settings for the project are very different.

For AMPSCZ, all branches of the pipeline are obviously being run daily for each site, and if no new transcripts have been returned by TranscribeMe for a particular site, the transcript branch of the pipeline will simply flag any pending transcriptions in the monitoring steps (if relevant) and gracefully exit. Nevertheless, I will now provide a parallel list of major code steps and their corresponding wrapping bash modules for the transcript branch of the pipeline, as I did for audio above:

* Check the TranscribeMe server for any pending transcriptions, and pull back all those that have been newly completed. Of the newly pulled transcripts, direct them either towards the site manual review process or towards the next steps of the processing pipeline (bypassing manual review), as per the procedure defined in the SOP. This step additionally takes note of the current status of each such transcript as it works through them, to compile an up to date monitoring email (if there is any actively relevant information to share for the site's transcriptions status).

* These tasks are handled by the run_transcription_pull.sh module, which primarily uses the interview_transcribeme_sftp_pull.py script (called for each subject ID and interview type with files awaiting transcripts) to handle the SFTP pull. More details on the download implementation can be found in security/privacy section <ref> below.

* The run_transcription_pull wrapper itself handles manual review assignment, first determining whether the site has had at least 5 interview transcriptions sent for manual review yet, and then if not assigning all new transcripts to manual review, otherwise assigning each new transcript to redaction review with 10% probability while allowing the rest to skip manual site checking.

* To keep track of transcripts that are still awaiting TranscribeMe transcription, the corresponding audio WAV files are kept in a pending_audio subfolder within each participant and interview type's PROTECTED side processed folder by the overall pipeline. These files should obviously not be deleted then, but as transcripts are successfully pulled back to the data aggregation server from TranscribeMe's SFTP server, this first step of the transcript branch of the pipeline will move them to a different (completed_audio) subfolder. The contents of those completed_audio subfolders can safely be deleted at any time if storage concerns arise.

* Identify any transcripts that were sent for manual site redaction review and have been newly returned, to copy them to the appropriate location on the data aggregation server for file organization and downstream pipeline steps. Also identify those transcripts (if any) that are still awaiting manual review for a given site, to put together a list for a daily email alert to the site contacts (as needed).

* The former task is handled by the run_transcription_review_update.sh module, and the latter by the run_transcription_review_alerts.sh module – both of which are simple bash scripts written for these AMPSCZ-specific accounting issues.

* Create redacted versions of any newly finalized transcripts, whether just returned by TranscribeMe and bypassing manual redaction review or just returned from the site review.

* This task is handled by the run_transcript_redaction.sh module, primarily through calling the redact_transcripts_func.py python helper on each new transcript identified to have a redacted copy generated.

* Recall that for the AMPSCZ project, TranscribeMe returns transcripts with redacted markings surrounding all PII/PHI words, but not obfuscating the words themselves (a good idea to consider for future projects). To mark words here we use curly braces as our convention, because they are not otherwise used in TranscribeMe's notation. Therefore TranscribeMe is the one performing the redacting, but this step is needed in the code to create shareable fully redacted versions by replacing all marked words with REDACTED.

* Convert any newly redacted transcript text files to CSV format, with each row a line in the transcript (here mapping to individual turns in the interview) and columns corresponding to TranscribeMe speaker ID, TranscribeMe turn start timestamp, and the verbatim text itself. This step will also confirm that the returned txt is correctly UTF-8 encoded and clean up excess white-space characters.

* These tasks are handled by the run_transcript_csv_conversion.sh module, a pure bash script written to parse transcript text files formatted according to TranscribeMe's convention(s).

* Compute interview-level transcript QC stats across the redacted transcript CSVs, to produce shareable QC outputs for transcription monitoring on the level of the subject ID and interview type – analogous to the final audio QC CSVs described in the preceding section.

* These tasks are handled by the run_transcript_qc.sh module, through calling the interview_transcript_qc.py python function for each participant ID and interview type with transcripts available. Details on the exact outputs produced by this QC function will be provided subsequently.

* Send the daily monitoring/alerting email(s) when applicable. The main monitoring email, to go to central monitors at present, will list all the transcripts that were successfully pulled from TranscribeMe for the current site, and those still awaiting transcription, as well as the transcripts newly returned from manual redaction review. The alerting email listing any transcripts actively awaiting manual redaction review goes directly to site contacts by contrast.

* Both of these emails were compiled as part of earlier pipeline steps described, and are only sent near the end of the transcript pipeline process because major warnings/errors that arise in later conversion and QC steps may be appended, to assure they are not missed.

* The sending of these emails (when created due to relevant updates being available) to the email addresses specified in the corresponding config occurs directly in the top level transcript pipeline branch wrapper (interview_transcript_process.sh), rather than occurring in a separated module.

* Update the file accounting log with information on current status of all of the newly recognized/moved/processed transcripts, including additional metadata like whether the transcript was marked for manual redaction review, the date that each major step in the transcript flow was completed (thus far), and specific transcript txt encoding info. This is an analogous step to the final accounting step on the audio branch of the pipeline, with similar downstream uses.

* These tasks are completed by the run_final_transcript_accounting.sh module, which calls the python script interview_transcript_process_account.py on each subject ID and interview type.

Note that to transfer files back to sites for the described manual review, separate code was written - by the Pronet IT team for push from their data aggregation server to Box and by DPACC for push from the Prescient data aggregation server to Mediaflux.

When my code designates a transcript for manual redaction review, it will put a copy of it into the PROTECTED/box_transfer/transcripts folder on the respective data aggregation server's main PHOENIX data structure. This folder is called box_transfer on both servers for simplicity. The supplemental transfer scripts (both found outside my repository) will then use information in the transcript file names along with Box/Mediaflux API tools to upload the transcripts for review to the corresponding site's Box (or Mediaflux) transcript review folder, moving them on the server from box_transfer/transcripts to box_transfer/completed to denote the transfer was successful. As my code checks this completed subfolder to see how many have been transferred so far for a given site, it is important that the transcripts stay there.

When my code designates a transcript for manual redaction review, it will also move the version of the transcript returned directly by TranscribeMe into a prescreening subfolder that is contained within the folder where both the raw transcripts that bypassed site review and the returned versions of manually reviewed transcripts are kept on the top level. This way we can always compare versions of transcripts before and after manual site review, in which sites should add curly braces around any words they feel were missed PII by TranscribeMe before returning them back to the data aggregation server.

To return successfully reviewed transcripts, the sites should simply move finished transcript text files from the "For review" subfolder they have in their transcript review Box/Mediaflux folder, to the corresponding spot under the "Approved" subfolder within the transcript review Box/Mediaflux – without changing the file path name under these subfolders when doing the move between the two. For more specific instructions on performing this transfer (and the manual review itself), please see the AMPSCZ project SOP. There is also some information on this process as part of the common site mistakes list provided in the main text section <ref>.

Lochness pulls any transcripts that have been correctly moved by sites to within "Approved" in this way, uploading them to the PROTECTED side of raw on the data aggregation server. Then my code copies newly returned transcripts into the corresponding places on the PROTECTED side of processed to maintain a complete set of final transcript txts with PII visible, handling the rest of the transcript processing steps from there (as were described above). Note that the only transcripts under raw will be those that underwent manual review and were subsequently returned, those that bypassed manual review will stay only under processed folders.

§.§.§ Key direct outputs of transcript branch of pipeline

The primary outputs of this branch of the pipeline include the transcripts at various stages of processing, as well as the final transcript-derived QC measures. In this section, I will provide more information about the contents of these outputs and their locations on the data aggregation servers.

We receive full verbatim transcriptions from TranscribeMe, with turn-level timestamps of second resolution and speaker identification included. As mentioned above, any PII in the transcription is marked by curly braces. Cost per minute varies by language. Saved transcripts can be found both with PII visible on the PROTECTED side of processed, and with marked PII fully redacted on the GENERAL side of processed. The former transcripts cannot be shared with the wider group but when needed can be further processed by those with access to the data aggregation server data structure. The latter transcripts are automatically pushed to predict by Lochness, and will eventually be staged for regular transfers to the NIH data repository. The redacted GENERAL side transcripts are also converted to CSV format for easier processing, as described above, which facilitates the transcript QC process as well.

The final PII-visible transcript txts can be found on the main PHOENIX data structure of each central aggregation server at paths of the form: PROTECTED/[siteID]/processed/[subject]/interviews/[type]/transcripts/[siteID]_[subject]_interviewAudioTranscript_[type]_day[day#]_session[session#].txt, where day number and session number are formatted as 4 and 3 digit integers respectively, defined based on the study day and interview number determined by the renaming function in the audio branch of the pipeline. Recall that there are many reasons (like split interview folders or backdated uploads) that could cause the automatically-determined session numbers to diverge from the true session number per the protocol for each interview type. This is discussed at greater length within main text section <ref>, but ultimately it does not affect the workings of the interview dataflow/QC pipeline, and can easily be taken into account in downstream interpretation.

The final redacted transcript txts can be found analogously at paths of the form: GENERAL/[siteID]/processed/[subject]/interviews/[type]/transcripts/[siteID]_[subject]_interviewAudioTranscript_[type]_day[d#]_session[s#]_REDACTED.txt. Under each GENERAL side processed transcripts folder, there is also a csv subfolder that holds the CSV-converted versions of each redacted transcript text file. In the future, AMPSCZ analysis work taking place on the data aggregation briefcase might include comparing the fully redacted transcripts to their PII-visible counterparts.

Of course, analogous to the audio branch of the pipeline, transcript QC features are stored on the GENERAL side of processed using DPDash naming conventions. Thus for all participants with interview transcript(s) of a particular type available, there will be a CSV containing interview-level transcript QC metrics for those sessions, found on the data aggregation servers at paths of the form: GENERAL/[siteFull]/processed/[subject]/interviews/[type]/[site2Digit]-[subject]-interviewRedactedTranscriptQC_[type]-day[start]to[end].csv. The overall construction/structure of these QC CSVs is similar to those for audio QC, and they are obviously also pushed by Lochness to predict and subsequently imported for viewing in DPDash. Additionally, they play a critical role in the quality monitoring processes that occur outside of DPDash, including some that utilize the data aggregation server more directly (such as email alerting).

The specific metrics calculated in the transcript quality control step are as follows:

* Metadata for DPDash formatting and related tracking (analogous to the audio QC step above).

* Number of unique speakers identified by TranscribeMe (labeled in the txt using speaker IDs "S1" to "SN" for N speakers, numbered in the order they first spoke).

* Note that per the AMPSCZ SOP, S1 should be the primary interviewer and S2 the participant, and open interviews in particular should only have 2 speakers total - though these instructions have likely not been perfectly followed by sites to date.

* For each of the first three speaker IDs (S1-S3), the following counting stats:

* Total number of conversational turns.

* Total number of words.

* Smallest number of words in a turn.

* Largest number of words in a turn.

* Number of times throughout the transcript a segment of the audio was marked [inaudible].

* Number of times throughout the transcript a segment was marked as uncertain (where TrancribeMe places the word/phrase followed by a ? in brackets).

* Number of times throughout the transcript [crosstalk] between two speakers was marked.

* Number of words marked by TranscribeMe as PII throughout the transcript.

* Total number of commas appearing in the transcript, as well as the total number of dashes appearing in the transcript (both related to TranscribeMe's verbatim notation, so this is a quick check on that).

* For more on TranscribeMe's verbatim convention and leveraging this in feature extraction steps in order to detect different categories of linguistic disfluency, see the audio journal pipeline documentation of supplemental section <ref> above.

* Timestamp for the start of the final turn, in minutes.

* Shortest gap between the starts of two subsequent turns, in seconds

* Longest gap between the starts of two subsequent turns, in seconds

* The shortest and longest such gaps weighted by the number of words in the intervening sentence.

Thus each column in an interviewRedactedTranscriptQC DPDash CSV is one of these individual metrics, and each row is a particular transcript (corresponding to one interview session). The specific column names used for these features are:

All of these features are imported into DPDash to assist in interview progress monitoring as needed. With results reported in the main chapter <ref>, a subset of the transcript QC features (in addition to normalized versions e.g. redactions per word) have been studied in more depth in the early AMPSCZ dataset, and where useful have subsequently been more closely monitored as part of regular data quality assurance.

As an aside, if you are in charge of the TranscribeMe SFTP account you will by default receive a separate email alert from them for every single completed transcript they upload to the SFTP server. You can turn these off (or redirect them to other addresses) by contacting your TranscribeMe rep. Regardless, please note that these emails have no direct relation to the interview pipeline described here, which has its own email alerts. Not every transcript returned by TranscribeMe will end up in manual site review, and there can be up to a one day delay between TranscribeMe returning a transcript and it hitting the data aggregation server, as my code runs on a once daily cron job to check for new transcriptions to pull back from the TranscribeMe SFTP server. From there, there could be an additional delay of up to a day for the external utility script to transfer any transcripts that were selected for manual review to Box/Mediaflux, and for transcripts not selected for manual review there could be an additional delay of up to a day for produced redacted transcripts on the GENERAL side of processed to be copied over to predict.

Usually gaps between these separated steps will be shorter than a full day, but please wait a reasonable amount of time before raising a potential missingness concern based on TranscribeMe's email alerts. More importantly, please do not raise a concern about missing transcripts based only on what gets transferred to the sites' manual review Box/Mediaflux folders, as the clear majority of transcripts do not get sent for manual review at all and thus never get synced back to Box/Mediaflux. Finally, if you are performing data monitoring for AMPSCZ, please do not contact TranscribeMe directly looking for transcript outputs unless you have first confirmed the missingness is a TranscribeMe issue. Transcripts for this project need to flow through the designed pipeline, for privacy and data integrity reasons, and as far as actual unintended missingness is concerned it is far more likely to be a site mistake or even a code bug – not things TranscribeMe has any control over. There have of course been a few times where we had to contact TranscribeMe over real issues with the way they returned a file, but these instances have been very rare. If you are not the person responsible for monitoring the operation of this software, you should probably not be the person initiating contact with TranscribeMe about a possible missing transcript.

§.§ Video processing components

The final relevant modality for interview recordings is video – assumed to be obtained from Zoom for the present code, and a required component of the open interview protocol (optional for psychs) in the AMPSCZ project.

The video parts of the pipeline do not rely on outputs from the audio/transcript branches to be available, so for basic data monitoring and quality assurance purposes it is possible to run only the video branch, if desired by a future project. Obviously for each AMPSCZ site all branches are run daily, and in the cron job we use the video modality comes third after the audio and transcript code has been run for that site.

Given the format of Zoom meetings, the main goal of video QC here is to determine how many faces are detectable throughout the interview, along with some other basic facts about detected faces like their estimated size in the frame. Of course the pipeline also handles basic metadata compilation for file renaming and monitoring purposes on the video branch too. For more clarity, I will now provide a parallel list of major code steps and their corresponding wrapping bash modules in the video branch of the pipeline, as I did for audio and transcripts above:

* Identify new video files that have been correctly uploaded to raw within a Zoom interview folder, additionally initializing an email body for the video update daily site message if there are indeed any new videos available for the current site. From each new video, extract 1 frame for every 4 minutes of recording. The extracted frames of course go into the PROTECTED side of processed, contained within a subfolder that is named using the date and time of the interview extracted from the Zoom title. The presence of such extracted frames is used in subsequent runs to determine whether a video has been analyzed yet or not.

* These tasks are handled by the run_new_video_extract.sh module, which relies on ffmpeg but does not utilize python.

* Note frames are extracted periodically so that face detection can be run whilst keeping the pipeline feasible to use on the lightweight data aggregation servers and quick to complete processing of even longer recordings.

* Run PyFeat's basic face detection model on each newly extracted frame, saving details about any detected faces on the frame level, as well as using those results to compile per interview summary stats about face detection. This step also utilizes site and interview metadata to map each raw video name to an appropriate processed file name based on our conventions, ultimately creating/updating a DPDash-formatted video QC CSV for each subject ID and interview type with new recordings. That info is used in adding details about successfully processed (or not) video recordings to the daily video site update email as well.

* These tasks are handled by the run_video_qc.sh module, through calling the interview_video_qc.py python function for each participant ID and interview type with newly extracted video frames. Details on the exact outputs produced by this QC function will be provided subsequently.

* If a daily video monitoring email has been constructed due to new video uploads being detected, send out that finalized email. This email will provide a list of all new video recordings processed for QC, in terms of their final renaming for processed outputs. It will also include any error messages that may have arisen that prevented QC from running on other newly detected video files.

* The email body was compiled (when relevant) as part of earlier pipeline steps described, and is then sent near the end of the top level pipeline branch wrapper (interview_video_process.sh), analogous to the transcript email alerts step.

* Update the file accounting log with metadata information related to the newly detected videos, very similarly to the final accounting steps at the ends of the audio and transcript branches of the pipeline.

* This task is completed by the run_final_video_accounting.sh module, which calls the python script interview_video_process_account.py on each subject ID and interview type.

Note that if the daily video update message sends for a site but includes no video names and no specific error messages, it means that a new video was detected but the code failed prior to reaching the video QC step - thus indicating a likely error with ffmpeg's frame extraction command on one of the site's uploaded interview recordings. For more interpreting possible edge case errors within the existing monitoring infrastructure (for all parts of the pipeline), please see the troubleshooting information in section <ref> below.

§.§.§ Key direct outputs of video branch of pipeline

The primary output of the video branch of the pipeline is again the DPDash CSVs for quality assurance monitoring, although the interview video renaming maps, as well as the extracted frame images and their corresponding PyFeat outputs, have possible downstream uses too. The details of these outputs follow, first file naming conventions and then additional content information where applicable.

When a new, correctly formatted, video has been uploaded by Lochness to raw within a (likely new) Zoom interview folder, the first step of the pipeline will create a corresponding subfolder for storing extracted frames from that interview on the PROTECTED side of processed. These folders can be found on the main PHOENIX data structure of each central aggregation server at paths of the form: PROTECTED/[siteID]/processed/[subject]/interviews/[type]/video_frames/[date]+[time], where date and time are extracted from the Zoom folder name based on expected conventions. Each extracted frame from the interview is saved as a JPG under such a folder and named based on its timestamp within the interview; the sampled frames are taken from pre-determined recording timestamps that occur literally every 4 minutes, starting from the 1st second of the video (to avoid the very first frame), and then proceeding with timestamp 00:04:00, 00:08:00, and so on. These extracted frames are expected to be kept on the PROTECTED side of processed for the duration of the study, for both accounting purposes and for quick spot checking videos as needed.

As part of video QC, more detailed PyFeat information is stored for each frame, to reference in possible expanded QC if needed in the future. The individual frame CSVs are named according to their corresponding frame JPGs, under a subfolder with the extracted frames i.e. video_frames/[date]+[time]/PyFeatOutputs. Also as part of the video QC there is a raw to processed file name link stored, analogous to the audio file name maps previously described. This mapping can be found at video_frames/[date]+[time]/[date]+[time].txt, and it is especially important for ongoing accounting that it is not deleted for any previously processed interview videos. The file name of the txt itself, when taken along with folder names contained in the rest of the file path structure, is directly linkable to a specific raw interview folder upload. The content of the txt is what that raw video should be renamed to in a processed setting, based on SOP conventions.

Of course all of the video outputs contained within a video_frames subfolder are on the PROTECTED side of the data aggregation server storage structure, and are thus only accessible to a limited number of individuals working on AMPSCZ interview recording collection. The primary output type that is pushed along to predict is the DPDash CSV with per interview video QC stats. Like audio and transcript QC, those metrics can now be visualized using the DPDash web interface, and they are additionally included in compiled form as part of the weekly summary updates with broader reach (reported on extensively in the main chapter <ref>). Again, there is one such DPDash CSV created by the video branch of the pipeline for each subject ID and interview type with at least one video recording available on PHOENIX. Each of these CSVs contains one row per detected interview session, and can be found on the data aggregation servers at paths of the form: GENERAL/[siteFull]/processed/[subject]/interviews/[type]/[site2Digit]-[subject]-interviewVideoQC_[type]-day[start]to[end].csv.

The interview QC metric columns for the video datatype include:

* Metadata for DPDash formatting and related tracking (analogous to the audio and transcript QC steps above).

* Number of frames extracted, and thus used for computing the QC summary stats (should equal the floor of interview minutes divided by 4).

* Minimum number of faces detected across the extracted frames from the interview.

* Maximum number of faces detected across the extracted frames from the interview.

* Mean number of faces detected across the extracted frames from the interview.

* Minimum confidence score across all faces that were detected from across all frames.

* Maximum confidence score across all faces that were detected from across all frames.

* Mean confidence score across all faces that were detected from across all frames.

* Minimum area across all faces that were detected from across all frames.

* Area for a particular face was computed by multiplying the height and width variables returned by the PyFeat face detection model, describing the face box associated with the detected face.

* Maximum area across all faces that were detected from across all frames.

* Mean area across all faces that were detected from across all frames.

The specific column names used for these features are:

We also have the CSVs (one per extracted frame) saved in associated PyFeatOutputs subfolders on the PROTECTED side. Those CSVs contain one row per detected face, with the following additional metrics provided as columns:

* Face ID (assigned by PyFeat).

* Face location X coordinate in frame.

* Face location Y coordinate in frame.

* Face box's rectangle width.

* Face box's rectangle height.

* Face's detection confidence score.

Recall that it is not possible to load the standard face pose or action unit detection models on the data aggregation servers (particularly for Pronet) at this time, and it would generally be incompatible with the lightweight philosophy of this pipeline. Therefore any QC code that might use information about e.g. FAUs would need to be separately written, and we focused only on basic detection here.

§.§ Site-specific daily monitoring additions

The final major branch of the pipeline, which also runs on a per-site (or per-study) basis using the config file system described, runs a set of tools added onto the codebase for more detailed file accounting and monitoring.  While parts of the individual modules within this file accounting branch may be useful to run independently on a different project, much of this work relies on existing pipeline outputs from the other three branches – primarily the outputs of the final accounting step within each of those branches, along with the DPDash QC CSVs for each modality.

The main goal in adding this branch to the pipeline was to call additional attention to SOP violations and clear cut quality red flags, as well as to create some more streamlined progress updates for each site that could assist in higher level monitoring than the detailed emails sent by the other three branches. As part of that, this step also improves the DPDash views available by creating QC CSVs that are combined across modalities for each subject/interview type, with a filtered down set of key columns.

Like with the other branches, I will now provide a list of major code steps and their corresponding wrapping bash modules for the additional monitoring components of the pipeline:

* Identify file formatting and organizational issues that would prevent anything newly uploaded to raw from being processed by the pipeline at all, so that monitoring can easily notice such violations.

* This task is handled by the run_raw_folder_check.sh module, through calling the raw_interview_sop_check.py python function on each relevant subject ID and interview type. Obviously it relies on assumptions about folder structure and file naming being used for the AMPSCZ interview data collection protocol, which has already been described at length.

* Many of the types of site mistakes that have prevented processing and were caught by this step in early AMPSCZ monitoring have been identified within the main chapter section <ref>. The exact info tracked by this module will be detailed subsequently.

* Using the SOP CSV from step 1 and the respective processed accounting CSVs created in the final step of each of the audio, transcript, and video branches of the pipeline, isolate all interview session records that have a newly detected upload issue or have some new processing update. If there are any processing updates, initialize (or add to) a summary stats update email with a header containing a list of subject IDs and interview types that have some processing updates from the current code run. Then for each specific interview record with updates, look up its QC metrics in the corresponding DPDash CSVs for each modality, checking for a few specified major QC red flags as well as some possible metadata inconsistencies (from both QC and accounting CSVs). If any warnings were triggered in that process, or if new basic upload SOP violations were detected, set up (or add to) a warning email alert to be sent today for this site. Ultimately, this step identifies a number of important warnings to be notified about, saving/updating a CSV for each subject ID and interview type where relevant with an account of all warnings detected to date. Also saves a combined version of the processed accounting CSV containing merged info from all 3 modalities (or whichever modalities are available) for each session.

* These tasks are handled by the run_processed_accounting_check.sh module, through calling the interview_warnings_detection.py python function for each subject and interview type with any available interview outputs under processed (including just a raw SOP violation log). A more specific list of the warnings that are compiled by this step can be found subsequently.

* If a summary stats email was initialized for this site as part of step 2, it means that there are new QC metrics for at least one modality of one interview for the site. In that case, the pipeline will compute new site-wide and per-subject summary stats for the current site, with select metrics from the former also being added into the summary stats update email to be sent. Regardless (to account for backlogged updates in a timely manner because it is an important output being pushed to predict), all subject IDs and interview types with DPDash QC CSVs available will then have a new modality-merged QC CSV saved containing the latest information.

* The wrapping script for this pipeline branch (interview_summary_checks.sh) checks for a new summary stats email body, and if one has been created it will use the run_qc_site_stats.sh module to handle all of these tasks. Otherwise, it will use the run_qc_combine_only.sh module to handle only the task of modality merging QC CSVs.

* Both run_qc_site_stats and run_qc_combine_only call the interview_qc_combine.py python function on all subjects/interview types with QC available. This is the only major functionality in the run_qc_combine_only module.

* The run_qc_site_stats module then also calls the interview_qc_statistics.py python script, first for all open interviews across the present site and then for all psychs interviews across the present site. The details of the outputs produced by this module will also be further detailed below.

* If a summary stats update and/or new issues warning email alert was generated, send them to the site-specific email update addresses specified in the corresponding config. Email sending is again performed directly in the wrapping script for this pipeline branch. Here it is the last step of the pipeline for a given site (when applicable).

Note that while the code is in progress, the described email bodies (when relevant) are drafted directly within the top level of the repository installation folder. After any finalized emails are successfully sent (or if the sending fails due to any issue with the mail command or server connectivity), the corresponding txt files will have a unique Unix timestamp appended to their name and will be moved into the logs folder of the repository for record-keeping purposes. These are of course also part of what will be excluded from any git pushes via the .gitignore. A similar process occurs for email alerts in all branches of the main pipeline.

§.§.§ Resulting accounting CSVs and monitoring alert details

A major result of this branch of the pipeline is the more salient daily site email updates produced, particularly the alerts that will highlight new warnings that arise (to be elaborated on next), which has been a key part of my monitoring workflow to date - especially for catching violations that prevent an interview upload from appearing in downstream QC records at all. Another major result is the creation of additional CSVs with new curated file accounting and quality control metrics, many of which are later used in server-wide (i.e. cross-study/site) summaries created by the utility scripts written for further AMPSCZ monitoring. Those utilities were introduced in more detail above, in section <ref>. Here I will elaborate on the specifics of both the emails and CSVs created by the site-specific monitoring branch of the main pipeline in the above mentioned steps.

The first step, utilizing raw_interview_sop_check, creates (or updates) a CSV ending with "RawInterviewSOPAccountingTable.csv" on the PROTECTED side of processed for each subject ID and interview type, i.e. directly within the folders of the form: PROTECTED/[siteID]/processed/[subject]/interviews/[type]/ The CSV contains one row for each problematic file or folder found under the raw interview uploads for that subject ID and interview. Note that only raw paths corresponding to an SOP violation are included in the list, and it focuses only on those SOP violations that completely stall processing of one or more modalities focused on by this interview pipeline (i.e. mono audio and/or video).

Thus the major issues caught here are:

* An interview upload is a standalone file but it is not named according to EVISTR convention.

* An interview upload is a folder but it is not named according to Zoom recording convention.

* An interview upload is a correctly named folder but it does not contain a top level M4A file with the expected mono audio naming convention for Zoom, or alternatively it contains more than one such file.

* An interview upload is a correctly named folder but it does not contain a top level MP4 with the expected video file naming convention for Zoom, or alternatively it contains more than one such file.

It is possible for a given upload to commit more than one of these violations, but because the various columns contained in the RawInterviewSOPAccountingTable will describe all such issues, there will still only be one row for that upload in the CSV. The columns of the CSV are as follows:

["raw_name", "folder_bool", "valid_folder_name_bool", "num_audio_m4a", "num_video_mp4", "num_top_level_files", "date_detected"]

Conversely, note that if the files from a Zoom folder are uploaded loosely (outside of their folder), there will be one row in this CSV for each such loose file. When the warnings email is constructed by the next step, it will list the raw name of each record newly added (i.e. detected today) to any RawInterviewSOPAccountingTable from across the given site, where applicable. All such issues are labeled as "SOP Violation" in that warnings email.

The other types of issues that will be included in the warnings email put together by the interview_warnings_detection function of the second step are:

* "Audio Rejected by QC"

* "Audio Failed SFTP Upload"

* "Missing Expected Speaker Specific Audios"

* "Transcript Encoding Not UTF-8"

* "English Transcript Encoding Not ASCII"

* "Video Day Inconsistent with Audio Day"

* "Video Number Inconsistent with Audio Number"

* "Consent Date Changed with New Files"

* "Session and Day Numbers Inconsistent"

* "Session Number Repeated"

* "Interview Under 4 Minutes"

* "No Faces Detected"

* "No Redactions Detected"

The first 10 problem types listed are derived from the processed file accounting CSV outputs, while the last 3 come from checking QC outputs. This list is of course not exhaustive, particularly when it comes to monitoring of quality metrics; other tools like DPDash and weekly histograms should also be used as part of the quality review workflow.

Recall that the above issues are only reported here when detected in interviews that were found to have had some update on the current pipeline run. If multiple problems are detected in such an interview though, each will be independently logged – in the email alert by listing the study day and code-labeled session number corresponding to each problem on a new line. The warnings generation step additionally creates (or updates) a CSV ending with "InterviewProcessWarningsTable.csv" on the PROTECTED side of processed for each subject ID and interview type i.e. under the same location as the RawInterviewSOPAccountingTable CSVs.

The InterviewProcessWarningsTable contains one row per detected processed accounting/QC warning from a run of this pipeline branch, with the following columns:

["day", "interview_number", "interview_date", "interview_time", "warning_text", "warning_date"]

As such, the CSV can be revisited at any time for a complete record of previously issued warnings for a given subject ID and interview type.

In the same processed folder, the code also saves a CSV merging the full set of file accounting information that has been compiled to date by other pipeline branches, under a file name ending with "InterviewAllModalityProcessAccountingTable.csv". Additionally, the interview_qc_combine function saves an analogous CSV that represents the merging of all DPDash QC CSVs produced from across modalities for each subject ID and interview type, under a file name ending with "combinedQCRecords.csv". However the combined QC metrics can be saved on the GENERAL side instead, so it may be pushed by Lochness to predict. That CSV is therefore found under GENERAL/[siteID]/processed/[subject]/interviews/[type]/ instead.

The final major step, utilizing interview_qc_statistics when new processing updates have occurred, creates (or updates) a CSV ending with "InterviewSummaryStatsLog.csv" on the GENERAL side of processed for each subject ID and interview type. This CSV has 2 new rows added each time it is updated, one for subject-specific summary stats and one for site-wide summary stats, both specific to the interview type and each with a column denoting the current computation date (and obviously a column indicating whether it is a subject summary or a study i.e. site-wide summary). Besides other basic metadata columns, the main columns of this CSV are summary stats over each QC metric available from across the interview modalities. The four summary stats calculated for every QC feature are the mean, standard deviation, minimum, and maximum of the feature to date.

While each InterviewSummaryStatsLog CSV covers summary stats for all QC features included by the pipeline, a select few on the site-wide level for each interview type (open and psychs here) are included in the daily summary stats update email, when there is new information to be conveyed. In that summary update email, the mean (+/- the standard deviation) are reported for open and then for psychs for the following features:

* "Interview Length in Minutes"

* "Mono Audio Decibels"

* "Audio Spectral Flatness"

* "Video Faces Detected per Frame" (mean per video)

* "Face Detection Confidence" (mean per video)

* "Transcript Number of Speakers"

* "Speaker 1 Word Count"

* "Speaker 2 Word Count"

* "Number of Inaudible Words"

* "Number of Redacted Words"

Note that each InterviewSummaryStatsLog CSV will get pushed to predict by Lochness as a GENERAL side output. Concatenated versions of these CSVs (filtered to be on subject and site levels separately) are also attached to a daily server-wide email sent by the AMPSCZ utility scripts described above (section <ref>).

The reason that site-wide stats are stored in every single one of these individual CSVs on the data aggregation server is because there is no good way to store data that is summarized on anything above the subject level with the PHOENIX data structure. Combined versions of CSVs on the site or server level that are computed for email alerts or other logging thus get stored within the repo installation folder on each server, in the logs subfolder. These outputs are of course protected from being pushed back to the central repository via the .gitignore defined in the repo. But still do take care of permissions on the installation folder, to ensure that summary info cannot be seen by anyone who should not have access to the server's interview data on even a summary level. In particular, real dates can be found via these files.

Ultimately, the participant-wide accounting branch of the pipeline has been important for the monitoring workflow of AMPSCZ. While the specific warning alerts are not exhaustive and sometimes can be triggered by different mistakes than they intend to encode, they have succeeded in capturing a great number of issues across sites, and continue to be checked regularly to find new SOP violations. For more on the specific warnings messages of greatest relevance and the likely root causes behind different types of warnings, please see the troubleshooting information within the future directions section below (<ref>).

The summary stat updates on the site-level sent by this branch of the pipeline can be of use as well, though for reviewing other possible QC issues I largely use the stats from cross-site server-wide information emailed out by the mentioned utility scripts (<ref>). Primarily I review the weekly histograms and tables described at length in main chapter <ref> to detect possible new problems, but I also often refer to the combined QC CSVs from daily email updates if more specific stats need to be investigated for a given interview.

§.§ Security review

Audio files are pushed to TranscribeMe using the following process. We connect to the TranscribeMe server via SFTP. The pySFTP python package, a wrapper around the Paramiko package specific for SFTP is used to automate this process. The host is sftp.transcribeme.com. The standard port 22 is used. The username is provided as a setting to the code, but will generally be one account for all sites on the Pronet side and one account for all sites on the Prescient side, with set up of this facilitated by TranscribeMe. These accounts are created only to be used with this code and will serve no other purpose.

The account password is input using a temporary environment variable. It was previously prompted for using read -s upon each run of the code, but for the purposes of automatic scheduled jobs, the password may now be stored in a hidden file with restricted permissions on a secure portion of our machine instead with permissions fully limited to only the account meant to run the interview pipeline. Presently this has been approved to be stored directly in the folder where the code is installed, but if needed it could instead be stored directly on the PROTECTED side of PHOENIX. In that case, there would be a further layer of guarantee, as anyone who could access the password file necessarily must also have access to the raw files themselves, so use of the TranscribeMe password would provide no additional benefit.

Interfacing with TranscribeMe is done using the same SFTP protocol as described in the audio section. Note that once the transcript side of the pipeline has detected a returned transcript on this SFTP server, the corresponding uploaded audio is deleted by our code on TranscribeMe's end using the same SFTP package. TranscribeMe also automatically deletes files on their server (including the transcripts this code moves to "archive" subfolders) after a certain amount of time has passed. TranscribeMe's server maintenance practices have also been reviewed and approved by Partners Healthcare.

The code itself (in the state presented here) was approved by both Pronet and Prescient central servers for safe installation and use on AMPSCZ data, and it is of course fully available for open source review as the need arises [Ennis, 2023]. The other privacy/security considerations relevant to AMPSCZ interview processing relate to other steps of the data collection and analysis process, which I will now touch on briefly as they pertain to possible future changes for this data management/QC pipeline.

§.§.§ Commentary on the use of Zoom

The use of HIPPA-compliant institutional Zoom accounts for recording AMPSCZ interviews (as detailed in the SOP instructions and reviewed in supplemental section <ref> above) has been IRB approved for all US-based sites and most foreign sites in this project.

Zoom was originally chosen as the remote interview recording platform because of its ability to save speaker-specific audio files, something absent from other major options (e.g. Microsoft Teams, Cisco WebEx). Of course, it is also desirable for sites across a collaborative study to agree on the recording platform. Fortunately, an acceptable secure protocol for using Zoom has largely been established across sites, and thus the pipeline was designed around Zoom for offsite (and onsite open) interviews. Practically speaking, the current implementation does not carry major risk of privacy violations and is considered suitably secure by institutions across the country (and world).

There are still a handful of sites that have not been permitted to use Zoom for recording participant interviews though, some of whom in fact may not be allowed to contribute video of participants at all. These restrictions are primarily a result of legal differences in some European countries, and for this project it is mostly German sites that are impacted. As an edge case, early work on the AMPSCZ interview pipeline did not prioritize accommodating these sites, but it will be necessary in the future to do so.