Psychotherapy research in the 21st century

TCP Vol. 76, Issue 1. Lead Article by Lorenzo Lorenzo-Luaces, PhD; John F. Buss, BS; Robinson De Jesús-Romero, BA, MsC; Allison Peipert, BS; Isabella Starvaggi, BS

Psychotherapy research in the 21st century

Mental disorders account for a substantial proportion of the disability attributable to health conditions (Whiteford et al., 2013). Depressive disorders specifically account for a considerable amount of that disability, partly due to their high prevalence (Patel et al., 2016). In the United States (U.S.), for example, 20% of individuals recall meeting the Diagnostic and Statistical Manual of Mental Disorders (DSM; American Psychiatric Association, 2013) criteria for major depressive disorder (MDD) at some point in their lives (Hasin et al., 2018). The prevalence of depression is not remarkably lower in other parts of the world and these prevalence rates are known to be underestimates because they are obtained from retrospective interviews which are subject to recall bias (i.e., people forget episodes of internalizing distress; Moffitt et al., 2010; Wells & Horwood, 2004). For example, in the Dunedin Birth Cohort Study, over 1,000 individuals were assessed at various timepoints from the age of 11 to the age of 45. The latest analysis showed that 86% of the cohort met criteria for psychopathology at some point during the follow-up period (Caspi et al., 2020). Other prospective epidemiological studies show high rates of depression and other forms of internalizing distress when assessed over the life course (Lorenzo-Luaces, 2015).

Cognitive-behavioral therapies (CBTs) are effective treatments for depression and other forms of internalizing distress and are considered the gold standard of psychological interventions (Lorenzo-Luaces, 2018). Although CBTs are effective, many individuals remain symptomatic after treatment. Moreover, it has been well articulated that individual face-to-face CBTs are unlikely to make a significant dent in the public health burden of depression (Jorm et al., 2017; Kazdin & Blase, 2011). Our current model of treatment allocation is largely based on trial and error or provider availability, which has led to a poor use of resources and limited dissemination of treatment to those who need it (Lorenzo-Luaces, Peipert, et al., 2021).

Furthering pessimism about the promise of psychological interventions, in comparative treatment studies the differences between CBTs and other interventions are small (Barkham et al., 2021; Cuijpers et al., 2012) and often so small that they are not statistically significant (Barth et al., 2016). This pattern of findings has been so commonly reported in research on depression and other forms of internalizing distress that it has a name: “the Dodo bird verdict” (Luborsky et al., 2002). The Dodo bird verdict has been taken to provide support for the idea that factors common to all therapies explain their efficacy, rather than factors specific to different types of therapy (Laska et al., 2014). In observational studies, the working alliance between the patient and the therapist, often defined as their agreement in treatment goals and their emotional bond, has been correlated with outcomes (r = 0.28, 95% CI: 0.26, 0.30), lending further support to this “common factors” account (Flückiger et al., 2018). Below, we summarize a program of research challenging the Dodo Bird verdict and the common factors theory. Additionally, we discuss future directions to increase the public health significance of psychotherapy research.

Depression heterogeneity

One challenge in “accepting” the Dodo Bird and common factors theory of psychotherapeutic change is the degree of heterogeneity in depression and other mental disorders. Often in the literature, heterogeneity is quantified by heterogeneity in symptom presentation. For example, there are over 10,000 symptom combinations that qualify for a diagnosis of MDD (Zimmerman et al., 2015).    In response to this problem, the DSM includes disorder subtypes and specifiers in an attempt to identify more homogeneous patient groups. While this is a sensible approach, many of the subtypes in the DSM add symptoms for consideration of the diagnosis, which increases heterogeneity rather than decreases it (Fried et al., 2020; Lorenzo-Luaces, Buss, et al., 2021). A critique of this kind of work on heterogeneity is that it quantifies heterogeneity in a very rough way: by unique combinations of symptoms endorsed (Zimmerman et al., 2015). In this approach, individuals could be said to differ if they have all symptoms in common but one. We (Buss et al., 2022) recently introduced a more sophisticated method of quantifying heterogeneity on a continuum, by using methods from information theory. The results are consistent with our prior work using less sophisticated approaches (Lorenzo-Luaces, Buss, et al., 2021) demonstrating that the atypical and melancholic subtypes of MDD do not reduce symptom heterogeneity.

Given such a high level of heterogeneity in symptoms, one may expect to find that symptoms moderate treatment outcomes such that some interventions are superior for some symptoms. The idea of moderation would imply that some individuals, with identifiable characteristics, experience better outcomes in some interventions (e.g., CBT) than others. Symptoms of depression have been widely studied as moderators of outcomes of CBTs vs. other interventions. Despite how frequently they have been studied,  there is little support for the ability of symptom constellations to predict differential treatment outcomes (Boschloo et al., 2019; Lorenzo-Luaces, Peipert, et al., 2021).

Patients with depression are also a heterogeneous group in regard to sociodemographics, comorbid features (Hasin et al., 2018), and psychological makeup. In our work, we have found that baseline characteristics moderate treatment outcomes when comparing CBTs to other interventions including: antidepressants (DeRubeis et al., 2014), positive psychotherapy (Lopez-Gomez et al., 2019), or interpersonal therapy (Van Bronswijk et al., 2021). These studies suggest that while CBT appears equally efficacious when compared to other interventions when one focuses on averages (e.g., the Dodo bird verdict), there are subgroups of patients who experience superior outcomes in CBTs vs. other interventions as well as the opposite (i.e., patients who experience better outcomes in other interventions than in CBTs).

Our work also suggests that patient characteristics moderate process-outcomes relationships in psychotherapy. For example, in one study of depressed patients (N = 60) undergoing CBT (Lorenzo-Luaces et al., 2014), we found that the working alliance was a stronger predictor of outcomes for patients with less recurrent depression (r = 0.52, 95% CI: 0.22, 0.73) than is traditionally reported in the literature (r = 0.28, 95% CI: 0.26, 0.30), but had no relationship with outcomes in patients with more recurrent forms of depression (r = -0.02, 95% CI: -0.41, 0.38). We replicated and extended these findings using data from a randomized controlled trial comparing CBT to psychodynamic therapy (Driessen et al., 2013). In CBT (n = 143), the alliance predicted outcome for patients with less recurrent depression (r = 0.39, 95% CI: 0.11, 0.60) but did not predict outcomes in patients with more recurrent forms of depression (r = 0.06, 95% CI: -0.16, 0.27). Interestingly, number of prior episodes did not moderate the alliance-outcome association in psychodynamic therapy (n = 141), such that the alliance predicted symptom change irrespective of prior episodes (r = 0.29, p < .001). These results suggest that to understand processes of change in psychotherapy, the field needs to move towards studies adequately powered to explore the effects of specific patient features like number of prior episodes, general therapeutic factors (e.g., alliance), and specific therapeutic factors.

In addition to being heterogeneous in its symptoms and contaminant features, depression is heterogeneous in its prognosis (Monroe & Harkness, 2011). While many cases in naturalistic samples remit within a 3-6 month period (~50%), many others have a chronic course (20%) or courses characterized by remission and subsequent relapse (~30%). Among individuals who relapse, repeated episodes are common (Monroe & Harkness, 2011). Given this level of heterogeneity in the prognosis, we have also attempted to predict prognosis and use the predicted prognosis as a potential guide to treatment allocation. For example, in one study, we calculated the predicted prognosis of 622 depressed patients based on baseline characteristics. We then examined whether the predicted prognosis moderated outcomes when patients were randomized to CBT, a brief therapy (BT) that was non-specific in techniques, or treatment as usual (TAU). For patients with a good prognosis (75% of the sample), there was no difference in outcome between the three treatment conditions (Lorenzo-Luaces et al., 2017). For patients with a good prognosis (the remaining 25%), CBT was superior to brief therapy and TAU These findings suggest that it may be possible to use risk stratification to triage individuals to different intensity of CBTs (see also Lorenzo-Luaces et al., 2020).  Most recently, a prospective trial by Jaime Delgadillo and colleagues supported the idea of prospective risk stratification following a machine learning algorithm (Delgadillo et al., 2022).

While our analyses and those of others are interesting and lend support to the idea that patients could be matched to CBT versus other interventions, these studies suffer from numerous limitations. Most notably, these studies have very small samples sizes versus benchmark recommendations from simulation studies (Luedtke et al., 2019). Additionally, our studies, as well as those by others, often lack validation samples, leaving them unable to rule out the possibility that seemingly interesting findings are the product of overfitting (Lorenzo-Luaces, Peipert, et al., 2021). Indeed, when my colleagues and I performed one of few studies that have used an external validation sample (van Bronswijk et al., 2021), we found inconsistent support for our prediction models’ generalizability outside the samples in which they were developed (Lorenzo-Luaces, Peipert, et al., 2021).

Thus, although exploring individual differences in processes and outcomes relevant to CBTs has yielded interesting findings that contradict the Dodo Bird and common factors theory, this research is still in its infancy, owing in part to the small samples that we are feasible to collect in traditional psychotherapy research.

Low intensity CBTs

One avenue we have explored that allows us to collect larger samples than in traditional psychotherapy research is the study of low intensity CBTs (LI-CBTs). LI-CBTs allow individuals to learn the information and skills they would obtain from face-to-face CBTs for internalizing distress by using books (i.e., bibliotherapy) or the internet (Bennett-Levy et al., 2010). LI-CBTs can be delivered with minimal support from a paraprofessional (i.e., guided), or a person can also complete them on their own (i.e., unguided). LI-CBTs are relatively inexpensive and scalable, such that they have the potential increase the uptake of mental health services, for example among communities that may not have equal access to CBTs like racial-and-ethnic minorities.

It is makes sense to think that LI-CBTs can reduce the public health burden of untreated mental health symptoms because they are relatively inexpensive. For example, during 2020-2021 and with minimal funding, we were able to treat 141 people from across the United States with guided LI-CBT (Lorenzo-Luaces, Howard, De Jesús-Romero, et al., 2022). During our open trial, participants experienced large improvements in internalizing distress, modest improvements in well-being and the use of cognitive reappraisal for emotion regulation, and relatively small improvements in expressive suppression, an avoidance strategy. Our secondary analyses of the data suggest that the improvements in internalizing distress were preceded and predicted by changes in cognitive reappraisal (De Jesús-Romero, Starvaggi, et al., 2022), a hypothesized mechanism of CBTs (Lorenzo-Luaces et al., 2015, 2016). Thus, studying LI-CBTs facilitates traditional psychotherapy process and outcome research.

As another example, in an eight-month period, and with a small amount of funding (~$20,000) from the Center for Rural Engagement at Indiana University, we used social media to recruit 216 adults throughout the state of Indiana for a randomized controlled trial comparing guided vs. unguided delivery of an LI-CBT that was developed by the World Health Organization (Tol et al., 2020). Our analyses of the trial are ongoing, but preliminary results suggest that participants in both conditions experienced large improvements in internalizing distress (e.g., depression, anxiety), well-being, and cognitive reappraisal. Individuals in the guided condition experienced better outcomes in internalizing distress (SMD = -0.39, 95% CI: -0.65, -0.11) and cognitive reappraisal (SMD = 0.32, 95% CI: 0.05, 0.59) than individuals in unguided LI-CBT.

In a recent study, we studied a single session intervention (see Schleider & Weisz, 2017) in a population that is at high risk for depression but easy to recruit: online workers (Lorenzo-Luaces & Howard, 2022). While this study was rather large (N = 828), we found no evidence of statistically or clinically-significant differences between the single-session intervention and a waiting list control. We are currently conducting further analyses investigating possible subgroup effects

One challenge with studying LI-CBTs is that while they are effective when used, individuals are hesitant to initiate and continue them (Cuijpers et al., 2019). Evidence from naturalistic studies suggests than the likelihood of dropping out of treatment altogether decreases with each specific intervention an individual tries but does not benefit from (Harris et al., 2020; Rush et al., 2006). We (Starvaggi & Lorenzo-Luaces, 2023) are currently working on methods to identify who may be most likely to initiate and complete LI-CBT by leveraging predictions about LI-CBT engagement from large samples of individuals recruited online.  Preliminary results suggest that while individuals can make confident predictions about their engagement with LI-CBT, and these predictions can be modelled, it is difficult to generalize such a model to make out-of-sample predictions about actual engagement in clinical trials. Better understanding of heterogeneity in engagement with LI-CBT has the potential to improve the scalability of these treatments, but novel research approaches may be required to do so.

Our studies are designed to be relatively high in external validity (e.g., very lax entry criteria, nationwide recruitment). While questions about the efficacy, mechanisms, and predictors of response to LI-CBTs in these high external validity contexts are important, prior work already supports the efficacy of LI-CBTs (Cuijpers et al., 2019). One of the core promises of LI-CBTs is the potential of their scalability to impact public health, but this claim is not always realized in LI-CBT research. Our work suggests that simply making LI-CBTs available to the public does not result in increased uptake. For example, as part of the WHO International College Student Initiative, my lab screened 2,534 Indiana University students during fall 2019 and fall 2020. We offered LI-CBT to students with an internalizing distress diagnosis (e.g., MDD). Although the rates of past-year internalizing distress diagnoses were quite high (~30%), especially in 2020, only a small subset of the students invited to complete LI-CBT actually entered treatment (23%), underscoring the need for more effective dissemination of LI-CBTs.

Our work suggests that the real-world reach of LI-CBTs has been rather limited. For example, analyses of app marketplaces (e.g., Google Play Store) in 2022 suggests that the top 3 mental health apps for depression accounted for 66% of all users (Wasil et al., 2020, 2021). Peipert et al. (2022) surveyed the perspectives of psychotherapists regarding LI-CBTs for patients on a waiting list, a natural place to disseminate LI-CBTs. Her work suggests that while therapists have positive attitudes towards LI-CBTs, very few (<15%) recommend them to patients who are on a waiting list for services. This is even though most (94%) had at least brief conversations with potential patients before putting them on a waiting list. In other words, psychotherapists can expand treatment access by recommending LI-CBTs. but they do not do so.

It’s not just in the “real world” that LI-CBTs have failed to fulfill the promise of more scalable treatment. For example, De Jesús-Romero (2022) conducted a meta-analysis of 69 studies of internet-based LI-CBT studies. He documented rather poor reporting of race-ethnicity in studies conducted outside the United States. Although reporting was relatively good within the U.S., racial-ethnic minorities appeared underrepresented in LI-CBT studies relative to their base rate in the general population and were underrepresented even relative to their base rate amongst in depressed outpatients. These data suggest that even when research programs are designed with the best of intentions (e.g., to reduce the public health burden of psychopathology), failure to critically evaluate study design and recruitment often leads to recreation of the conditions that researchers aim  to solve. A similar argument could be made about research on mechanisms and novel interventions: we aim to use this research to improve outcomes, but we are far from accomplishing that goal (Lorenzo-Luaces, 2022).

Conclusion

Depression and other forms of internalizing distress are common and can be very impairing. This makes questions about treatment outcomes and processes very important. However, symptoms of internalizing distress are heterogeneous in their presentation, their prognosis, and the populations they affect. Questions about heterogeneity are very exciting and allow us to apply novel and interesting statistical methods. However, the public health reach of this kind of work (e.g., parsing heterogeneity in the alliance-outcome correlation) may be rather limited.

If psychotherapy research is to remain relevant in the 21st century, we need to adopt study designs across the clinical-translational spectrum, especially reaching out to practicing providers in community settings. One principle our lab has followed, for example, is trying to “go where the people are.” In the United States most people belong to at least one social media platform (Pew Research Center: Internet & Technology, 2019). Social media can be used for participant recruitment, even in clinical studies, and  can facilitate nationwide research. We have even used social media to study purported mechanisms of depression including circadian rhythm disturbances (Thij et al., 2020) and cognitive distortions (Bathina et al., 2021). We have also leveraged large samples to triangulate self-report and data acquired via social media from the same individuals (Lorenzo-Luaces, Howard, Edinger, et al., 2022).

References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders: DSM-5 (5th ed.). Author.

Barkham, M., Saxon, D., Hardy, G. E., Bradburn, M., Galloway, D., Wickramasekera, N., Keetharuth, A. D., Bower, P., King, M., Elliott, R., et al. (2021). Person-centred experiential therapy versus cognitive behavioural therapy delivered in the English Improving Access to Psychological Therapies service for the treatment of moderate or severe depression (PRaCTICED): A pragmatic, randomised, non-inferiority trial. The Lancet Psychiatry, 8(6), 487–499.

Barth, J., Munder, T., Gerger, H., Nüesch, E., Trelle, S., Znoj, H., Jüni, P., & Cuijpers, P. (2016). Comparative efficacy of seven psychotherapeutic interventions for patients with depression: A network meta-analysis. PLOS Medicine, 10, e1001454. https://doi.org/10.1371/journal.pmed.1001454

Bathina, K. C., Ten Thij, M., Lorenzo-Luaces, L., Rutter, L. A., & Bollen, J. (2021). Individuals with depression express more distorted thinking on social media. Nature Human Behaviour, 5(4), 458–466. https://doi.org/https://doi.org/10.1038/s41562-021-01050-7

Bennett-Levy, J., Richards, D., Farrand, P., Christensen, H., Griffiths, K., Kavanagh, D., Klein, B., Lau, M. A., Proudfoot, J., & Ritterband, L. (2010). Oxford Guide To Low Intensity CBT Interventions. OUP Oxford.

Boschloo, L., Bekhuis, E., Weitz, E. S., Reijnders, M., DeRubeis, R. J., Dimidjian, S., Dunner, D. L., Dunlop, B. W., Hegerl, U., Hollon, S. D., et al. (2019). The symptom-specific efficacy of antidepressant medication vs. Cognitive behavioral therapy in the treatment of depression: Results from an individual patient data meta-analysis. World Psychiatry, 18(2), 183–191. https://doi.org/10.1002/wps.20630

Buss, J. F., Watts, A., & Lorenzo-Luaces, L. (2022). Methods for quantifying diagnostic heterogeneity in psychopathology: The example of depression and its specifiers. Manuscript Submitted for Publication.

Caspi, A., Houts, R. M., Ambler, A., Danese, A., Elliott, M. L., Hariri, A., Harrington, H., Hogan, S., Poulton, R., Ramrakha, S., et al. (2020). Longitudinal assessment of mental health disorders and comorbidities across 4 decades among participants in the dunedin birth cohort study. JAMA Network Open, 3(4), e203221–e203221.

Cuijpers, P., Driessen, E., Hollon, S. D., Oppen, P. van, Barth, J., & Andersson, G. (2012). The efficacy of non-directive supportive therapy for adult depression: A meta-analysis. Clinical Psychology Review, 32(4), 280–291. https://doi.org/10.1016/j.cpr.2012.01.003

Cuijpers, P., Noma, H., Karyotaki, E., Cipriani, A., & Furukawa, T. A. (2019). Effectiveness and acceptability of cognitive behavior therapy delivery formats in adults with depression: A network meta-analysis. JAMA Psychiatry, 76(7), 700–707. https://doi.org/10.1001/jamapsychiatry.2019.0268

De Jesús-Romero, R., Holder-Dixon, A., & Lorenzo-Luaces, L. (2022). Reporting and representation of racial and ethnic diversity in randomized controlled trials of internet-based cognitive-behavioral therapy (iCBT) for depression. PsyArXiv Pre-Print. Available at: https://psyarxiv.com/kfnhm/.

De Jesús-Romero, R., Starvaggi, I., Howard, J., Peipert, A., Buss, J., Lind, C. M., Riley, T. N., & Lorenzo-Luaces, L. (2022). Cognitive reappraisal as a mechanism of change in transdiagnostic low-intensity cognitive behavioral therapy for internalizing distress: Disaggregating within vs between individual variability. PsyArXiv Pre-Print. Available at: https://Psyarxiv.com/T6ymw/.

Delgadillo, J., Ali, S., Fleck, K., Agnew, C., Southgate, A., Parkhouse, L., Cohen, Z. D., DeRubeis, R. J., & Barkham, M. (2022). Stratified care vs stepped care for depression: A cluster randomized clinical trial. JAMA Psychiatry, 79(2), 101–108.

DeRubeis, R. J., Cohen, Z. D., Forand, N. R., Fournier, J. C., Gelfand, L. A., & Lorenzo-Luaces, L. (2014). The personalized advantage index: Translating research on prediction into individualized treatment recommendations. A demonstration. PLOS ONE, 9(1).

Driessen, E., Van, H. L., Don, F. J., Peen, J., Kool, S., Westra, D., Hendriksen, M., Schoevers, R. A., Cuijpers, P., Twisk, J. W., et al. (2013). The efficacy of cognitive-behavioral therapy and psychodynamic therapy in the outpatient treatment of major depression: A randomized clinical trial. American Journal of Psychiatry, 170(9), 1041–1050. https://doi.org/10.1176/appi.ajp.2013.12070899

Flückiger, C., Del Re, A. C., Wampold, B. E., & Horvath, A. O. (2018). The alliance in adult psychotherapy: A meta-analytic synthesis. Psychotherapy, 55(4), 316.

Fried, E. I., Coomans, F., & Lorenzo-Luaces, L. (2020). The 341737 ways of qualifying for the melancholic specifier. The Lancet Psychiatry, 7(6), 479–480. https://doi.org/10.1016/S2215-0366(20)30169-3

Fried, E. I., & Nesse, R. M. (2015). Depression is not a consistent syndrome: An investigation of unique symptom patterns in the STAR* D study. Journal of Affective Disorders, 172, 96–102. https://doi.org/10.1016/j.jad.2014.10.010

Harris, M. G., Kazdin, A. E., Chiu, W. T., Sampson, N. A., Aguilar-Gaxiola, S., Al-Hamzawi, A., Alonso, J., Altwaijri, Y., Andrade, L. H., Cardoso, G., et al. (2020). Findings from world mental health surveys of the perceived helpfulness of treatment for patients with major depressive disorder. JAMA Psychiatry, 77(8), 830–841. https://doi.org/10.1001/jamapsychiatry.2020.1107

Hasin, D. S., Sarvet, A. L., Meyers, J. L., Saha, T. D., Ruan, W. J., Stohl, M., & Grant, B. F. (2018). Epidemiology of adult DSM-5 major depressive disorder and its specifiers in the United States. JAMA Psychiatry, 75(4), 336–346. https://doi.org/10.1001/jamapsychiatry.2017.4602

Jorm, A. F., Patten, S. B., Brugha, T. S., & Mojtabai, R. (2017). Has increased provision of treatment reduced the prevalence of common mental disorders? Review of the evidence from four countries. World Psychiatry, 16(1), 90–99. https://doi.org/https://doi.org/10.1002/wps.20388

Kazdin, A. E., & Blase, S. L. (2011). Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspectives on Psychological Science, 6(1), 21–37.

Laska, K. M., Gurman, A. S., & Wampold, B. E. (2014). Expanding the lens of evidence-based practice in psychotherapy: A common factors perspective. Psychotherapy, 51(4), 467.

Lopez-Gomez, I., Lorenzo-Luaces, L., Chaves, C., Hervas, G., DeRubeis, R. J., & Vazquez, C. (2019). Predicting optimal interventions for clinical depression: Moderators of outcomes in a positive psychological intervention vs. Cognitive-behavioral therapy. General Hospital Psychiatry, 61, 104–110. https://doi.org/10.1016/j.genhosppsych.2019.07.004

Lorenzo-Luaces, L. (2015). Heterogeneity in the prognosis of major depression: From the common cold to a highly debilitating and recurrent illness. Epidemiology and Psychiatric Sciences, 24(6), 466–472. https://doi.org/10.1017/S2045796015000542

Lorenzo-Luaces, L. (2018). The evidence for cognitive behavioral therapy. JAMA, 319(8), 831–832. https://doi.org/10.1001/jama.2017.20826

Lorenzo-Luaces, L. (2022). Identifying active ingredients in cognitive-behavioral therapies: What if we didn’t? (Or couldn’t?). PsyArXiv Pre-Print. Available at: https://doi.org/10.31234/osf.io/8eqnx.

Lorenzo-Luaces, L., Buss, J. F., & Fried, E. I. (2021). Heterogeneity in major depression and its melancholic and atypical specifiers: A secondary analysis of STAR* D. BMC Psychiatry, 21(1), 1–11.

Lorenzo-Luaces, L., DeRubeis, R. J., Straten, A. van, & Tiemens, B. (2017). A prognostic index (PI) as a moderator of outcomes in the treatment of depression: A proof of concept combining multiple variables to inform risk-stratified stepped care models. Journal of Affective Disorders, 213, 78–85. https://doi.org/10.1016/j.jad.2017.02.010

Lorenzo-Luaces, L., DeRubeis, R. J., & Webb, C. A. (2014). Client characteristics as moderators of the relation between the therapeutic alliance and outcome in cognitive therapy for depression. Journal of Consulting and Clinical Psychology, 82(2), 368–373. https://doi.org/10.1037/a0035994

Lorenzo-Luaces, L., German, R. E., & DeRubeis, R. J. (2015). It’s complicated: The relation between cognitive change procedures, cognitive change, and symptom change in cognitive therapy for depression. Clinical Psychology Review, 41, 3–15. https://doi.org/10.1016/j.cpr.2014.12.003

Lorenzo-Luaces, L., & Howard, J. (2022). Efficacy of a single session intervention for depression in online workers: A randomized controlled trial with transdiagnostic mental health outcomes. PsyArXiv Pre-Print. Available at: https://psyarxiv.com/tx7nh/.

Lorenzo-Luaces, L., Howard, J., Edinger, A., Yan, H. Y., Rutter, L. A., Valdez, D., Bollen, J., et al. (2022). Sociodemographics and transdiagnostic mental health symptoms in SOCIAL (studies of online cohorts for internalizing symptoms and language) i and II: Cross-sectional survey and Botometer analysis. JMIR Formative Research, 6(10), e39324.

Lorenzo-Luaces, L., Howard, J., De Jesús-Romero, D., Peipert, A., Buss, J. F., Lind, C., Botts, K., & Starvaggi, I. (2022). Acceptability and outcomes of transdiagnostic guided self-help bibliotherapy for internalizing disorder symptoms in adults: A fully remote nationwide open trial. Cognitive Therapy and Research, 1-14.

Lorenzo-Luaces, L., Keefe, J. R., & DeRubeis, R. J. (2016). Cognitive-behavioral therapy: Nature and relation to non-cognitive behavioral therapy. Behavior Therapy, 47(6), 785–803. https://doi.org/10.1016/j.beth.2016.02.012

Lorenzo-Luaces, L., Peipert, A., De Jesús- Romero, R., Rutter, L. A., & Rodriguez-Quintana, N. (2021). Personalized medicine and cognitive behavioral therapies for depression: Small effects, big problems, and bigger data. International Journal of Cognitive Therapy, 14(1), 59–85.

Lorenzo-Luaces, L., Rodriguez-Quintana, N., Riley, T. N., & Weisz, J. R. (2020). A placebo prognostic index (PI) as a moderator of outcomes in the treatment of adolescent depression: Could it inform risk-stratification in treatment with cognitive-behavioral therapy, fluoxetine, or their combination? Psychotherapy Research, 1–14. https://doi.org/10.1080/10503307.2020.1747657

Luborsky, L., Rosenthal, R., Diguer, L., Andrusyna, T. P., Berman, J. S., Levitt, J. T., Seligman, D. A., & Krause, E. D. (2002). The dodo bird verdict is alive and well–mostly. Clinical Psychology: Science and Practice, 9, 2–12.

Luedtke, A., Sadikova, E., & Kessler, R. C. (2019). Sample size requirements for multivariate models to predict between-patient differences in best treatments of major depressive disorder. Clinical Psychological Science, 7(3), 445–461. https://doi.org/10.1177/2167702618815466

Moffitt, T. E., Caspi, A., Taylor, A., Kokaua, J., Milne, B. J., Polanczyk, G., & Poulton, R. (2010). How common are common mental disorders? Evidence that lifetime prevalence rates are doubled by prospective versus retrospective ascertainment. Psychological Medicine, 40(6), 899–909. https://doi.org/https://doi.org/10.1017/S0033291709991036

Monroe, S. M., & Harkness, K. L. (2005). Life stress, the “kindling” hypothesis, and the recurrence of depression: Considerations from a life stress perspective. Psychological Review, 112(2), 417–445. https://doi.org/10.1037/0033-295X.112.2.417

Monroe, S. M., & Harkness, K. L. (2011). Recurrence in major depression: A conceptual analysis. Psychological Review, 118(4), 655–674. https://doi.org/10.1037/a0025190

Patel, V., Chisholm, D., Parikh, R., Charlson, F. J., Degenhardt, L., Dua, T., Ferrari, A. J., Hyman, S., Laxminarayan, R., Levin, C., et al. (2016). Addressing the burden of mental, neurological, and substance use disorders: Key messages from disease control priorities. The Lancet, 387(10028), 1672–1685. https://doi.org/0.1016/S0140-6736(15)00390-6

Peipert, A., Krendl, A. C., Lorenzo-Luaces, L., et al. (2022). Waiting lists for psychotherapy and provider attitudes toward low-intensity treatments as potential interventions: Survey study. JMIR Formative Research, 6(9), e39787.

Pew Research Center: Internet & Technology. (2019). Social Media Fact Sheet. https://www.pewresearch.org/internet/fact-sheet/social-media/

Rush, A. J., Trivedi, M. H., Wisniewski, S. R., Nierenberg, A. A., Stewart, J. W., Warden, D., Niederehe, G., Thase, M. E., Lavori, P. W., Lebowitz, B. D., et al. (2006). Acute and longer-term outcomes in depressed outpatients requiring one or several treatment steps: A STAR*D report. American Journal of Psychiatry, 163(11), 1905–1917.

Schleider, J. L., & Weisz, J. R. (2017). Little treatments, promising effects? Meta-analysis of single-session interventions for youth psychiatric problems. Journal of the American Academy of Child & Adolescent Psychiatry, 56(2), 107–115.

Starvaggi, I., & Lorenzo-Luaces, L. (2023). Using machine learning to predict dropout from a guided self-help intervention despite low sample size: Testing a novel method. Manuscript in Preparation.

Thij, M. ten, Leemput, I. A. van de, Bathina, K., Rutter, L. A., Lorenzo-Luaces, L., Scheffer, M., & Bollen, J. (2020). Depression alters circadian pattern of online activity. Scientific Reports, 17272. https://doi.org/https://doi.org/10.1038/s41598-020-74314-3

Tol, W. A., Leku, M. R., Lakin, D. P., Carswell, K., Augustinavicius, J., Adaku, A., Au, T. M., Brown, F. L., Bryant, R. A., Garcia-Moreno, C., et al. (2020). Guided self-help to reduce psychological distress in south sudanese female refugees in uganda: A cluster randomised trial. The Lancet Global Health, 8(2), e254–e263.

Van Bronswijk, S. C., Bruijniks, S. J., Lorenzo-Luaces, L., Derubeis, R. J., Lemmens, L. H., Peeters, F. P., & Huibers, M. J. (2021). Cross-trial prediction in psychotherapy: External validation of the personalized advantage index using machine learning in two dutch randomized trials comparing CBT versus IPT for depression. Psychotherapy Research, 31(1), 78–91.

Wasil, A. R., Gillespie, S., Patel, R., Petre, A., Venturo-Conerly, K. E., Shingleton, R. M., Weisz, J. R., & DeRubeis, R. J. (2020). Reassessing evidence-based content in popular smartphone apps for depression and anxiety: Developing and applying user-adjusted analyses. Journal of Consulting and Clinical Psychology. 88(11), 983.

Wasil, A. R., Gillespie, S., Schell, T., Lorenzo-Luaces, L., & DeRubeis, R. J. (2021). Estimating the real-world usage of mobile apps for mental health: Development and application of two novel metrics. World Psychiatry, 20(1), 137.

Wells, J. E., & Horwood, L. J. (2004). How accurate is recall of key symptoms of depression? A comparison of recall and longitudinal reports. Psychological Medicine, 34(6), 1001–1011. https://doi.org/https://doi.org/10.1017/S0033291703001843

Whiteford, H. A., Degenhardt, L., Rehm, J., Baxter, A. J., Ferrari, A. J., Erskine, H. E., Charlson, F. J., Norman, R. E., Flaxman, A. D., Johns, N., et al. (2013). Global burden of disease attributable to mental and substance use disorders: Findings from the global burden of disease study 2010. The Lancet, 382(9904), 1575–1586.

Zimmerman, M., Ellison, W., Young, D., Chelminski, I., & Dalrymple, K. (2015). How many different ways do patients meet the diagnostic criteria for major depressive disorder? Comprehensive Psychiatry, 56, 29–34.