se_ratio_rate_ratio <- (ratio_rate_ratio_ci_upper - ratio_rate_ratio_ci_lower) / (2 * 1.96)Housing and Support Interventions for Homeless Youth in High-Income Countries: a Systematic Review — supplementary material
1 About
This page contains supplementary material for a systematic review titled Housing and Support Interventions for Homeless Youth in High-Income Countries: a Systematic Review.
The following information is provided here:
PRISMA 2020 Checklist — includes details of how this review meets the PRISMA 2020 reporting guidelines
SWiM Checklist — includes details of how this review meets the SWiM reporting guidelines
Search methods — includes search strategies and results for each published database and grey sources
Data extraction — includes the template and information extracted from each included study used to transform into a common effect size
Effect size transformation — includes detailed description of the methods used to transform reported results into a common effect size
Studies excluded at full text review — details of studies excluded at full text review with reasons for exclusion
Housing First components of included interventions — includes details of how the components of each included intervention map on to Housing First components
Instruments used in included studies — includes details of the instruments used to assess outcomes for participants in included studies
Risk of bias in included studies — includes the template and assessments used to assess the risk of bias in included studies
Barriers and Facilitators — includes additional details on the barriers and facilitators affecting implementation identified in included studies
The code to produce this supplementary material, along with the analysis code for this review is available in a GitHub repository.
2 Reporting Standards
2.1 PRISMA 2020 checklist
The PRISMA 2020 checklist (Page et al., 2021) is included in Table S1.
| Section and Topic | Item # | Checklist item | Location where item is reported |
|---|---|---|---|
| Title | |||
| Title | 1 | Identify the report as a systematic review. | In the article title |
| Abstract | |||
| Abstract | 2 | See the PRISMA 2020 for Abstracts checklist. | Abstract is fully compliant |
| Introduction | |||
| Rationale | 3 | Describe the rationale for the review in the context of existing knowledge. | Article section: ‘Why is it important to do this review?’ |
| Objectives | 4 | Provide an explicit statement of the objective(s) or question(s) the review addresses. | Article section: ‘Objectives’ |
| Methods | |||
| Eligibility criteria | 5 | Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses. | Article section: ‘Selection criteria’ and Table 1 |
| Information sources | 6 | Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted. |
Article section ‘Search methods’ Supplementary material: Section 3.1. |
| Search strategy | 7 | Present the full search strategies for all databases, registers and websites, including any filters and limits used. | Supplementary material: Table S3 and Table S4. |
| Selection process | 8 | Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process. | Article section: ‘Data collection and extraction’ |
| Data collection process | 9 | Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process. | Article section: “Data collection and extraction” |
| Data items | 10a | List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g. for all measures, time points, analyses), and if not, the methods used to decide which results to collect. | Supplementary material: Table S6 |
| Data items | 10b | List and define all other variables for which data were sought (e.g. participant and intervention characteristics, funding sources). Describe any assumptions made about any missing or unclear information. | Supplementary material: Table S5 |
| Study risk of bias assessment | 11 | Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and if applicable, details of automation tools used in the process. | Article section: ‘Assessment of risk of bias’ |
| Effect measures | 12 | Specify for each outcome the effect measure(s) (e.g. risk ratio, mean difference) used in the synthesis or presentation of results. | Article section: ‘Measures of treatment effect’ |
| Synthesis methods | 13a | Describe the processes used to decide which studies were eligible for each synthesis (e.g. tabulating the study intervention characteristics and comparing against the planned groups for each synthesis (item #5)). | Article section: “Synthesis of results” |
| Synthesis methods | 13b | Describe any methods required to prepare the data for presentation or synthesis, such as handling of missing summary statistics, or data conversions. |
Article section: “Measure of treatment effect” Supplementary material: Section 3.3 |
| Synthesis methods | 13c | Describe any methods used to tabulate or visually display results of individual studies and syntheses. | Article section: ‘Synthesis of results’ |
| Synthesis methods | 13d | Describe any methods used to synthesize results and provide a rationale for the choice(s). If meta-analysis was performed, describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used. | Article section: ‘Synthesis of results’ |
| Synthesis methods | 13e | Describe any methods used to explore possible causes of heterogeneity among study results (e.g. subgroup analysis, meta-regression). | Not applicable, quantitative synthesis was not undertaken. |
| Synthesis methods | 13f | Describe any sensitivity analyses conducted to assess robustness of the synthesized results. | Not applicable, quantitative synthesis was not undertaken. |
| Reporting bias assessment | 14 | Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases). | Article section: ‘Risk of bias assessment’ |
| Certainty assessment | 15 | Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome. |
Article section: “Assessment of bias and confidence in results” Supplementary material: Section 4.4 |
| Results | |||
| Study selection | 16a | Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram. | Article: Figure 1 |
| Study selection | 16b | Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded. | Supplementary material: Table S7 |
| Study characteristics | 17 | Cite each included study and present its characteristics. | Article: Table 2 |
| Risk of bias in studies | 18 | Present assessments of risk of bias for each included study. |
Article: Figure 2 and Figure 3 Supplementary material: Table S10 & Table S11 |
| Results of individual studies | 19 | For all outcomes, present, for each study: (a) summary statistics for each group (where appropriate) and (b) an effect estimate and its precision (e.g. confidence/credible interval), ideally using structured tables or plots. | Article: Tables 3, 4, 5, 6, 7, 8 & 9 |
| Results of syntheses | 20a | For each synthesis, briefly summarise the characteristics and risk of bias among contributing studies. | Article section: ‘Impact of included interventions’ |
| Results of syntheses | 20b | Present results of all statistical syntheses conducted. If meta-analysis was done, present for each the summary estimate and its precision (e.g. confidence/credible interval) and measures of statistical heterogeneity. If comparing groups, describe the direction of the effect. | Not applicable, quantitative syntheses was not undertaken. |
| Results of syntheses | 20c | Present results of all investigations of possible causes of heterogeneity among study results. | Not applicable, quantitative syntheses was not undertaken. |
| Results of syntheses | 20d | Present results of all sensitivity analyses conducted to assess the robustness of the synthesized results. | Not applicable, quantitative syntheses was not undertaken. |
| Reporting biases | 21 | Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed. | Not applicable, quantitative syntheses was not undertaken. |
| Certainty of evidence | 22 | Present assessments of certainty (or confidence) in the body of evidence for each outcome assessed. | Not applicable |
| Discussion | |||
| Discussion | 23a | Provide a general interpretation of the results in the context of other evidence. | Article section: ‘Discussion and Applications to Practice’ |
| Discussion | 23b | Discuss any limitations of the evidence included in the review. | Article section: ‘Discussion and Applications to Practice’ |
| Discussion | 23c | Discuss any limitations of the review processes used. | Article section: ‘Discussion and Applications to Practice’ |
| Other Information | |||
| Registration and protocol | 24a | Provide registration information for the review, including register name and registration number, or state that the review was not registered. | Article section: ‘Method’ |
| Registration and protocol | 24b | Indicate where the review protocol can be accessed, or state that a protocol was not prepared. | Article section: ‘Method’ |
| Registration and protocol | 24c | Describe and explain any amendments to information provided at registration or in the protocol. | Article section: ‘Deviations from the protocol’ |
| Support | 25 | Describe sources of financial or non-financial support for the review, and the role of the funders or sponsors in the review. | Article section: ‘Funding statement’ |
| Competing interests | 26 | Declare any competing interests of review authors. | Article section: ‘Declaration of conflicting interest’ |
| Availability of data, code and other materials | 27 | Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review. | Data collection forms: available in supplementary material and Github repository Data extracted from included studies: available in Github repository Data used for all analyses: available in Github repository Analytic code: in supplementary material and Github repository Other materials used in the review: all material required to replicate this review is available in the Github repository |
2.2 SWiM Checklist
The Synthesis Without Meta-analysis (SWiM) checklist (Campbell et al., 2020) is included in Table S2.
| SWiM reporting item | Item description | Location where item is reported |
|---|---|---|
| Methods | ||
| 1. Grouping studies for synthesis | 1a) Provide a description of, and rationale for, the groups used in the synthesis (eg, groupings of populations, interventions, outcomes, study design) | See sectopn 'Synthesis of results' |
| 1. Grouping studies for synthesis | 1b) Detail and provide rationale for any changes made subsequent to the protocol in the groups used in the synthesis | See section 'Deviations from the protocol' |
| 2. Describe the standardised metric and transformation methods used | Describe the standardised metric for each outcome. Explain why the metric(s) was chosen and describe any methods used to transform the intervention effects, as reported in the study, to the standardised metric, citing any methodological guidance consulted | See section 'Measures of treatment effect' |
| 3. Describe the synthesis methods | Describe and justify the methods used to synthesise the effects for each outcome when it was not possible to undertake a meta-analysis of effect estimates | See section: 'Data analysis and synthesis' |
| 4. Criteria used to prioritise results for summary and synthesis | Where applicable, provide the criteria used, with supporting justification, to select the particular studies, or a particular study, for the main synthesis or to draw conclusions from the synthesis (eg, based on study design, risk of bias assessments, directness in relation to the review question) | Not applicable |
| 5. Investigation of heterogeneity in reported effects | State the method(s) used to examine heterogeneity in reported effects when it was not possible to undertake a meta-analysis of effect estimates and its extensions to investigate heterogeneity | Not applicable, heterogeneity was no assessed |
| 6. Certainty of evidence | Describe the methods used to assess the certainty of the synthesis findings | Not applicable, certainty of findings was not assessed |
| 7.Data presentation methods | Describe the graphical and tabular methods used to present the effects (eg, tables, forest plots, harvest plots) | Results are presented in Tables 3-9 |
| 7.Data presentation methods | Specify key study characteristics (eg, study design, risk of bias) used to order the studies, in the text and any tables or graphs, clearly referencing the studies included | Study characteristics are presented in Table 2 |
| Results | ||
| 8. Reporting results | For each comparison and outcome, provide a description of the synthesised findings and the certainty of the findings. Describe the result in language that is consistent with the question the synthesis addresses, and indicate which studies contribute to the synthesis | See section 'Impact of included interventions' |
| Discussion | ||
| 9. Limitations of the synthesis | Report the limitations of the synthesis methods used and/or the groupings used in the synthesis and how these affect the conclusions that can be drawn in relation to the original review question | See section 'Discussion and Applications to Practice' |
3 Supplementary methodology
This section contains additional information on the methodology used in this review.
3.1 Search methods
3.1.1 Search strategy and results for database
The databases and search terms are detailed in Table S3 below.
| Database | Search terms | Date searched | Record # |
|---|---|---|---|
| Cochrane CENTRAL Registed of Controlled Trials via Ovid | 1. exp Homeless Youth/ or exp Homeless Persons/ or Homeless*.mp or Couchsurf*.mp or (Unstab* adj2 hous*).mp or Runaway.mp or Rough sleep*.mp or Insecur* hous*.mp 2. exp Young Adult/ or exp Adolescent/ or (Emerging adj2 (Adult or Youth)).mp or adolescent.mp or Youth.mp or (young adj1 (adult or person or people)).mp 3. 1 AND 2 4. exp Randomized Controlled Trial/ or exp Clinical Trial/ or exp Controlled Clinical Trial/ or exp Comparative Study/ 5. (RCT or trial* or randomi* or random* allocat* or random* assign* or (control* adj1 intervention*) or (treatment* adj1 control*) or (control adj1 (group or condition*)) or (comparison adj1 (group* or condition*)) or int* time series or comparative effectiv* or experiment* or difference in difference* or instrumental variable* or (propensity adj1 (score or match*)) or (match* adj2 (control or comparison)) or (control* adj1 treat*) or wait* list or quasi exp*).mp. 6. 4 OR 5 7. 3 AND 6 |
7 June 2023 | 234 |
| CINAHL via EBSCO | 1. (MH “Homeless Persons”) OR (MH “Homelessness”) OR TI Homeless* OR AB Homeless* OR TI Couchsurf* OR AB Couchsurf* OR TI(Unstab* n2 hous*) OR AB(Unstab* n2 hous*) OR TI Runaway OR AB Runaway OR TI Rough sleep* OR AB Rough sleep* OR TI Insecur* hous* OR AB Insecur* hous* 2. (MM “Young Adult”) OR (MH “Adolescence”) OR TI (Emerging n2 (Adult OR Youth)) OR AB (Emerging n2 (Adult OR Youth)) OR TI Adolescent OR AB Adolescent OR TI Youth OR AB Youth or TI (Young n1 (Adult or Person or People)) OR AB (Young n1 (Adult or Person or People)) 3. 1 AND 2 4. (MH “Randomized Controlled Trials”) OR (MH “Clinical Trials”) OR (MH “Experimental Studies”) OR (MH “Interrupted Time Series Analysis”) OR (MH “Quasi-Experimental Studies”) 5. TI RCT OR AB RCT OR TI Trial* OR AB Trial* OR TI Randomi* OR AB Randomi* OR TI Random* Allocat* OR AB Random* Allocat* OR TI Random* Assign* OR AB Random* Assign* OR TI (Control* n1 Intervention*) OR AB (Control* n1 Intervention*) OR TI (Treatment* n1 Control*) OR AB (Treatment* n1 Control*) OR TI (Control n1 (Group OR Condition*)) OR AB (Control n1 (Group OR Condition*)) OR TI (Comparison n1 (Group* OR Condition*)) OR AB (Comparison n1 (Group* OR Condition*)) OR TI Int* Time Series OR AB Int* Time Series OR TI Comparative Effectiv* OR AB Comparative Effectiv* OR TI Experiment* OR AB Experiment* OR TI Difference in Difference* OR AB Difference in Difference* OR TI Instrumental Variable* OR AB Instrumental Variable* OR TI (Propensity n1 (Score OR Match*)) OR AB (Propensity n1 (Score OR Match*)) OR TI (Match* n2 (Control OR Comparison)) OR AB (Match* n2 (Control OR Comparison)) OR TI (Control* n1 Treat*) OR AB (Control* n1 Treat*) OR TI Wait* list OR AB Wait* list OR TI Quasi exp* OR AB Quasi exp* 6. 4 OR 5 7. 3 AND 6 |
7 June 2023 | 429 |
| Criminal Justice Abstracts via EBSCO | 1. SU Homeless OR SU Homeless Persons OR SU Homeless Youth OR SU Runaway Teenagers OR TI Homeless* OR AB Homeless* OR TI Runaway OR AB Runaway 2. SU Youth OR SU Teenagers OR SU Adolescence OR TI (Emerging n2 (Adult OR Youth)) OR AB (Emerging n2 (Adult OR Youth)) OR TI Adolescent OR AB Adolescent OR TI Youth OR AB Youth OR TI (Young n1 (Adult or Person or People)) OR AB (Young n1 (Adult or Person or People)) 3. 1 AND 2 4. SU Treatment Effectiveness OR SU Randomized Controlled Trials 5. TI RCT OR AB RCT OR TI Trial* OR AB Trial* OR TI Randomi* OR AB Randomi* OR TI Random* Allocat* OR AB Random* Allocat* OR TI Random* Assign* OR AB Random* Assign* OR TI (Control* n1 Intervention*) OR AB (Control* n1 Intervention*) OR TI (Treatment* n1 Control*) OR AB (Treatment* n1 Control*) OR TI (Control n1 (Group OR Condition*)) OR AB (Control n1 (Group OR Condition*)) OR TI (Comparison n1 (Group* OR Condition*)) OR AB (Comparison n1 (Group* OR Condition*)) OR TI Int* Time Series OR AB Int* Time Series OR TI Comparative Effectiv* OR AB Comparative Effectiv* OR TI Experiment* OR AB Experiment* OR TI Difference in Difference* OR AB Difference in Difference* OR TI Instrumental Variable* OR AB Instrumental Variable* OR TI (Propensity n1 (Score OR Match*)) OR AB (Propensity n1 (Score OR Match*)) OR TI (Match* n2 (Control OR Comparison)) OR AB (Match* n2 (Control OR Comparison)) OR TI (Control* n1 Treat*) OR AB (Control* n1 Treat*) OR TI Wait* list OR AB Wait* list OR TI Quasi exp* OR AB Quasi exp* 6. 4 OR 5 7. 3 AND 6 |
8 June 2023 | 131 |
| EMBASE via Ovid | 1. exp Homeless Person/ or exp Homelessness/ or exp Homeless Youth/ or Homeless*.mp or Couchsurf*.mp or (Unstab* adj2 hous*).mp or Runaway.mp or Rough sleep*.mp or Insecur* hous*.mp 2. exp Young Adult/ or exp Adolescent/ or exp Juvenile/ or (Emerging adj2 (Adult or Youth)).mp or adolescent.mp or Youth.mp or (young adj1 (adult or person or people)).mp 3. 1 AND 2 4. exp Randomized Controlled Trial/ or exp Clinical Trial as Topic/ or exp Controlled Clinical Trial/ or exp Comparative Study/ or exp Quasi Experimental Study or exp Experimental Study/ 5. (RCT or trial* or randomi* or random* allocat* or random* assign* or (control* adj1 intervention*) or (treatment* adj1 control*) or (control adj1 (group or condition*)) or (comparison adj1 (group* or condition*)) or int* time series or comparative effectiv* or experiment* or difference in difference* or instrumental variable* or (propensity adj1 (score or match*)) or (match* adj2 (control or comparison)) or (control* adj1 treat*) or wait* list or quasi exp*).mp. 6. 4 OR 5 7. 3 AND 6 |
8 June 2023 | 933 |
| ERIC via Proquest | 1. MAINSUBJECT.EXACT(“Homeless People”) OR MAINSUBJECT.EXACT.EXPLODE(“Runaways”) OR noft(Homeless*) OR noft(Couchsurf*) OR noft(Unstab* N/2 hous*) OR noft(Runaway) OR noft(Rough sleep*) OR noft(Insecur* hous*) 2. MAINSUBJECT.EXACT(“Young Adults”) OR MAINSUBJECT.EXACT.EXPLODE(“Late Adolescents”) OR noft(Emerging N/2 (Adult or Youth)) OR noft(Adolescent) OR noft(Youth) OR noft(Young N/1 (Adult OR Person OR people)) 3. 1 AND 2 4. MAINSUBJECT.EXACT(“Randomized Controlled Trials”) OR MAINSUBJECT.EXACT(“Quasiexperimental Design”) OR MAINSUBJECT.EXACT(‘Program Evaluation’) 5. noft(RCT) OR noft(Trial*) OR noft(Randomi*) OR noft(Random* allocat*) OR noft(Random* assign*) OR noft(Control* N/1 Intervention*) OR noft(Treatment* N/1 Control*) OR noft(Control N/1 (Group OR Condition*)) OR noft(Comparison N/1 (Group* OR Condition*)) OR noft(Int* Time Series) OR noft(Comparative Effectiv*) OR noft(Experiment*) OR noft(Difference in Difference*) OR noft(Instrumental Variable*) OR noft(Propensity N/1 (Score OR Match*)) OR noft(Match* N/2 (Control OR Comparison)) OR noft(Control* N/1 Treat*) OR noft(Wait* list) OR noft(Quasi exp*) 6. 4 OR 5 7. 3 AND 6 |
8 June 2023 | 304 |
| APA PsycINFO via Ovid | 1. exp Homeless Youth/ or exp Homeless/ or exp Homeless Mentally Ill or exp Runaway Behavior/ or exp Shelters/ or Homeless*.mp or Couchsurf*.mp or (Unstab* adj2 hous*).mp or Runaway.mp or Rough sleep*.mp or Insecur* hous*.mp 2. exp Emerging Adulthood/ or exp Late Adolescence/ or (Emerging adj2 (Adult or Youth)).mp or adolescent.mp or Youth.mp or (young adj1 (adult or person or people)).mp 3. 1 AND 2 4. exp Randomized Controlled Trials/ or exp Clinical Trials/ or exp Treatment Effectiveness Evaluation/ or exp Quasi Experimental Methods/ or exp Experimental Methods/ 5. (RCT or trial* or randomi* or random* allocat* or random* assign* or (control* adj1 intervention*) or (treatment* adj1 control*) or (control adj1 (group or condition*)) or (comparison adj1 (group* or condition*)) or int* time series or comparative effectiv* or experiment* or difference in difference* or instrumental variable* or (propensity adj1 (score or match*)) or (match* adj2 (control or comparison)) or (control* adj1 treat*) or wait* list or quasi exp*).mp. 6. 4 OR 5 7. 3 AND 6 |
8 June 2023 | 360 |
| Ovid MEDLINE | 1. exp Homeless Youth/ or exp Ill-Housed Persons/ or Homeless*.mp or Couchsurf*.mp or (Unstab* adj2 hous*).mp or Runaway.mp or Rough sleep*.mp or Insecur* hous*.mp 2. (limit 1 to (“all child (0 to 18 years)” or “adolescent (13 to 18 years)” or “young adult (19 to 24 years)”) 3. exp Young Adult/ or exp Adolescent/ or (Emerging adj2 (Adult or Youth)).mp or adolescent.mp or Youth.mp or (young adj1 (adult or person or people)).mp 4. 1 AND 3 5. 2 OR 4 6. exp Randomized Controlled Trial/ or exp Clinical Trial/ or exp Controlled Clinical Trial/ or exp Controlled before-after Studies/ or exp Comparative Study/ or exp Evaluation Study/ or exp Feasibility Studies/ or exp Follow-up Studies/ or exp Interrupted Time Series Analysis/ 7. (RCT or trial* or randomi* or random* allocat* or random* assign*).mp or (control* adj1 intervention*).mp or (treatment* adj1 control*).mp or (control adj1 (group or condition*)).mp or (comparison adj1 (group* or condition*)).mp or int* time series.mp or comparative effectiv*.mp or experiment*.mp or difference in difference*.mp or instrumental variable*.mp or (propensity adj1 (score or match*)).mp or (match* adj2 (control or comparison)).mp or (control* adj1 treat*).mp or wait* list.mp or quasi exp*.mp 8. 6 OR 7 9. 5 AND 8 |
8 June 2023 | 959 |
| Sociological Abstracts via Proquest | 1. MAINSUBJECT.EXACT(“Homeless People”) OR MAINSUBJECT.EXACT.EXPLODE(“Runaways”) OR noft(Homeless*) OR noft(Couchsurf*) OR noft(Unstab* N/2 hous*) OR noft(Runaway) OR noft(Rough sleep*) OR noft(Insecur* hous*) 2. MAINSUBJECT.EXACT(“Young Adults”) OR MAINSUBJECT.EXACT(“Adolescents”) OR noft(Emerging N/2 (Adult or Youth)) OR noft(Adolescent) OR noft(Youth) OR noft(Young N/1 (Adult OR Person OR people)) 3. 1 AND 2 4. MAINSUBJECT.EXACT(“Experiments”) OR MAINSUBJECT.EXACT(“Effectiveness”) OR MAINSUBJECT.EXACT(“Evaluation”) 5. noft(RCT) OR noft(Trial*) OR noft(Randomi*) OR noft(Random* allocat*) OR noft(Random* assign*) OR noft(Control* N/1 Intervention*) OR noft(Treatment* N/1 Control*) OR noft(Control N/1 (Group OR Condition*)) OR noft(Comparison N/1 (Group* OR Condition*)) OR noft(Int* Time Series) OR noft(Comparative Effectiv*) OR noft(Experiment*) OR noft(Difference in Difference*) OR noft(Instrumental Variable*) OR noft(Propensity N/1 (Score OR Match*)) OR noft(Match* N/2 (Control OR Comparison)) OR noft(Control* N/1 Treat*) OR noft(Wait* list) OR noft(Quasi exp*) 6. 4 OR 5 7. 3 AND 6 |
7 June 2023 | 327 |
| Social Services Abstracts via Proquest | 1. MAINSUBJECT.EXACT(“Homelessness”) OR MAINSUBJECT.EXACT(“Runaways”) OR noft(Homeless*) OR noft(Couchsurf*) OR noft(Unstab* N/2 hous*) OR noft(Runaway) OR noft(Rough sleep*) OR noft(Insecur* hous*) 2. MAINSUBJECT.EXACT(“Young Adults”) OR MAINSUBJECT.EXACT(“Adolescents”) OR noft(Emerging N/2 (Adult or Youth)) OR noft(Adolescent) OR noft(Youth) OR noft(Young N/1 (Adult OR Person OR people)) 3. 1 AND 2 4. MAINSUBJECT.EXACT(“Evaluation Research”) OR MAINSUBJECT.EXACT(“Evaluation”) 5. noft(RCT) OR noft(Trial*) OR noft(Randomi*) OR noft(Random* allocat*) OR noft(Random* assign*) OR noft(Control* N/1 Intervention*) OR noft(Treatment* N/1 Control*) OR noft(Control N/1 (Group OR Condition*)) OR noft(Comparison N/1 (Group* OR Condition*)) OR noft(Int* Time Series) OR noft(Comparative Effectiv*) OR noft(Experiment*) OR noft(Difference in Difference*) OR noft(Instrumental Variable*) OR noft(Propensity N/1 (Score OR Match*)) OR noft(Match* N/2 (Control OR Comparison)) OR noft(Control* N/1 Treat*) OR noft(Wait* list) OR noft(Quasi exp*) 6. 4 OR 5 7. 3 AND 6 |
8 June 2023 | 538 |
| Violence and Abuse Abstracts via EBSCO | 1. SU Homelessness OR SU Homeless Persons OR SU Homeless Youth OR SU Runaway Teenagers OR TI Homeless* OR AB Homeless* OR TI Runaway OR AB Runaway 2. SU Youth OR SU Teenagers OR SU Adolescence OR TI (Emerging n2 (Adult OR Youth)) OR AB (Emerging n2 (Adult OR Youth)) OR TI Adolescent OR AB Adolescent OR TI Youth OR AB Youth OR TI (Young n1 (Adult or Person or People)) OR AB (Young n1 (Adult or Person or People)) 3. 1 AND 2 4. SU Treatment Effectiveness OR SU Randomized Controlled Trials 5. TI RCT OR AB RCT OR TI Trial* OR AB Trial* OR TI Randomi* OR AB Randomi* OR TI Random* Allocat* OR AB Random* Allocat* OR TI Random* Assign* OR AB Random* Assign* OR TI (Control* n1 Intervention*) OR AB (Control* n1 Intervention*) OR TI (Treatment* n1 Control*) OR AB (Treatment* n1 Control*) OR TI (Control n1 (Group OR Condition*)) OR AB (Control n1 (Group OR Condition*)) OR TI (Comparison n1 (Group* OR Condition*)) OR AB (Comparison n1 (Group* OR Condition*)) OR TI Int* Time Series OR AB Int* Time Series OR TI Comparative Effectiv* OR AB Comparative Effectiv* OR TI Experiment* OR AB Experiment* OR TI Difference in Difference* OR AB Difference in Difference* OR TI Instrumental Variable* OR AB Instrumental Variable* OR TI (Propensity n1 (Score OR Match*)) OR AB (Propensity n1 (Score OR Match*)) OR TI (Match* n2 (Control OR Comparison)) OR AB (Match* n2 (Control OR Comparison)) OR TI (Control* n1 Treat*) OR AB (Control* n1 Treat*) OR TI Wait* list OR AB Wait* list OR TI Quasi exp* OR AB Quasi exp* 6. 4 OR 5 7. 3 AND 6 |
8 June 2023 | 17 |
3.1.2 Search strategy and results for grey sources
Sources and search terms used for grey sources are detailed in Table S4 below.
| Sources | Date searched | Search process |
|---|---|---|
| Australian Housing and Urban Research Institute (AHURI) | 17 October 2023 | Review all publications under “Homelessness” |
| Analysis and Policy observatory (APO) database (Informit) | 16 October 2023 | Search for “Youth Homeless*” and review all results |
| Australian Criminology Database (CINCH) (Informit) | 16 October 2023 | Search for “Youth Homeless*” and review all results |
| Australian Family and Society Abstracts database (Informit) (AIFS) | 17 October 2023 | Search for “Youth Homeless*” and review all results |
| Chapin Hall at the University of Chicago | 17 October 2023 | Review all publications under: “Youth Homelessness” and type of work “Research” |
| Social Care Online (SCIE) | Database no longer maintained/accessible | Not applicable |
| Washington State Institute for Public Policy (WSIPP) | 16 October 2023 | Reviewed all “Publications” returned after “homeless” was searched in “any field” |
3.2 Data extraction
Pairs of reviewers performed data extraction (DT and SV), with one reviewer checking the other’s results.
We developed a data extraction template using Google’s Sheets app to facilitate collaboration between team members. Each tab of the sheet contained specific information.
All reported measures corresponding to outcomes of interest at all available time points were extracted into a shared spreadsheet which is available in Table S9.
The template in Table S5 was used to extract information about the study.
×
The template in Table S6 was used to extract information about reported outcomes that would allow us to transform each of the reported results into a common effect size.
×
3.3 Effect size transformation
In most cases, formulas from Borenstein, Hedges, Higgins, & Rothstein (2009) were used to convert reported results into a common effect size (Hedges’ g). This was done using the R Project for Statistical Computing (R Core Team, 2025), and specifically the esc package (Lüdecke, 2019).
In cases where studies lacked sufficient data for effect size computation or conversion, the study’s authors were approached to obtain the missing information. If this information was unavailable or insufficient to calculate an effect size, reasonable assumptions were made.
The results in the study by Kozloff, Adair, et al. (2016) are one such case where assumptions were made. Results in this study were presented as treatment effects (i.e., mean differences, ratios or rate ratios and ratios of odds ratios and their confidence intervals). The review team was unable to obtain the necessary information from the authors to transform these using the formulas in the esc package. Accordingly we approximated these results using information reported in the included studies. The approach used for each type of result, and the assumptions they rely on are detailed below.
3.3.1 Estimating Cohen’s d or Hedges’ g from a Ratio of Rate Ratios
The esc_ratio_rate_ratio function estimates Cohen’s d or Hedges’ g effect sizes from a reported ratio of rate ratios, its standard error or confidence interval, sample sizes, and the population baseline rate. The method relies on several assumptions and approximations to convert the ratio of rate ratios to a standardised mean difference.
3.3.1.1 Assumptions
- The ratio of rate ratios is a measure of the relative difference in event rates between the treatment and comparison groups, with the comparison group as the reference (Lash, VanderWeele, Haneause, & Rothman, 2021).
- The standard error of the ratio of rate ratios is either provided or can be calculated from the confidence interval assuming a normal distribution (Altman & Bland, 2011).
- The population baseline rate represents the proportion of the population experiencing the event at baseline and is known or can be estimated (Lash et al., 2021).
- The event counts in the treatment and comparison groups follow a Poisson distribution (Fleiss, Levin, & Paik, 2003).
- The sample sizes of the treatment and comparison groups are known.
3.3.1.2 Step 1: Obtain the standard error of the ratio of rate ratios
The function first obtains the standard error of the ratio of rate ratios either directly from the provided ratio_rate_ratio_se argument or by calculating it from the confidence interval using the following formula (Altman & Bland, 2011):
\[ SE_{RRR} = \frac{RRR_{upper} - RRR_{lower}}{2 \times 1.96} \]
where \(SE_{RRR}\) is the standard error of the ratio of rate ratios, \(RRR_{upper}\) and \(RRR_{lower}\) are the upper and lower bounds of the ratio of rate ratios confidence interval, respectively.
3.3.1.3 Step 2: Estimate the event counts
The event count for the comparison group is estimated using the population baseline rate and the comparison group sample size:
\[ E_C = n_C \times BR \]
where \(E_C\) is the event count for the comparison group, \(n_C\) is the comparison group sample size, and \(BR\) is the population baseline rate.
comparison_event_count <- comparison_n * population_baseline_rateThe event count for the treatment group is then estimated using the ratio of rate ratios and the comparison event count:
\[ E_T = E_C \times RRR \]
where \(E_T\) is the event count for the treatment group, and \(RRR\) is the ratio of rate ratios.
treatment_event_count <- comparison_event_count * ratio_rate_ratio3.3.1.4 Step 3: Calculate the pooled event rate and standard deviation
The pooled event count is calculated as the average of the treatment and comparison event counts:
\[ E_P = \frac{E_T + E_C}{2} \]
where \(E_P\) is the pooled event count.
pooled_event_count <- (treatment_event_count + comparison_event_count) / 2The pooled event rate is then estimated by dividing the pooled event count by the average of the treatment and comparison sample sizes:
\[ R_P = \frac{E_P}{\frac{n_T + n_C}{2}} \]
where \(R_P\) is the pooled event rate, and \(n_T\) and \(n_C\) are the treatment and comparison group sample sizes, respectively.
pooled_event_rate <- pooled_event_count / ((treatment_n + comparison_n) / 2)The pooled standard deviation is estimated using the pooled event rate, assuming a Poisson distribution (Fleiss et al., 2003):
\[ SD_P = \sqrt{R_P \times (1 - R_P)} \]
where \(SD_P\) is the pooled standard deviation.
pooled_sd <- sqrt(pooled_event_rate * (1 - pooled_event_rate))3.3.1.5 Step 4: Calculate Cohen’s d
Cohen’s d is calculated using the estimated event rates for the treatment and comparison groups and the pooled standard deviation (Cohen, 1988):
\[ d = \frac{\frac{E_T}{n_T} - \frac{E_C}{n_C}}{SD_P} \]
where \(d\) is Cohen’s d.
cohens_d <- (treatment_event_count / treatment_n - comparison_event_count / comparison_n) / pooled_sd3.3.1.6 Step 5: Calculate the standard error and confidence interval for Cohen’s d
The standard error for Cohen’s d is estimated using the event rates and sample sizes:
\[ SE_d = \sqrt{\frac{E_T}{n_T^2} + \frac{E_C}{n_C^2}} \times \frac{1}{SD_P} \]
where \(SE_d\) is the standard error for Cohen’s d, \(E_T\) and \(E_C\) are the event counts in the treatment and comparison groups, respectively, \(n_T\) and \(n_C\) are the sample sizes of the treatment and comparison groups, respectively, and \(SD_P\) is the pooled standard deviation.
standard_error_d <- sqrt((treatment_event_count / treatment_n^2) + (comparison_event_count / comparison_n^2)) / pooled_sdThe confidence interval for Cohen’s d is then calculated using the standard error (Nakagawa & Cuthill, 2007):
\[ CI_d = [d - 1.96 \times SE_d, d + 1.96 \times SE_d] \]
where \(CI_d\) is the confidence interval for Cohen’s d.
cohens_d_ci_lower <- cohens_d - (1.96 * standard_error_d)
cohens_d_ci_upper <- cohens_d + (1.96 * standard_error_d)3.3.1.7 Step 6: Calculate Hedges’ g (optional)
If the esc_type argument is set to ‘g’, the function also calculates Hedges’ g, which applies Hedges’ correction to an estimate of the standardised mean difference (Hedges, 1981):
\[ g = d \times \left(1 - \frac{3}{4 \times (n_T + n_C) - 9}\right) \]
where \(g\) is Hedges’ g.
hedges_g <- cohens_d * (1 - (3 / (4 * (treatment_n + comparison_n) - 9)))The confidence interval for Hedges’ g is calculated using the same standard error as Cohen’s d.
3.3.2 Estimating Cohen’s d or Hedges’ g from a Mean Difference
The esc_mean_difference function estimates Cohen’s d or Hedges’ g effect sizes from a reported mean difference, its standard error or confidence interval, and the sample sizes of the treatment and comparison groups. The function calculates the pooled standard deviation, effect size, standard error, confidence interval, and p-value.
3.3.2.1 Assumptions
- The mean difference between the treatment and comparison groups is provided.
- The standard error of the mean difference is either provided or can be calculated from the confidence interval assuming a normal distribution (Altman & Bland, 2011).
- The sample sizes of the treatment and comparison groups are known.
- The variance of the outcome is assumed to be equal in both groups.
3.3.2.2 Step 1: Obtain the standard error of the mean difference
The function first obtains the standard error of the mean difference either directly from the provided mean_diff_se argument or by calculating it from the confidence interval using the following formula (Altman & Bland, 2011):
\[ SE_{MD} = \frac{MD_{upper} - MD_{lower}}{2 \times 1.96} \]
standard_error <- (mean_diff_ci_upper - mean_diff_ci_lower) / (2 * 1.96)where \(SE_{MD}\) is the standard error of the mean difference, \(MD_{upper}\) and \(MD_{lower}\) are the upper and lower bounds of the mean difference confidence interval, respectively.
3.3.2.3 Step 2: Estimate the pooled standard deviation
The pooled standard deviation is estimated using the standard error and the sample sizes of the treatment and comparison groups:
\[ SD_{pooled} = \frac{SE_{MD}}{\sqrt{\frac{1}{n_T} + \frac{1}{n_C}}} \]
where \(SD_{pooled}\) is the pooled standard deviation, \(SE_{MD}\) is the standard error of the mean difference, and \(n_T\) and \(n_C\) are the sample sizes of the treatment and comparison groups, respectively.
standard_deviation_pooled <- standard_error / sqrt((1 / treatment_n) + (1 / comparison_n))3.3.2.4 Step 3: Calculate Cohen’s d
Cohen’s d is calculated by dividing the mean difference by the pooled standard deviation (Cohen, 1988):
\[ d = \frac{MD}{SD_{pooled}} \]
where \(d\) is Cohen’s d, \(MD\) is the mean difference, and \(SD_{pooled}\) is the pooled standard deviation.
cohens_d <- mean_difference / standard_deviation_pooled3.3.2.5 Step 4: Calculate the standard error and confidence interval for Cohen’s d
The standard error for Cohen’s d is calculated by dividing the standard error of the mean difference by the pooled standard deviation:
\[ SE_d = \frac{SE_{MD}}{SD_{pooled}} \]
where \(SE_d\) is the standard error for Cohen’s d.
standard_error_d <- standard_error / standard_deviation_pooledThe confidence interval for Cohen’s d is then calculated using the standard error (Nakagawa & Cuthill, 2007):
\[ CI_d = [d - 1.96 \times SE_d, d + 1.96 \times SE_d] \]
where \(CI_d\) is the confidence interval for Cohen’s d.
mean_diff_ci_lower_d <- cohens_d - (1.96 * standard_error_d)
mean_diff_ci_upper_d <- cohens_d + (1.96 * standard_error_d)3.3.2.6 Step 5: Calculate the p-value
The t-statistic is calculated by dividing the mean difference by its standard error:
\[ t = \frac{MD}{SE_{MD}} \]
where \(t\) is the t-statistic.
t_stat <- mean_difference / standard_errorThe degrees of freedom are calculated as the sum of the sample sizes minus 2:
\[ df = n_T + n_C - 2 \]
where \(df\) is the degrees of freedom.
df <- treatment_n + comparison_n - 2The p-value is then calculated using the t-distribution with the calculated degrees of freedom:
p_value <- 2 * (1 - pt(abs(t_stat), df))3.3.2.7 Step 6: Calculate Hedges’ g (optional)
If the esc_type argument is set to ‘g’, the function also calculates Hedges’ g, which is an unbiased estimate of the standardised mean difference (Hedges, 1981):
\[ g = d \times \left(1 - \frac{3}{4 \times (n_T + n_C) - 9}\right) \]
where \(g\) is Hedges’ g.
hedges_g <- cohens_d * (1 - (3 / (4 * (treatment_n + comparison_n) - 9)))The confidence interval for Hedges’ g is calculated using the same standard error as Cohen’s d.
3.3.3 Estimating Cohen’s d or Hedges’ g from a Ratio of Odds Ratios
The esc_ratio_odds_ratio function estimates Cohen’s d or Hedges’ g effect sizes from a reported ratio of odds ratios, its standard error or confidence interval, sample sizes, and the population baseline rate. The method relies on several assumptions and approximations to convert the ratio of odds ratios to a standardised mean difference.
3.3.3.1 Assumptions
- The ratio of odds ratios is a measure of the relative difference in odds between the treatment and comparison groups, with the comparison group as the reference (Szumilas, 2010).
- The standard error of the ratio of odds ratios is either provided or can be calculated from the confidence interval assuming a normal distribution of the log ratio of odds ratios (Altman & Bland, 2011).
- The population baseline rate represents the proportion of the population experiencing the event at baseline and is known or can be estimated (Lash et al., 2021).
- The event counts in the treatment and comparison groups follow a binomial distribution (Fleiss et al., 2003).
- The sample sizes of the treatment and comparison groups are known.
3.3.3.2 Step 1: Obtain the standard error of the log ratio of odds ratios
The function first obtains the standard error of the log ratio of odds ratios either directly from the provided ratio_odds_ratio_se argument or by calculating it from the confidence interval using the following formula (Altman & Bland, 2011):
\[ SE_{log(ROR)} = \frac{log(ROR_{upper}) - log(ROR_{lower})}{2 \times 1.96} \]
se_log_ratio_odds_ratio <- (log_ratio_odds_ratio_ci_upper - log_ratio_odds_ratio_ci_lower) / (2 * 1.96)where \(SE_{log(ROR)}\) is the standard error of the log ratio of odds ratios, \(ROR_{upper}\) and \(ROR_{lower}\) are the upper and lower bounds of the ratio of odds ratios confidence interval, respectively.
3.3.3.3 Step 2: Convert the ratio of odds ratios to the log scale
The ratio of odds ratios is converted to the log scale:
\[ log(ROR) = log(ROR) \]
log_ratio_odds_ratio <- log(ratio_odds_ratio)where \(log(ROR)\) is the log ratio of odds ratios.
3.3.3.4 Step 3: Estimate the event probabilities and odds
The event probability in the comparison group is estimated using the population baseline rate:
\[ P_C = BR \]
where \(P_C\) is the event probability in the comparison group, and \(BR\) is the population baseline rate.
comparison_event_prob <- population_baseline_rateThe odds in the comparison group are estimated using the event probability:
\[ O_C = \frac{P_C}{1 - P_C} \]
where \(O_C\) is the odds in the comparison group.
comparison_odds <- comparison_event_prob / (1 - comparison_event_prob)The odds in the treatment group are estimated using the ratio of odds ratios and the comparison odds:
\[ O_T = O_C \times ROR \]
where \(O_T\) is the odds in the treatment group, and \(ROR\) is the ratio of odds ratios.
treatment_odds <- comparison_odds * ratio_odds_ratioThe event probability in the treatment group is estimated using the treatment odds:
\[ P_T = \frac{O_T}{1 + O_T} \]
where \(P_T\) is the event probability in the treatment group.
treatment_event_prob <- treatment_odds / (1 + treatment_odds)3.3.3.5 Step 4: Convert event probabilities to approximate event counts
The event probabilities are converted to approximate event counts using the sample sizes:
\[ E_T = n_T \times P_T \\ E_C = n_C \times P_C \]
where \(E_T\) and \(E_C\) are the event counts in the treatment and comparison groups, respectively, and \(n_T\) and \(n_C\) are the sample sizes of the treatment and comparison groups, respectively.
treatment_event_count <- treatment_n * treatment_event_prob
comparison_event_count <- comparison_n * comparison_event_prob3.3.3.6 Step 5: Calculate the pooled event rate and standard deviation
The pooled event count is calculated as the average of the treatment and comparison event counts:
\[ E_P = \frac{E_T + E_C}{2} \]
where \(E_P\) is the pooled event count.
pooled_event_count <- (treatment_event_count + comparison_event_count) / 2The pooled event rate is then estimated by dividing the pooled event count by the average of the treatment and comparison sample sizes:
\[ R_P = \frac{E_P}{\frac{n_T + n_C}{2}} \]
where \(R_P\) is the pooled event rate.
pooled_event_rate <- pooled_event_count / ((treatment_n + comparison_n) / 2)The pooled standard deviation is estimated using the pooled event rate, assuming a binomial distribution (Fleiss et al., 2003):
\[ SD_P = \sqrt{R_P \times (1 - R_P)} \]
where \(SD_P\) is the pooled standard deviation.
pooled_sd <- sqrt(pooled_event_rate * (1 - pooled_event_rate))3.3.3.7 Step 6: Calculate Cohen’s d
Cohen’s d is calculated using the estimated event probabilities for the treatment and comparison groups and the pooled standard deviation (Cohen, 1988):
\[ d = \frac{P_T - P_C}{SD_P} \]
where \(d\) is Cohen’s d.
cohens_d <- (treatment_event_prob - comparison_event_prob) / pooled_sd3.3.3.8 Step 7: Calculate the standard error and confidence interval for Cohen’s d
The standard error for Cohen’s d is estimated using the event probabilities and sample sizes (Hasselblad & Hedges, 1995):
\[ SE_d = \sqrt{\frac{P_T \times (1 - P_T)}{n_T} + \frac{P_C \times (1 - P_C)}{n_C}} \times \frac{1}{SD_P} \]
where \(SE_d\) is the standard error for Cohen’s d, \(P_T\) and \(P_C\) are the event probabilities in the treatment and comparison groups, respectively, \(n_T\) and \(n_C\) are the sample sizes of the treatment and comparison groups, respectively, and \(SD_P\) is the pooled standard deviation.
standard_error_d <- sqrt((treatment_event_prob * (1 - treatment_event_prob)) / treatment_n + (comparison_event_prob * (1 - comparison_event_prob)) / comparison_n) / pooled_sdThe confidence interval for Cohen’s d is then calculated using the standard error (Nakagawa & Cuthill, 2007):
\[ CI_d = [d - 1.96 \times SE_d, d + 1.96 \times SE_d] \]
where \(CI_d\) is the confidence interval for Cohen’s d.
cohens_d_ci_lower <- cohens_d - (1.96 * standard_error_d)
cohens_d_ci_upper <- cohens_d + (1.96 * standard_error_d)3.3.3.9 Step 8: Calculate the p-value
The Z-statistic is calculated using the log ratio of odds ratios and its standard error:
\[ Z = \frac{log(ROR)}{SE_{log(ROR)}} \]
where \(Z\) is the Z-statistic.
z_stat <- log_ratio_odds_ratio / se_log_ratio_odds_ratioThe p-value is then calculated using the standard normal distribution:
p_value <- 2 * (1 - pnorm(abs(z_stat)))3.3.3.10 Step 9: Calculate Hedges’ g (optional)
If the esc_type argument is set to ‘g’, the function also calculates Hedges’ g, which is an unbiased estimate of the standardised mean difference (Hedges, 1981):
\[ g = d \times \left(1 - \frac{3}{4 \times (n_T + n_C) - 9}\right) \]
where \(g\) is Hedges’ g.
hedges_g <- cohens_d * (1 - (3 / (4 * (treatment_n + comparison_n) - 9)))The confidence interval for Hedges’ g is calculated using the same standard error as Cohen’s d.
3.3.4 Analytic Code Implementation
The code chunk below details the R function that implements the methods described above.
Code
# Purpose: Most common transformations can be handled using those in the esc() R package.
# However, the results in the study by Kozloff (2016) are presented as treatment effects (i.e., mean differences, ratios or rate ratios and ratios of odds ratios and their confidence intervals).
# These three functions transform these results into a common measure of the standardised mean difference: Cohen's d or Hedge's g
## Transform ratio of odds ratios
# Declare function to estimate either Cohen's d or Hedges' g, along with their 95% confidence interval and derived p-value from reported ratio of odds ratios and baseline rates
esc_ratio_odds_ratio <- function(
ratio_odds_ratio,
ratio_odds_ratio_se = NULL,
ratio_odds_ratio_ci_lower = NULL,
ratio_odds_ratio_ci_upper = NULL,
treatment_n,
comparison_n,
population_baseline_rate,
esc_type = 'd') {
# Use reported standard error if provided, otherwise calculate standard error from confidence intervals
if (!is.null(ratio_odds_ratio_se)) {
se_log_ratio_odds_ratio <- ratio_odds_ratio_se / ratio_odds_ratio
} else if (!is.null(ratio_odds_ratio_ci_lower) & !is.null(ratio_odds_ratio_ci_upper)) {
log_ratio_odds_ratio_ci_lower <- log(ratio_odds_ratio_ci_lower)
log_ratio_odds_ratio_ci_upper <- log(ratio_odds_ratio_ci_upper)
se_log_ratio_odds_ratio <- (log_ratio_odds_ratio_ci_upper - log_ratio_odds_ratio_ci_lower) / (2 * 1.96)
} else {
stop("Either ratio_odds_ratio_se or both ratio_odds_ratio_ci_lower and ratio_odds_ratio_ci_upper must be provided.")
}
# Convert ratio of odds ratios to log scale
log_ratio_odds_ratio <- log(ratio_odds_ratio)
# Estimate the event probability in the comparison group using the population baseline rate
comparison_event_prob <- population_baseline_rate
# Estimate the odds in the comparison group
comparison_odds <- comparison_event_prob / (1 - comparison_event_prob)
# Estimate the odds in the treatment group using the ratio of odds ratios and the comparison odds
treatment_odds <- comparison_odds * ratio_odds_ratio
# Convert odds to event probabilities
treatment_event_prob <- treatment_odds / (1 + treatment_odds)
# Convert event probabilities to approximate event counts
treatment_event_count <- treatment_n * treatment_event_prob
comparison_event_count <- comparison_n * comparison_event_prob
# Calculate the pooled event count
pooled_event_count <- (treatment_event_count + comparison_event_count) / 2
# Estimate the pooled event rate
pooled_event_rate <- pooled_event_count / ((treatment_n + comparison_n) / 2)
# Estimate the pooled standard deviation using the pooled event rate
pooled_sd <- sqrt(pooled_event_rate * (1 - pooled_event_rate))
# Calculate Cohen's d
cohens_d <- (treatment_event_prob - comparison_event_prob) / pooled_sd
# Calculate the standard error for Cohen's d using the event probabilities and sample sizes
standard_error_d <- sqrt((treatment_event_prob * (1 - treatment_event_prob)) / treatment_n + (comparison_event_prob * (1 - comparison_event_prob)) / comparison_n) / pooled_sd
# Calculate the confidence interval for Cohen's d
cohens_d_ci_lower <- cohens_d - (1.96 * standard_error_d)
cohens_d_ci_upper <- cohens_d + (1.96 * standard_error_d)
# Calculate the Z-statistic
z_stat <- log_ratio_odds_ratio / se_log_ratio_odds_ratio
# Calculate the p-value using the standard normal distribution
p_value <- 2 * (1 - pnorm(abs(z_stat)))
# Check if Hedges' g is requested
if (esc_type == 'g') {
# Calculate Hedges' g
hedges_g <- cohens_d * (1 - (3 / (4 * (treatment_n + comparison_n) - 9)))
return(list(effect_size = hedges_g, conf_interval = c(cohens_d_ci_lower, cohens_d_ci_upper), p_value = p_value))
} else {
return(list(effect_size = cohens_d, conf_interval = c(cohens_d_ci_lower, cohens_d_ci_upper), p_value = p_value))
}
}
## Transform mean differences
# Declare function to estimate either Cohen's d or Hedges' g, along with their 95% confidence interval and derived p-value from reported mean differences
esc_mean_difference <- function(
mean_difference,
mean_diff_se = NULL,
mean_diff_ci_lower = NULL,
mean_diff_ci_upper = NULL,
treatment_n,
comparison_n,
esc_type) {
# Use reported standard error if provided, otherwise calculate standard error from confidence intervals
if (!is.null(mean_diff_se)) {
standard_error <- mean_diff_se
} else if (!is.null(mean_diff_ci_lower) & !is.null(mean_diff_ci_upper)) {
standard_error <- (mean_diff_ci_upper - mean_diff_ci_lower) / (2 * 1.96)
} else {
stop("Either mean_diff_se or both mean_diff_ci_lower and mean_diff_ci_upper must be provided.")
}
# Estimate the pooled standard deviation using the standard error
standard_deviation_pooled <- standard_error / sqrt((1 / treatment_n) + (1 / comparison_n))
# Calculate Cohen's d
cohens_d <- mean_difference / standard_deviation_pooled
# Calculate the standard error for Cohen's d
standard_error_d <- standard_error / standard_deviation_pooled
# Calculate the confidence interval for Cohen's d (95% confidence level, Z = 1.96)
mean_diff_ci_lower_d <- cohens_d - (1.96 * standard_error_d)
mean_diff_ci_upper_d <- cohens_d + (1.96 * standard_error_d)
# Calculate the t-statistic
t_stat <- mean_difference / standard_error
# Calculate degrees of freedom
df <- treatment_n + comparison_n - 2
# Calculate the p-value using the t-distribution
p_value <- 2 * (1 - pt(abs(t_stat), df))
# Check if Hedges' g is requested
if (esc_type == 'g') {
# Calculate Hedges' g
hedges_g <- cohens_d * (1 - (3 / (4 * (treatment_n + comparison_n) - 9)))
return(list(effect_size = hedges_g, conf_interval = c(mean_diff_ci_lower_d, mean_diff_ci_upper_d), p_value = p_value))
} else {
return(list(effect_size = cohens_d, conf_interval = c(mean_diff_ci_lower_d, mean_diff_ci_upper_d), p_value = p_value))
}
}
## Transform ratio of rate ratios
# Declare function to estimate either Cohen's d or Hedges' g, along with their 95% confidence interval and derived p-value from reported differences in rate ratios and baseline rates
esc_ratio_rate_ratio <- function(
ratio_rate_ratio,
ratio_rate_ratio_se = NULL,
ratio_rate_ratio_ci_lower = NULL,
ratio_rate_ratio_ci_upper = NULL,
treatment_n,
comparison_n,
population_baseline_rate,
esc_type = 'd') {
# Use reported standard error if provided, otherwise calculate standard error from confidence intervals
if (!is.null(ratio_rate_ratio_se)) {
se_ratio_rate_ratio <- ratio_rate_ratio_se
} else if (!is.null(ratio_rate_ratio_ci_lower) & !is.null(ratio_rate_ratio_ci_upper)) {
se_ratio_rate_ratio <- (ratio_rate_ratio_ci_upper - ratio_rate_ratio_ci_lower) / (2 * 1.96)
} else {
stop("Either ratio_rate_ratio_se or both ratio_rate_ratio_ci_lower and ratio_rate_ratio_ci_upper must be provided.")
}
# Estimate the event count for the comparison group using the population baseline rate
comparison_event_count <- comparison_n * population_baseline_rate
# Estimate the event count for the treatment group using the rate ratio and the comparison event count
treatment_event_count <- comparison_event_count * ratio_rate_ratio
# Calculate the pooled event count
pooled_event_count <- (treatment_event_count + comparison_event_count) / 2
# Estimate the pooled event rate
pooled_event_rate <- pooled_event_count / ((treatment_n + comparison_n) / 2)
# Estimate the pooled standard deviation using the pooled event rate
pooled_sd <- sqrt(pooled_event_rate * (1 - pooled_event_rate))
# Calculate Cohen's d
cohens_d <- (treatment_event_count / treatment_n - comparison_event_count / comparison_n) / pooled_sd
# Calculate the standard error for Cohen's d using the event rates and sample sizes
standard_error_d <- sqrt((treatment_event_count / treatment_n^2) + (comparison_event_count / comparison_n^2)) / pooled_sd
# Calculate the confidence interval for Cohen's d
cohens_d_ci_lower <- cohens_d - (1.96 * standard_error_d)
cohens_d_ci_upper <- cohens_d + (1.96 * standard_error_d)
# Calculate the Z-statistic
z_stat <- (ratio_rate_ratio - 1) / se_ratio_rate_ratio
# Calculate the p-value using the standard normal distribution
p_value <- 2 * (1 - pnorm(abs(z_stat)))
# Check if Hedges' g is requested
if (esc_type == 'g') {
# Calculate Hedges' g
hedges_g <- cohens_d * (1 - (3 / (4 * (treatment_n + comparison_n) - 9)))
return(list(effect_size = hedges_g, conf_interval = c(hedges_g - (1.96 * standard_error_d), hedges_g + (1.96 * standard_error_d)), p_value = p_value))
} else {
return(list(effect_size = cohens_d, conf_interval = c(cohens_d_ci_lower, cohens_d_ci_upper), p_value = p_value))
}
}
# testing sandpit
diff <- 0.67
diff_lower <- 0.22
diff_upper <- 2.07
treatment_n <- 87
comparison_n <- 69
population_baseline_rate <- 0.2
esc_mean_difference(
mean_difference = diff,
mean_diff_ci_lower = diff_lower,
mean_diff_ci_upper = diff_upper,
treatment_n = treatment_n,
comparison_n = comparison_n,
esc_type = "g")
esc_ratio_rate_ratio(
ratio_rate_ratio = diff,
ratio_rate_ratio_ci_lower = diff_lower,
ratio_rate_ratio_ci_upper = diff_upper,
population_baseline_rate = population_baseline_rate,
treatment_n = treatment_n,
comparison_n = comparison_n,
esc_type = "g")
esc_ratio_odds_ratio(
ratio_odds_ratio = diff,
ratio_odds_ratio_ci_lower = diff_lower,
ratio_odds_ratio_ci_upper = diff_upper,
population_baseline_rate = population_baseline_rate,
treatment_n = treatment_n,
comparison_n = comparison_n,
esc_type = "g")The code below details the complete process for effect-size transformation.
Code
# load custom es transformation functions
source("./review/code/es_transformation/custom_es_transformation_functions.R")
# read raw data
data_extraction_sheet_location <- "./review/inputs/yh_review_data_extraction_sheet.xlsx"
data_extraction_sheet <- read_excel(
path = data_extraction_sheet_location,
sheet = "outcome_data",
col_names = TRUE
)
# clean data
clean_data_extraction_sheet <- data_extraction_sheet |>
# remove instructions
filter(!row_number() == 1) |>
# remove superfluous columns
select(
-primary_extractor,
-secondary_extractor,
-meta,
-required_information,
-comment,
-additional_extraction_items
) |>
# specify variable types
mutate(
across(
c(
outcome_timing,
grp1n,
grp2n,
prop1event,
prop2event,
grp1m,
grp1sd,
grp1se,
grp2m,
grp2sd,
grp2se,
pre1mean,
pre1sd,
post1mean,
post1sd,
pre2mean,
pre2sd,
post2mean,
post2sd,
gain1mean,
gain1se,
gain2mean,
gain2se,
or,
f,
se,
totaln,
b,
beta,
sdy,
chisq,
t,
t_pvalue,
diff,
diff_lower,
diff_upper,
diff_se,
population_baseline_rate
),
as.numeric
)
) |>
# exclude baseline measurements
filter(
!outcome_timing == 0
)
# group data by effect size type
mean_gain <- clean_data_extraction_sheet |>
filter(
esc_type == "Mean Gain"
) |>
mutate(
id = row_number()
)
mean_sd <- clean_data_extraction_sheet |>
filter(
esc_type == "Mean SD"
) |>
mutate(
id = row_number()
)
binary_proportions <- clean_data_extraction_sheet |>
filter(
esc_type == "Binary proportions"
) |>
mutate(
id = row_number()
)
mean_difference <- clean_data_extraction_sheet |>
filter(
esc_type == "Mean Difference"
) |>
mutate(
id = row_number()
)
ratio_rate_ratio <- clean_data_extraction_sheet |>
filter(
esc_type == "Ratio of Rate Ratios"
) |>
mutate(
id = row_number()
)
ratio_odds_ratio <- clean_data_extraction_sheet |>
filter(
esc_type == "Ratio of Odds Ratios"
) |>
mutate(
id = row_number()
)
# transform results reported as mean gains to hedges' g using esc package
mean_gain_hedges_g <- mean_gain |>
# run es function
effect_sizes(
study = study_id,
fun = "esc_mean_gain",
grp1n = grp1n,
grp2n = grp2n,
pre1mean = pre1mean,
pre1sd = pre1sd,
post1mean = post1mean,
post1sd = post1sd,
pre2mean = pre2mean,
pre2sd = pre2sd,
post2mean = post2mean,
post2sd = post2sd,
es.type = "g") |>
# rename var
rename(
study_id = study,
es_ci_low = ci.lo,
es_ci_high = ci.hi,
sample_size = sample.size,
) |>
# add row number as id
mutate(
id = row_number()
) |>
# merge back with raw data
left_join(
mean_gain,
by = c(
"id",
"study_id")
) |>
# select vars for reporting
select(
study_id,
outcome_domain,
outcome_construct,
outcome_measure,
outcome_timing,
estimand,
es,
es_ci_low,
es_ci_high,
sample_size,
favourable_direction
)
# transform results reported as mean and standard deviation gains to hedges' g using esc package
mean_sd_hedges_g <- mean_sd |>
# run es function
effect_sizes(
study = study_id,
fun = "esc_mean_sd",
grp1m = grp1m,
grp1sd = grp1sd,
grp1n = grp1n,
grp2m = grp2m,
grp2sd = grp2sd,
grp2n = grp2n,
es.type = "g") |>
# rename var
rename(
study_id = study,
es_ci_low = ci.lo,
es_ci_high = ci.hi,
sample_size = sample.size,
) |>
# add row number as id
mutate(
id = row_number()
) |>
# merge back with raw data
left_join(
mean_sd,
by = c(
"id",
"study_id")
) |>
# select vars for reporting
select(
study_id,
outcome_domain,
outcome_construct,
outcome_measure,
outcome_timing,
estimand,
es,
es_ci_low,
es_ci_high,
sample_size,
favourable_direction
)
# transform results reported as binary proportions to hedges' g using esc package
binary_proportions_hedges_g <- binary_proportions |>
# run es function
effect_sizes(
study = study_id,
fun = "esc_bin_prop",
prop1event = prop1event,
grp1n = grp1n,
prop2event = prop2event,
grp2n = grp2n,
es.type = "g") |>
# rename var
rename(
study_id = study,
es_ci_low = ci.lo,
es_ci_high = ci.hi,
sample_size = sample.size,
) |>
# add row number as id
mutate(
id = row_number()
) |>
# merge back with raw data
left_join(
binary_proportions,
by = c(
"id",
"study_id")
) |>
# select vars for reporting
select(
study_id,
outcome_domain,
outcome_construct,
outcome_measure,
outcome_timing,
estimand,
es,
es_ci_low,
es_ci_high,
sample_size,
favourable_direction
)
# transform results reported as mean differences from Kozloff 2016 et al that reports CI's using custom function
mean_difference_hedges_g_kozloff_2016 <- mean_difference |>
# filter results from Gilmer 2016
filter(
study_id == "kozloff_2016"
) |>
# run custom function
rowwise() |>
mutate(result = list(
esc_mean_difference(
mean_difference = diff,
mean_diff_ci_lower = diff_lower,
mean_diff_ci_upper = diff_upper,
treatment_n = grp1n,
comparison_n = grp2n,
esc_type = "g"
)
))
# transform results reported as mean differences from Gilmer 2016 that reports SE's using custom function
mean_difference_hedges_g_gilmer_2016 <- mean_difference |>
# filter results from Gilmer 2016
filter(
study_id == "gilmer_2016"
) |>
# run custom function
rowwise() |>
mutate(result = list(
esc_mean_difference(
mean_difference = diff,
mean_diff_se = diff_se,
treatment_n = grp1n,
comparison_n = grp2n,
esc_type = "g"
)
))
# merge results back together and clean up
mean_difference_hedges_g <- bind_rows(
mean_difference_hedges_g_kozloff_2016,
mean_difference_hedges_g_gilmer_2016
) |>
ungroup() |>
unnest_wider(
result
) |>
# Convert the character column to a list of numeric vectors
mutate(
conf_interval = as.character(conf_interval),
conf_interval = str_extract_all(conf_interval, "-?[0-9.]+") |>
map(~ as.numeric(.))
) |>
# split the list-column into two new columns
unnest_wider(
conf_interval,
names_sep = "_") |>
rename(
es = effect_size,
es_ci_low = conf_interval_1,
es_ci_high = conf_interval_2
) |>
# add sample size var
mutate(
sample_size = grp1n + grp2n,
p_value = round(p_value, 4)
) |>
# select vars for reporting
select(
study_id,
outcome_domain,
outcome_construct,
outcome_measure,
outcome_timing,
estimand,
es,
es_ci_low,
es_ci_high,
p_value,
sample_size,
favourable_direction
)
# transform results reported as ratios of rate ratios using custom function
ratio_rate_ratio_hedges_g <- ratio_rate_ratio |>
# run custom function
rowwise() |>
mutate(result = list(
esc_ratio_rate_ratio(
ratio_rate_ratio = diff,
ratio_rate_ratio_ci_lower = diff_lower,
ratio_rate_ratio_ci_upper = diff_upper,
population_baseline_rate = population_baseline_rate,
treatment_n = grp1n,
comparison_n = grp2n,
esc_type = "g"
)
)) |>
# clean up results
ungroup() |>
unnest_wider(
result
) |>
# Convert the character column to a list of numeric vectors
mutate(
conf_interval = as.character(conf_interval),
conf_interval = str_extract_all(conf_interval, "-?[0-9.]+") |>
map(~ as.numeric(.))
) |>
# split the list-column into two new columns
unnest_wider(
conf_interval,
names_sep = "_") |>
rename(
es = effect_size,
es_ci_low = conf_interval_1,
es_ci_high = conf_interval_2
) |>
# add sample size var
mutate(
sample_size = grp1n + grp2n,
p_value = round(p_value, 4)
) |>
# select vars for reporting
select(
study_id,
outcome_domain,
outcome_construct,
outcome_measure,
outcome_timing,
estimand,
es,
es_ci_low,
es_ci_high,
p_value,
sample_size,
favourable_direction
)
# transform results reported as ratios of odds ratios using custom function
ratio_odds_ratio_hedges_g <- ratio_odds_ratio |>
# run custom function
rowwise() |>
mutate(result = list(
esc_ratio_odds_ratio(
ratio_odds_ratio = diff,
ratio_odds_ratio_ci_lower = diff_lower,
ratio_odds_ratio_ci_upper = diff_upper,
population_baseline_rate = population_baseline_rate,
treatment_n = grp1n,
comparison_n = grp2n,
esc_type = "g"
)
)) |>
# clean up results
ungroup() |>
unnest_wider(
result
) |>
# Convert the character column to a list of numeric vectors
mutate(
conf_interval = as.character(conf_interval),
conf_interval = str_extract_all(conf_interval, "-?[0-9.]+") |>
map(~ as.numeric(.))
) |>
# split the list-column into two new columns
unnest_wider(
conf_interval,
names_sep = "_") |>
rename(
es = effect_size,
es_ci_low = conf_interval_1,
es_ci_high = conf_interval_2
) |>
# add sample size var
mutate(
sample_size = grp1n + grp2n,
p_value = round(p_value, 4)
) |>
# select vars for reporting
select(
study_id,
outcome_domain,
outcome_construct,
outcome_measure,
outcome_timing,
estimand,
es,
es_ci_low,
es_ci_high,
p_value,
sample_size,
favourable_direction
)
# clean up data for export
effect_sizes <- dplyr::bind_rows(
mean_gain_hedges_g,
mean_sd_hedges_g,
binary_proportions_hedges_g,
mean_difference_hedges_g,
ratio_rate_ratio_hedges_g,
ratio_odds_ratio_hedges_g
) |>
# round quant outcomes for reporting and combine in single column
mutate(
es = round(es, 2),
es_ci_low = round(es_ci_low, 2),
es_ci_high = round(es_ci_high, 2)
)
# add flags for discussion
effect_sizes_info <- effect_sizes |>
mutate(
# create variable that determines the precision of the estimate based on whether or not it crosses zero
estimate_precision = case_when(
es_ci_low < 0 & es_ci_high > 0 ~ "imprecise",
TRUE ~ "precise"
),
# create variable that determines the size of the effect based on Cohen's benchmarks
es_magnitude = case_when(
abs(es) < 0.2 ~ "very small",
abs(es) >= 0.2 & abs(es) < 0.5 ~ "small",
abs(es) >= 0.5 & abs(es) < 0.8 ~ "medium",
abs(es) > 0.8 ~ "large"
),
# create variable that shows if direction of effect favours the intervention or comparison
direction_of_effect = case_when(
es < 0 & favourable_direction == "Negative" ~ "Favours intervention",
es > 0 & favourable_direction == "Negative" ~ "Favours comparison",
es < 0 & favourable_direction == "Positive" ~ "Favours comparison",
es > 0 & favourable_direction == "Positive" ~ "Favours intervention"
)
) |>
# drop redundant vars
select(
-p_value
)
# export effect size data
saveRDS(
effect_sizes_info,
"./review/output/tables/effect_size_results.RDS"
)
write_csv(
effect_sizes_info,
"./review/output/tables/effect_size_results.csv"
)4 Supplementary Results
4.1 Studies excluded at full text review
Studies excluded at full-text review stage are detailed in Table S7, with a reason and rationale for their exclusion.
| Reference | Reason for exclusion | Rationale |
|---|---|---|
| Aparicio et al. (2019) | Wrong Study Design | Intervention does not include a housing component |
| Aubry et al. (2019) | Wrong Population | Mean age treatment group ~29; Mean age comparison group ~22 |
| Baer, Garrett, Beadnell, Wells, & Peterson (2007) | Wrong Intervention | Intervention does not include a housing component |
| Baggett et al. (2018) | Wrong Intervention | Intervention does not include a housing component |
| Balasuriya & Dixon (2021) | Wrong Study Design | Not a study |
| Bender et al. (2018) | Wrong Intervention | Intervention does not include a housing component |
| Brown, Rowe, Cunningham, & Ponce (2018) | Wrong Population | Study population mean age ~48.8 |
| Burt (2012) | Wrong Population | Only ~2% of treatment group and ~5% of comparison group aged 18-24 at baseline |
| Chum et al. (2020) | Wrong Population | Only 13.6% of treatment group and 11.3% of comparison group aged <25 at baseline |
| Ciaranello et al. (2006) | Wrong Population | Mean age treatment group ~41.6; Mean age comparison group ~41.3 |
| D’Amico, Houck, Tucker, Ewing, & Pedersen (2017) | Wrong Intervention | Intervention does not include a housing component |
| DiGuiseppi et al. (2021) | Wrong Intervention | Intervention does not include a housing component |
| Dizon-Ross (2020) | Wrong Population | Intervention provided to families, outcomes measured for children |
| Doré-Gauthier, Miron, Jutras-Aswad, Ouellet-Plamondon, & Abdel-Baki (2020) | Wrong Study Design | Counterfactual relied on historical comparison group, did not meet study design inclusion criteria |
| Drake, Yovetich, Bebout, Harris, & Mchugo (1997) | Wrong Population | Mean age treatment group ~36.2; Mean age comparison group ~34.4 |
| Dunt et al. (2022) | Wrong Population | Only ~33% of treatment group and ~31% of comparison group aged 18-30 at baseline |
| Ferguson, Xie, & Glynn (2012) | Wrong Intervention | Intervention does not explicitly include a housing component |
| Ferguson (2018) | Wrong Intervention | Intervention does not include a housing component |
| Glendening et al. (2020) | Wrong Population | Intervention designed for families in child welfare system |
| Grace & Gill (2014) | Wrong Intervention | Intervention does not include a housing component |
| Grace & Gill (2016) | Wrong Intervention | Intervention does not include a housing component |
| Guo, Slesnick, & Feng (2016) | Wrong Population | Secondary report of Slesnick & Erdem (2013) |
| Guo & Slesnick (2017) | Wrong Intervention | Intervention does not include a housing component |
| Hawk & Davis (2012) | Wrong Study Design | Study does not include a counterfactual, also adult population with mean age ~46. |
| Hyun, Chung, & Lee (2005) | Wrong Intervention | Intervention does not include a housing component |
| Kennedy et al. (2018) | Wrong Intervention | Intervention does not include a housing component |
| Kidd et al. (2020) | Wrong Intervention | Intervention does not include a housing component |
| Kidd (2021) | Wrong Intervention | Intervention does not include a housing component |
| Krabbenborg, Boersma, & Wolf (2013) | Wrong Intervention | Intervention does not include a housing component |
| Krabbenborg, Boersma, Beijersbergen, Goscha, & Wolf (2015) | Wrong Intervention | Intervention does not include a housing component |
| Krabbenborg et al. (2017) | Wrong Intervention | Intervention does not include a housing component |
| C. T. Lee et al. (2018) | Wrong Population | Only 12.9% of treatment group and 14.6% of comparison group aged 18-29 at baseline |
| Linnemayr et al. (2021) | Wrong Intervention | Intervention does not include a housing component |
| Lund et al. (2024) | Wrong Study Design | Implementation study, secondary report of Kidd et al. (2020) |
| MacDonald (2020) | Study protocol | Ongoing study that would likely meet our inclusion criteria |
| Mackelprang, Collins, & Clifasefi (2014) | Wrong Population | Study population mean age ~48.4 |
| McCay et al. (2011) | Wrong Intervention | Intervention does not include a housing component |
| McCay et al. (2015) | Wrong Intervention | Intervention does not include a housing component |
| Medalia, Saperstein, Huang, Lee, & Ronan (2017) | Wrong Intervention | Intervention does not include a housing component |
| Metraux, Marcus, & Culhane (2003) | Wrong Population | Only 25.7% of treatment group and 30.4% of comparison group aged 18-29 at baseline |
| Nolan (2006) | Wrong Study Design | Study does not include a counterfactual |
| O’Campo et al. (2017) | Wrong Population | Only ~10% of treatment group and ~9% of comparison group aged 18-24 at baseline |
| Osilla, Kennedy, Hunter, & Maksabedian (2016) | Wrong Study Design | Study does not include a counterfactual, Intervention does not include a housing component |
| Padgett, Gulcur, & Tsemberis (2006) | Wrong Population | Only ~19.1% of study population was aged 18-30 at baseline |
| Padgett, Stanhope, Henwood, & Stefancic (2011) | Wrong Population | Mean age treatment group ~44; Mean age comparison group ~40 |
| Parpouchi, Moniruzzaman, McCandless, Patterson, & Somers (2016) | Wrong Population | Only ~7% of study population was aged 18-24 at baseline |
| Parpouchi, Moniruzzaman, & Somers (2021) | Wrong Study Design | Study does not include a counterfactual, or test an intervention. |
| Patterson, Currie, Rezansoff, & Somers (2015) | Wrong Population | Adult population in Housing First Trial |
| Pochetti (2018) | Wrong Study Design | Study does not include a counterfactual, or test an intervention. |
| Poremski, Rabouin, & Latimer (2017) | Wrong Population | Mean age treatment group ~45.2; Mean age comparison group ~47.1 |
| Samuels, Fowler, Ault-Brutus, Tang, & Marcal (2015) | Wrong Population | Study population mean age ~32.5 |
| Schick, Wiginton, Crouch, Haider, & Isbell (2019) | Wrong Population | Only 5.3% of study population was aged 18-29 at baseline |
| Simard, Chouinard-Thivierge, & Tanguay (2023) | Wrong Study Design | Study does not include a counterfactual, or test an intervention. |
| Slesnick & Erdem (2012) | Wrong Study Design | Study does not include a counterfactual |
| Slesnick & Erdem (2013) | Wrong Population | Study population mean age ~26.3 |
| Slesnick et al. (2021) | Study protocol | Ongoing study that would likely meet our inclusion criteria |
| Smith, North, & Fox (1996) | Wrong Population | Study population mean age ~29.8 |
| Sullivan et al. (2023) | Wrong Population | Study population mean age ~34.5 |
| Thompson & Hasin (2017) | Wrong Intervention | Intervention does not included housing component |
| Tsemberis, Gulcur, & Nakae (2004) | Wrong Population | Only ~19% of study population was aged 18-30 at baseline |
| Tucker, D’Amico, Ewing, & Pedersen (2016) | Wrong Intervention | Intervention does not include a housing component |
| Tucker, D’Amico, Ewing, Miles, & Pedersen (2017) | Wrong Intervention | Intervention does not include a housing component |
| Tucker, D’Amico, Pedersen, Rodriguez, & Garvey (2020) | Wrong Intervention | Intervention does not include a housing component |
| Tucker, Kennedy, Osilla, & Golinelli (2021) | Wrong Intervention | Intervention does not include a housing component |
| Tucker et al. (2022) | Wrong Intervention | Intervention does not include a housing component |
| Upshur (1986) | Wrong Study Design | Study design does not include a counterfactual |
| Wolitski et al. (2010) | Wrong Population | Only 11.1% of treatment group and 9.6% of comparison group aged 18-29 at baseline |
4.2 Housing First components of included interventions
Details of Housing First components in included studies are detailed in Table S8.
| Components | Kozloff, Adair, et al. (2016) | Thulien et al. (2019) | Slesnick et al. (2023) | Raithel, Yates, Dworsky, Schretzman, & Welshimer (2015) | Gilmer (2016) |
|---|---|---|---|---|---|
| Housing-related components | |||||
| Secure access to housing | Sought to place youth in secure housing | Provision of rental subsidies is deemed as securing access to housing | Sought to place youth in housing of their choice and provided utility and rental assistance | Provision of housing for up to 2 years. During the program they assisted individuals to secure more stable housing | Sought to place youth in permanent housing |
| Support to maintain tenancy | Sought to support the maintenance of tenancy through development of life skills | Rent subsidies to help maintain tenancy | Youth were provided services to assist with maintaining tenancy as well as utility and rental assistance | Sought to support the maintenance of tenancy through development of life skills, money management skills, etc | — |
| Clients are re-housed | Sought to provide alternative housing for youth who were unable to maintain tenancy | Rent subsidies are portable and therefore was not limited to a specific location | Utility and rental assistance are portable and therefore they were not tied to a specific location | Sought to help youth secure stable housing | — |
| Conditionality of access to housing | Not mandated to achieve or maintain sobriety or accept psychiatric treatment | There were no conditions to receiving rent subsidies | There were no prerequisites for receiving housing support | — | There were no conditions to access housing |
| Relationship between housing and other supports | Housing and support services were not contingent on one another - mobile, off-site support services | Rent subsidies were not contingent on mentorship and vice versa | Housing placement and rental assistance were not contingent on support sessions and vice versa | — | — |
| Varied housing options available | Scattered-site housing was offered at all sites, one site offered transitional/congregate housing option also | — | Rent subsidies are portable and not tied to a specific location | — | — |
| Clients are matched to housing based on their preferences | Youth were able to choose from available scattered-site housing | Youth had control over where they wanted to live | Youth were able to choose from available apartment options | — | — |
| Support services components | |||||
| Support to re-engage in services | — | — | — | — | — |
| Young people are supported to reunify with their families | Assistance in helping youth establish family connections as desired by the individual | — | — | — | Sought to assist youth with individualised goals, of which can include reunification with family |
| Uses a recovery-focused approach | Sought to improve social and family connections, development of life-skills, etc | Sought to support recovery through enhancing and encouraging choice and control | Sought to support recovery via the building self-efficacy, problem-solving skills, etc | Sought to assist youth to develop an individualised “action plan” outlining educational/vocational, money management, housing, health/wellness, personal goals, life skills development, community building, etc | Focus of case management was rehabilitation and recovery |
| Support services encourage social and community integration | Sought to provide services that work toward recovery and reintegration into the community including educational, employment and other life skills training | Mentors sought to work towards recovery and reintegration into the community | Sought to provide strengths-based advocacy support and sought to connect youth with other needed support in the community | Sought to assist youth to develop life skills, build community networks, educational/vocational support services, etc | Sought to provide services that work toward recovery and reintegration into the community including educational, employment and other life skills training |
| Services are provided at suitable intensity | Service delivery varied in terms of intensity depending on need. Youth had weekly contact with mental health worker | — | — | — | — |
| Type of case management engagement | ACT and ICM was used | — | — | — | ICM was used. A multidisciplinary treatment team model was also used but did not specify ACT approach |
| Components applicable to both housing and support services | |||||
| Involvement of youth in consumer choice and self-determination | Sought to promote individual choice and control over decisions, housing and support services | Youth had high degree of control over the housing and support they received | Sought to encourage and facilitate youth choice in the housing they were placed | — | Sought to encourage and facilitate choice in the supports and services they received |
| Access to housing or services is ongoing, for as long as required | No limited duration, services ongoing | — | — | Access to transitional housing for up to 2 years, during this time staff assisted individuals to secure more stable housing. Sought to provide aftercare | — |
| Case managers have sufficient time to devote to clients | — | — | — | — | — |
4.3 Instruments used in included studies
Instruments used to measure outcomes in included studies are detailed in Table S9.
| Construct | Measure | Publicly available | Evidence of validation identified | Evidence of validation among population with complex needs identified | Evidence of validation for homeless youth population identified | Free to use | Included studies using instrument | Reference |
|---|---|---|---|---|---|---|---|---|
| Stable Housing | ||||||||
| Housing Stability | Residential Time-Line Follow-Back Inventory | No | Yes | Yes | No | Unclear | Kozloff, Adair, et al. (2016) | Tsemberis, McHugo, Williams, Hanrahan, & Stefancic (2007) |
| Housing Quality | Perceived Housing Quality Scale | Yes | Yes | No | No | Unclear | Thulien et al. (2019) | Caffaro, Galati, & Roccato (2016) |
| Health | ||||||||
| Mental Health | GAIN Short Screener (GAIN-SS) – Substance Use | Yes | Yes | Yes | No | Yes | Kozloff, Adair, et al. (2016) | Dennis, Chan, & Funk (2006) |
| Quality of Life | EuroQoL 5 Dimensions | Yes | Yes | No | No | Yes | Kozloff, Adair, et al. (2016) | Lamers, Bouwmans, van Straten, Donker, & Hakkaart (2006) |
| Quality of Life | Lehman Quality of Life Interview 20 (QOLI-20) | Yes | Yes | Yes | No | Unclear | Kozloff, Adair, et al. (2016) | Lehman (1996) |
| Recovery | Recovery Assessment Scale | Yes | Yes | Yes | No | Yes | Kozloff, Adair, et al. (2016) | Corrigan, Giffort, Rashid, Leary, & Okeke (1999) |
| Self-rated Physical Health | Short Form 12 – Physical Component | Yes | Yes | No | No | No | Kozloff, Adair, et al. (2016) | Ware, Kosinski, & Keller (1996) |
| Self-rated Mental Health | Short Form 12 – Mental Component | Yes | Yes | No | No | No | Kozloff, Adair, et al. (2016) | Ware et al. (1996) |
| Mental Health Symptoms | Colorado Symptom Index | Yes | Yes | Yes | No | Yes | Kozloff, Adair, et al. (2016); Thulien et al. (2019) | Conrad et al. (2001) |
| Health and Criminal Justice System Involvement | Health, Social and Justice Service Use Inventory | No | No | No | No | Unclear | Kozloff, Adair, et al. (2016) | Goering et al. (2011) |
| Substance Use | Form-90 | Yes | Yes | No | No | Yes | Slesnick et al. (2023) | Miller (1996) |
| Social and emotional wellbeing | ||||||||
| Global Self-Worth | Rosenberg Self-Esteem Scale | Yes | Yes | No | No | Unclear | Thulien et al. (2019) | Rosenberg (1965) |
| Belongingness | Social Connectedness Scale | Yes | Yes | No | No | Unclear | Thulien et al. (2019) | R. M. Lee & Robbins (1995) |
| Motivation and expectations about future | Beck Hopelessness Scale | Yes | Yes | Yes | No | No | Thulien et al. (2019) | Beck, Weissman, Lester, & Trexler (1974) |
| Degree to which self-concept is defined by homelessness | Modified Engulfment Scale | Yes | Yes | Yes | No | Unclear | Thulien et al. (2019) | McCay & Seeman (1998) |
| Self-efficacy | Mastery Scale | Yes | Yes | No | No | Yes | Slesnick et al. (2023) | Pearlin & Schooler (1978) |
| Employment and earnings | ||||||||
| Employment | Vocational Time-Line Follow-Back Inventory | No | No | No | No | Unclear | Kozloff, Adair, et al. (2016) | Goering et al. (2011) |
| Legal and justice | ||||||||
| Victimisation | Canadian General Social Survey | Yes | No | No | No | Yes | Kozloff, Adair, et al. (2016) | Statistics Canada (2019) |
| Living skills and independence | ||||||||
| Community Integration | Community Integration Scale | Unclear | No | No | No | Unclear | Kozloff, Adair, et al. (2016); Thulien et al. (2019) | Stergiopoulos et al. (2014) |
| Community Functioning | Multnomah Community Ability Scale | Yes | Yes | Yes | No | No | Kozloff, Adair, et al. (2016) | Barker, Barron, McFarland, & Bigelow (1994) |
4.4 Risk of bias in included studies
In a similar fashion to the data extraction templates, we used a shared Google Sheet to assess the risk of bias in each reported result.
For randomised controlled trials, we used the Cochrane Risk of Bias 2 tool to assess the risk of bias in all reported results Sterne et al. (2019). The template and results are available in Table S10.
×
For non-randomised studies of interventions, we used the ROBINS-I tool Sterne et al. (2016). The template and results are available in Table S11.
×
4.5 Barriers and Facilitators
Barriers and facilitators grouped by EPIS domain are detailed in Table S12.
| EPIS domain | Inner context | Outer context | Inner context | Outer context | Inner context | Outer context |
|---|---|---|---|---|---|---|
| Exploration | ||||||
| Barriers | — | Legal status and informed consent issues complicate implementing Housing First with youth under 18 (Kozloff, Stergiopoulos, et al., 2016). | — | — | — | — |
| Facilitators | — | — | — | — | — | — |
| Preparation | ||||||
| Barriers | — | — | — | — |
Designing training that was relevant across sites proved challenging due to the heterogenous skills and capabilities of participants and the varying local, social and systemic factors unique to each location (Nelson et al., 2012). Poor communication about the limited scope for local adaptation affected staff buy-in to the intervention (Nelson et al., 2013). Challenges were faced in hiring culturally competent staff to accommodate the needs of Aboriginal participants in the program (Nelson et al., 2012; Nelson et al., 2014) . Tight timelines impacted ability to engage staff with lived experience in the development of the intervention, and led to issues such as mistrust and poor communication between sites and head office (Nelson et al., 2013). |
Discomfort arose for some stakeholders regarding the project’s relationship to existing local services, particularly the head-to-head comparisons between the externally developed interventions, Housing First and local approaches (Nelson et al., 2013). Some sites faced challenges in completing proposals due to a lack of a local housing and homelessness services infrastructure, leadership, service capacity, or human resources (Nelson et al., 2013). Sites who were otherwise well equipped felt they had insufficient time to consult with the existing service providers during the proposal phase (Nelson et al., 2013). |
| Facilitators | — | — | — | The strength of existing services in the community was a key factor in the project’s success (Nelson et al., 2012; Nelson et al., 2014). |
Effective leaders and strong staff teams with the right combination of technical and interpersonal skills, including decision-making abilities, clear direction, fostering a shared learning, and extensive experience working with various consumer populations, were crucial for success (Macnaughton et al., 2015; Nelson et al., 2012; Nelson et al., 2014). Timelines and expectation promoted teamwork and productivity the during planning stages (Nelson et al., 2013). Site-specific programs addressing the needs of CALD and Indigenous communities facilitated engagement and success (Nelson et al., 2012; Nelson et al., 2014). Flexibility for sites to develop unique interventions or third arms was valuable in adjusting to individual needs (Nelson et al., 2013). At the national level, Site Coordinators with a strong understanding of the project-wide logic model and the ability to move their site towards implementation of that model were critical to success (Nelson et al., 2013, 2012). A site operations team, local advisory committee and various work groups facilitated good organisational structure and governance (Nelson et al., 2012). |
Stakeholders deemed the program a valuable opportunity with potential for transformative change, and the initial excitement and enthusiasm were beneficial for moving the project forward (Nelson et al., 2013). The community valued collaboration and collective effort which aligned with the project (Nelson et al., 2012). Recruiting consumers in a manner that allowed them to maintain existing service relationships with established community service providers was important (Nelson et al., 2012) . Collaboration between project programs and the major shelters facilitated recruitment of participants (Nelson et al., 2012). |
| Implementation | ||||||
| Barriers |
The lack of established program protocols prior to implementation made it difficult to manage the intake of new consumers while simultaneously finding housing for participants, supporting existing consumers and helping them maintain housing (Nelson et al., 2012; Nelson et al., 2014). Bottlenecks in the referral process from screening to referral slowed down access to housing (Macnaughton, Goering, & Nelson, 2012). ACT and ICM teams needed closer relationships with housing team to better identify and engage yet-to-be housed individuals (Macnaughton et al., 2012). The lack of a clear governance model sometimes led to delays in obtaining housing for consumers and a lack of cohesion and contact among clinical housing teams (Nelson et al., 2014). There was a lack of after-hours staff support to assist with crisis situations and issues with landlords (Nelson et al., 2012; Nelson et al., 2014). |
Housing availability was limited due to a lack of affordability and the reluctance of some housing agencies to risk established relationships with landlords by housing higher-risk Housing First clients (Macnaughton et al., 2012, 2015; Nelson et al., 2012; Nelson et al., 2014) In small communities, privacy was a concern when certain information about the project or consumers was shared. For example, landlords would sometimes share information about difficult consumers within their network, making it more challenging to secure housing for those individuals (Nelson et al., 2012). Some landlords exhibited stigma and racism, particularly towards Aboriginal people with mental health issues or substance use disorders, which led to them avoiding housing these individuals (Macnaughton et al., 2012; Nelson et al., 2012; Nelson et al., 2014) Program staff had to work hard to sustain/repair relationships with landlords, as they were unable to fulfill all the guarantees that initially attracted them to the program, such as consistent visits with the participants or prompt repairs to damaged property (Nelson et al., 2012; Nelson et al., 2014). |
There was insufficient planning and discussion on how to assist people not doing well in the program due to the diversity of consumer needs and functioning (Nelson et al., 2014). Varying support needs meant that vocational specialists’ work was often dominated crisis management for a significant minority of unstable clients, rather than focusing on proactive, long-term goal planning with participants ready to move further in their recovery (Macnaughton et al., 2015). ACT and ICM teams struggled to reconcile a recovery approach with heavy workload demands (Nelson et al., 2012). The high consumers to staff ratio resulted in reduced one-to-one attention for participants, the impact of travel time between staff offices and participants’ homes was a particular burden for service teams (Nelson et al., 2012; Nelson et al., 2014). Busy team members found it difficult to find time to participate in training and improve their practice approaches without further increasing workload and stress (Nelson et al., 2012). Service providers’ irregular work hours and lack of opportunities for debriefing concerns were challenging (Nelson et al., 2012). |
Lack of public transport was a barrier for consumers to get to appointments to receive health and social services, visit food bank, maintain relationships with family and friends (Nelson et al., 2012; Nelson et al., 2014). Better collaboration and communication with local health care providers in the community were needed to address challenged in partnerships with community agencies (Nelson et al., 2012). Important service programs, such as addiction treatment, vocational and educational support, were lacking in the community (Nelson et al., 2014). COVID pandemic restrictions impacted the ability to have face to face meetings with mentors (Thulien et al., 2022, 2019). |
The project design made transferring participants between service teams and housing types very difficult, even when a different model would better suit clients’ needs (Nelson et al., 2012). There was a lack of training and difficulty meeting the cultural and linguistic requirements of diverse populations (Nelson et al., 2012; Nelson et al., 2014). Limited opportunities for informal communication between teams, housing and support services being at different sites, and tensions between support teams due to competition for housing and perceived comparisons of project outcomes hindered collaboration and cohesion (Nelson et al., 2012; Nelson et al., 2014). High staff turnover was experienced, possibly driven by weak leadership and poor team dynamics (Macnaughton et al., 2015). Peer specialists were not seen as equal team members and were not meaningfully integrated, with a general lack of communication about their role. One site noted the importance of supporting peers to adopt a full-service role through proper training and workplace support, especially given their susceptibility to re-traumatisation during work (Macnaughton et al., 2015; Nelson et al., 2012) |
— |
| Facilitators | — |
Rent subsidies, housing team procurement strategies and community relationships were crucial in both expanding potential housing supply and achieving quicker access to housing (Macnaughton et al., 2012). Collaborating with landlords, landlord associations and property management companies was beneficial for implementation. These partnerships: a) made landlords more likely to consult with service team members to resolve issues rather than notifying police or initiating eviction proceedings, and b) led to successful, high quality housing procurement (Macnaughton et al., 2015; Nelson et al., 2012; Nelson et al., 2014). Providing consistent support to landlords, including prompt responses to problematic tenancy issues, paying for any damage, and guaranteeing rent payment, were strong incentives for landlord participation (Macnaughton et al., 2015). |
Partnerships with consumers enabled peer-driven initiatives like social gatherings, peer support programs for individuals with substance use issues, participant-produced newsletters, and participant-led focus groups on eviction prevention. Consumers could help fellow consumers and offer staff insider information about community resources (Nelson et al., 2012; Nelson et al., 2014). | Across sites, participants highlighted the importance of having a diverse group of partners in providing a comprehensive continuum of care to participants (Nelson et al., 2012). |
Implementation support from project’s national implementation team was beneficial (Macnaughton et al., 2015). Staff members with lived experience were able to build trust, relate and empathise with participants (Nelson et al., 2012). Organisational structure and governance were critical in defining roles, responsibilities, facilitating collaboration, partnership building, effective communication and conflict resolution (Nelson et al., 2012; Nelson et al., 2014). A good organisational culture facilitated team dynamics – where the values were consistent with Housing First principles and common attitude to do whatever it takes to help participants achieve their goals, teams worked better together and had higher fidelity (Macnaughton et al., 2015). Team diversity, fostering cross-team learning and sharing, breaking down hierarchies, along with team cohesion through structured meetings, formal training, all-team events and shared office space, nurtured teamwork and supported service delivery that leveraged expertise across areas (Nelson et al., 2012; Nelson et al., 2014). Staff stability allowed service providers to integrate new approaches, practices and tools effectively (Nelson et al., 2012). Incorporating additional expertise into the team, such as a psychiatrist to support clients with complex needs, or a home economist to help participants develop tenancy skills, was beneficial (Macnaughton et al., 2015). |
Partnerships with government agencies and departments enhanced the project’s ability to secure consumer access to housing units, mental health and homelessness services, and government income supports (Nelson et al., 2012; Nelson et al., 2014). Participants stated that local partners, including business owners, police, hospital staff, community mental health teams, shelters, meal providers, churches, the United Way, and mobile crisis teams, were crucial in contributing expertise and experience. This increased the project’s ability to integrate into the community’s network of supports and services (Nelson et al., 2012; Nelson et al., 2014). The small size of the community facilitated easier contact and collaboration with existing community service providers, enabling efficient information transfer among professionals, the public and potential participants (Nelson et al., 2012). Training and technical assistance provided by the Mental Health Commission of Canada were responsive, fair, generous and supportive. Positive and productive communication, funding, practical guidance and training opportunities were beneficial (Nelson et al., 2012; Nelson et al., 2014). |
| Sustainment | ||||||
| Barriers | Short project funding horizon reduced landlords’ willingness to sign year-long leases as the project neared its end (Macnaughton et al., 2015) | — | — | — |
Program costs were identified as a barrier to program sustainability (Nelson et al., 2012) Staff expressed significant concerns about the lack of information regarding project sustainability. They felt a critical need for knowledge dissemination about the steps being taken to ensure that consumers were not abandoned once the research project ended (Nelson et al., 2013, 2012). |
— |
| Facilitators | — | — | — | Forming and maintaining good community partnerships with various stakeholders and existing agencies with various social service systems, (e.g., justice system personnel, employment/income assistance agencies, and wider mental health and substance use systems). These collaborations led host agencies to take “ownership” over the project and this was deemed as potentially important for sustaining the project long-term (Macnaughton et al., 2015) | Staff demonstrated a high-level of commitment to the project and its values, despite the challenging nature of the work (Macnaughton et al., 2015) | — |
References
Citation
@online{taylor2026,
author = {Taylor, David and Vecchio, Stephanie and Baidawi, Susan and
Shlonsky, Aron},
title = {Housing and {Support} {Interventions} for {Homeless} {Youth}
in {High-Income} {Countries:} A {Systematic} {Review} —
Supplementary Material},
date = {2026-01-15},
url = {https://davetayl-r.github.io/youth-homelessness-review},
doi = {10.5281/zenodo.14503767},
langid = {en}
}