Risk Assessment Validation
Table 9 presents the range of risk values (-5 to +5) for the pre-trial assessment, the quantity and proportion of the sample that was assigned to each value, and the failure rates of the outcome measure. When the table is examined in terms of the distribution of assessment scores, the largest group of the sample (n=56) scored a -1 on the assessment, with the second largest group scoring either a zero (n=46) or a -2 (n=43). The overall distribution of scores is bell-shaped, with most individuals falling into the center of the distribution, with fewer cases at either extreme end. A higher score corresponds to a greater likelihood of failing to appear or supervision failure. For example, 33.3 percent of the individuals who received a score of 1 were arrested or failed to appear. Comparatively, none of the individuals who received a score of -3 received a FTA, or were arrested for a new crime during their pre-trial period.
TABLE 9 Distribution of Failure Rates Across Total Assessment Score
A series of cross-tabulation analyses were used to create cut-off values in the distribution of risk scores of the sample. From these cross-tabulations, an optimal cut-off score was devised with three categories. Individuals who scored a negative number on the risk assessment were classified as low risk, or individuals unlikely to either receive an FTA or to commit a new offense. Individuals who scored a zero on the risk assessment were classified as medium risk, and were substantively more likely than low-risk individuals to re-offend in either outcome measure.
High-risk individuals scored a positive number on the assessment and were the most likely classification to recidivate.
A majority of the defendants in the sample were classified as low risk (n=114), followed by medium risk (n=48), with fewer defendants classified as high risk (n=34). Table 10 reports the number of defendants assigned to each risk level and failure rates attributed to those risk levels.
TABLE 10 Distribution of Construction Sample & Outcome Across Risk Category
TABLE 11 Bivariate Correlations for Total Score & Recidivism
The practical utility of a risk assessment lies in its ability to accurately distinguish between risk groups of defendants (low, medium, and high) for the purposes of case planning, resource allocation, and supervision. As demonstrated by the analysis, there is considerable difference in the failure rates between risk categories. Only 1.8 percent of those classified as low-risk defendants were arrested, compared to 6.3 percent of medium-risk defendants, and 32.4 percent for high-risk groups. These failure rates are illustrated in Table 10.
Validation Analysis
With the assessment constructed and risk level cutoffs created, the next analyses have the purpose of testing the assessment on the sample. To test the linear relationship between the pre-trial assessment score and outcomes, a bivariate correlation analysis and a Receiver Operating Characteristic (ROC) analysis were conducted. Under the bivariate analysis, the total score was significantly correlated with the recidivism measure (either a new arrest for criminal conduct or a FTA) at a score of.377 (see Table 11.)
The Receiver Operating Characteristic (ROC) curve plots the true positives and the false positives at each level of the risk scale and the Area Under the Curve (AUC) statistic can be calculated and used as a measure of predictive accuracy. A ROC curve is a graphical representation of the trade off between the false negative and false positive rates, i.e. it is the trade off between sensitivity (Sn) and specificity (Sp). The ROC analysis, therefore, plots sensitivity against specificity. You can quantify the accuracy of a ‘test’ using an ROC curve by measuring the area under the 8 pt curve, (commonly referred to as the Area Under the Curve or AUC). The area under the curve is a useful value because if the area under the curve is 1.0, this means that you have an ideal test (though in the real world, this is very unlikely to happen), because it achieves both 100% sensitivity and 100% specificity. If the area under the curve is 0.5, then the test has 50% sensitivity and 50% specificity, which would be no better than flipping a coin. In general terms, an AUC between 0.90 and 1 is considered excellent, between 0.80 and 0.90 good, between 0.70 to 0.80 fair, between 0.60 to 0.70 poor and between0.50 and 0.60 a fail (i.e. a reveals a test that fails to identify whatever it is supposed to identify).
Validity of Tool
Under the Area Under the Curve statistic, the model scored a value of .823, which would suggest the tool does a good job of predicting recidivism during the pre-trial period.
Limitations
The current study has some methodological limitations that should be addressed. The first is the relatively small size of the sample. Due to the size of the sample and the voluntary nature of the research, arguments could be made that this sample is likely to represent a
subgroup of pre-trial defendants who are lower risk or who are more apt to comply with supervision requirements. If this is the case, there is a chance that the sample characteristics could have limited the robustness of the predictive ability of items and the assessment in its entirety. This problem is inherent in the initial study of any assessment and only additional studies and samples will confirm either argument.
Another limitation is that screeners relied on self reporting for some variables. While criminal history, employment, and residency could be verified, substance use could not. Some defendants may be hesitant to provide information about their alcohol or drug use, and missing data on many of these variables led to their exclusion from the analysis.
