“Measuring and Reporting Model Fits for Theory and Practice: The A3 Method”
Introduced by Andy Lyons
When models are used to guide policy or scientific decisions, it is important that the accuracy of the underlying models and the strength of their support for a decision be understood. A wide range of metrics have been developed to assess model results for either prediction or inference. Such metrics include the classic coefficient of determination and p value, risk functions such as MSE, and newer approaches such as information theoretic techniques like AIC or BIC. Often these metrics are difficult to understand and they may frequently be misinterpreted and misused.
This talk presents the new A3 method which is designed to be accessible, accurate and adaptable. In addition to using familiar concepts, the method attempts to deemphasize the oft-misinterpreted p value in favor of a new metric: the added R^2. This metric indicates how much the model improves when a given variable is added to the model. The A3 method uses robust, resampling-based techniques for the calculation of the quality of a model. These techniques are adaptable to different modeling algorithms allowing A3 to be used to directly compare the results from different modeling approaches.
Stepping away from the jargon and technical details, the A3 method makes it easier to understand the quality of a model fit. In practice, this could lead to better models being used and more accurate inferences and conclusions being made based on these models.