Multiple Principled Solutions

stats
workflow
social justice
science
Author

Andy Grogan-Kaylor

Published

May 11, 2024

One of my most recent ideas about statistical workflows is that there are certainly wrong decisions that one can make with data.

For example, I would not want to write the paper that says that smoking prevents lung cancer, nor would I want to write a paper saying spanking is good for children.

That being said, I think there are often multiple principled ways forward.

Often the key is not so much to make the 100% correct decision, but to make one of several possible principled decisions.

Then after making a principled decision, one is transparent and thorough about describing the decision that one made.

For example, to analyze clustered data, I could employ multilevel models, fixed effects regression, clustered standard errors, or generalized estimating equations. Each of these methods of analysis would carry with it different assumptions.

To further illustrate this point, within the domain of multilevel models, I would have further choices: I could estimate only a random intercept; estimate one or more random slopes; or estimate all possible random slopes. These random effects could be correlated or uncorrelated. I could estimate only main effects, or could estimate interactions of several variables. Each of these would be a different, yet principled, approach to analyzing the data.

As another example, to more robustly estimate causal effects, I could employ fixed effects, propensity scores, or cross-lagged regression models. Again, each of these methods of analysis would be a different, yet principled, approach to analyzing the data.