Statistical Associates Publishers
MANOVA/MANCOVA: 10 Worst Pitfalls and Mistakes
- Inappropriate model specification.
As in almost all statistical procedures, MANOVA and MANCOVA coefficients may change markedly if important predictor variables are added to the model. Proper model specification is assumed, including exclusion of variables which are correlated with variables in the model but which are spurious from a causal point of view.
- Reporting only significance and not effect size also.
As in other statistical procedures, a result may be weak to the point of not mattering even though it is statistically significant. Partial eta-square is the main effect size measure in many of the tables associated with multiple analysis of variance. Other effect size measures include R-squared and omega-squared.
- Performing post-hoc and univariate tests when multivariate tests fail to show the model to be significant. .
It is inappropriate to run post-hoc, multiple comparison, and custom hypotheses univariate tests if the mode is not significant. These are only run as a follow-up to a significant multivariate test result.
- Not adjusting for multiple univariate tests..
If multiple independent univariate tests are run, the reported probability level will be inaccurate, inflating the Type I error rate (false positives). It is necessary to penalize the reported probability when conducting such tests. There is a wide variety of adjustments and degrees of penalty from which the researcher must choose.
- Using ordinal dependent variables.
This is the same violation as is common in multiple linear regression. MANOVA and MANCOVA analyse variance and covariance, which require continuous interval dependent variables. Violating this assumption attenuates effect sizes and inflates the Type II error rate (false negatives) to a degree dependent on how much the ordinal scale differs from an interval one (often unknown).
- Failing to test for homogeneity of error variances .
The assumption of homogeneity of error variances (also known as "homoscedasticity" or "homoskedasticity") is a critical one in MANOVA/MANCOVA. Within each group formed by the categorical independent variables, the error variance of each dependent variable should be similar, as tested by Levene's test. Violation of the homogeneity of error variances assumption is a form of measurement error and as such is likely to attenuate effect size measures and inflate the Type II error rate false negatives).
- Confusing the meaning of different MANOVA/MANCOVA significance tests .
MANOVA and MANCOVA report three major classes of significance findings: multivariate tests of the model, significance of parameter estimates, and significance of between-subject effects. Each has a different meaning, which is by a predictor variable might be significant as a between-subjects effect but not as a parameter estimate. Accounting for differences of significance tests is critical to proper statistical inference.
- Not examining outlier diagnostic measures and plots .
As in regression, outliers may have a strong effect on results. Analysis of residuals and outliers is an integral part of multivariate analysis of variance and covariance.
- Not testing for homogeneity of regressions. .
While testing for homogeneity of variance is widely observed among researchers, testing for homogeneity of regressions in MANCOVA is important also. MANCOVA assumes that the covariate coefficients (the slopes of the regression lines) are the same for each group formed by the categorical predictor variables (the factors). The more this assumption is violated, the more conservative MANCOVA becomes and the more likely it is to make Type II errors, which are false negatives.
- Not meeting the assumptions of multiple analysis of variance and covariance, including linearity, multivariate normality, sphericity, cell count adequacy, amd low mesurement error..
Our book, listed below, enumerates many assumptions of MANOVA and MANCOVA, clearly listed in the "Assumptions" section.
Want to learn more about all this and much more?
"GLM Multivariate, MANOVA, & Canonical Correlation" on Amazon, Kindle format. After purchase, send your receipt to sa.publishers@gmail.com to get also a free pdf version if you wish.
"GLM Multivariate, MANOVA, & Canonical Correlation" Preview, PDF format
"GLM Multivariate, MANOVA, & Canonical Correlation" Information and table of contents
"Statistical Associates Library" of 50 Statistics E-books on Amazon, no-password .PDF format