The permutation testing approach: a review


  • Fortunato Pesarin Università degli Studi di Padova
  • Luigi Salmaso Università degli Studi di Padova



In recent years permutation testing methods have increased both in number of applications and in solving complex multivariate problems. A large number of testing problems may also be usefully and effectively solved by traditional parametric or rank-based nonparametric methods, although in relatively mild conditions their permutation counterparts are generally asymptotically as good as the best ones. Permutation tests are essentially of an exact nonparametric nature in a conditional context, where conditioning is on the pooled observed data as a set of sufficient statistics in the null hypothesis. Instead, the reference null distribution of most parametric tests is only known asymptotically. Thus, for most sample sizes of practical interest, the possible lack of efficiency of permutation solutions may be compensated by the lack of approximation of parametric counterparts. There are many complex multivariate problems (quite common in biostatistics, clinical trials, engineering, the environment, epidemiology, experimental data, industrial statistics, pharmacology, psychology, social sciences, etc.) which are difficult to solve outside the conditional framework and outside the nonparametric combination (NPC) method for dependent permutation tests. In this paper we review this method along with a number of applications in different experimental and observational situations (e.g. multi-sided alternatives, zero-inflated data and testing for a stochastic ordering) and we present properties specific to this methodology, such as: for a given number of subjects, when the number of variables diverges and the noncentrality of the combined test diverges accordingly, then the power of combination-based permutation tests converges to one.


E.F ABD-ELFATTAH,. R.W. BUTLER, (2007). The weighted log-rank class of permutation tests: p-values and confidence intervals using saddlepoint methods. “Biometrika”, 94, 543-551.

A. AGRESTI, B. KLINGENBERG, (2005). Multivariate tests comparing binomial probabilities, with application to safety studies for drugs. “Journal of the Royal Statistical Society”, Series C, 54, 691-706.

R. ARBORETTI, S. BONNINI, F. PESARIN, L. SALMASO (2008). One-sided and two-sided nonparametric tests for heterogeneity comparisons. “Statistica”, LXVIII;57-69.

F. BARZI, G. CELANT, A. DI CASTELNUOVO, F. PESARIN, L. SALMASO (2001). Test di permutazione multidimensionali per misure ripetute: applicazioni alle curve di crescita tumorali in modelli animali. “Statistica”, LXI, 533-543.

D. BASSO, F. PESARIN, L. SALMASO, A. SOLARI, (2009). Permutation tests for stochastic ordering and ANOVA: theory and applications in R. Springer, New York.

D. BASSO, L. SALMASO, (2009). A permutation test for umbrella alternatives. Statistics and Computing, (DOI 10.1007/s11222-009-9145-8).

F. BERTOLUZZO, F. PESARIN, L. SALMASO, (2011). Multi-sided permutation tests: an approach to random effects. “Journal of Statistical Planning and Inference” (forthcoming).

A. BIRNBAUM, (1954). Combining independent tests of significance. “Journal of the American Statistical Association”, 49, 559-574.

A. BIRNBAUM, (1955). Characterizations of complete classes of tests of some multiparametric hypotheses, with applications to likelihood ratio tests. The Annals of Mathematical Statistics, 26, 21-36.

F.L. BOOKSTEIN, (1991). Morphometric Tools For Landmark Data: Geometry and Biology. Cambridge University Press, Cambridge.

D. BOSQ, (2005). Inférence et Prévision en Grande Dimensions. Economica, Paris.

C. BROMBIN, L. SALMASO, (2009). Multi-aspect permutation tests in shape analysis with small sample size. “Computational Statistics & Data Analysis”, 53, 3921-3931.

G. CELANT, F. PESARIN, (2000). Alcune osservazioni critiche riguardanti l.analisi Bayesiana condizionata. Statistica, LX, 25-37.

G. CELANT, F. PESARIN, (2001). Sulla de.nizione di analisi condizionata. Statistica, LXI, 185-194.

L. CORAIN, L.SALMASO (2003). An empirical study on new product development process by onparametric combination (NPC) testing methodology and post-strati.cation. “Statistica”, LXIII, 335-357.

D.D. COX, J.S. LEE, (2008). Pointwise testing with functional data using the Westfall-Young randomization method. “Biometrika”, 95, 621-634.

D.R. COX, D.V. HINKLEY, (1974). Theoretical Statistics. Chapman and Hall, London.

J.H. CHUNG, D.A.S. FRASER, (1958). Randomization tests for a multivariate two-sample problem. “Journal of the American Statistical Association”, 53, 729-735.

H.A. DAVID, (2008). The beginnings of randomization Tests. “The American Statistician”, 62, 70-72.

P.J. DIGGLE, K.Y. LIANG, S.L. ZEGER, (2002). Analysis of Longitudinal Data. Oxford University Press, Oxford.

A. DI CASTELNUOVO, D. MAZZARO, PESARIN F., SALMASO L., (2000). Test di permutazione multidimensionali in problemi di inferenza isotonica: un.applicazione alla genetica. “Statistica”, LX, 692-700.

R. DOWNER, (2002). A permutation alternative and other test procedures for spatially correlated data in one-way ANOVA. “Journal of Statistical Computation and Simulation”, 72, 747-757

S. DRAY, (2008). On the number of principal components: a test of dimensionality based on measurements of similarity between matrices. “Computational Statistics & Data Analysis”, 52, 2228-2237.

I.L. DRYDEN, K.V. MARDIA, (1998). Statistical Shape Analysis. John Wiley & Sons, London.

E.S. EDGINGTON, P. ONGHENA, (2007). Randomization Tests (4th ed.). Chapman and Hall/CRC, London.

F. FERRATY, P. VIEU (2006). NonParametric Functional Data Analysis: Theory and Practice. Springer, New York.

L. FINOS, L. SALMASO, (2006). Weighted methods controlling the multiplicity when the number of variables is much higher than the number of observations. “Journal of Nonparametric Statistics”, 18, 245-261.

L. FINOS, L. SALMASO, A. SOLARI, (2007). Conditional inference under simultaneous stochastic ordering constraints. “Journal of Statistical Planning and Inference”, 137, 2633-2641.

R.A. FISHER, (1936). “The coefficient of racial likeness” and the future of craniometry. “Journal of the Royal Anthropological Institute” of Great Britain and Ireland, 66, 57-63.

G.M. FITZMAURICE, S.R. LIPSITZ, J.G. IBRAHIM, (2007). A note on permutation tests for variance components in multilevel generalized linear mixed models. “Biometrics”, 63, 942-946.

J.L. FOLKS, (1984). Combinations of independent tests. In P.R. Krishnaiah and P.K. Sen (eds.), Handbook of Statistics, 4, 113-121, North-Holland, Amsterdam.

A. FRIMAN, C.F. WESTIN, (2005). Resampling fMRI time series. “NeuroImage”, 25, 859-867.

M.L. GOGGIN, (1986). The “Too Few Cases/Too Many Variables” Problem in Implementation Research. “The Western Political Quarterly”, 39, 328-347.

P. GOOD, (2005), Permutation, Parametric, and Bootstrap Tests of Hypotheses (3rd ed.), Springer-Verlag, New York.

C. HIROTSU, (1986). Cumulative chi-squared statistic or a tool for testing goodness of fit. “Biometrika”, 73, 165-173.

C. HIROTSU, (1998a). Max t test for analysing a dose-response relationship - an efficient algorithm for p value calculation. In L. Pronzato (ed.), Volume of Abstracts of MODA-5 “5th International Conference on Advances in Model Oriented Data Analysis and Experimental Design”, CIRM, Marseille.

C. HIROTSU, (1998b). Isotonic inference. In Encyclopedia of Biostatistics, 2107 - 2115, Wiley, New York.

W. HOEFFDING, (1952). The large-sample power of tests based on permutations of observations. Annals of Mathematical Statistics, 23, 169-192.

G.A. HOSSEIN-ZADEH, H. SOLTANIAN-ZADEH,B.A. ARDEKANI, (2003). Multiresolution fMRI activation detection using translation invariant wavelet transform and statistical analysis based on resampling. IEEE Transactions on Medical Imaging, 22, 302-314.

B. KLINGENBERG, A. SOLARI, L. SALMASO, F. PESARIN, (2009). Testing marginal homogeneity against stochastic order in multivariate ordinal data. “Biometrics”, 65, 452-462.

A. JANSSEN, (2005). Resampling student’s t-type statistics. Annals of the Institute of Statistical Mathematics, 57, 507-529.

H. JOE, (1997). Multivariate Models and Dependence Concepts. Chapman & Hall, London.

S. JUNG, H. BANG, S. YOUNG, (2005). Sample size calculation for multiple testing in microarray data analysis. “Biostatistics”, 6, 157-169.

P.A. LACHENBROOK, (1976). Analysis of data with clumping at zero. “Biometrical Journal”, 18, 351-356.

O. KEMPTHORNE, (1955). The randomization theory of experimental inference. “Journal of the American Statistical Association”, 50, 964-967.

E.L. LEHMANN, (2009). Parametric versus nonparametrics: two alternative methodologies. “Journal of Nonparametric Statistics”, 21, 397-405.

E.L. LEHMANN, J.P. ROMANO, (2005). Testing statistical hypotheses (3rd ed.). Springer, New York.

E.L. LEHMANN, H. SCHEFFÉ, (1950). Completeness similar regions, and unbiased estimation. Sankhyā, 10, 305-340.

E.L. LEHMANN, H. SCHEFFÉ, (1955). Completeness similar regions, and unbiased estimation - part II. Sankhyā, 15, 219-236.

I. LIPTAK, (1958). On the combination of independent tests. Magyar Tudomanyos Akademia Matematikai Kutato Intezenek Kozlomenyei, 3, 127-141.

J. LUDBROOK, H. DUDDLEY, (1998). Why permutation tests are superior to t anf F tests in biomedicar research. “American Statistician”, 52, 127-132.

H. MANSOURI, (1990). Rank tests for ordered alternatives in analysis of variance. “Journal of Statistical Planning and Inference”, 24, 107-117.

MAZZARO, D., PESARIN, F., SALMASO, L., (2001). A discussion on multi-way ANOVA using a permutation approach. “Statistica”, LXI, 15-26.

C.R. MEHTA, N.R. PATEL, (1980). A network algorithm for the exact treatment of the 2 K  contingency table. “Communications in Statistics”, Simulation and Computation, 9, 649-664.

C.R. MEHTA, N.R. PATEL, (1983). A network algorithm for performing Fisher’s exact test in r  c contingency tables. “Journal of the American Statistical Association”, 78, 427-434.

P.W. MIELKE, K.J. BERRY, (2007). Permutation Methods, A Distance Function Approach, 2nd Ed. Springer, New York.

K. MODER, D. RASCH, K.D. KUBINGER, (2009). Don’t use the two-sample t-test anymore! Proceedings of the “6th St. Petersburg Workshop on Simulation”, Edited by S.M. Ermakov, V.B. Melas and A.N. Pepelyshev, 258-264.

F. PESARIN, (2001). Multivariate Permutation tests: with application in “Biostatistics”. John Wiley & Sons, Chichester, UK.

F. PESARIN, (2002). Extending permutation conditional inference to unconditional one. “Statistical Methods and Applications”, 11, 161-173.

F. PESARIN, (1995). An almost exact solution for the univariate Behrens-Fisher problem. “Statistica”, LV,131-146.

F. PESARIN, L. SALMASO, (2009). Finite-sample consistency of combination-based permutation tests with application to repeated measures designs. “Journal of Nonparametric Statistics”, DOI 10.1080/10485250902807407.

F. PESARIN, L. SALMASO, (2010). Permutation Tests for Complex Data. Theory, Applications and Software. Wiley Series in Probability and Statistics, John Wiley & Sons, Chichester, UK.

F. PESARIN, L. SALMASO, (2011). A new characterization of weak consistency of permutation tests. Journal of Statistical Planning and Inference (forthcoming).

J.O. RAMSAY, B.W. SILVERMAN, (1997). Functional Data Analysis. Springer-Verlag, New York.

J.O. RAMSAY, B.W. SILVERMAN, (2002). Applied Functional Data Analysis. Springer-Verlag, New York.

R.H. RANDLES, D.A. WOLFE, (1979). Introduction to the Theory of Nonparametric Statistics. Wiley, New York.

P. RÉVÉSZ, (1968). The Laws of Large Numbers. Academic Press, New York.

J.P. ROMANO, (1990). On the behaviour of randomization tests without group variance assumption. “Journal of the American Statistical Association”, 85, 686-692.

L. SALMASO, A. SOLARI, (2005). Multiple Aspect Testing for case-control designs. “Metrika”, 12, 1-10.

L. SALMASO, A. SOLARI, (2006). Nonparametric iterated combined tests for genetic differentiation. “Computational Statistics & Data analysis”, 50, 1105-1112.

H. SCHEFFÉ, (1943). Statistical inference in the non-parametric case. Annals of Mathematical Statistics, 14, 305-332.

G.R. SHORACK, (1967). Testing against ordered alternative in model I analysis of variance: Normal theory and non-parametrics. Annals of Mathematical Statistics, 38, 1740-1753.

L. TANG, N. DUAN, R. KLAP, J. ROSENBAUM ASARNOW, T.R. BELIN, (2009). Applying permutation tests with adjustment for covariates and attrition weights to randomized trials of health-services interventions. “Statistics in Medicine”, 28, 65-74.

R. XU, X. LI, (2003). A Comparison of Parametric versus Permutation Methods with Applications to General and Temporal Microarray Gene Expression Data. “Bioinformatics”, 19, 1284-1289.

P.H. WESTFALL, S.S. YOUNG, (1993). Resampling-based multiple testing: Examples and methods for pvalues adjustment. Wiley, New York.

L. ZHANG, J. WU, W. JOHNSON, (2010). Empirical study of six tests for equality of populations with zeroinflated continuous distributions. “Communications in Statistics”, Simulation and Computation, 39, 1181-1196.




How to Cite

Pesarin, F., & Salmaso, L. (2010). The permutation testing approach: a review. Statistica, 70(4), 481–509.