Some remarks about the number of permutations one should consider to perform a permutation test

Abstract

The main practical drawback of permutation testing is that, except for very small sample sizes, the number of all possible permutations is usually impractically large. Although now fast and relatively cheap computing facilities are at our disposal, this problem is still interesting, in particular for applied statisticians. The main idea is that it is not necessary to compute all possible permutations to obtain a reliable p-value estimate of the test. To deal with this problem, one may approximate the exact p-value of the test by using a random sample from all permutations. The aim of this paper is to reply to this question: how many permutations should be considered in the p-value estimation procedure? We suggest to use 500-1000 permutations to estimate the size and power of a permutation test, via Monte Carlo simulations, at the a significance level of 5% and 2000-5000 when a = 1%. Moreover, we suggest to use 5000 permutations in actual applications when a = 5% and 10000 when a = 1%. These suggestions are based on a review of many papers, a simulation study and two applications to actual data sets.