Some comments on the runs test function

[colinpmillar]

Hi,

While looking into this function, and its equivalent in a4adiags package, I noticed one or two things:

1. Very small pedantic thing - the line:
   https://github.com/PIFSCstockassessments/ss3diags/blob/796279840feb73bd1fec9b6c96e21f22337778cf/R/SSplotRunstest.R#L32

does not need to subtract the mean, as diff() removes the mean anyway., so the line could be simplified to

mr <- abs(diff(x))

2. the standard deviation estimated here is an estimate of the process error, that is the variability of the rate of change of the residuals over time. This variance is used in ndividual control plots (See https://en.wikipedia.org/wiki/Shewhart_individuals_control_chart) to test if a process is "out of control". It wasnt so clear to me that this was the technical defition of the standard deviation and 3 sigma being applied in the runs tests plots.

https://github.com/PIFSCstockassessments/ss3diags/blob/796279840feb73bd1fec9b6c96e21f22337778cf/R/SSplotRunstest.R#L31-L43

So really it seems this function does two things, a one sided runs test for positive autocorrelation (or more simply as stated in the randtests documentation, "a trend"), but also provides additional ranges to test whether the residuals are "out of control" :)

3. In terms of assessing pass or fail in the plots, I think only the runs tests are considered, and are always compared to the p < 0.05 for significance. However, given than there are many tests being conducted, it would be better to correct the p values for multiple comparisons. There are lots of options here: https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/p.adjust

4. The runs test p values are not very accurate for small sample sizes, there is a nice example here: https://stats.stackexchange.com/questions/144598/one-sample-run-test-but-with-p-neq-frac-12 where the runs.test function gives a significant result (p=0.035) when the exact P value is 0.08. Perhaps a warning could be given when sample sizes are small.

thanks!

[MOshima-PIFSC]

Hi @colinpmillar, thanks for the insightful comments and suggestions. We can definitely improve the documentation for the runs test and look into the alternative p-values. Realistically, we probably won't have time to make these improvements in the next few months but if you would like to, feel free to submit a PR. Also, just in case you missed it, our repository has moved to nmfs-ost/ss3diags now. Thanks!