Standardized Minimum Detectable Effect (MDES) Calculations

Benchmark Calculation

The following calculation assumes a sample size of 150, a two-sided alpha value of 0.05, and a power of 0.80. The plot below shows the MDES as a function of the proportion of treated units.

t_c <- qt(0.975, 148, lower.tail=T) # critical t-value
t_a <- qt(0.80, 148, lower.tail=T)  # t-value for alternative

N <- 150  # sample size
p <- 0.10 # probability of treatment

mdes <- (t_c + t_a) * sqrt(1/(N*p*(1-p)))
mdes

## [1] 0.7675555

# now with varying probabilities of treatment

p_vals <- seq(0.05, 0.40, by = 0.01)
mdes_vals <- rep(NA, length(p_vals))

for(i in 1:length(p_vals)){
  mdes_vals[i] <- (t_c + t_a) * sqrt(1/(N*p_vals[i]*(1-p_vals[i])))
}

plot1_df <- cbind.data.frame(p_vals,mdes_vals)

ggplot(plot1_df, aes(x=p_vals,y=mdes_vals)) +
  geom_point(col="gray40") +
  geom_line(y=0.8, col="red", linetype="dashed") +
  geom_line(y=0.5, col="forestgreen", linetype="dashed") +
  xlab("Probability of Treatment") +
  ylab("MDES") +
  theme_bw()

The commonly used threshold for a ‘’large’’ MDES is 0.80. This is met as long as p > 0.10. The threshold for a ‘’medium’’ MDES is 0.50. This is met only when p > 0.30.

Blocking

We now estimate the MDES, but with a stratified (block) design. For simplicity, assume that we have 4 blocks. We fix the proportion of treated units p at 0.10 and vary the proportion of explained variation in the outcome predicted by the blocks from 0.1 to 0.8.

t_c <- qt(0.975, 145, lower.tail=T) # critical t-value
t_a <- qt(0.80, 145, lower.tail=T)  # t-value for alternative

N <- 150
p <- 0.10

r2_vals <- seq(0.1, 0.8, by=0.025)
mdes_vals <- rep(NA, length(r2_vals))

for(i in 1:length(r2_vals)){
  mdes_vals[i] <- (t_c + t_a) * sqrt((1-r2_vals[i])/(N*p*(1-p)))
}

plot2_df <- cbind.data.frame(r2_vals, mdes_vals)

ggplot(plot2_df, aes(x=r2_vals,y=mdes_vals)) +
  geom_point(col="gray40") +
  geom_line(y=0.8, col="red", linetype="dashed") +
  geom_line(y=0.5, col="forestgreen", linetype="dashed") +
  xlab("R-squared from block dummy variables") +
  ylab("MDES") +
  theme_bw()

When the blocks remove a little over half of the variation in the outcome, we are able to lower the MDES close to 0.5 while holding p at 0.10.