17 Mc Nemar’s Test

McNemar test

The McNemar test (Quinn McNemar, 1947) is a statistical test used to analyze paired nominal data or matched data collected from two related groups or conditions. It assesses whether there is a significant difference in proportions or frequencies of a categorical outcome between paired observations. The McNemar test is particularly useful when dealing with data that are not independent, such as before-and-after measurements on the same subjects or matched pairs in case-control studies.

17.1 Understanding the McNemar Test:

1. Null and Alternative Hypotheses:

Null Hypothesis (H0): There is no difference in the proportions or frequencies of the categorical outcome between the two conditions.
Alternative Hypothesis (H1): There is a significant difference in the proportions or frequencies of the categorical outcome between the two conditions.

2. Test Statistic:

The McNemar test statistic is calculated based on the number of discordant pairs, i.e., pairs where the outcomes differ between the two conditions.
It follows a chi-squared distribution with one degree of freedom under the null hypothesis.

3. Calculation of Test Statistic:

Let a be the number of discordant pairs where a subject has a positive outcome in the first condition and a negative outcome in the second condition.
Let b be the number of discordant pairs in the opposite direction.
The uncorrected McNemar test statistic is:

\[\chi^2_{M} = \frac{(b - a)^2}{b + a}\]

The corrected McNemar test statistic (Yates’ continuity correction to adjust for the discrete nature of the data)is:

\[\chi^2_{M, corrected} = \frac{(|b - a| - 1)^2}{b + a}\]

4. Interpretation of Results:

If the calculated \(\chi^2_M\) value is greater than the critical value from the chi-squared distribution with 1 degree of freedom at the chosen significance level (commonly \(\alpha = 0.05\)), we reject the null hypothesis, indicating a significant difference.
The continuity-corrected value is slightly lower than the uncorrected value, but in most practical cases, both lead to the same conclusion.

17.1.1 McNemar Test: Corrected and Uncorrected

The McNemar test is designed to analyze paired nominal data, particularly when the outcomes are dichotomous (e.g., pass/fail). There are two approaches to calculate the test statistic: uncorrected and continuity-corrected.

Uncorrected McNemar Test

Calculates the chi-squared statistic directly from the discordant pairs.
Formula:

\[ \chi^2_{uncorrected} = \frac{(b - a)^2}{b + a} \]

a = number of pairs where the first condition is positive and the second is negative.
b = number of pairs where the first condition is negative and the second is positive.
The uncorrected test shows the theoretical difference but may slightly overestimate significance with small samples.

Continuity-Corrected McNemar Test

Also known as Yates’ correction, adjusts for small sample discrete data.
Formula:

\[ \chi^2_{corrected} = \frac{(|b - a| - 1)^2}{b + a} \]

This correction reduces the test statistic slightly, making it more conservative.
Recommended for small discordant pairs.

Summary: The uncorrected McNemar test is straightforward, while the continuity-corrected version provides a more conservative estimate. Both are valid, but the corrected test is often preferred for small sample sizes or when using standard software implementations.

Considerations:

The McNemar test assumes that the data are paired and dichotomous.
It is sensitive to small sample sizes, especially when the number of discordant pairs is small.
Extensions exist for analyzing categorical data with more than two categories.

17.1.2 Applications of McNemar Test

A. Medical Research:

In clinical trials, the McNemar test is used to assess whether there is a significant difference in treatment outcomes between two treatment groups or before and after treatment within the same group.

B. Education:

Educational researchers may use the McNemar test to evaluate the effectiveness of teaching methods or interventions by comparing pre-test and post-test scores of students.

C. Epidemiology:

Epidemiologists use the McNemar test to analyze matched case-control studies or cohort studies where data are collected from the same subjects at different time points.

17.2 Example problem: McNemar test.

A study was conducted to evaluate the effectiveness of a new teaching method aimed at improving students’ performance in a mathematics examination. Data were collected from 50 students, each of whom took a pre-test (before the introduction of the new method) and a post-test (after its implementation). In both assessments, students’ outcomes were classified as either “Pass” or “Fail.”

The results are summarized as follows:

In the pre-test, 25 students passed and 25 students failed.
In the post-test, 35 students passed and 15 students failed.
Of the students who passed the pre-test, 20 also passed the post-test.
Of the students who failed the pre-test, 15 subsequently passed the post-test.

These results can be represented in the following contingency table:

	Passed Post-test (Yes)	Passed Post-test (No)
Passed Pre-test (Yes)	20	5
Passed Pre-test (No)	15	10

The objective of this analysis is to determine whether there is a significant change in the proportion of students who passed before and after the introduction of the new teaching method.

This comparison involves paired categorical data, making it suitable for testing with McNemar’s Test, which assesses whether the proportion of discordant pairs (i.e., students who changed from pass→fail or fail→pass) differs significantly.

17.2.1 Calculation of McNemar Test (Continuity-Corrected)

Create a Contingency Table:

	Passed Post-test (Yes)	Passed Post-test (No)
Passed Pre-test (Yes)	20	5
Passed Pre-test (No)	15	10

Calculate the McNemar Test Statistic (Continuity-Corrected):
- \(a\) = Number of discordant pairs where a student passed the pre-test but failed the post-test = 5
- \(b\) = Number of discordant pairs where a student failed the pre-test but passed the post-test = 15
\[\chi^2_{M, corrected} = \frac{(|b - a| - 1)^2}{b + a}\] \[\chi^2_{M, corrected} = \frac{(|15 - 5| - 1)^2}{15 + 5} = \frac{(10 - 1)^2}{20} = \frac{81}{20} = 4.05\]
Determine the Critical Value:
- With 1 degree of freedom and \(\alpha = 0.05\), the critical value of the chi-squared distribution is approximately 3.84.
Compare the Test Statistic to the Critical Value:
- Since \(\chi^2_{M, corrected} = 4.05 > 3.84\), we reject the null hypothesis.

Interpretation:

Based on the McNemar test, we conclude that there is a significant difference in the proportions of students passing the exam before and after the teaching method was introduced. In this example, the teaching method appears to have a significant positive effect on students’ performance in the mathematics exam.

17.3 McNemar Test in R and Python

Python

Summary

Concept	Description
Foundations
McNemar Test	Non-parametric test for paired dichotomous outcomes measured on the same subjects
Paired Nominal Data	Data collected twice on the same units, such as pre-test and post-test pass/fail outcomes
Test Statistic
Discordant Pairs	Pairs whose outcomes disagree between the two conditions, the only pairs that carry information for the test
a and b	Counts of the two kinds of discordant pairs used in the McNemar formula
Uncorrected Statistic	(b minus a) squared divided by (b plus a), the direct chi-squared statistic
Yates' Continuity Correction	(\|b minus a\| minus 1) squared divided by (b plus a), preferred for small samples
Chi-squared with 1 df	Reference distribution for the McNemar statistic under the null hypothesis
Hypotheses and Table
Null Hypothesis	States that the paired proportions are equal, so the treatment has no effect
Alternative Hypothesis	States that the paired proportions differ, indicating a real change between conditions
Contingency Table	Two-by-two table of paired categorical outcomes that feeds the McNemar formula
Applications
Medical Research	Compares treatment outcomes before and after therapy in the same patients
Education	Compares student pass rates before and after a new teaching method
Epidemiology	Applies to matched case-control and cohort designs with paired categorical data
Psychology	Tests whether survey or preference responses shift after an intervention
In R and Python
R mcnemar.test()	Built-in R function that performs the McNemar test with optional continuity correction
Python statsmodels mcnemar()	Python function from statsmodels that performs exact and corrected McNemar tests