Testing Claims About Presidential Results

Some notes from my notes

Do you want to credit or slag a president or his administration? Apply these tests:

IDENTIFY THE ACTION

Select a specific implemented policy, not the president’s rhetoric.

What changed in government policy or practice?

Examples might include an executive order, a new regulation, legislation the president signed, a change in enforcement, a funding decision, a grant program, a tariff, a change in benefits or services, or a change in federal personnel or deployment.

ESTABLISH THE MECHANISM

Explain how that action could produce the measured outcome through any of these actions. Trace the chain from presidential action to the claimed result.

ESTABLISH IMPLEMENTATION

Did the action actually change what government did?

An executive order, for example, may direct agencies to act. But issuing the order does not establish that agencies implemented it, how quickly they did so or at what scale. Measure what actually happened on the ground.

CHECK TIMING AND REACH

Did the intervention precede the measured outcome? Or was the thing already happening? Did the policy have enough time to work and reach the people or places being measured? (Running to the front of a parade to claim credit for trends doesn’t count).

ESTABLISH THE COUNTERFACTUAL

What probably would have happened without the policy? For example, has the Iran war won more than diplomacy would have?

Did affected people or places improve more than credible comparison groups, after accounting for earlier trends and relevant differences in the places you’re comparing?

Where no comparison group exists, look for other evidence capable of distinguishing the policy’s effect from the existing trend.

LOOK FOR DIFFERENCES IN EXPOSURE

If some people or places received more of the intervention than others, did greater exposure correspond with a larger effect?

A federal grant program, enforcement initiative or funding increase should generally leave some evidence where its reach or intensity differed.

TEST COMPETING EXPLANATIONS

Examine local policies, inherited federal programs, economic changes, reporting changes and other plausible causes. Ask whether those explanations fit the timing, geography and size of the change as well as or better than the presidential action.

REPORT THE LIMIT

Distinguish a demonstrated contribution, suggestive evidence and an effect that has not been established. (But just because a presidential connection has not been established does not mean “zero,” it might mean you just don’t have the data.

The critical question: What changed because of the policy, beyond what was already happening?

A before-and-after comparison to an inauguration alone cannot answer that, no matter how hard a supporter or detractor tries.

IS THE FIGURE A CONTINUATION OR A TURN?

Every figure used to support a claim needs at least four comparisons:

LEVEL

How high or low is it now, compared with a meaningful historical baseline, not just a conveniently selected peak?

DIRECTION

Was it already rising or falling before the administration or intervention? Or is the rise or fall determined by causes an administration has nothing to do with, like population growth.

PACE

Has that rise or fall accelerated, slowed or reversed? Compare equivalent periods and consistent populations.

BREADTH AND DURABILITY

Is the change widespread and sustained, or concentrated in a few places or months?

TIMING OF THE TURN

When did the change in direction or pace actually begin? Did it precede the presidential action, coincide with its implementation or follow it by a plausible interval?

For economic measures, that also means separating inflation-adjusted from nominal values, per-person from aggregate totals, and median households from averages.

For crime, it means separating homicide from other violence and national estimates from selected cities.

REMINDER

A result can occur during a presidency without resulting from presidential action. Conversely, a trend that began earlier could still have been accelerated, slowed or otherwise altered by later presidential action.