I wear a lot of hats in my day to day. I’m a pollster. I use data to raise money for Democrats. I’m doing my best to foster a culture of activism here in Texas.
I’m also a bit of a nut, who has fun doing things like writing hundreds of postcards and designing an experiment around them all in my spare time.
I began my political career at about the same time as the “test everything” ethos began permeating the Democratic technology space and that thinking has informed everything I do. The willingness (indeed, the urgency) to test new things I think has helped my career along, and given me some things to say I don’t think others got to first.
And so as we approached the crucial November elections in 2022, in addition to my day job helping Democrats win, on weekends I volunteered for candidates, knocked doors, made phone calls, raised money, and even made some merch.
I also wrote some postcards.
Specifically, I wrote postcards to voters in Pennsylvania who I thought were unlikely to vote, but if they did, would vote for Democratic candidates. Specifically, I sampled from the Pennsylvania pre-election voterfile (i.e., the version of the file that would be current as of October 13, 2022) subpopulation that included voters who
- Are registered Democrats
- Did not vote in a primary election in 2016, 2018, or 2020
- Did not vote in the 2016 or 2020 Presidential elections
I didn’t impose a rule concerning 2018 vote. This felt like the right sample at the time, though in hindsight I might’ve noodled on it a bit. Because I conducted this experiment in a personal capacity (i.e., only really had to shell out twenty bucks for the voterfile export and a couple hundred bucks for stamps and postcards) I did not have access to any sort of scores that could’ve allowed for more fine-grained sampling.
In total, I sampled 400 of these low-propensity Democratic voters from the overall Pennsylvania population of about 1.17 million of this type of voter and then divided them into two populations I cleverly called Population A and Population B, set the seed to my lucky number, drew a sample of 400, drew another random number, and split that sample 200/200 into each messaging condition. I did some but not a ton of sample balancing tests, including regressing treatment assignment on variables available on the voterfile including county, age, and sex. I didn’t go much further than that because imputing additional variables against which to check balance is a highly fraught practice. I’m skittish about walking through the R code I used to cut these samples as any work with a voterfile puts personally identifiable information at risk, but if you are curious to learn more you know how to reach me.
Then, it was time to actually write the damn things. Armed with a pen and some stamps, over the course of a couple of weeks I knocked out 400 postcards that each contained one of these two messages, written by hand:
Treatment A (“Roads and schools”) Hey, [first name]! This November vote for Democrats John Fetterman and Josh Shapiro because here in Pennsylvania, we value good schools, clean water, and affordable healthcare. We deserve leaders who fight for a stronger Pennsylvania. Vote by November 8 for Democrats John Fetterman and Josh Shapiro. Best, John
Treatment B (“Trump is on the ballot”) Hey, [first name!] This November vote for Democrats John Fetterman and Josh Shapiro because here in Pennsylvania, Donald Trump is on the ballot. We deserve leaders who stand up to traitors and fight for our glorious Republic. Vote by November 8 for Democrats John Fetterman and Josh Shapiro. Best, John
In both treatments, the postcards had a plain white side for the handwritten message and the same glossy backside I printed using VistaPrint.
The text of the glossy backside is based on the benefits Democrats delivered to Pennsylvania in the Bipartisan Infrastructure Framework. (I have since updated the design for future projects to include additional benefits for Pennsylvania from, among other things, the Inflation Reduction Act - Help show off the benefits of Democratic leadership in your state now! wink wink, nudge nudge, stamps are expensive don’t ya know, etc.)
At the time, I felt these two messages captured a messaging dynamic being heavily debated at the time: specifically, “Should Democrats message about the economy, or should they message about Trump?” This experiment lacks the sample size, completeness of sampling strategy, completeness of design and (spoiler alert) clarity of results to settle this debate, but I can at least add to it.
I ended up writing a lot of postcards!
How did I manage to sit there and get through such a pile in just a few weeks, you ask? The answer may surprise you:
(Or it may not)
The results are about what I should’ve expected from such a small message test, but even with the underpowered design, I am pleasantly surprised to see the results confirm my suspicion that talking about Democratic benefits is, for this specific subset of the population, more effective than the threat of Donald Trump. (The polling I was doing in my day job at the same time was saying this, so this is a positive result overall as far as I’m concerned, unimpressive though it may be)
Here’s the average treatment effect compared to the sample population. Indeed, this sample universe’s turnout was relatively low (about 51 percent of the voterfile, compared to statewide turnout approaching 70 percent of the voterfile). Turnout was a statistically indistinguishable 50 percent among those in the “Roads and schools” condition, and a statistically distinguishable, and lower, 41 percent among those in the “Trump is on the ballot” condition.
There isn’t enough sample to do much in the way of analyze heterogeneous treatment effects and so I won’t dither with the attempt. The two treatments are not statistically distinguishable from one another, but the “Trump is on the ballot” message resulted in lower turnout than the sample-wide average.
Like all such projects, that was a lot of work for a short write-up of results that I find informative but not earth-shattering.
The specific findings of the experiment are
- Postcards do not radically alter turnout (duh)
- When it comes to low-propensity voters, a positive vision for concrete improvements to ones’ material conditions outperforms scaring them with the alternative (perhaps less duh)
The longterm learnings of the experiment are
- If you want to commit to doing a message test, compose messages short enough to both make a point and that you won’t drive yourself nuts by writing hundreds of times
- The Post Office’s hours are more limited even than you already thought they were
- Designing and creating your own campaign materials is a lot of fun even if the finesse can be tedious
In conclusion, this is just one test among many more to come in the future…
And so if you have ideas for experiments you’d like to see in the future, don’t hesitate to reach out!