Back in 2013, the Pew Research Center released a “Portrait of Jewish Americans.” That survey and many of the numbers listed in it received quite a bit of attention. Pew seems to have decided they liked the response, so they’ve apparently done it again. They are having a press event for the release of their new survey on May 11th. I do not know anything about the new results. Still, if the 2013 survey was any guide, there will be a swarm of Jewish opinionators highlighting all sorts of numbers in the new survey. 

In advance of their new release, I want to highlight the most important number from their earlier survey: 16%. That’s the overall response rate for the survey. Don’t worry if you missed it–it was conveniently on page 135 in appendix A. That means about 16% of the people they contacted took the survey, through which they identified approximately 3500 Jews total. A response rate like this might be acceptable for an opinion poll, which is much of what Pew does. Such a low response rate plus the small total sample is never ideal. It is ok, if you’re repeatedly sampling a population and trying to understand trends, but it’s more than a little problematic for a infrequent census or demographic survey.

Additionally, their 2013 survey over-sampled densely Jewish regions, particularly regions with high Orthodox populations, because, with their surveying methods, they acknowledged the difficulty of randomly calling a single Orthodox Jew outside of these regions (Page 127 in Appendix A). As a consequence, the 2013 survey measured Orthodox Jews in NY & NJ and compared them to Jews in the rest of the country. They had reasonable reasons to do this, but it generated a substantial geographic bias that was neither acknowledged in Pew’s summary nor in the many commentaries talking about denominational differences. Even overlooking this oversampling of certain regions, the margin of error for Orthodox responses was 9.1%, vs a still high 4.8-6.5% for others. That means any statistic on the Orthodox subsample from NY & NJ could easily have been off in either direction by 9.8%. Many of the commentaries on denominational subsamples were focusing on differences within these margins of error. With these margins of error, any claim based on numbers for a sub-group of a sub-group (i.e. young Orthodox Jews) is effectively meaningless.

I’m certainly not the first to note these problems (Samuel Klausner wrote about much of this back in 2013), but I want to bring your attention to them in advance of the new report. The 2013 Pew survey results were a numbers-based Rorschach test, where many Jewish opinionators found some numbers that fit their existing opinions and used them to try and make their pre-existing opinions seem fact-based. Don’t fall for it this time.

The 2013 survey results did not give us meaningful additional information about the demographics of Jewish communities. I hope the new survey corrects some of the methodological issues from the 2013 version. I hope Jewish publications find experts to help them understand the limits of claims that can be made from the new Pew data and shares those limits with their readers. Perhaps some publications will think twice before publishing another flood of opinion pieces making claims that are unsubstantiated by the data.

The fundamental problem with Pew’s first Jewish survey is that they are a reputable opinion polling firm, but the types of questions asked in their 2013 survey were designed for a demographic survey, such as the many previous demographic studies of Jews funded in part by Jewish Federations. Those studies were flawed and expensive, but they were serious attempts at actually counting Jewish populations. The Pew poll may be cheaper, but it’s a poor substitute for a genuine census. 

If Jewish leaders care about these types of data, it will take investment. I’ll leave it to those with the money to decide if this is where they want to invest their money, but don’t pretend that the Pew opinion polls are an acceptable substitute.