Kosher Stats 201: Taglit-Birthright Israel might not change interest in in-marriage for program participants who were raised secular or “just Jewish”
I’m going to try something a bit new here and take a close look at the data analyses from a single study. I chose this particular study, The Impact and Lessons of Taglit‐Birthright Israel by Saxe et al, because someone asked my opinion about it and I thought it did some things very well. I think the data supports some interesting findings, although it includes some all-to-common misinterpretations of statistical results. I want to say at the outset that, although I have some critiques of this study, topics worth studying rarely give easily interpretable results. The authors make a positive contribution to the discussion. I’m also impressed that Taglit-Birthright Israel has worked to include data collection and analysis as part of their mission. Their data collection and fairly frequent publications are what make quantitative discussion of Birthright Israel possible.
The primary goal of this specific study was to examine whether participation in Taglit-Birthright Israel affected attitudes towards in-marriage vs intermarriage (and later marriage rates) and views on raising children as Jews regardless of the spouse’s religion. This examination of actual marriage rates is now possible because the 2001-2004 cohort of Birthright attendees now have a sufficient population of participants who’ve married to run statistical analyses on their marriage choices. Most of the examinations of attitudes come from surveys conducted 3 months before and 3 months after 2008 Birthright trips.
The article’s starts with a nuanced discussion that puts concerns that intermarriage will destroy Judaism in the context of existing research. I was surprised to learn that 15.4% of the 2001-2004 applicants to Birthright Israel had a non-Jewish parent while 24% of the 2008 applicants had a non-Jewish parent. That large of a jump in just a few years means that Birthright is increasingly attracting children who doomsayers consider lost to Judaism.
Their main analyses use something called ordinal logistic regression. This is a kind of analysis of ranked responses (i.e. “Thinking about the future, how important is it to you to marry someone Jewish?” 0=Not Important, 1=A little Important, 2=Somewhat Important, 3=Very Important or “ For how many years did you attend a [supplemental or day] school?”) If you know nothing about a person besides their response to a single question, the results specify the odds that an increase in one level from one questions predicts an increase in one level for another question. For example, how much does an additional year of supplemental school or participation in Birthright change the odds that a person will consider inmarriage as “A little important” rather than “Not important.”
The authors present two types of results: (1)The effect of Birthright on people’s actions or attitudes. (2) The relative effect of Birthright compared to other Jewish education and social programs.
(1)The effect of Birthright on people’s actions or options.
Pretty much any way the authors slice their data, Birthright participation increased the odds that someone from the 2001-4 participation cohort married a Jew, with a greater odds increase if the participant has a non-Jewish parent (Table 2). Participation also increased the odds that participants wanted to raise Jewish children, even if the person had a non-Jewish parent (Table 3). However, some of this affect is probably driven by the fact that they didn’t ask the same questions before the Israel trips in 2001-4, and so the people who decided to participate may be the people who cared more about marrying a Jew to begin with.
In 2008, they asked the same questions before the trip. By far, the biggest predictor of how much participants valued in-marriage in the post-trip survey was how much they valued in-marriage in the pre-trip survey. This is a reminder that we need to be careful not to over-interpret the 2001-4 results or results from any other survey that doesn’t ask similar questions before whatever intervention or program is being evaluated.
The data they collected in 2008 allows the authors to control for many factors, including years and types of Jewish education, ritual practice levels, movement affiliations, and Jewish social connections. Even after including all of these other factors, Taglit-Birthright Israel participation shows an effect on the participants. The left columns of tables 4 and 5 show an increase in interest in in-marriage and in raising Jewish children from 3 months before to 3 months after the trip. The right column of table 4 even shows an increase in interest in finding a Jewish spouse for people with a non-Jewish parent. The authors don’t state the level of statistical significance values for any of these results, but the size of the Birthright participation effect seems large enough to be of interest.
I do want to note here that ordinal logistic regression assumes that it is equally important for someone to jump between any adjacent levels of a response (i.e. someone is equally likely to change from “not important” to “a little important” as from “somewhat important” to “very important”). If this isn’t true and the non-participants had a different distribution of responses, it has the potential to bias the results. Figure 10 in a related publication The Impact of Taglit-Birthright Israel: 2010 Update shows a clear distribution difference on in-marriage interest between participants and non-participants. 48% of non-participants think in-marriage is not or a little important compared to 32% of participants. The authors could show whether this distribution difference is a problem by dividing these data into subsamples based on pre-trip responses to separately show the distribution of changes for every group. They could also show the distribution of responses (or the distribution of changes) pre AND post trip for participants and non-participants. Since only a fraction of people would show any change over 6 months, this would help better understand which groups of people are most heavily affecting the results.
One other thing that stands out to me is they include having a non-Jewish parent in the model of what predicts intermarriage (Table 4), but not in the model of what predicts wanting to raise children Jewish (Table 5). The omission without an explanation seems odd.
(2) The relative effect of Birthright compared to other Jewish education and social programs.
The authors fall short in comparing Birthright to other programs. They properly designed their analyses to model the unique effects of Birthright. They then used the same models to make claims like, “These data indicate that in order to equal the impact of Taglit on importance of marrying a Jew, one would need to attend… Jewish overnight summer camp for 12.8 years (well beyond the number of years most Jewish summer camps offer sessions)” If I found a result showing that two weeks in Israel on a Birthright tour is so transformative that it changes people more than an unobtainable number of years in Jewish summer camp (or 4.8 years of day school or 12.0 years of supplemental school), I’d worry. Unbelievable results are usually wrong.
These numbers bother me more because I can’t figure out how they calculated them. The authors say they simply divided the coefficients on participation by the coefficients on Jewish education (footnote 16), but I can’t get near these values with the results presented in table 4. But even using the values from table 4 (middle model), 2 weeks of Birthright has the same effect as 10 years of Jewish Summer camp. This doesn’t make sense.
I suspect that the core problem is that this analysis was designed to test for a Birthright effect, after controlling for many other factors, and this isn’t a good way to identify how the Birthright effect compares to these other factors. A good demonstration of this problem is in the left-most column of table 4, where the authors also include the pre-trip importance of in-marriage as a control variable. This model is comparing the attitudes of people who went on the Birthright trip with people who didn’t go on the trip, but we can probably assume that the attitudes of non-participants didn’t change much during the 6 months between surveys without a trip. Thus, the left and right columns of Table 4 and the left column of table 5 are actually showing the interactions of Birthright participation with schooling, childhood affiliations, and so forth rather than the unique contributions of each of these. If there is little or no change among people who didn’t participate in Birthright, then the results in these tables show that the impact of Birthright participation on attitudes towards intermarriage are heavily driven by Orthodox and Conservative participants those with more years of schooling & camping, and those who had more Jewish high school friends. If my assumptions are correct, based on the left and right columns of figure 4, there’s no clear evidence of a Birthright effect on in-marriage attitudes for participants who were raised secular or just Jewish. The authors could easily test this by running analyses only on the Birthright participants and presenting mean differences across various populations.
The authors do try to deal with the effect of pre-trip attitudes by removing them from the middle model in table 4. This removes the biggest cause of undesirable interactions. In these results, the impact of pre-trip life on in-marriage attitudes is much greater, and, strangely, the effect of Birthright drops by 30%. Still, the other factors still interact. Most of that increase in the Birthright effect on attitudes comes from participants with childhood Orthodox or Conservative affiliation, parental organizational ties, high school ritual practice, and high school Jewish friends. If there was minimal relationship between all these things, it wouldn’t affect the authors’ conclusions about which of these things are more important, but we all know these things are strongly correlated. For example, if 80% of dayschoolers were also Orthodox and 15% of dayschoolers were Conservative, this analysis would be unable to distinguish day school participation from affiliation. This is called muticollinearity, and it isn’t an easy problem to address. One approach would be to throw many possible interactions between factors into the analysis. Even if is possible to model everything, it would be much harder to combine those results into simple interpretations. Another approach would be to look at subsets of the population. For example, look at the effects of schooling choices vs Birthright participation for only those with a Reform affiliation in childhood. If the sample size isn’t big enough to do this, we have to just accept the limits of what we can conclude based on this data.
This is something of an aside, but I thought it was interesting. The authors show that Taglit participants were not more likely to date Jews, but the participants were significantly less likely to be married before age 30 compared with nonparticipants (marriage rates equalized for those aged 30+). They follow with some speculation, “One possible explanation for this phenomenon is that Taglit participants are more likely to want to marry a Jewish person and consequently spend a longer time searching for a suitable partner.” The Impact of Taglit-Birthright Israel: 2010 Update, a related publication, contains the same speculation, but adds a footnote that, “Another possible explanation for the disparate marriage rates among respondents under the age of 30 is that married individuals, or those who are engaged or about to become engaged, are more likely to remain nonparticipants, potentially due to less flexible schedules (and therefore declining a trip and/or not re-applying). This is supported by the finding that over three percent of nonparticipants were married at the time of application to Taglit, compared to less than one percent of participants.” When there are two possible explanations for a finding and they only want to mention one, it would be more appropriate to include the one supported by their data.
This is a great analysis, and sadly really the proper response is: duh. Did anyone really think a week of tourism would make a real difference? This has been known as a waste of money for decades.
KRG, I think you’re being a bit too harsh. This specific study is focusing on an outcome that isn’t a primary goal of the trip (although it was important enough for them to survey).
There are also some fascinating hints of results with potential policy applications. To speculate a bit, it seems like any effect they are seeing is dominated by people who come from more observant backgrounds or who have more formal Jewish education. This means, that a free trip to Israel for someone from an actively Jewish background who hadn’t been to Israel before strengthens their interest in a Jewish family. A trip to Israel for someone with little or no existing connections to Jewish practice or knowledge doesn’t alter them much & might even negatively change their opinions on these questions.
This means there might be benefits to narrowing the Birthright target population or investing in a much larger pre-trip education component for people coming of less-connected backgrounds.
I also did a quick look and this seems like the first study that published results from the Summer 2008 participants. I don’t think the older years were asked the same survey questions before and after the trip. Thus, only data from the 2008 group (or later) allows for perfect comparisons of how individuals’ opinions changed rather than comparisons to imperfectly matched non-participants. It will be very interesting to see results from the 2008 on the issues that ARE the core goals of Birthright.
The data collection must have been completed by the end of 2008 (three months after the Summer trip). That they’ve only published one finding on a non-core goal of the trip makes me very curious to see the rest of the results.
BTW – awesome analysis. Thanks!
@ Dan Ab: “This specific study is focusing on an outcome that isn’t a primary goal of the trip (although it was important enough for them to survey).”
Two thoughts on this:
1 – Whether it is a primary goal or not, the in-marriage numbers have been a main feature of Birthright PR for a couple of years.
2 – Based on the editorials from the main funders, it is not clear to me what Birthright is for besides feeling “more Jewish” and having a better connection to the State of Israel. The summer of Lebanon and the North getting bombed (’06, I think) the editorial about Taglit at the end of the summer trumpeted how the participants supported Israel.
In general, I find the goals of Birthright to be very lowest-common-denominator. Also, the study which they have not done (and I suspect will never do) is whether the existence of Birthright affects a family’s decision whether or not to send a teen on a 4-6 week trip while in High School. The youth groups and camps have anecdotal evidence of this. If Birthright is REALLY interested in honesty about the trip’s impact, they would begin to ask the question.
Also – this XKCD came to mind – http://xkcd.com/552/
@Gnatie,
I agree that marriage & child-raising might not be their primary goal, but it is clearly one of their goals and one of very few topics they deemed worth asking multiple questions about in their survey.
As for the more general goal of Birthright, feeling more Jewish & a connection to Israel is accurate. Here’s the statement from their website: “Taglit-Birthright Israel’s founders created this program to send thousands of young Jewish adults from all over the world to Israel as a gift in order to diminish the growing division between Israel and Jewish communities around the world; to strengthen the sense of solidarity among world Jewry; and to strengthen participants’ personal Jewish identity and connection to the Jewish people.”
It would be really interesting to see how Birthright affects longer and earlier trips to Israel. I don’t expect Birthright to do this study because it would really be an independent data-set from what they could otherwise collect from their participants. The better approach would be to go to the organizations that host many of those longer visits to see how their participation numbers changed with the growth of Birthright. The relative drop in Orthodox participation makes me wonder if Orthodox programs have actually stepped up support for longer non-Birthright visits to Israel.
I don’t think this study would be good or bad for Birthright. If it is shown that Birthright is causing more people to have shallower Israel trips, they could change to subsidize some of these longer-trip with the money that would have paid for a shorter trip. Birthright isn’t a static program and they could be able to adapt to what works if the researchers they fund are willing to point out program weaknesses.
That’s also one of my favorite XKCD cartoons, which should tell you a bit about my personality.
In my day job I am a statistician in medical research. It is so hard to let go of the criticisms you raise on their methodology, explanations and reporting of it ( and more that I thought of as well) to really read the article, but I did.
I completely agree with your objections on multicolinarity ( a paper like this would not get published in my area, though we do a lot of in-the-field surveys.)
Beyond that, if half of Jews who intermarry raise their children as Jews (exact numbers disputed) , then mathematically, how are we losing Jews children to intermarriage? (Say I had 4 kids all Jews and so do you – say the all marry each other – four brides for 4 brothers say to simplify the actual in population/out-population pattern. Lets say they all have 2 kids each. Then we have from 8 Jews we have 8 Jewish grandchildren. Now lets Say we each have 4 Jewish children. Two of mine marry two of yours as before but 2 of yours and 2 of mine marry non Jews. Again let’s say they have 2 kids each. We now have 2 Jewish couples who raise 4 Jewish children, 2 intermarried couples raising 4 non-Jewish and 2 intermarried couples raising 4 Jewish each for a total of 8 Jewish grandchildren). This does work if you scale it up to population levels and use real models.
Also, I know this has been discussed before but the whole survival for the sake of survival thing in the context of a modern liberal society seems misguided.
@RTB
To echo your point with silly a silly anecdote: while waiting on the stroller line with my family in Brooklyn for the Governor’s Island Ferry, we struck up a conversation with another young family. It turned out that the couple were both children of intermarriage. Incidentally, they were both of Jewish and Irish heritage, just like me. My wife is of Jewish heritage. Our friends (a third couple) that we were meeting on the Island for a picnic include another child of intermarriage with a Jewish spouse. We all had various levels of observance (within progressive Judaism or none).
What was funny about the experience is just how common it is among my demographic. From a cultural point of view, we’re all of a tribe. From a genetic point of view, we’re all of a tribe. From a heritage point of view, our Jewishness is only minimally “diluted”. “Out-marriage” is only the beginning of the story. Considering it “the end” is not only offensive, it’s dumb.
Families like mine used to cluster in Unitarian Universalist Churches because that’s where we were welcomed. That’s not so anymore, and so the disincentive for intermarried families to raise children Jewish no longer exists. Most Jewish organizations have learned from their mistake, but some are obviously slower on the uptake.
@RTB & Dan O,
I think the issue of the future religion of intermarried families is one place where we agree with the authors. In fact, I think they are trying to present evidence to challenge that view with their child-rearing questions (I didn’t focus on that section too much because it has some of the same statistical issues & this was already too long for a typical blog post).
The authors clearly state that intermarriage is not the end of Jewish life. This has definitely changed in the past century. The question is why. The biggest factor is probably communal acceptance (i.e. someone with a non-Jewish spouse can now be part of many Jewish communities including their extended family). Still, the chance that a child of intermarriage remains connected to Judaism IS lower than a child of inmarriage. What causes that difference and what are there things that the larger Jewish community can do to influence that choice? Based on this research, it seems like Birthright might make a diference, but I wouldn’t call anything in this paper definitive.
@RTB, While I agree with the general concept of your example, the challenges are in the proportions. Communal factors clearly affect the percent of intermarried couples who raise children with strong connections to Judaism. You created sustainability by setting this to 50%, but I suspect that’s a bit high. This study doesn’t present the data cleanly, but if you look at figures 10 & 11 in the 2010 Update (also linked above), they show 54% nonparticipants considered inmarriage somewhat or very important while 74% of nonparticipants considered raising Jewish children somewhat or very important. If we guess that anyone who considered inmarriage important also considered raising Jewish children important, then 43% of people without interest in inmarriage had interest in raising Jewish children. The same number is 56% for Birthright participants, but there very well could be a selection bias there.
Since even non-participants who applied to Birthright have a selection bias over the larger Jewish population, I wouldn’t be surprised if the actual interest in raising Jewish children is slightly lower than 43%, but I’m speculating on speculations here.
Hi
I got the 50% figure from the first study The Impact of Taglit-Birthright Israel page 2 and the references therein. I acknowledge that the number might be lower, but even still, even at 40% the loss to intermarriage is much less than portrayed ( because in-marriage takes 2 Jews to create children and out-marriage might take a little more than 2, bit not 3). And logically if the number of intermarried couples who raise their kids as Jews is greater than 50% ( and the intermarriage rate is 50% and we assume equal birth rates, ) it actually creates MORE Jewish children than if there was only in-marriage.
OK enough math…
Whether you assume that 50% of children of intermarriages are raised Jewish, or whether you assume slightly lower than 43%, either way you’re going to get much more accurate results than the famous “Will Your Grandchildren Be Jewish?” chart (which some people actually take seriously, because, you know, it has numbers ‘n shit). Among its many flaws, it assumes that ZERO children of intermarriages will be raised Jewish! (It’s particularly ironic that this chart is sometimes brandished by adherents of Jewish streams that hold the position that any child of a Jewish mother is Jewish. From that definition of Jewishness, the intermarriage rate is completely irrelevant in determining the number of Jews in the next generation! All that matters is how many children each Jewish woman has, regardless of who her partner is, and regardless of anything that Jewish men do.)
@Dan Ab
“Still, the chance that a child of intermarriage remains connected to Judaism IS lower than a child of inmarriage. What causes that difference and what are there things that the larger Jewish community can do to influence that choice?”
The change (i.e. the removal of culturally enforced disincentives for intermarried families participating in Jewish communities) hasn’t happened in the last century. It’s happened in the last 30 years or so, and it isn’t complete. It’s loads better, and, yeah, participation is never going to be the same, regardless of disincentives. Still the effect of simply removing disincentives is remarkable.
But, regardless of short-term numbers, I don’t think it’s a great idea to bet the future of Jewish engagement of intermarried families by a harried trip to Israel.
Let’s suppose two things you rightly dispute: (i) that there is a causal relationship between Birthright participation and attitudes toward marriage, and (ii) that Birthright participation is more causally effective in effecting Jewish participation, than, say, a whole bunch of years going to Camp Kinderland.
If I want Jewish participation for my progeny (and I do), I’d still rather bet on Camp Kinderland. Attitudes toward Israel, because of events, are just too volatile to stake the future of Jewish participation upon. I mean, what does it mean that this study was conducted upon trips that occurred during the height of the 2nd Intifada? What effect did the Dolphinarium terror attack have on Birthright participants, specifically, in the year and years following it? (I ask because the empathetic connection between peers is obviously strong.)
Those aren’t rhetorical questions, they’re just really hard questions I don’t expect to be answered. My point is that the question is, simply, wrongheaded. It assumes that something needs to be done to snap us all out of assimilated dissolution. It’s just so Kristol, and so risky.
I just looked at the “Will your grandchildren be jewish chart” and it seems like they DO account for intermarried families raising children as Jews. See http://www.simpletoremember.com/articles/a/WillYourGrandchildrenBeJews/
The text says that 28% of intermarried families raise their children as Jews.
To get close to their numbers, I take (population size)*(avg children/woman)*( (inmarriage rate)/2 + (intermarriage rate)*0.28/2)
For reform Jews, that’s:
100*1.36*(0.54/2+0.46*0.28/2) = 45.478 (their number is 46)
45.478*1.36*(0.54/2+0.46*0.28/2) = 20.682 (their number is 21)
20.682*1.36*(0.54/2+0.46*0.28/2) = 9.406 (their number is 10)
You’ll note a major mistake that’s exactly what RTB highlighted above. They’re still dividing intermarried families by 2 even though each family contains only one Jewish child from the previous Jewish generation.
If you correct for this (remove the 2nd division by 2), the numbers become 100, 54, 29, & 16.
The other interesting thing is that they took the secular, conservative, & reform childbirth numbers from a national population survey, which probably includes families without children and people who never married, but the orthodox numbers are from a 2000 census of 8th graders in Jewish day schools where every respondent has children! While I get that they wanted to separate Centrist from Chassic Orthodox, that’s a completely invalid use of data. I can’t quite figure out where they got their exact #’s in that paper, but the childbirth rates for Schecter (Conservative) families are 2.65 and are 2.7 for families in Reform schools.
Redoing the above reform calculations with the inflated numbers from day school families the Reform generations become 100, 108, 116, & 125.
While their original Conservative numbers were 100, 66, 44, & 29, correcting the divide by 2 error & using the inaccurate rates from the day school survey results in growth of: 100, 114, 148, & 168.
The other thing that fascinates me is that they claim 28% of intermarried families raising their children as Jewish in a 1990 population study. That percentage definitely seems to be rising. This explains what people might have been very concerned with intermarriage in the recent past, but it’s considered less of a demographic tragedy today.
Ah, thanks. I didn’t see the 28% before, and when I tried to reconstruct their numbers, I got to something close to it (but not quite as close as you got, which I just attributed to rounding error) by taking (population size/2)*(avg children/woman)*(inmarriage rate), where the inmarriage rate isn’t just (1 – intermarriage rate), because I assumed (in order to get the numbers to work) that intermarriage rate wasn’t defined as “percentage of Jews who intermarry” but as “percentage of marriages involving a Jew that are intermarriages” (which I’ve seen elsewhere), so that a “50% intermarriage rate” actually means that 33% of Jews are intermarried, and 67% are married to each other. (Of course, there are Jews who don’t marry anyone! But this can be accounted for with the average number of children per woman. But there’s still the divide-by-2 problem as you point out.) But your way makes more sense.
The other interesting thing is that they took the secular, conservative, & reform childbirth numbers from a national population survey, which probably includes families without children and people who never married, but the orthodox numbers are from a 2000 census of 8th graders in Jewish day schools where every respondent has children!
Wow. Amazing.
I think that Birthright has a huge impact! At least it did on my life. Although I am going to intermarry – after coming back from birthright I knew I had to raise my children Jewish (my fiance agrees). And I was raised by my NON jewish parent only and was raised secularly.
It had a huge impact on my jewish identity and I thank G*d everyday for the opportunity