Paul Sullins, defending a flawed paper by Mark Regnerus in 2013.
The Federalist website is giving this research significant play:
The study by sociology professor Paul Sullins found that “[a]t age 28, the adults raised by same-sex parents were at over twice the risk of depression as persons raised by man-woman parents.” In addition, there was an “elevated risk associated with imbalanced closeness and parental child abuse in family of origin; depression, suicidality, and anxiety at age 15; and stigma and obesity.”
Given these findings, Sullins concluded that “[m]ore research and policy attention to potentially problematic conditions for children with same-sex parents appears warranted.” This study is significant, Sullins writes, because other studies that have “reported ‘no differences’ in well-being” most often use “psychometric measures of depression or anxiety,” which has led to “a lapse in policy attention to the potential needs of such children.” Sullins’ research challenges the “benign findings” of these other studies.
“Reanalyses have confirmed, not surprisingly, the presence in such samples of strong ascertainment bias, social desirability bias, and/or positive reporting bias” in studies that have concluded there are no differences between children of same-sex couples and those of opposite-sex parents.
Let’s go to that paper, shall we? First of all, the good thing here is that the paper, published in the Egyptian journal Depression Research and Treatment, is available online for free. That’s also the bad thing, which I’ll get to later. Second, this paper has many of the same problems with the widely-panned 2012 paper by Mark Regnerus that purported to show that children of parents in same-sex relationships fared significantly more poorly than children who were raised in homes by their biological opposite-sex parents. In fact, Regnerus’s paper found no such thing, although he did his best to make his turd look nice and shiny.
Like Regnerus’s paper, Sullins says that he based his research on a nationally-represented sample from the US National Survey of Adolescent to Adult Health. He combed through 15,701 respondents, at ages 15, 22, and 28. But like Regnerus, Sullins quickly ran into a problem. Remember, Regnerus could only find two same-sex couples in his sample who had actually raised their children as a couple. Two! So he stacked the deck by re-defining “children,” “raised,” “by,” and “same-sex couples.” Sullins had a similar problem: out of a sample of 15,701 respondents, he could only find 23 adolescents raised by just 20 same-sex couples, “consisting of 17 lesbian partners and 3 gay male partners.”
And what do we know about those families? Not much. We don’t know how much time those children spent with their same-sex parents. We don’t know whether they spent the most formative periods of their lives with opposite-sex parents who then got divorced, or whether they were children of a single parent who had gone through multiple parters over time. We know nothing about the stability, or lack there-of, of either the opposite-sex or same-sex parenting that these children experienced.
But let those numbers sink in: Sullins is basing his entire eight page on those tiny numbers. Twenty same-sex couples.
Out of how many? In 2013, Gary J. Gates at the Williams Institute combed through census data and found that “more than 111,000 same-sex couples are raising an estimated 170,000 biological, step, or adopted children.” Sullins found just twenty out of 111,000 couples. That would be about 0.018% of the total. Which means that if this were a poll, Sullins’s margin of error would be, at best, plus or minus 18.4 percent at a 90% confidence level.
Click to enlarge.
But let’s also look at it a different way. The Standard Error (SE) that Sullins calculated for outcome measures of the children of those twenty same-sex couples is huge when compared to the the data set for opposite-sex couples. What this means is that it only takes one or two individual adverse scores coming from one or two dysfunctional families to throw off those averages. When you see such large swings in the data, you know right away that you need a larger sample to get a clearer picture of what’s going on. Any small sample can sweep up significant anomalies that diminish or disappear once the sample size gets larger. A sample size of 2,000 gay- or lesbian-led families could decrease that standard error by a factor of ten. At least then, you’d start to look at something that can approximate the rest of the 109,000 same-sex couples raising children. Running the numbers again, an opinion poll of 2,000 respondents would have a margin of error of ±1.8%.
How does this scale up to the 111,000 same-sex couples raising children? Well, because it’s supposed to be a peer-reviewed paper — more on that in a moment — Sullins covers himself here somewhat:
Limitations. Despite the signal strengths of Add Health as a large nationally representative longitudinal dataset and notwithstanding the strong significance for contrast effects reported above, the very small size of the sample of children raised by lesbians imposes important limits and prompts great caution regarding the conclusions of this study. As with all observational studies, causal inference is not possible. Moreover, many subtle distinctions and pathways of influence simply cannot be addressed with only 20 cases, and unobserved differences between the parent comparison groups may well confound some or all of the child differences observed. In particular, the lack of useful measures for parent mental distress, depression, family history of violence, alcohol consumption, and substance abuse precluded examination of important familial risk factors which may be associated with child distress. For these reasons, the findings of this study should be considered only provisional and exploratory until and unless they are confirmed by further research.
I wonder if Sullins would be so careful when he’s interviewed about this. Somehow, I doubt it. After all, he’s a research fellow at the Marriage and Religion Research Institute, which is an arm of the Family “Research” Council.
So how did a paper with such sweeping conclusions make it into a professional journal? Well the biggest problem with professional publishing these days is that there are literally thousands of medical and social science journals around the world begging for papers to fill their issues. A few are influential because they can attract the best of the best. And because they can attract the best of the best, they can by very selective about which papers they publish. Many of them reject far more papers than they publish. Other journals however are left fighting over scraps.
Cairo-based Hindawi Publishing Corporation, which publishes Depression Research and Treatment, has been criticized for using spam email to solicit manuscripts (PDF: 111KB/2 pages). What’s more, it’s a pay-to-publish journal, charging its authors about US$1,000 as an “article processing fee.” Truly reputable journals don’t rely on charging authors to submit their manuscripts for their profits, not when they can charge a subscription or they sell the articles on a per-article basis because the quality of their content justifies the price.But Hindawi is among the new class of “Open Access” free journals, which advertises its bug as a feature. It gives away its articles for free, in exchange for charging authors exorbitant “fees” on the front end. Naturally, this creates an incentive to publish more articles already paid for by authors –especially from those who can’t get their papers published elsewhere — with little regard to whether the article has any merit in the first place.
And Hindawi goes a step further. They increase their profit margins by not employing editors for their journals. Editors fill a critical function. An editor is ordinarily a recognized subject matter expert who can act as a gatekeeper to ensure the quality of the journal’s content. More critically, that editor is also tasked with overseeing the peer-review process, which involves knowing which reviewers are qualified to review a paper. Instead, the editorial role at Hindawi is handled by staff members at the company’s headquarters in Cairo, which leaves the most critical task of peer review in the hands of those who may know little or nothing about the paper’s subject. That is, if Hindawi actually has a peer review process. Many open access journals use the practice of publishing now (and collecting the author’s “processing fee”) and letting people ask questions later.
Those problems at Hindawi are evident not only with Sullins’s paper itself, but with the overall reputation of Depression Research and Treatment. Among the 1,061 psychology journals listed at Scimago Journal and Country Rank, Depression Research and Treatment is ranked at number 380. From a top-to-bottom perspective, they can at least brag that they’re in the top half. Unfortunately, Hindawi’s poor publishing practices have become far more common in the publishing world as newcomers to professional publishing scramble to find manuscripts to publish.
So the real question is this: what do other professionals think of the kinds of papers that Hindawi publishes in Depression Research and Treatment. One key measure is to count how many times other researchers cite papers in a given journal when they’re writing their papers. Articles in the top twenty psychology journals are cited, on average, 8.8 times for each paper. For the top fifty, each paper on average gets 6.1 cites. For Depression Research and Treatment, that average is just 1.7.
So there you have it: Sullins had to pay $1,000 to publish a flawed paper using flawed methodology in a pay-to-publish journal with no editor to oversee a questionable peer-review process, and that is generally ignored by his peers.
