Wednesday, May 29, 2013

Mark Regnerus methodological failures and breach of AAPOR ethical guidelines in NFSS

One need not look past the high school level methodological failures in this "research" paper to discard it as the toilet paper it is. I hope that if he ever bothered to join the professional organization tasked with standards in social science research that they have removed him from the roles for gross intentional breach of ethics and scholarship.

The first article I reviewed was published in Social Science Research 41, no. 4(2012) by Mark Regnerus, "How Different are the Adult Children of Parents Who have Same-Sex Relationships? Findings from the New Family Structures Study."

The target population for this survey was adults in the United States from the ages 18-39. The data was collected by Knowledge Networks, described as an online survey company that maintains a panel randomly recruited via phone and mail surveys with no self selection allowed. The sampling frame does not exclude those without phones or computers/online access. Computers and/or online access were provided when necessary. Details regarding response rates ,completion rates, panel recruitment are not included in this article, but readers are directed to contact Knowledge Networks for the information. Selected demographics from the sample are compared to the Current Population Survey, National Longitudinal Study of Adolescent Health, National Survey of Family Growth, and the National Study of Youth and Religion. Not being familiar with any of those surveys, the chart is not particularly helpful for me in the scope of this assessment.

The author's intent was to compare outcomes of children raised by non-heterosexual parents to children whose birth parents (both contributed genetic material) never divorced to outcomes for children of gay and lesbian parents. This necessitated locating children who were raised by gay fathers and/or lesbian mothers, and in an attempt to do so a screening process was applied to the larger sample to increase the numbers of those sub-populations. The screening process lasted from July 2011 and February 2012, in order to attempt to locate current panelists and recruit new panelists. The screening process is not described. Former panelists were also contacted to attempt to find more respondents for the sub-populations of children who were adopted before the age of 2, as well as those who could answer the question "From when you were born until age 18 (or until you left home to be on your own), did either of your parents ever have a romantic relationship with someone of the same sex?" affirmatively.

It is reported that after screening a total of 15,058 panelists, 2988 total surveys were completed with the author's key sub-populations of interest including 163 respondents coded as LM (respondent had lesbian mother) and 75 coded as GF (respondent had gay father), and 101 respondents reporting that they were adopted before the age of 2. It is not clear how the quota for sub-populations was determined, or if time constraints finally determined the cutoff. No response rate is included, and it is not made clear how large the potential pool of respondents was following the screening process. A within survey response rate of 65% is reported as an average for Knowledge Network surveys, but none is reported for this particular survey. The survey was completed by respondents online, with computer access provided for those who needed it.

The sponsor of the project is not listed, and incomplete information is provided for all of the funding sources. Two key sponsors are identified as the Witherspoon Institute and the Bradley Foundation. The design team was assembled from family researchers from Penn State, Brigham Young, San Diego State, University of Virginia, as well as the hosting university and home of the principal investigator, University of Texas at Austin. The external consultants are not identified, but the reader is assured that the funding organizations were not involved in any way with the survey, beyond providing financial support.

This report becomes somewhat problematic once best practices are addressed. The purpose of the study seems to be to address results of existing (mostly qualitative) research that show little or no difference in outcomes for children of non-heterosexual parents when compared to their counterparts who were raised by heterosexuals, by collecting a large, nationally representative random sample. A further concern seems to be the absence of studies that show that gay and lesbian parents are qualitatively inferior to heterosexual parents who both contribute genetic material to the child. The dataset was meant to gather information from adult offspring of gays and lesbians, 18-39, about their experiences, as opposed to existing studies that interviewed parents who self identified as being part of a gay or lesbian headed families. The youngest respondent turned 18 in 2011. Unfortunately, the concepts are poorly operationalized in the design, and although data was collected, it does not capture what the author claims to be looking at. Despite the recognition in section 1.1, that "Any claims about a population based on a group that does not represent it will be distorted ", that is exactly what the research design and even more so, the report proceeded to do.

The author has put all respondents into 8 different categories, that he refers to as family-of-origin structure/and or experience. These categories are based on the now adult child's answers to screening questions and measure the respondent's perceptions. IBF – intact biological family , LM – lesbian mother, GF -gay father, Adopted, Divorced, Stepfamily. Single parent, All Others. In creating these variables, he is conflating family structure or configuration – two parent ,one parent, married, divorced, adopted – with categories of LM and GF, which are based entirely on an affirmative answer to the question

''From when you were born until age 18 (or until you left home to be on your own), did either of your parents ever have a romantic relationship with someone of the same sex?'' Response choices were ''Yes, my mother had a romantic relationship with another woman,'' ''Yes, my father had a romantic relationship with another man,'' or ''no.

In order to boost the numbers in those hard to find categories of LM and GF– second order information on a child's understanding of their parents' "romantic relationships", categories were made mutually exclusive – with all respondents with "lesbian" and "gay" parents being placed in the LM and GF categories regardless of the family structure (single, married, adopted before the age of 2, divorced, other). On top of the conceptual error of treating the LM or GF category as a family structure, it also created only one category – the LM category – made up of exclusively female headed households. The rest of the 7 categories are a mix of male and female headed households. When discussing the sub-optimal outcomes that seem most pronounced for children in the LM category (and contradict the majority of existing research), there is no recognition of this fact, nor the possibility what is actually being measured is the difference between single female headed households and every other configuration including male headed households.

I find the language "romantic relationship" to be problematic, a notion with no clear cut, universally understood meaning. What is described by the author as a key population of interest (children of non heterosexual parents) is constructed using second order information in response to a poorly worded question. The findings report that more respondents mothers everhad "a romantic relationship" with another woman than were adopted, as well as more than children of divorced parents. At one point in the report the number of respondents coded as LM is 175, later in the paper it is 163. Twelve respondents report that both their mother and father had ever had a romantic relationship with someone of the same sex and were coded to increase the numbers in the so called gay father category.

Although the author acknowledges that comparing outcomes of households supported by single mothers to two parent /never divorced households is unfair, that is the method of analysis that is employed – in fact it seems to be the focus and purpose of the study. A list of what is referred to as outcomes are run against LM and IBF categories, and the author cautions that causation cannot be assigned, he simply wants to "highlight the differences" between the outcomes reported. Some of the concepts categorized as outcomes seem more appropriate as demographic or influences rather than outcomes. There is a set of questions regarding inappropriate sexual contact All respondents were asked if ''a parent or other adult caregiver ever touched you in a sexual way, forced you to touch him or her in a sexual way, or forced you to have sexual relations?'' Possible answers were: no, never; yes, once; yes, more than once; or not sure. A broader measure about forced sex was asked before it, and read as follows: ''Have you ever been physically forced to have any type of sexual activity against your will?' It is implied that the perpetrator could be identified (possibly) by who the child was living with at different times of their lives, but the question does not allow for that process. The question is overly broad – caretaker could be understood as anyone from grandma to someone at a daycare or other temporary custodial situation, and does not ask the location or time period in which the assault occurred. It is beyond the scope of this assignment to address the tone and politically motivated assumptions throughout the report, but it was at times difficult to wade through to find questions of design and process.

I am unable to answer questions of the cost/benefit analysis of the design, whether the questionnaire was pre-tested, or answers regarding data collection methods during the screening survey, or training of Knowledge Networks staff. It is unclear how many respondents were contacted for the actual survey, and I cannot assess the response rate, nor the methods used to convert refusals. It is unclear how, as it is with mail surveys, how to determined if the email invitations were ever seen by the potential interviewees.

The author recognizes that it is difficult to find the population the report wanted to study, and that he could not say that the coding of LM and GF reflect the sexual orientation of the parent, or even the child's perception of their parent's sexual orientation. His fix was to code them as such regardless, and report the findings as data on Lesbian and Gay parents. I found his designations to be confusing in the text, speaking at times of same-sex parent (implying a couple) when there is only one parent being referenced, and at times sounding as if the data reflects the parent, rather than the child of a parent. The underlying assumption being re-enforced throughout the study is that heterosexuality is the only acceptable orientation, and homosexuality is a negative, chosen behavior, and I believe it led to ambiguous and obfuscating language and bad design choices.

More information I feel would be necessary to assess the analysis is a description of the weighting variables. The reader is told that "each case in the NFSS sample was assigned a weight based on the sampling design and their probability of being selected", but not what those might be. Also, one of the control variables in comparisons between children whose mother may have kissed a woman at some point and those who grew up in a family headed by their still married heterosexual (both parents contributed genetic material) was household income. This variable is a second order report of what a child perceived their household income to be when they were growing up. There are too many conceptual /operational disconnects in the variable definitions for me to see this survey to be particularly useful as a scholarly instrument as it exists