The Slow Work of Evaluating Abstinence Education
Does abstinence-only sex education work? Buzz Pruitt and Pat Goodson, two professors of health education at Texas A&M, can’t say. Not yet, anyway. In the last two years, they have received nearly $360,000 from the Texas Department of Health to do a rigorous scientific evaluation of the state’s 31 so-called “abstinence contractors.” “Every-one wants to know the bottom line,” Pruitt said. “We don’t have that information yet.”
They’re not alone. Since 1997, over half a billion in federal funds has flowed into the states to teach abstinence. President Bush has long been a backer of such programs. “We must convince youngsters to resist opening the Pandora’s box,” he told an audience at the 1999 Governor’s Conference on Right Choices for Youth, “not just to avoid death or disease, but to embrace a life that is physically, and morally, and emotionally healthy.” In his 2003 budget, Bush requested an increase of $33 million for abstinence education, for a total of $135 million–all of it without a lick of scientific evidence that these programs do what people want them to: lower the teen pregnancy rate and reduce sexually transmitted infections.
What’s taking so long to get answers? Measuring the outcome of a public health intervention like abstinence-only is always tricky, as much for political reasons as for methodological ones. Fortunately for Pruitt and Goodson, their funding comes from the Texas Department of Health’s (TDH) research division, not the abstinence program itself, so they are objective and somewhat insulated. But not entirely. They’ve had to build relationships with suspicious contractors, who tend to look on them as “TDH police” and who, all things considered, would rather see the money spent on their programs.
“You have to look at the fact that there’s got to be some conflict of interest in an organization that also funds family planning programs,” says Marilyn Ammon, director of the McClennan County Collaborative Abstinence Project (McCAP) in Waco. Ammon, who has called safe sex campaigns conspiracies to create customers for condom manufacturers, is considered something of a hardliner in the community. Back in 1999, the Waco ISD school board refused McCAP’s services, arguing that their existing sexuality curriculum–which teaches contraception–works better than Ammon’s hardline version of abstinence-only-until-marriage. The combative Ammon even called for an audit of abstinence money–which stopped Pruitt’s work for several months–because she didn’t trust TDH.
Ammon isn’t the only one suspicious. “Philosophically, they’re not a good fit,” says Starla Kelley, a contractor in Amarillo, referring to Pruitt and Goodson. “They came in with a preconceived notion of what abstinence was, and that it didn’t work.” She does welcome the opportunity to find out what parts of her program can be improved. But she’s not happy that a prominent researcher named Doug Kirby is a consultant to the evaluation. Widely considered in the abstinence community to be a “condom pusher,” Kirby’s name is anathema in some circles.
Pruitt and Goodson showed up “four years after that train left the station,” as Pruitt likes to say. As a consequence several contractors had already begun collecting data on their own, but in an unscientific mish-mash. “How are they going to get us to abandon the efforts we’ve already come up with?” says Bonnie Auburn, the director of a 10-county abstinence-only program in Paris, Texas. “They’re going to have their work cut out for them.” McCAP in Waco and Worth the Wait in Pampa have already hired Jeff Tanner, a professor of marketing at Baylor University, to evaluate their programs.
One of Pruitt and Goodson’s first steps was to spend about $10,000 to purchase 40 different sets of abstinence-only curricula, which are used in various ways by the state’s 31 contractors. It is now the largest such collection in the state. In a small white office in College Station, two floor-to-ceiling bookcases are crammed with large plastic tubs, each of which contains an abstinence-only curriculum and its teacher’s guide, activity packets, videos, and games. The curricula have titles like, “Baby, Think it Over,” “Not Me, Not Now,” and “Character Counts,” and they’re not cheap–prices ranged from $350 to $1,500, highlighting the business side of abstinence-only. Three schoolteachers spent a summer evaluating each curriculum, then assigning it a quality score. As a sign of the times, they borrowed criteria from both sides of the teen sex debates–from ETR Associ-ates, the California-based educational publishing company where Doug Kirby works, and from the pro-abstinence-only Medical Institute, based in Austin. The move is less ecumenical than it appears: Without its point of view represented, each side is more apt to cry foul.
Next the researchers broke down the contractors’ 31 grant proposals to TDH, which represented a variety of approaches–some work in schools, others in health care settings, while some gather youth after school or educate parents. Pruitt and Goodson boiled it all down by asking one question: How will the programs teach and promote the factors that lead to sexually abstinent behavior? To this end the researchers produce a map of all the various “inputs” to abstinent behavior, which they gathered from the scientific literature in the field. A student questionnaire will help the researchers evaluate how well the programs enhance each of these inputs for students.
Developing that questionnaire has proved to be no small task. Each question has had to run a gauntlet of interested parties. Many parents object to “graphic” terms on questionnaires, such as questions about anal and oral sex. “That’s what gets everybody’s dander raised,” says abstinence contractor Bonnie Auburn. “Even though all the reports we get from kids is that it’s going on in the parking lots.” Nevertheless, conservatives remain attached to the idea that children have a fragile “latency period,” and that puts them in conflict with evaluators who are obligated to ask explicit questions about sexual behavior.
Pruitt and Goodson deleted questions about oral sex on a middle school questionnaire. “Both the [sex education] field and TDH felt they were too ‘frank’ and would not yield high quality data,” Pruitt says. On the other hand, there is support for asking such questions of high schoolers. “We are all–including the Governor’s office–interested in getting to the question of whether youth are participating in activities such as oral sex as a means of preserving ‘virginity,’” Pruitt says.
Others in the abstinence community insist that the order of questions can bias a test. If you’re teaching abstinence-only-until-marriage, they argue, the question about intending to remain abstinent should come before the one about their sexual behavior. Some don’t want questions about contraceptive use, even if such questions are motivated by public health concerns. “If I’m really teaching abstinence-to-marriage,” says Amy Stephens at Focus on the Family, “I’m not interested in the success of the condom message.” Focus on the Family, an influential, multi-million dollar conservative Christian organization based in Colorado Springs, is run by nationally known radio personality James Dobson. The group, which claims millions of listeners across the heartland, recently thanked Bush for his increased funding support for abstinence-only education.
To answer such objections, Pruitt and Goodson say they’re combing their questionnaire for phrasings that might bias the data, vetting questions with the Governor’s office, clearing them with the contractors, and checking them with experts. Even then, the gauntlet’s not over. Before they field-test the questionnaire, they plan to review the items with parents, although they won’t get to see the entire questionnaire. Once they pass that trial, no item is secure, even if it’s scientifically sound. “We will decide on an item-by-item basis whether to keep, modify, or toss,” Pruitt says.
Once they get a test that works, they have to find groups willing to take it. Many public schools don’t want to participate in sex research. Pruitt and Goodson will have to secure six levels of permission to give their questionnaire: two institutional review boards to clear human subject test protocols, the school board, the superintendent, the principal, the parent, and the student. If they receive a third year of funding, they plan to begin collecting data in the fall of 2002. Until then, they are resigned to talking a lot about how they don’t have the bottom line. “There’s no story in, ‘We don’t know,’” Pruitt acknowledges.
Nevertheless, Pruitt and Goodson now know more about the abstinence community–or culture, as Goodson prefers to call it–than they once did. For many health educators, the abstinence movement is easy to criticize as theoretically unsound. In fact, Goodson says, though most of the contractors aren’t trained health educators, their work is not without theoretical footing. Pruitt and Goodson were surprised that the contractors’ map of inputs to abstinence largely matched that of professional health educators. The one big exception, they found, is that the contractors tended to have an inordinate fondness for supposing that when kids know the risks of sexual behavior, they won’t have sex. “The notion that knowledge equals behavior is not supported by the literature,” Goodson says.
What’s also been surprising, they say, is how heterogeneous the abstinence culture is in method, motive, and previous experience. One contractor is a mother of three. Another is a former English teacher. Several gave up productive careers elsewhere to start non-profits. “These are good folks, well-meaning, and well-intentioned,” Pruitt said. “Most of them are people you wouldn’t mind having dinner with.” Only a few of them, such as Marilyn Ammon, are politically active.
Pruitt and Goodson also found that the contractors don’t share a uniform definition of abstinence. For some, it means eschewing all risky behaviors, not just sex; for others, it means teaching kids to embrace smart decision-making. In other words, not all definitions of abstinence treat sex, or sexual feelings, negatively. But this is overshadowed–and perhaps explained–by the discovery that many abstinence programs aren’t actually sexuality programs at all. While they hold dances for teens, they don’t teach the basic “facts of life”: how babies are made, how puberty works, all the interesting ways that women are different from men. “They might be youth development, they might be character development,” Pruitt says, “but it’s an absolute mistake to call it abstinence-only sex education.” Goodson, a trained sexuality educator, says she raises the point with the contractors. “Do they expect that kids on their wedding nights will suddenly become experts on sex?” she asks rhetorically.
To be fair, some programs do teach basic human sexuality, but there’s no explicit charge in the federal program to do so, particularly not to middle school kids, where pregnancies and infections are startlingly high. The absence of a federal position on sexuality education means that local communities have to pick up the slack. In many Texas communities–where the word “condom” sends school boards into paroxysms of disgust–that simply does not happen.
Ironically, Pruitt and Goodson credit Bush for giving them the institutional support to produce a scientifically valid and politically durable evaluation. “As the governor, he was very concerned with validity and also willing to give us time. They genuinely want to know whether this stuff works.” It makes sense: The CEO governor (now the CEO president) wanted to make sure that taxpayer dollars were well spent. The problem is, there’s no guarantee that anyone will listen to the science, no matter how conscientious the scientists have been. Pruitt and Goodson both acknowledge that evidence, or lack thereof, has little bearing on policy and law–look at the original 1996 welfare reform act, which started the federal push for abstinence education in the first place. For right now, they don’t think about how the best scientific findings aren’t safe in the funhouse of politics. They just want to do their job right. “Our motive is to publish papers,” Pruitt said, “and you can’t publish crap.”