This book focuses on evaluative strategies that, in effect, explore the translation of plausible concepts into functioning policies and programs. The techniques further evaluate how well the policies and programs work in all their administrative and management dimensions and, in some instances, examine how the policies and programs are experienced both by intended beneficiaries and by those who carry them out. This group of techniques is known by a variety of names: implementation studies, process evaluations, discrepancy analyses, and even qualitative studies. The various labels connote subtle differences in the underlying research question. For example, implementation studies are thought to focus on newer policies and programs, whereas process evaluations look at their more established counterparts. But the lack of common terminology can be confusing. Because much of the content of this book focuses on the implementation of recent changes in welfare policies and programs, we label the entire array of techniques implementation analysis throughout the book.

Purposes of the Book

The book has three salient motivations. These are to encourage a dialogue about developing a conceptual framework for understanding the role of implementation analysis, including its scientific and intellectual contributions; to bring together recent advances in implementation analysis that have yet to be widely recognized as having important common characteristics and goals; and to recognize the importance of timing. Passage of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) in 1996 accelerated fundamental policy and program changes in the social welfare arena in ways that are transforming the institutions and organizations underpinning our systems of social assistance. Arguably, this revolution substantially increases the importance of implementation analysis in the program evaluation world.

The Translation of Proposal into Program

Public programs translate policy intent into action in ways designed to achieve specified goals and reach intended target populations. Among other research questions of interest, program evaluations provide the means for learning whether programs have been implemented as intended by the policy, how they actually work, how well they are achieving the goals envisioned for them, and how well they might work in other environments. Program evaluations have two distinct conceptual dimensions: implementation analysis and impact analysis.

Implementation analysis seeks to understand the program in its own right. It does so by exploring a number of fundamental questions about how systems function—what the real (as opposed to nominal) goals of the policy are, whether those goals are internally consistent and shared among multiple stakeholders, what administrative and management procedures are engaged in the pursuit of the goals, whether they reflect the intent of the program designers, how well the program is achieving its goals, how policy is changed by frontline staff decisions, and whether clients are reacting to the program in ways intended by its designers.

Impact analysis encompasses the research question most commonly associated with program evaluations—whether the program or policy has the desired effect on critical outcomes. In the welfare reform context, the traditional outcomes have been economic—hours worked, earnings, welfare receipt—but they have also included social outcomes such as marital stability and fertility decisions (U.S. Department of Health and Human Services 1983). Because impact analysis is designed to attribute causation, this approach is in essence comparative. It requires a measure of the counterfactual—what would have happened to the outcomes of interest if the program had not existed or if different services, policies, or programs were in place. The gold standard of impact analysis is a design that randomly assigns members of a target population either to the program being evaluated or to control (counterfactual) status. The outcomes of the program being evaluated can then be compared with the outcomes under the alternative program configuration, and the net impact derived. These experimental designs, if done well, typically can deal with selectivity issues (e.g., only motivated individuals "choose" to participate in a welfare-to-work program) that bedevil causal attributions in most nonexperimental analyses.

But impact analyses, even when using rigorous experimental methods, are not absent cautionary concerns. Unless the impact of a program is large, dramatic, and immediate, net impact evaluations need data collected at regular intervals over a considerable period from a large number of participants to reliably assess program-induced changes in behavior. Moreover, it is often difficult to control cross-group contamination—the spillover of an intervention to the control group. In addition, there are some interventions that intend to alter the policy signals experienced by target populations. Random assignment, in most cases, dilutes the new signals by making exposure to the new program more like a lottery—people might get to play by the new rules or they might not. Both contamination effects and diluted signaling effects result in a conservative bias in measuring impacts.

One of the most persistent cautions attached to conventional experiments has to do with the difficulties associated with untangling the causal chain. For example, if we do not obtain significant experimental effects, that is, non-chance differences between the experimental and control group, we cannot really be sure that the intervention is a failure. Perhaps the new policy or program was never really implemented as intended, or was implemented in ways that the sponsors would hardly recognize. On the other hand, positive findings can also leave us with unanswered questions. Which component of an intervention might be disproportionately affecting outcomes? Is the nominal policy or program change really causing the effect, or is the cause some associated change in the way things are done outside the "experimental" intervention? Is there a way to tease out evidence that different dosages (strength of the intervention) or delivery mechanisms or contextual attributes would enhance performance? These questions, and many others, require that we peer into the "black box" of program and policy implementation—the way things really work.

When done well, implementation analyses can supplement and complement impact evaluations. They can yield more immediate evidence—suggestive evidence on program-induced change and rather detailed hypotheses on how different components of the program might contribute to overall effects. But as we shall see in the subsequent chapters in this work, implementation analysis encompasses a variety of research questions and techniques that make unique contributions to our knowledge base. And although the range of questions pursued and techniques used is varied, good implementation work should be rigorous and should conform to the accepted standards of science.

In short, both implementation and net impact evaluation techniques are essential weapons in the evaluation arsenal necessary for a comprehensive understanding of a policy or program, each with its own role to play.¹ The methods and standards for impact analysis are well established, however, whereas major recent advances in implementation methods still need to be compiled into a systematic body of knowledge. This book begins to fill that gap, using as its primary examples implementation approaches as applied to evaluations of the 1996 welfare reform initiative. The next two sections of chapter 1 provide context for subsequent chapters. We shall first review the changes that helped produce the new reform environment and then summarize the welfare evaluation history of the 1980s and 1990s.

The AFDC Waiver Programs: A Brief History of Evaluation Expectations

The growing importance of implementation studies in welfare research reflects the revolution experienced over the past decade or so in the way we deliver social assistance.

The welfare reform ideas of strengthening incentives for recipients to work, paying more attention to preparing them for work, and instituting work requirements for cash recipients and/or penalties for noncompliance had been germinating at the federal level and in state capitols well before PRWORA's passage in 1996. States were eager to put these ideas into practice. Federal officials wanted to learn about their impacts without exploding the federal welfare budget. Out of these mutual interests was born an era of waivers where variations of certain federal laws and regulations governing Aid to Families with Dependent Children (AFDC) were systematically explored. Under that system, the U.S. Department of Health and Human Services granted waivers to states to implement changes to their welfare systems under demonstration conditions, with the requirement that the waiver programs be rigorously evaluated.

The waiver program allowed two important things to happen. First, it devolved some decisionmaking to the states while retaining some control of federal matching funds because waivers required that states not spend more on their new systems than they would have under their previous AFDC program. The matching fund issue is important here. The AFDC program had always provided an open-ended subsidy to states that mitigated the financial consequence to the state's budget of its own decisions regarding its AFDC program. It needs to be remembered that state variation did not start with PRWORA, or even with the waiver program. State welfare programs had always had a hand in setting their own eligibility rules and benefit levels above federal minimums, and the federal subsidy had always been open-ended (i.e., with no federal limits on the total amount of matching funds a state could receive for its AFDC program). That is unlike the block grant system now in place, which provides a fixed amount of federal funding to states. The block grant amount is based on a state's AFDC caseload in the mid-1990s, in effect shifting fiscal responsibility to the states if caseloads and costs were to dramatically increase.

The second thing the waiver program accomplished was to relax the principle that all members of the target population in a state should be treated the same. This change permitted states to implement their reforms for particular groups of welfare recipients and in particular parts of the state. Therefore, the federally mandated evaluations could compare the outcomes for recipients of the waiver program with a good measure of the counterfactual—outcomes for recipients with the same characteristics on average as waiver participants who were still under the state's regular AFDC program. Similarity between the two groups was ensured through a client selection process that assigned AFDC caseload members or applicants randomly either to the waiver program or to the traditional AFDC status in that state. This is the classic standard for impact analysis, as noted earlier in the chapter. Random assignment ensures that the two groups do not differ systematically in characteristics that could affect program outcomes (Orr 1999; Rossi and Freeman 1993).

The waiver evaluations contributed valuable information on the average net impacts of the specific waiver programs. Because the basic components of the AFDC program were similar across states, suggestive conclusions could also be drawn about the differential impacts of different state waiver programs. The conclusions about differential impacts, in turn, allowed conclusions about which types of waiver programs were the most effective in producing the desired impacts on work effort, earnings, and welfare receipt.² The waiver evaluations also taught us more than we knew before about the impacts of particular components of some of the programs—such as various job search assistance techniques, positive and negative financial incentives tied to certain behaviors (the latter called sanctions in the welfare context), mandatory versus voluntary program participation, and generous versus less generous earnings exemptions (called disregards in the welfare context), among other things.

But the impact findings were years in the making and did not always identify the particular processes through which waiver interventions actually influenced the outcomes of interest, that is, which elements were the most (and least) powerful parts of the program being evaluated. Knowing that a combination of program factors together led to an average program impact, but not having the information necessary to identify which factor(s) caused the impact (or, indeed, whether the intervention as implemented was close to what was intended, either in design or in intensity) illustrates the black box issue introduced earlier.

The waiver experiments used the combined efforts of implementation and impact analysts to address the black box problem. But the implementation parts of the analysis were typically not funded generously enough to provide rich process information. They were also, for the most part, done in isolation from one another and without any general analytic consensus about how they should be done. Further, the results of the implementation components of the evaluations were given considerably less public attention than the impact results and were virtually ignored as having potentially important contributions to make, in their own right, to the public policy debate.

In the current evaluation environment, impact evaluations are virtually never mounted without an implementation evaluation component. But the implementation work is too often done by analysts who have been trained primarily in impact evaluation techniques and have learned whatever they know about implementation analysis on the job. Some of the most intensive implementation studies, in contrast, have been done with no impact component, by evaluators who not only have little experience with impact studies but also often have little interest in them. The evaluation community now has the task of identifying the complementarities of the two approaches and highlighting opportunities for them to come together.

The New Policy World

PRWORA formally ushered in the new welfare policy world presaged by states during the waiver era of reform. It made important changes to the Supplemental Security Income (SSI) program for children, Food Stamps, the child support enforcement and child care systems, and child nutrition programs. Other changes were directed at policy domains that intersected existing programs and policies, for example, restricting assistance to immigrants and reducing nonmarital births. Most policy and public attention, however, has focused on the Temporary Assistance for Needy Families (TANF) program, which has replaced AFDC, the primary cash income support system for poor families with children that was established as part of the Social Security Act in 1935.

TANF is a block grant program that gives states substantially increased responsibility, within more limited federal rules, both for development of their own welfare programs and for the budgetary consequences of their policy and program choices. The most significant federal provisions include five-year lifetime limits on cash assistance receipt, work requirements for welfare recipients after two years of continuous cash support, and financial penalties for states that fail to meet steadily more stringent federal requirements for the proportion of the cash assistance caseload in an allowable work activity. PRWORA also made substantial cuts in several purely federal support programs—including Food Stamps and Supplemental Security Income—and mandated restrictions on assistance formerly provided to immigrants legally allowed to live and work in the United States (some of the latter cuts were subsequently relaxed).

The sweeping reforms embodied in PRWORA are, however, more than a set of new policies. They are, at least in part, the culmination of a set of profound changes transforming the character of government and the structure of social policy in the United States. These deep currents of change are captured by a new set of three R's: the interrelated themes of redirection, reinvention, and reallocation.

Redirection

The redirection theme involves the transformation of public assistance for poor families with children from a system of cash support on the basis of economic need to one aimed at fostering individual and community change. The goals attached to the new concept of support for poor families have multiplied over time. At the local level, they may include communities coming together to improve parenting, family formation, family functioning, and decisionmaking with respect to fertility and sexual activity. Some proponents of reform also anticipate that the redirection of welfare policy will help heal dysfunctional communities, restore public confidence in social assistance programs, and eventually result in better transitions into adult roles by poor and disadvantaged children.

Reinvention

The reinvention theme embraces the transformation of public management from a focus on inputs and protocols for action (tightly allocated resources for specified purposes and explicitly spelled out procedures for which tasks should be done and how) to outputs (how organizations choose to allocate the resources they have to work with and how well the choices they make accomplish the goals they set). Proponents of the new focus argue that greater freedom for managers to structure the processes by which they shape and deliver services will enhance program efficiency and responsiveness. The reinvention theme also emphasizes accountability for results and working within market forces. Some proponents go so far as to challenge the presumption of a public sector monopoly on the provision of assistance.

Reallocation

The reallocation theme encompasses the shifting of program and policy authority from higher levels of government to levels closer to the problems being addressed, not only from the federal level to the states but also from states to localities. The new flexibility afforded to lower levels of authority, proponents suggest, substantially increases both cost-effectiveness and the responsiveness of programs to need—by putting the decisionmaking closer to the particular circumstances of the need. At the extreme, this theme is reflected in a new sense of professionalism on the front lines of welfare agencies where workers now exercise considerable discretion rather than merely carry out the rules formulated in Washington or the state capitol.

This combination of changes is producing a fundamental shift from a bureaucratic, centralized mode characteristic of an income transfer program to a professional mode suited to programs directed at complex behavioral change. Outcomes of interest in the new policy environment include not only the more traditional economic outcomes—poverty, earnings, work effort, welfare receipt—but a variety of other dimensions of behavior. These, as implied by the redirection theme discussed above, include new client attitudes to work versus welfare, new attitudes on the part of program personnel in helping clients reach long-term independence (rather than issues related to benefit payment calculation), and new attitudes of the nation's poorer parents toward themselves, their children, and the quality of their family lives. Also of importance are child well-being, fertility, and family formation.

The fundamental reforms to the nation's support system for low-income families embodied in PRWORA have provoked intense debate. Advocates predict changes for the better, not only in people's behavior with respect to employment, earnings, and welfare receipt, but also in their psychological and social well-being and, as a result, family and community functioning. Other observers predict changes for the worse, particularly in poverty among families with children and the strains associated with it. A strong public consensus has emerged from the debate that we, as a society, must begin at once to understand the implications of these changes. That task is made more complex by expected major differences among state approaches, given increased state flexibility in determining policy and program design. Because of the pressing need for feedback on such broad trends in society's approach to public assistance, the authors of the subsequent chapters anchor their discussions of implementation evaluation issues and techniques in the new welfare reform context.

Implications for Evaluation

Under the old welfare regime, policy functioned in a more set-piece fashion. Uniform federal rules and processes were modified by states only at the margin, and usually only under selected circumstances. Change was slow and linear. Consequently, the research questions were relatively limited in nature and typically could be answered by conventional experiments. For example, would a new job search policy lead to increased employment and lower welfare utilization?

Now, program and policy purposes, including the institutional mechanisms for achieving these ends, are exploding across the social welfare landscape. In essence, reform has become more of an evolutionary process and less a periodic, distinct event. Over the past several years, state and local officials have been quite successful in moving low-income adults into the labor market. But many recognize this as only the first step in the reform agenda unleashed by PRWORA. New entrants must be kept in the labor market through a variety of work supports; functioning two-parent families need to be nurtured and promoted; and the communities in which low-income families reside must be strengthened.

Early on, more farsighted policymakers considered the ultimate purpose of reform to be the prevention of dependence in the first instance, by promoting independence through prudent investments in children, families, and communities. Today's reform landscape has moved closer to this early vision. For example, in seven upper Midwest states, proportional spending on what has been termed family stability issues increased threefold between fiscal year 1996 and fiscal year 2000. Many of those dollars are directed toward investments in children and youth or efforts to stabilize families. Moreover, the future of reform is even more likely to emphasize family functioning if the goals embodied in the administration's TANF reauthorization proposal eventually become law. The proposal, titled Working Toward Independence, stresses child well-being and healthy marriages as important ends to be pursued by the states.

We can see that TANF agencies increasingly concentrate on complex, behavior-focused interventions, which tend to be dynamic and longitudinal—seeking fundamental change over time in those served. Interventions tend to be multidimensional—addressing several issues simultaneously as they seek to encourage positive behaviors and discourage those that are counterproductive. If welfare workers in the emerging era are to be effective, they must eschew bureaucratic rules and adopt professional norms.

As this happens, the organizations within which workers function will become structurally flatter and less hierarchical. Agency boundaries themselves will become porous as interagency agreements and one-stop models emerge. Malleable and plastic institutional forms that can respond quickly to new challenges, that are more entrepreneurial in their approaches, will supplant traditionally static and risk-aversive welfare systems. Distinct program and funding streams, the "silo" approach to social policy, will be merged into networks of social assistance. Within agencies, horizontal patterns of communication—dialogue among peers—will become more prominent, and discretion at the operational level will replace traditional command-and-control organizational strategies.³

What does all this mean for those responsible for evaluating reform? Basically, a new set of questions now moves to the front burner. We must know much more about the institutional responses to reform, about the new service delivery systems that emerge, and about how policies are filtered through workers empowered with new discretion. At the extreme, policy and rules may be less important than people and organizational cultures in the new world of social assistance—how agencies and workers and clients interact and communicate. In consequence, we must learn much more about the character of interactions at the front lines, how workers and service consumers relate to one another.

These questions are not easily addressed by impact analyses. Rather, we must invest heavily in evaluation strategies that get inside the skin of the new welfare world. This, in turn, requires that we have the methods and strategies to do implementation studies well, with the kind of rigor expected of science. But implementation studies seldom have received the attention devoted to conventional experiments. Often, they are considered little more than an add-on, a nicety to be done as resources permit. But in the new world of welfare, they can easily be seen as an essential analytic tool for understanding what is going on.

Unfortunately, standards for what constitutes good work are not universally accepted. Too often, implementation work is viewed as subjective and interpretive, the purveyors of implementation methods as practicing craft rather than science. We believe it is time to enhance the reputation of implementation work and to improve the quality of that practice. We intend this work as a starting point toward rectifying those ends.

The Content of the Book

The potential of implementation research is clear—as is the pressing need for the evaluation community (1) to define both the strengths and the weaknesses of various approaches to implementation analysis and (2) to promote quality standards. The contributors to this volume have written their chapters with these goals in mind.

• Part One (chapters 2-4) presents the perspectives of policymakers and program managers, provides an overview of the concepts and analytical approaches that guide implementation research generally, and gives a sense of the historical development of implementation analysis.
• Part Two (chapters 5-9) discusses how particular implementation analysis approaches can be used to help understand a variety of dimensions of program implementation.
• Part Three (chapters 10-11) deals with types of data, methods of collecting them, and their utility for implementation research.
• Part Four (chapter 12) concludes the book with a discussion of what the foregoing chapters tell us about how to pursue quality implementation analysis and with a suggested set of priorities for future research efforts.

Together, the conceptual and empirical contributions of the chapters highlight the critical importance of implementation analysis in understanding how social policy is translated into practice, and the fundamental value of this evaluative approach to understanding the range of program and policy changes embodied in recent changes in welfare policies. We believe that the authors demonstrate the utility and centrality of implementation analyses within the spectrum of evaluation research methods.

Notes

1. Some analysts contest this perspective. They argue that the nature of the PRWORA changes has increased the need for implementation analysis but reduced the need for net impact evaluation, because the primary focus has changed from client impacts after the program to compliance during it. For example, see Mead's discussion in chapter 6.

2. Some waiver experiments introduced more than one program variant into the experimental design.

3. Recently, states have been experiencing increasing economic constraints. We are not sure how this will affect state variation in welfare regimes. However, we expect that these changes should increase attention to administrative and management practices that are effective, underscoring the importance of implementation research of the types laid out in this volume.

References

Orr, Larry L. 1999. Social Experiments: Evaluating Public Programs with Experimental Methods. Thousand Oaks, Calif.: Sage Publications.

Rossi, Peter H., and Howard E. Freeman. 1993. Evaluation: A Systemic Approach. Newbury Park, Calif.: Sage Publications.

U.S. Department of Health and Human Services. 1983. Overview of the Seattle-Denver Income Maintenance Experiment Final Report. Washington, D.C.: Office of Income Security Policy, Assistant Secretary for Planning and Evaluation. May: 1-17.

Return to Top

---

Comments and questions may be
sent via email.