What impedes and fosters implementation success when evidence-based programs are applied in real world settings?  This was the overarching question in a two-year study conducted by the Blueprints for Violence Prevention initiative, launched by the Center for the Study and Prevention of Violence (CSPV), Boulder, Colorado.  After a succinct historical review of the literature and research regarding "what works" in the field of criminal justice, the authors note that "what is missing is a systematic analysis of what can make or break implementation efforts across a variety of contexts and programs." 

[Mihalic, S. F. and Irwin, K. (2003). Blueprints for Violence Prevention: From Research to Real-World Settings-Factors Influencing the Successful Replication of Model Programs. Youth Violence and Juvenile Justice, 1(1): 1-23.]

Their study evaluates eight violence-prevention programs, which demonstrated significant decreases in violence/problem behaviors during their research phase, and were identified as "model" programs by Blueprints.  The eight covered various forms of violence prevention measures such as prenatal-postpartum pregnancy, family therapy, foster care, mentoring and school-based interventions.  They include:

1. Big Brothers/Big Sisters of America (BBBSA)
2. Bullying Prevention Program
3. Functional Family Therapy (FFT)
4. Multidimensional Treatment Foster Care (MTFC)
5. Multisystemic Therapy (MST)
6. Nurse-Family Partnership
7. Promoting Alternative Thinking Strategies (PATHS)
8. Quantum Opportunities Program

After site assessments by CSPV to ensure that sites could successfully implement the programs for the long-term (at least two years), 42 sites were selected.  The sites included non-profit community groups, schools, state health agencies and private treatment centers. 

The authors note the conceptual framework for this study draws from diffusion theory (Rogers, 1983, 1995), which holds that innovation occurs in four stages:

1. awareness gained through dissemination;
2. innovation adoption;
3. innovation implementation; and
4. maintenance.

 It is during the implementation phase when an innovative program is put into practice and behavior change is required that most problems arise, according to Rogers (1995). 

Problem Studied

The authors argue that much of the current research in this area is anecdotal, and that findings insinuate that almost "everything matters" in the implementation phase.  Factors they note via the limited research, which are considered relevant to successful implementation include:

• Organizational features of the adopting agency such as structure, history, philosophical traditions, economic saliency, stability, clear-open communication;
• Staff characteristics such as motivation, support, and involvement in program design;
• Adequate financial support for "direct line" staff;
• Strong leadership/key people to motivate, advocate, and coordinate the program;
• Practitioner training
• Community support - especially for school-based implementation;
• Program characteristics such as structure, complexity (i.e., evidence-based, theoretically-supported models) and flexibility;
• Pre-planning.

Measurements

The author-researchers designed a two-year process evaluation study of the aforementioned selected program replication sites throughout the U.S.  The study aimed first to identify research-based programs demonstrating effectiveness outcomes, and secondly to nationally replicate these same programs in non-research settings.  A grant was secured via the Office of Juvenile Justice and Delinquency Prevention (OJJDP) to fund training and technical assistance efforts by the Blueprints/CSVP group at the replication sites.  Both quantitative and qualitative research methods were used for the process evaluation.  The primary goals of the process evaluation were to identify:

• whether a program reaches its intended target population;
• whether it is being implemented as designed;
• which obstacles most interfere with completing implementation; and
• what are the implementers' perceptions of the program's worthiness.

Three data collection/site visits were conducted yearly at regular intervals at each site, in addition to phone visits. The evaluation metrics consisted of implementation factors (independent variables) such as:

1. ideal program characteristics,
2. Community support,
3. number of technical assistance (TA) visits,
4. TA (or training) quality,
5. time to implement,
6. ideal staff characteristics,
7. ideal champion characteristics,
8. ideal agency characteristics,
9. inconsistent staffing, and
10. inconsistent funding. 

The dependent variables (success outcomes) included :

1. adherence to core program components,
2. percentage of core programs components achieved,
3. dosage, and
4. sustainability. 

The measures of the independent and dependent variables were deduced from the literature review, as well as the problem studied. 

Some of the key measurements with brief descriptions included:

• Qualitative (open-ended questions) and quantitative questionnaires, informal interviews during the on-site and telephone visits to the sites to assess how program implementation was going at each site with reference to the dependent variables;
• Summary staff observations after the site visits; 
• Questions were created by examining "the prescribed program delivery" in  Blueprints documentation for design of the model programs;
• Quantitative questions also assessed the aforementioned primary goals of the evaluation (i.e., did programs reach their intended target populations);
• To assess independent variables such as program, community, staff, leadership  and agency, a set of five-point scales were developed and given to site coordinators to rate the effectiveness (asset versus barrier) of each variable. The scales had reliability ratios ranging from .69 to .88 based on Cronbach's alpha.
• A combination of bivariate and multivariate analyses was performed on aggregated site data to determine the significance and strength of the relationships between the independent and dependent variables. Four multiple regression models were created to identify the independent effects of all of the independent variables on the dependent variables. 

Findings

The key findings include (refer to the journal article for comprehensive details):

• The dominating arguments in the literature were corroborated in this study;
• 74% of the 42 sites implemented all core components (adherence) of their programs;
• 11 sites failed in implementing all core components -- 4 ended implementation prematurely, and 5 implemented 75-92% of the core components;
• An average of 85% of the core components was achieved by the sampled sites;
• Several bivariate relationships appeared significant including Ideal Program and Ideal Staff Characteristics, both of which related significantly to all four dependent variables, while Community Support, Time, Ideal Champion and Ideal Agency Characteristics and TA Visits (effects opposite of predictions) related significantly to three dependent variables;
• 9 of the 10 independent variables were significantly associated with one or both of the adherence measures (dependent variables #1 and #2);
• Several of the bivariate relationships regarding success disappeared at the multivariate level. For example, for sustainability - i.e., Community Support, Number of TA visits, Time, Ideal Staff, Champion, and Agency Characteristics had no significant direct relationship with the ability of programs to sustain for 30 months (6 months after the grant ended).
• TA quality, program characteristics and inconsistent staffing were the three most consistent predictors across various success measures.

 Limitations & Implications

The authors point out that the implementation study was conducted with a pre-selected group of sites, and due to this site culling, "real world" contexts may not be represented. They also note that, while the study implies that providing continuous TA to agencies and schools requires significant coordination and resources, the precise type of TA that lends itself to implementation success was not discerned in this study.  

They call for more studies to continue assessing the direct and indirect effects of the implementation factors identified in this study, in particular, examinations with diverse contexts, random-sample techniques, and multiple respondents. 

Tools/Knowledge Objects/Resources/Contacts/Etc: 

•  Rogers, E. M. (1983 and 1995). Diffusion of Innovation. New York: Free Press.
• Sharon Mihalic, MA, Director, Blueprints for Violence Prevention Initiative, Center for the Study and Prevention of Violence, University of Colorado.  303/492-8465:  www.colorado.edu.cspv/blueprints