Search results matching tags 'Transfer of innovation' and 'Social System'

"How Technology Enables Transformation of Human Service Administration" by M. Geffen and J. Kost

jstengel — Mon, 16 Feb 2009 15:29:24 GMT

This article focuses on human service organizations whose ultimate service mission to public clientele is attainment of self-sufficiency. The authors cite various reasons for an ever-evolving 21^st Century human services model of enormous complexity that must exponentially increase its service delivery capability to keep up with increasing and varied client demands; demands which, in turn, are fraught with an increase in the pace and frequency of individual needs. Thus, that ultimate goal of users of social services realistically achieving self-sufficiency becomes still more elusive. Simply bisected, the current service delivery evolution supports innovative change on two orchestrated flanks: (1) technology and (2) human support. The latter flank must undergo reverberating changes in front-line service worker jobs and that of their case managers as a result of admittedly late-arriving technical changes in the former flank.

The authors offer a well-defined “categorical grip” to interested readers for getting a conceptual handle on the technologically-based tools involved in exponentially keeping apace of diverse human service needs: integrated case management solutions (e.g., residing on CRM platforms), sophisticated eligibility (etc) rules engines, workflow tools that integrate and make seamless multiple (triaged) services, knowledge management and collaboration tools, computer tablets and other devices for users’ remote access to services, and advanced data integration/business intelligence tools providing real-time data where imminently required.

Some of this recommended technology comes by way of budget-friendly COTS (commercial off-the-shelf) software and hardware; some of which also lends itself to increasing client participation in needed services (self-sufficiency), while raising the service performance bar for human service agencies. The authors add three recommended caveats: (1) overhauling the procurement process (which inhibits multiyear turnkey projects in general); (2) acting on the foreknowledge that vendors of COTS solutions are as important as the products they promote (i.e., issues regarding installation, integration, and maintenance); and (3) having internal technical support readily available to create easier and more meaningful ways for diverse users to use an upgraded system to its fullest (operations side of knowledge management).

Citation

Geffen, M. and J. Kost (2006). "How Technology Enables Transformation of Human Service Administration." Policy & Practice of Public Human Services 64(4): 14-17.

Toward a Sociology of the Network Society

jstengel — Mon, 09 Feb 2009 16:30:00 GMT

This article invokes a provocative view of a changed society poised on the cusp of high potential for innovation because all of us (in developed nations) now function amidst a vast complex of macro-level social networks (and thusly a new framework for the study of sociology needs to be erected). Just when consortium members might think they’re ready to spin hypotheses about promising innovative inroads in technology and organizational culture, Castell comes along with fresh insight into our changed society in ways that possibly alter the rules of engagement in innovation generation.

The author offers a glimpse of some macro-level social and technological networks that amend our current view of society enough to seemingly need to be accounted for in present thinking (yet give sufficient pause to hamper forward progress?).

The author cites information technology, globalization, the WWW, and the dilution of the sovereign nation-state as four axes of significant societal restructuring we’ve undergone. He further describes examples of macro-level networks impacting our lives in ways that have indirectly reshaped society as we’ve known it; these changes brought on by current technology, the author seems to suggest, have spawned a diffusion of innovative “side effects” on the social structure itself.

Citation

Castells, M. (2000). "Toward a Sociology of the Network Society." Contemporary Sociology 29(5): 693-699.

“Futures That Work: Using Search Conferences to Revitalize Companies, Communities, and Organizations”

jstengel — Wed, 28 Jan 2009 16:13:21 GMT

A search conference is a participative planning event that enables people to create a plan for the most desirable future of their community or organization, a plan they carry out themselves. Search conferencing has a long, rich history. The first search conference happened in 1960 in Great Britain when leaders of two aircraft engine manufacturers came together in the first search conference to merge their two companies into one. The inventors of the method and the leaders of that first search were social scientists Fred Emery and Eric Trist. The search conference is a practical way to build communities of people who step up to the challenges of our turbulent times and take responsibility for making change happen in a purposeful way. As the world becomes more and more turbulent, the need is great for people to form communities to search for their desirable futures together. The search conference puts people in the driver’s seat of change, so they can steer together toward the future they want for their system, making adjustments as they go forward.

This book has a detailed description of what happens in a search conference, identifies the many ways it can be used, the principles that it is based on, how to plan for a search conference, and stories from many different organizations, from a Palestinian YMCA to Microsoft. It is a good guide for those who want to bring together a group of people to plan strategically and holistically and, importantly, in an accountable and sustainable way to implement what they plan. It also begins to shift organizations to a more open and participative culture.

Citation

Rehm, R., Cebula, N., Ryan, F., Large, M. (2002), “Futures That Work: Using Search Conferences to Revitalize Companies, Communities, and Organizations”, New Society Publishing and Hawthorn Press.

"Counselor perceptions of organizational factors and innovations..." by Joe, G. et al.

Donna Cooper — Fri, 22 Feb 2008 00:45:00 GMT

Researchers from Texas Christian University studied a large population of drug abuse counselors focusing on the individuals as treatment counselors with respect to training and organization, a departure from the norm of studying counselors in the aggregate to ascertain the behavior of the organization.

[Joe, G. W., Broome, K. M., Simpson, D., & Rowan-Szal, G. A. (2007). Counselor perceptions of organizational factors and innovations training experiences. Journal of Substance Abuse Treatment, 33 (2), 171-182.]

Three primary questions were considered:

How do counselors' perceptions of themselves and their working environment relate to readiness for and responsivity to innovations training?
How do they rate their previous experiences in adopting workshop training materials?
Will patterns (types) of counselors be predicted and identified from organizational climate perceptions?

The primary assessment tool utilized for data collection was the TCU-developed Organizational Readiness for Change (ORC), a 129-item assessment measuring 18 categories across 4 primary areas: Program needs/pressures (indicative of training motivation); program resources; staff attributes; and organizational climate (Lehman, Greener and Simpson, 2002).

Hypotheses

The ORC scales have been proven to be significant predictors of innovation at the program level (Simpson, et al, 2007), treatment program process indicators (Greener, et al., 2007), and predictors of attitudes toward evidence-based practice (EBP) at the individual level (Saldana et al. 2007).

Therefore, it was hypothesized that:

As measured by the ORC, several patterns or types of counselors could be described from their perceptions of organizational climate and perceived job performance abilities;
These descriptive patterns should be applicable across a variety of treatment settings;
Counselors who have more "favorable" profiles should be more engaged in efforts to improve work performance, such as innovative training opportunities.

Research Design Method

Treatment programs were recruited by four Addiction Technology Transfer Center networks (ATTC) sponsored by the Substance Abuse and Mental Health Services Administration. Data collection and administration procedures were approved by the Institutional Review Board at TCU. Participation was solicited by contacting treatment programs which provided mailing lists and letters of support encouraging participation in subsequent ATTC-sponsored workshop trainings. Counseling staff completed the ORC assessment 3 months prior to the training conferences. An average of three assessments was completed per treatment unit.

Confidentiality was ensured by providing participants postage-paid, pre-addressed envelops that could be sealed and mailed directly to TCU. One state offered electronic collection of the ORC via the internet. Overall, the paper version of the ORC was administered to counselors in 180 programs, while the electronic version was administered in 165 programs.

Sample Population

A total of 1,047 counselors voluntarily participated in the assessment in 10 states.
Data was collected between 6/2000 and 6/2003.
35% of the participants were male;
Average age was 45;
71% were Caucasian, 17% African American, and 12% were Hispanic.
60% were certified counselors, 12% working to receive certification, 2% previously certified, and 26% not certified.
Counseling experience varied -- 16% less than one year, 20% 1-3 years, 14% 3-5 years, and 49% more than 5 years.

Measures and Reliability

The staff attributes and organizational climate domains of the ORC comprised the profile of measures identifying the latent classes of counselors.

Staff attributes addressed growth (5 items with a coefficient α = .68); efficacy (5 items with a coefficient α = .68); influence (6 items = with a coefficient α = .64).
Organizational climate addressed clarity of mission and goals (5 items with a coefficient α = .74); staff cohesiveness (6 items with a coefficient α = .88); staff autonomy (5 items with a coefficient α = .56); open communication (5 items with a coefficient α = .83); stress (4 items with coefficient α = .82); and openness to change (5 items with coefficient α = .73).

The other two ORC domains - needs and pressures, and resources - were used as predictors of the counselor groups.

Needs and pressures addressed program need for improvement (8 items, coefficient α = .90); immediate training needs (8 items, coefficient α = .88); and pressures for change (7 items, coefficient α = .68).
Resources addressed offices (4 items, coefficient α = .74); staffing (6 items, coefficient α = .74); training resources (4 items, coefficient α = .63); equipment (computer access with 7 items, coefficient α = .77); and internet (electronic communication with 4 items, coefficient α = .78).

There were 14 questions on the ORC which addressed previous workshop training experiences as related to the expectation that there would be a correlation between counselor job-related efficacy/climate and job engagement.

Training exposure consisted of 5 items with a coefficient α of .62;
Individual adoption (pertaining to prior workshops) consisted of 4 items with a coefficient α of .78;
Program adoption (pertaining to agency wide use of prior workshop training) consisted of 3 items with a coefficient α = .81;
Training satisfaction (pertaining to prior year trainings and training opportunities in general) consisted of 2 items with a coefficient α = .77).

Analyses

Latent profile analysis (LPA) of staff attributes and organizational climate measures were conducted using MPlus software. As counselors were nested within clinics, a bi-level model for estimating patterns with emphasis on within-program counselor differences was used. The advantage of the LPA, the authors note, is that it has the ability to address incomplete data, using information from all cases (complete or incomplete), and assuming that missing information is unrelated to the values that are missing.

The researchers based their decisions about the appropriate number of latent classes on the Bayesian information criterion (BIC), adjusted for sample size and entropy index (Muthen & Muthen, 2004).

Multinomial logistic regressions of the latent grouping variable on program resources, program motivation and counselor background were performed in a second set of analyses, again with the MPlus software. This was done to identify key predictors of the latent classification patterns.

A third set of analyses using 2 models in a chi-square difference test, addressed the hypothesis that the latent groups were significantly different regarding innovations training and adoption.

Limitations

The following limitations may inhibit the researchers' ability to generalize findings across a wide array of treatment programs:

counselors and programs represent a convenience sample gathered from a large number of treatment programs covered by the ATTC;
the data are cross-sectional in nature;
some participating programs were motivated by the innovation training offered.

Findings

Three classes of counselor perceptions of organizational dynamics in work environments were discovered and labeled:

isolated (43%),
integrated (43%), and
exceptional (14%).

The ‘isolated' group was identified as having higher stress and also had lower cohesiveness and communication scores. The ‘integrated' group had a positive view of the programs' climates and moderate stress. The ‘exceptional' group showed low stress and high scores for other indicators, with cohesiveness the highest.

Considering the combined percentages (57%) for integrated and exceptional classes, a majority of the counselors had positive perceptions of their work environment. Also, a majority of the counselors feel confident about their job efficacy and adaptability (scores of near or above 40 out of a 50-point scale).

The level of organizational functioning in the work environment was found to be related significantly to various training-experience measures.

The source of strongest discrimination came from the program resources domain. All of these scales showed that those perceiving better offices, program staffing, training resources, equipment, and internet were more likely to be in the two higher treatment organizational dynamics work environment groups when compared to the isolated group.

The finding that experienced counselors were less likely to report integrated and supportive environments, a valuable ingredient for adopting new techniques, raises "special concerns" that should be studied further, note the authors.

Summary Review

The present research establishes the relationship of organizational climate as an aid in identifying whether counselors may be ready to engage in the process of adoption of innovations.

The results emphasize the importance of communication, staff cohesiveness, clarity of mission, and work stress, which are key components of the organizational dynamics of the treatment programs.

Tools/Knowledge Objects/Resources

Greener, J. M., Joe, G. W., Simpson, D., Rowan-Szal, G. A., & Lehman, W. E. K. (2007). Influence of organizational functioning on client engagement in treatment. Journal of Substance Abuse Treatment, 33(2), 139-147.
Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment, 22(4), 197-209.
Muthen, L. K., & Muthen, B. O. (2004). Mplus user's guide: Statistical analysis with latent variables. Los Angeles: Muthen & Muthen.
Saldana, L., Chapman, J. E., Henggeler, S. W., & Rowland, M. D. (2007). The Organizational Readiness for Change scale in adolescent programs: Criterion validity. Journal of Substance Abuse Treatment, 33(2), 159-169.
Simpson, D. D., & Flynn, P. M. (2007). Moving innovations into treatment: A stage-based approach to program change. Journal of Substance Abuse Treatment, 33(2), 111-120.

"Organizational readiness for change and opinions toward treatment..." by Fuller, B. E. et al.

Donna Cooper — Thu, 21 Feb 2008 00:01:00 GMT

The authors noted a significant gap between evidence-based practice (EBP) innovations and their adoption in the mental health care profession.

[Fuller, B. E., Rieckmann, T., Nunes, E. V., Miller, M., Arfken, C., Edmundson, E., et al. (2007). Organizational readiness for change and opinions toward treatment innovations. Journal of Substance Abuse Treatment, 33(2), 183-192.]

Based on an in-depth review of implementation research, six core elements were identified as critical to adoption of and maintaining EBP programs:

selection of staff capable of implementing such programs;
staff training;
on-going coaching;
supervision and assessment measures to provide staff performance feedback;
program evaluation;
fostering of implementation/maintenance via administrative supports (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005).

They assessed the utility of the Organizational Readiness for Change (ORC) scale (Lehman, Greener & Simpson, 2002) using a regression model on four types of EBP:

manuals,
medication,
integrated mental health services, and
motivational enhancements. -

They also provided a literature review of these four EBP types.

Research Design Method

Three interrelated self-report surveys focusing on organizational, treatment unit, and workforce issues were utilized and completed sequentially. Opinion items were measured on a five-point Likert scale. Complete details on collection methods are provided in other literature (McCarty, Fuller, Arfen, Miller, Nunes, Edmundson, et al, 2007). Surveys were completed online or via paper transmission. The workforce survey was distributed via sealed correspondence to specified employees, and, most often, distributed and completed during staff meetings.

The Oregon Health & Science University Institutional Review Board (IRB) reviewed and approved study procedures, and 40 IRBs reviewed and approved the protocol. Respondent participation could be declined.

Items for the organizational self-report survey included information on ownership status, primary service setting, and company size. Response rate was 95%. Information in the treatment unit survey covered care and service levels, facility accreditation and licensure, patient characteristics, revenue sources, staffing, staff retention, and scales assessing treatment environment and philosophy. A response rate of 91% was reported.

The workforce survey assessed such items as training, credentials, education, years of experience, and opinions toward specific treatment types. It also included the 115-item ORC scale (Lehman, Greener & Simpson, 2002) which assesses factors influencing organizational change. Four categories covering 18 topics were addressed:

Motivation for change based on three factors -- program need for improvement, perceived training needs and pressure to change;
institutional resources measured five areas covering adequacy of resources - office space, staffing, training resources, computer access; and e-mail/Internet useage;
staff attributes influencing organizational change - professional growth potential, counseling efficacy, influence on co-workers, and adaptability;
organizational climate, which included six topics -- agency mission and goal clarity; staff cohesion; staff autonomy; management openness to staff communication; perception of stress level; and openness to change.

A response rate of 71% was reported for the workforce survey.

An ORC scale initial validation study indicated that the 18 topics were generally single factors and that the psychometric properties were strong (Lehman, et al., 2002).

Sample Population

The National Drug Abuse Treatment Clinical Trials Network (CTN) includes 17 regional research centers and approximately 150 community-based groups which provide AOD treatment services, and whose membership fluctuates over time as protocols dictate. The network was founded to provide support for innovations in EBP and evidence about efficacy of those practices. The sample population included:

participants from 249 treatment units in this network including
outpatient treatment programs (n = 79)
residential and inpatient programs (n = 95)
detoxification programs (n = 34) and
methadone programs (n = 41)
Treatment units with at least five staff members were required to allow for the use of models using group mean centering
Respondents included administrators and staff

Measurements

Factor analysis was used on opinion items to form factor scores. Hierarchical linear modeling assessed the influence of organizational characteristics and ORC domains on factor scores. (Comprehensive details of the methods and results are provided in the full journal article.) Five factor scores were generated and modeled as the outcome variable in a bi-level random effects regression model. The five score categories are:

EBP and manualized treatment;
Confrontation and non-compliance discharge;
medications;
mental health services;
motivational incentives.

Level 1 predictors were the ORC scales centered on the group mean to note within-group variation. Level 2 predictors included eight organizational variables: 1) independent treatment centers; 2) for-profit status; 3) annual revenue; 4) national accreditation; 5) residential treatment centers; 6) outpatient treatment services; 7) methadone services; 8) detoxification centers. The algebraic framework of the random-effects mixed model is provided in this article.

Findings

A large and heterogeneous population for assessing the influence of the ORC scales in EBP was provided by the CTN.
Staff attributes appeared to have positive and negative effects with regard to EBP.
Within the ORC domains, perceived need for improvement was key to motivation for change, and staff members scoring high here also endorsed treatment manuals, medications and motivational incentives.
Staff internet access surfaced as a significant institutional resource.
Staff who rated internet access as adequate showed more support for treatment manuals, medications and motivational incentives and less support for confrontation and non-compliance discharge.
Staff who rated themselves as more influential with peers indicated more support of manuals, integrated mental health services and motivational incentives.
Staff indicating less influence with peers appeared more supportive of confrontation and non-compliance discharge.
Perception of opportunities for professional growth affected positively willingness to use medications, and negatively influenced the use of confrontation and non-compliance discharge.

Organizational climate measures were also rated influential:

Perceived stress ratings were lower when support for using confrontation and non-compliance discharges was evidenced.
Stress ratings were higher with more support for integrated mental health services - apparently reflecting change and integration pressures.
A strong sense of mission clarity was also associated with use of treatment manuals and integrated mental health services.

The authors note that many of the 13,000+ specialty treatment centers for drug and alcohol abuse in the U.S. are under pressure to improve economic efficiency while providing EBP-proven treatment. Analysis of organizational and workforce variables added to the validity of the ORC, and indicate that the ORC measures important factors in organizational readiness for change. The analysis of the ORC dimensions within the CTN indicates that treatment center leaders should systematically involve their staff in the change process. The authors recommend subsequent examination of ORC influence on patient outcomes and program performance in clinical trials.

Tools/Knowledge Objects/Resources/Contacts:

Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment, 22, 197-209.
McCarty, D., Fuller, B., Arfken, C. L., Miller, M., Nunes, E. V., & Edmundson, E. (2007). Direct care workers in the National Drug Abuse Treatment Clinical Trials Network: Characteristics, opinion, and beliefs. Psychiatric Services, 58, 181-190.

"Assessing program needs and planning change" by Rowan-Szal, G. A. et al.

Donna Cooper — Wed, 20 Feb 2008 21:54:00 GMT

Personnel from the Texas Christian University (TCU) examined the organizational functioning of alcohol and drug abuse treatment centers across two states in the Gulf Coast with reference to strategic planning, in particular staff training needs and transfer of evidence-based practice into clinical settings.

[Rowan-Szal, G. A., Greener, J. M., Joe, G. W., & Simpson, D. D. (2007). Assessing program needs and planning change. Journal of Substance Abuse Treatment, 33(2), 121-129.]

Two TCU-developed assessments were used for their evaluation:

The Organizational Readiness for Change (ORC) assessment, which evaluates organizational traits -- needs and pressures, resources, staff characteristics, and organizational environment -- that predict readiness for change;
The Program Training Needs (PTN) assessment helps identify innovative treatments based on staff input from seven areas -- staff training needs, training content and training strategy preferences, training barriers and satisfaction with training, program facilities/climate and computer resources. The PTN was created as a complementary assessment to the ORC to provide staff with more detailed information regarding treatment issues than that provided by the ORC.

Problem Studied

Their quantitative analysis had three goals:

To determine the basic psychometric properties of the seven scales in the PTN, its factor and internal structure (reliability and dimensionality), and to explore relationships among its seven scales;
To ascertain differences and similarities between perceptions of training needs as reported by clinical supervisors and staff.
To explore training needs identified by the PTN assessment within a more "global" context and with relation to the ORC assessment.

Sample Population

PTN assessment data was collected from 589 counselors at 194 treatment centers (members of the Gulf Coast Addiction Technology Transfer Center) in two states.
A total of 579 counselors completed the ORC assessments, which were collected from 231 treatment programs in both states.
State A had 192 completed PTN surveys (33% of the data sent) while State B had 397 completed (67%).
There were 174 ORC assessments collected (30%) from State A and 405 ORC assessments (70%) collected from State B.
Caucasians represented 63% of respondents, African Americans 24%, Hispanics 14%.
Sixty-nine percent of all respondents were female.
Regarding years of counseling experience, 45% of the participants had at least 5 years experience in drug and alcohol counseling; 21% held their current jobs for at least 5 years and 43% of respondents had been employed in their current position for less than one year.
Clinical supervisors accounted for 14% of PTN completion rate while 86% of the assessments were completed by counseling staff (81 and 502 PTNs, respectively).

Research Design Methods

Quantitative data collection from both PTN and ORC surveys was conducted over a two-year period spanning 2003 and 2004. The TCU Institutional Review Board approved all protocols used, and funding was provided by the National Institute on Drug Abuse.

State A: Fifty-nine treatment units were sent self-addressed, stamped envelopes for voluntary survey return. For PTN protocols, there was a 59% participation rate, or 192 responses out of 330 forms sent. The ORC return rate was 52%, with 174 out of 330 assessments returned to TCU researchers. The PTN assessment forms were sent four months after the mailing of the ORC scales.

State B: Researchers used PsychData to provide online ORC assessments in June 2003, and PTN surveys were conducted via the internet in November and December 2003. PTN assessments were sent to 135 programs within the state, and ORC forms were sent to 173 treatment centers. TCU researchers said they were unable to calculate response rates for these assessments from State B because they were online responses.

Due to the researchers' need to restrict information to that data which was linked between the PTN and ORC surveys, the final sample was reduced to one-third of that originally collected to 195 PTN assessments and 174 ORC assessments from a total of 97 matched treatment centers from both states.

Assessments

The seven PTN areas covering 54 items are identified as follows:

Program facilities & climate - includes 7 items assessing the physical office space available and the overall treatment staff morale;
Program computer resources - includes 5 items covering the use of and adequacy of computer equipment including e-mail and internet access;
Staff training needs - incorporates 10 items assessing the skills such as cognitive and behavioral interventions, assessments and monitoring techniques which counselors think would help them perform their jobs better;
Training content preferences - (8 items) assesses perceptions of need for certain types of training such as diagnostic training tools, dual diagnosis and special populations;
Preferences for training strategy - (10 items) centers on types of training that staff prefer to receive along with logistics, such as full-day training, onsite consultation and using conceptual models;
Training roadblocks - (10 items) on staff thoughts about barriers to training such as budget constraints, applying ideas from workshops, and workload pressures that could interfere with a counselor's ability to attend training events;
Training satisfaction - (4 items) reports on the satisfaction level of staff regarding the sources, quality and types of training offered during the year.

Scoring for the PTN was as follows: 10=strongly disagree; 20=disagree; 30=undecided; 40=agree; 50=strongly agree, with mean scores higher than 30 indicating overall agreement.

For a description of the ORC's psychometric properties, scales and rationale for the assessment, please refer to Lehman et al. (2002). The ORC covered 115 items representing 18 subscales spanning the four aforementioned organizational traits of needs and pressures, resources, staff characteristics, and organizational environment. Responses were graded on a five point scale (disagree strongly, disagree, uncertain, agree, agree strongly) with scores being reverse-scored computing the mean score and multiplying by 10. Higher scores represented more agreement while lower scores represented levels of disagreement, while 30 represented a neutral score.

ORC composite scores were computed for each of the four domains using the 18 domain subscales:

The Needs and Pressures Index (NPI: motivation for treatment) is the mean value of the scores on the 3 needs/pressures subscales - training needs; program needs and pressures for change;
The Organizational Climate Index (OCI), a composite measure, is the mean value of scores on 6 subscales - mission, staff cohesiveness, staff autonomy, communication, stress and openness to change;
The Institutional Resources Index (IRI) is the mean value of scores on 5 subscales - offices, staffing, training resources, equipment and internet;
The Staff Attributes Index (SAI) is the mean value of the scores on the 4 staff characteristics subscales - influence, adaptability, efficacy and growth.

Measurements

Confirmatory factor analysis (CFA) using Proc CALIS (SAS, 2003) software evaluated the fit of the 54 PTN items to the hypothesized scales. Cronbach's α was used to compute reliability of PTN scales, and the relationships among seven PTN scale composites were examined using Pearson's correlations.

Comparisons between clinical supervisors and staff responses were made by using analysis of variance to examine mean score differences between the two groups: Staff responses in a treatment unit were averaged and compared to supervisor's responses from the same unit.

The relationship between the PTN scales and the ORC's organizational functioning scales were assessed based on analyses at the treatment unit level. Pearson's correlations were used to evaluate the relationships between the PTN and ORC domain indexes.

Findings

For the seven-factor CFA of the 54 PTN items, surveyed by the PTN, the RMR (root mean square residual) was .065 while the RMSEA (root mean square error of approximation) was .06. which suggests that the seven factors are "appropriate" for describing the PTN, but not a good fit. Browne and Cudeck (1993) note that an RMSEA of .05 or less indicates a close fit in relation to degrees of freedom, while values up to .08 indicates a fit "having reasonable errors of approximation in the population." Hu and Bentler (1999) present evidence that RMSEA scores of approximately .06 and SRMR (standard root mean square residual) results of close to .08 indicate a relatively good fit between a hypothesized model and the observed data.

Loadings from the CFA indicate that all items in the Program facilities and climate, Training satisfaction, Staff training needs and Training barriers would be "marker variables for the scales because all loadings are higher than .40. However, some of the items for the Training content and Training strategy preferences along with Program computer resources did not meet that cutoff, which explains why the fit was not better.

The reliability of the PTN scales was computed using Cronbach's α. Except for the Program Computer Resources scale, the scales had a reliability of .70 or higher ranging from .71 to .92.

These psychometric analyses indicate that the PTN adequately fits the pre-conceptualized seven-factor structure and that its scales have acceptable reliabilities.

The differences and similarities between the perceptions of supervisors and subordinate staff members revealed significant differences in only one area of the PTN survey, Satisfaction with Training. Clinical supervisors rated outside training and state funded drug and alcohol agencies significantly higher than did the staff. No other significant differences were reported.

Finally, the relationship between ORC NPI composite index was shown to be positively related to two PTN areas, Staff Training and Preferences for Training. This indicates that agencies which reported higher training needs and preferences for specific training content also received greater needs and pressures reflecting motivation for organizational change. Results also showed that treatment units with higher PTN scores for Program Facilities and climate scale had higher ORC scores for those corresponding measures. Importantly, those centers with the greatest number of resources and most favorable climate scores also, perhaps not surprisingly, reported fewer barriers to success and greater satisfaction with training strategies and opportunities.

Implications

It has been suggested (Backer, 1993; Peterson, 1998; Goldstein, 1991, 2002) that any attempt to initiate organizational change and successfully transfer innovation into evidence-based practice must be preceded by an organizational sense of urgency based on a needs assessment of both the current organizational structure and the needs and skills of the individuals working within that organization. The researchers involved in this survey and assessment of PTN and ORC surely subscribe to that paradigm.

In an effort to successfully transfer evidence-based treatment programs into the everyday practice of hundreds of drug and alcohol counselors and supervisors, Rowan-Szal, et al., in conjunction with TCU, have built a template of indexes designed to measure both the needs of specific counseling centers and their openness to accepting organizational change. This "systematic assessment of needs and staff readiness for innovations" is necessary before any program change or knowledge acquisition can occur. Organizations which have had the opportunity to identify their own particular challenges and successes are, according the authors, much more likely to adapt systematic reformation, based on research, and initiate those reforms over time.

Tools/Knowledge Objects/Resources/Contacts:

Backer, T. E. (1993). Information alchemy: Transforming information through knowledge utilization. Journal of the American Society for Information Science, 44, 217-221.
Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage.
Goldstein, I. L. (1991). Training in work organizations. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed.). Palo Alto, CA: Consulting Psychologists Press.
Goldstein, I. L., & Ford, J. K. (2002). Training in organizations: Needs assessment, development, and evaluation (4th ed.). Belmont, CA: Wadsworth.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment, 22, 197-209.
Peterson, R. (1998). Training needs assessment: Meeting the training needs for quality performance. Sterling, VA: Kogan Page.
SAS Institute. (2003). SAS/STAT user's guide. Cary, NC: SAS Institute.

"Blueprints for Violence Prevention: From Research to Real-World..." by Mihalic & Irwin.

Donna Cooper — Tue, 12 Feb 2008 21:16:00 GMT

What impedes and fosters implementation success when evidence-based programs are applied in real world settings? This was the overarching question in a two-year study conducted by the Blueprints for Violence Prevention initiative, launched by the Center for the Study and Prevention of Violence (CSPV), Boulder, Colorado. After a succinct historical review of the literature and research regarding "what works" in the field of criminal justice, the authors note that "what is missing is a systematic analysis of what can make or break implementation efforts across a variety of contexts and programs."

[Mihalic, S. F. and Irwin, K. (2003). Blueprints for Violence Prevention: From Research to Real-World Settings-Factors Influencing the Successful Replication of Model Programs. Youth Violence and Juvenile Justice, 1(1): 1-23.]

Their study evaluates eight violence-prevention programs, which demonstrated significant decreases in violence/problem behaviors during their research phase, and were identified as "model" programs by Blueprints. The eight covered various forms of violence prevention measures such as prenatal-postpartum pregnancy, family therapy, foster care, mentoring and school-based interventions. They include:

Big Brothers/Big Sisters of America (BBBSA)
Bullying Prevention Program
Functional Family Therapy (FFT)
Multidimensional Treatment Foster Care (MTFC)
Multisystemic Therapy (MST)
Nurse-Family Partnership
Promoting Alternative Thinking Strategies (PATHS)
Quantum Opportunities Program

After site assessments by CSPV to ensure that sites could successfully implement the programs for the long-term (at least two years), 42 sites were selected. The sites included non-profit community groups, schools, state health agencies and private treatment centers.

The authors note the conceptual framework for this study draws from diffusion theory (Rogers, 1983, 1995), which holds that innovation occurs in four stages:

awareness gained through dissemination;
innovation adoption;
innovation implementation; and
maintenance.

It is during the implementation phase when an innovative program is put into practice and behavior change is required that most problems arise, according to Rogers (1995).

Problem Studied

The authors argue that much of the current research in this area is anecdotal, and that findings insinuate that almost "everything matters" in the implementation phase. Factors they note via the limited research, which are considered relevant to successful implementation include:

Organizational features of the adopting agency such as structure, history, philosophical traditions, economic saliency, stability, clear-open communication;
Staff characteristics such as motivation, support, and involvement in program design;
Adequate financial support for "direct line" staff;
Strong leadership/key people to motivate, advocate, and coordinate the program;
Practitioner training
Community support - especially for school-based implementation;
Program characteristics such as structure, complexity (i.e., evidence-based, theoretically-supported models) and flexibility;
Pre-planning.

Measurements

The author-researchers designed a two-year process evaluation study of the aforementioned selected program replication sites throughout the U.S. The study aimed first to identify research-based programs demonstrating effectiveness outcomes, and secondly to nationally replicate these same programs in non-research settings. A grant was secured via the Office of Juvenile Justice and Delinquency Prevention (OJJDP) to fund training and technical assistance efforts by the Blueprints/CSVP group at the replication sites. Both quantitative and qualitative research methods were used for the process evaluation. The primary goals of the process evaluation were to identify:

whether a program reaches its intended target population;
whether it is being implemented as designed;
which obstacles most interfere with completing implementation; and
what are the implementers' perceptions of the program's worthiness.

Three data collection/site visits were conducted yearly at regular intervals at each site, in addition to phone visits. The evaluation metrics consisted of implementation factors (independent variables) such as:

ideal program characteristics,
Community support,
number of technical assistance (TA) visits,
TA (or training) quality,
time to implement,
ideal staff characteristics,
ideal champion characteristics,
ideal agency characteristics,
inconsistent staffing, and
inconsistent funding.

The dependent variables (success outcomes) included :

adherence to core program components,
percentage of core programs components achieved,
dosage, and
sustainability.

The measures of the independent and dependent variables were deduced from the literature review, as well as the problem studied.

Some of the key measurements with brief descriptions included:

Qualitative (open-ended questions) and quantitative questionnaires, informal interviews during the on-site and telephone visits to the sites to assess how program implementation was going at each site with reference to the dependent variables;
Summary staff observations after the site visits;
Questions were created by examining "the prescribed program delivery" in Blueprints documentation for design of the model programs;
Quantitative questions also assessed the aforementioned primary goals of the evaluation (i.e., did programs reach their intended target populations);
To assess independent variables such as program, community, staff, leadership and agency, a set of five-point scales were developed and given to site coordinators to rate the effectiveness (asset versus barrier) of each variable. The scales had reliability ratios ranging from .69 to .88 based on Cronbach's alpha.
A combination of bivariate and multivariate analyses was performed on aggregated site data to determine the significance and strength of the relationships between the independent and dependent variables. Four multiple regression models were created to identify the independent effects of all of the independent variables on the dependent variables.

Findings

The key findings include (refer to the journal article for comprehensive details):

The dominating arguments in the literature were corroborated in this study;
74% of the 42 sites implemented all core components (adherence) of their programs;
11 sites failed in implementing all core components -- 4 ended implementation prematurely, and 5 implemented 75-92% of the core components;
An average of 85% of the core components was achieved by the sampled sites;
Several bivariate relationships appeared significant including Ideal Program and Ideal Staff Characteristics, both of which related significantly to all four dependent variables, while Community Support, Time, Ideal Champion and Ideal Agency Characteristics and TA Visits (effects opposite of predictions) related significantly to three dependent variables;
9 of the 10 independent variables were significantly associated with one or both of the adherence measures (dependent variables #1 and #2);
Several of the bivariate relationships regarding success disappeared at the multivariate level. For example, for sustainability - i.e., Community Support, Number of TA visits, Time, Ideal Staff, Champion, and Agency Characteristics had no significant direct relationship with the ability of programs to sustain for 30 months (6 months after the grant ended).
TA quality, program characteristics and inconsistent staffing were the three most consistent predictors across various success measures.

Limitations & Implications

The authors point out that the implementation study was conducted with a pre-selected group of sites, and due to this site culling, "real world" contexts may not be represented. They also note that, while the study implies that providing continuous TA to agencies and schools requires significant coordination and resources, the precise type of TA that lends itself to implementation success was not discerned in this study.

They call for more studies to continue assessing the direct and indirect effects of the implementation factors identified in this study, in particular, examinations with diverse contexts, random-sample techniques, and multiple respondents.

Tools/Knowledge Objects/Resources/Contacts/Etc:

Rogers, E. M. (1983 and 1995). Diffusion of Innovation. New York: Free Press.
Sharon Mihalic, MA, Director, Blueprints for Violence Prevention Initiative, Center for the Study and Prevention of Violence, University of Colorado. 303/492-8465: www.colorado.edu.cspv/blueprints