Personnel from the Texas Christian University (TCU) examined the organizational functioning of alcohol and drug abuse treatment centers across two states in the Gulf Coast with reference to strategic planning, in particular staff training needs and transfer of evidence-based practice into clinical settings.

[Rowan-Szal, G. A., Greener, J. M., Joe, G. W., & Simpson, D. D. (2007). Assessing program needs and planning change. Journal of Substance Abuse Treatment, 33(2), 121-129.]

Two TCU-developed assessments were used for their evaluation: 

• The Organizational Readiness for Change (ORC) assessment, which evaluates organizational traits -- needs and pressures, resources, staff characteristics, and organizational environment -- that predict readiness for change;
• The Program Training Needs (PTN) assessment helps identify innovative treatments based on staff input from seven areas -- staff training needs, training content and training strategy preferences, training barriers and satisfaction with training, program facilities/climate and computer resources. The PTN was created as a complementary assessment to the ORC to provide staff with more detailed information regarding treatment issues than that provided by the ORC.

Problem Studied

Their quantitative analysis had three goals:

1. To determine the basic psychometric properties of the seven scales in the PTN, its factor and internal structure (reliability and dimensionality), and to explore relationships  among its seven scales;
2. To ascertain differences and similarities between perceptions of training needs as reported by clinical supervisors and staff.
3. To explore training needs identified by the PTN assessment within a more "global" context and with relation to the ORC assessment. 

Sample Population

• PTN assessment data was collected from 589 counselors at 194 treatment centers (members of the Gulf Coast Addiction Technology Transfer Center) in two states.
• A total of 579 counselors completed the ORC assessments, which were collected from 231 treatment programs in both states.
•  State A had 192 completed PTN surveys (33% of the data sent) while State B had 397 completed (67%).
• There were 174 ORC assessments collected (30%) from State A and 405 ORC assessments (70%) collected from State B.
• Caucasians represented 63% of respondents, African Americans 24%, Hispanics 14%. 
• Sixty-nine percent of all respondents were female. 
• Regarding years of counseling experience, 45% of the participants had at least 5 years experience in drug and alcohol counseling; 21% held their current jobs for at least 5 years and 43% of respondents had been employed in their current position for less than one year.
• Clinical supervisors accounted for 14% of PTN completion rate while 86% of the assessments were completed by counseling staff (81 and 502 PTNs, respectively).

Research Design Methods

Quantitative data collection from both PTN and ORC surveys was conducted over a two-year period spanning 2003 and 2004.  The TCU Institutional Review Board approved all protocols used, and funding was provided by the National Institute on Drug Abuse. 

State A:  Fifty-nine treatment units were sent self-addressed, stamped envelopes for voluntary survey return.  For PTN protocols, there was a 59% participation rate, or 192 responses out of 330 forms sent.  The ORC return rate was 52%, with 174 out of 330 assessments returned to TCU researchers.  The PTN assessment forms were sent four months after the mailing of the ORC scales.

State B:  Researchers used PsychData to provide online ORC assessments in June 2003, and PTN surveys were conducted via the internet in November and December 2003.  PTN assessments were sent to 135 programs within the state, and ORC forms were sent to 173 treatment centers.  TCU researchers said they were unable to calculate response rates for these assessments from State B because they were online responses. 

Due to the researchers' need to restrict information to that data which was linked between the PTN and ORC surveys, the final sample was reduced to one-third of that originally collected to 195 PTN assessments and 174 ORC assessments from a total of 97 matched treatment centers from both states.

 Assessments

The seven PTN areas covering 54 items are identified as follows:

1. Program facilities & climate - includes 7 items assessing the physical office space available and the overall treatment staff morale;
2. Program computer resources - includes 5 items covering the use of and adequacy of computer equipment including e-mail and internet access;
3. Staff training needs - incorporates 10 items assessing the skills such as cognitive and behavioral interventions, assessments and monitoring techniques which counselors think would help them perform their jobs better;
4. Training content preferences - (8 items) assesses perceptions of need for certain types of training such as diagnostic training tools, dual diagnosis and special populations;
5. Preferences for training strategy - (10 items) centers on types of training that staff prefer to receive along with logistics, such as full-day training, onsite consultation and using conceptual models;
6. Training roadblocks - (10 items) on staff thoughts about barriers to training such as budget constraints, applying ideas from workshops, and workload pressures that could interfere with a counselor's ability to attend training events;
7. Training satisfaction - (4 items) reports on the satisfaction level of staff regarding the sources, quality and types of training offered during the year.

Scoring for the PTN was as follows: 10=strongly disagree; 20=disagree; 30=undecided; 40=agree; 50=strongly agree, with mean scores higher than 30 indicating overall agreement.  

For a description of the ORC's psychometric properties, scales and rationale for the assessment, please refer to Lehman et al. (2002). The ORC covered 115 items representing 18 subscales spanning the four aforementioned organizational traits of needs and pressures, resources, staff characteristics, and organizational environment.  Responses were graded on a five point scale (disagree strongly, disagree, uncertain, agree, agree strongly) with scores being reverse-scored computing the mean score and multiplying by 10.  Higher scores represented more agreement while lower scores represented levels of disagreement, while 30 represented a neutral score.

ORC composite scores were computed for each of the four domains using the 18 domain subscales: 

• The Needs and Pressures Index (NPI: motivation for treatment) is the mean value of the scores on the 3 needs/pressures subscales - training needs; program needs and pressures for change;
• The Organizational Climate Index (OCI), a composite measure, is the mean value of scores on 6 subscales - mission, staff cohesiveness, staff autonomy, communication, stress and openness to change;
• The Institutional Resources Index (IRI) is the mean value of scores on 5 subscales - offices, staffing, training resources, equipment and internet;
• The Staff Attributes Index (SAI) is the mean value of the scores on the 4 staff characteristics subscales - influence, adaptability, efficacy and growth.

Measurements

Confirmatory factor analysis (CFA) using Proc CALIS (SAS, 2003) software evaluated the fit of the 54 PTN items to the hypothesized scales. Cronbach's α was used to compute reliability of PTN scales, and the relationships among seven PTN scale composites were examined using Pearson's correlations. 

Comparisons between clinical supervisors and staff responses were made by using analysis of variance to examine mean score differences between the two groups:  Staff responses in a treatment unit were averaged and compared to supervisor's responses from the same unit.  

The relationship between the PTN scales and the ORC's organizational functioning scales were assessed based on analyses at the treatment unit level.  Pearson's correlations were used to evaluate the relationships between the PTN and ORC domain indexes.  

Findings

For the seven-factor CFA of the 54 PTN items, surveyed by the PTN, the RMR (root mean square residual) was .065 while the RMSEA (root mean square error of approximation) was .06. which suggests that the seven factors are "appropriate" for describing the PTN, but not a good fit. Browne and Cudeck (1993) note that an RMSEA of .05 or less indicates a close fit  in relation to degrees of freedom, while values up to .08 indicates a fit "having reasonable errors of approximation in the population."   Hu and Bentler (1999) present evidence that RMSEA scores of approximately .06 and SRMR (standard root mean square residual) results of close to .08 indicate a relatively good fit between a hypothesized model and the observed data.

Loadings from the CFA indicate that all items in the Program facilities and climate, Training satisfaction, Staff training needs and Training barriers would be "marker variables for the scales because all loadings are higher than .40.  However, some of the items for the Training content and Training strategy preferences along with Program computer resources did not meet that cutoff, which explains why the fit was not better.   

The reliability of the PTN scales was computed using Cronbach's α.  Except for the Program Computer Resources scale, the scales had a reliability of .70 or higher ranging from .71 to .92.

These psychometric analyses indicate that the PTN adequately fits the pre-conceptualized seven-factor structure and that its scales have acceptable reliabilities. 

The differences and similarities between the perceptions of supervisors and subordinate staff members revealed significant differences in only one area of the PTN survey, Satisfaction with Training.  Clinical supervisors rated outside training and state funded drug and alcohol agencies significantly higher than did the staff.  No other significant differences were reported. 

Finally, the relationship between ORC NPI composite index was shown to be positively related to two PTN areas, Staff Training and Preferences for Training.  This indicates that agencies which reported higher training needs and preferences for specific training content also received greater needs and pressures reflecting motivation for organizational change.   Results also showed that treatment units with higher PTN scores for Program Facilities and climate scale had higher ORC scores for those corresponding measures.  Importantly, those centers with the greatest number of resources and most favorable climate scores also, perhaps not surprisingly, reported fewer barriers to success and greater satisfaction with training strategies and opportunities. 

Implications 

It has been suggested (Backer, 1993; Peterson, 1998; Goldstein, 1991, 2002) that any attempt to initiate organizational change and successfully transfer innovation into evidence-based practice must be preceded by an organizational sense of urgency based on a needs assessment of both the current organizational structure and the needs and skills of the individuals working within that organization.  The researchers involved in this survey and assessment of PTN and ORC surely subscribe to that paradigm. 

In an effort to successfully transfer evidence-based treatment programs into the everyday practice of hundreds of drug and alcohol counselors and supervisors, Rowan-Szal, et al., in conjunction with TCU, have built a template of indexes designed to measure both the needs of specific counseling centers and their openness to accepting organizational change. This "systematic assessment of needs and staff readiness for innovations" is necessary before any program change or knowledge acquisition can occur. Organizations which have had the opportunity to identify their own particular challenges and successes are, according the authors, much more likely to adapt systematic reformation, based on research, and initiate those reforms over time. 

Tools/Knowledge Objects/Resources/Contacts:

•  Backer, T. E. (1993). Information alchemy: Transforming information through knowledge utilization. Journal of the American Society for Information Science, 44, 217-221.
• Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage.
• Goldstein, I. L. (1991). Training in work organizations. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed.). Palo Alto, CA: Consulting Psychologists Press.
• Goldstein, I. L., & Ford, J. K. (2002). Training in organizations: Needs assessment, development, and evaluation (4th ed.). Belmont, CA: Wadsworth.
• Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
• Lehman, W. E. K., Greener, J. M., & Simpson, D. D. (2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment, 22, 197-209.
• Peterson, R. (1998). Training needs assessment: Meeting the training needs for quality performance. Sterling, VA: Kogan Page.
• SAS Institute. (2003). SAS/STAT user's guide. Cary, NC: SAS Institute.