Academic Skills Problems

Direct Assessment and Intervention
By Edward S. Shapiro

The Guilford Press

Copyright © 2004 The Guilford Press
All right reserved.

ISBN: 1-57230-968-7


Chapter One

Choosing Targets for Academic Assessment and Remediation

When a child is referred for academic problems, the most important questions facing the evaluator are "What do I assess?" and "What behavior(s) should be targeted for intervention?" As simple and straightforward as these questions seem to be, the correct responses to these inquiries are not as logical as one might think. For example, if a child is reported to be showing difficulties in sustaining attention to task, it seems logical that one would assess the child's on-task behavior and design interventions to increase attention to task. The literature, however, suggests that this would not be the most effective approach to solving this type of problem. If a child is not doing well in reading skills, one obviously should assess reading skills to determine whether the selected intervention is effective. But reading is an incredibly complex skill consisting of many subskills. What is the most efficient way to determine student progress? Clearly, the selection of behaviors for assessment and intervention is a critical decision in remediating academic problems.

The complexity of target behavior selection for assessment and intervention is reflected in a number of articles published in a special issue of Behavioral Assessment (1985, Vol. 7, No. 1). For example, Evans (1985) suggested that identifying targets for clinical assessment requires an understanding of the interactions among behavioral repertoires. He argued for the use of a systems model to plan and conduct appropriate target behavior selection in the assessment process. Kazdin (1985) likewise pointed to the known constellations of behavior, which suggest that focus on single targets for assessment or remediation would be inappropriate. Kratochwill (1985b) has discussed the way in which target behaviors are selected through behavioral consultation. He noted the issues related to using verbal behavior as the source for target behavior selection as problematic.

Nelson (1985) has pointed to several additional concerns in the selection of targets for behavioral assessment. The choice of behaviors may be based on both nonempirical and empirical guidelines, for example, choosing positive behaviors that need to be increased over negative behaviors that need to be decreased. Likewise, when a child presents several disruptive behaviors, the one chosen for intervention may be the one that is the most irritating to the teacher or causes the most significant disruption to other students.

Target behaviors may also be selected empirically by using normative data. Those behaviors that are evident in peers but not in the targeted child may then become the targets for assessment and intervention (Ager & Shapiro, 1995; Hoier & Cone, 1987; Hoier, McConnell, & Palley, 1987). Other empirical methods for choosing target behaviors have included use of regression equations (McKinney, Mason, Perkerson, & Clifford, 1975), identification of known groups of children who are considered to be demonstrating effective behavior (Nelson, 1985), use of functional analysis that experimentally manipulates different behaviors to determine which result in the best outcomes (e.g., Broussard & Northrup, 1995; Cooper, Wacker, Sasso, Reimers, & Donn, 1990; Cooper et al., 1992; Ervin, DuPaul, Kern, & Friman, 1998; Finkel, Derby, Weber, & McLaughlin, 2003; Lalli, Browder, Mace, & Brown, 1993), and use of the triple-response mode system (Cone, 1978, 1988). Clearly, selection of behaviors for assessment and intervention for nonacademic problems considers both the individual behavior and the environmental context in which it occurs to be equally important. There appears to be a tendency in assessing academic problems, however, not to consider the instructional environment, or to consider it only rarely, when a child is referred for academic problems (but see Christenson & Ysseldyke, 1989; Greenwood, 1996; Greenwood, Carta, & Atwater, 1991; Lentz & Shapiro, 1986). Traditional assessment measures, both norm- and criterion-referenced, and many models of CBA (e.g., Deno, 1985), make decisions about a child's academic skills without adequate consideration of the instructional environment in which these skills have been taught. Unfortunately, a substantial literature has demonstrated that a child's academic failure may reside in the instructional environment rather than in the child's inadequate mastery of skills (Lentz & Shapiro, 1986; Thurlow, Ysseldyke, Wotruba, & Algozzine, 1993; Ysseldyke, Spicuzza, Kosciolek, & Boys, 2003). Indeed, if a child fails to master an academic skill, it directly suggests potential failure in the instructional methodologies.

SELECTING TARGETS FOR ASSESSMENT

We (Lentz & Shapiro, 1985) have listed several basic assumptions in assessing academic problems. Each assumption is consistent with a behavioral approach to assessment and recognizes the important differences between assessment for academic problems and assessment for behavioral-emotional problems (in which behavioral methods more typically are used).

1. Assessment must reflect an evaluation of the behavior in the natural environment. Behavioral assessment emphasizes the need for collecting data under conditions that most closely approximate the natural conditions under which the behavior originally occurred. A child can perform academically in many ways, including individual seatwork, teacher-led small-group activities, teacher-led large-group activities, independent or small peer groups at learning centers, teacher-led testing activities, cooperative groups, peer-tutoring dyads, and so forth. Each of these instructional arrangements may result in differential academic performance under the same task. Whatever method is chosen for the assessment of academic skills, the procedure should be closely related to the way in which the behavior of interest occurs during the regular instructional period.

2. Assessment should be idiographic rather than nomothetic. The concerns that often drive the assessment process are the identification and evaluation of potential intervention procedures that may assist the remediation process. In assessing academic skills, it is important to determine how the targeted student is performing against a preintervention baseline rather than normative comparisons. In this way, any changes in performance subsequent to interventions can be observed. Although normative comparisons are important in making eligibility decisions and for setting goals, intra-individual rather than interindividual comparisons remain the primary focus of the direct assessment of academic skills.

3. What is taught and expected to be learned should be what is tested. One of the significant problems with traditional norm-referenced testing, as noted in Chapter 1, is the potential lack of overlap between the instructional curriculum and the content of achievement tests (Bell et al., 1992; Good & Salvia, 1988; Jenkins & Pany, 1978; Martens, Steele, Massie, & Diskin, 1995; Shapiro & Derr, 1987). In the behavioral assessment of academic skills, it is important that there be significant overlap between the curriculum and the test. Without such overlap, it is difficult to separate a child's failure on these tests due to inadequate mastery of the curriculum from failure to teach material covered on the test.

4. The results of the assessment should be strongly related to planning interventions. A primary purpose of any assessment is to identify those strategies that may be successful in remediating the problem. When assessing academic skills, it is important that the assessment methods provide some indications of potential intervention procedures.

5. Assessment methods should be appropriate for continuous monitoring of student progress, so that intervention strategies can be altered as indicated. Because the assessment process is idiographic and designed to evaluate behavior across time, it is critical that the measures employed be sensitive to change. Indeed, whatever methods are chosen to assess academic skills, these procedures must be capable of showing behavioral improvement (or decrements), regardless of the type of intervention selected. If the intervention chosen is effective at improving a child's math computation (e.g., single-digit subtraction), the assessment method must be sensitive to any small fluctuations in the student's performance. It is also important to note that, because of the frequency with which these measures are employed, they must be brief, repeatable, and usable across types of classroom instructors (e.g., teachers, aides, peers, parents).

6. Measures used need to be based upon empirical research and to have adequate validity. Like all assessment measures, methods used to conduct direct assessment of academic skills must meet appropriate psychometric standards. From a traditional perspective, this would require that the measures display adequate test-retest reliability and internal consistency, sufficient content validity, and demonstrated concurrent validity. In addition, because these measures are designed to be consistent with behavioral assessment, the measures should also meet standards of behavioral assessment, such as interobserver agreement, treatment validity, and social validity. Although there have been substantial research efforts to provide a traditional psychometric base for direct assessment measures (e.g., Shinn, 1988, 1998) there have been few efforts to substantiate the use of the measures from a behavioral assessment perspective (Lentz, 1988); however, see Derr and Shapiro (1989) and Derr-Minneci and Shapiro (1992).

7. Measures should be useful in making many types of educational decisions. Any method used for assessing academic skills should contribute across different types of decisions (Salvia & Ysseldyke, 2001). Specifically, the assessment should be helpful in screening, setting individual educational plan (IEP) goals, designing interventions, determining eligibility for special services, and evaluating special services.

The keys to selecting the appropriate behaviors for assessing academic problems are their sensitivity to small increments of change, their ability to reflect improvement in more molar areas of academic skills (e.g., reading), the curriculum validity of the observed behaviors (match between the assessment measure and the instructional objectives), their ability to assist in the development of intervention strategies, the ability to meet appropriate psychometric standards, and the inclusion of both the academic environment and individual skills in the assessment process. Interestingly, an examination of the literature from somewhat different perspectives (cognitive psychology, educational psychology, applied behavior analysis, and special education) provides significant support for the selection of specific classes of behavior from which one should choose the appropriate targets for evaluation of both the academic environment and the individual's skills.

Assessing the Academic Environment

Academic Engaged Time

Considerable effort has been given to the identification of the critical instructional variables affecting student mastery of basic skills. Much of this research was derived from Carroll's (1963) model of classroom learning, which hypothesized that learning is a function of time engaged in learning relative to the time needed to learn. Although a few researchers have examined issues related to the time needed for learning (e.g., Gettinger, 1985), most efforts have concentrated on the relationship of engaged time to academic performance (Caldwell, Huitt, & Graeber, 1982; Goodman, 1990; Karweit, 1983; Karweit & Slavin, 1981).

One of the most significant projects that examined relationships between time and academic achievement was the Beginning Teacher Evaluation Study (BTES; Denham & Lieberman, 1980). Observations were conducted on the entire instructional day in second- and fifth-grade classrooms across a 6-year period. Data were collected on the amount of time allocated for instruction, how the allocated time was actually spent, and the proportion of time that students spent actively engaged in academic tasks within the allocated time. From the BTES was derived the concept of "academic learning time" (ALT), a variable that incorporates allocated time, engaged time, and success rate.

Berliner (1979), in data reported from the BTES study, compared the amount of allocated time (time assigned for instruction) and engaged time (time actually spent in academic tasks) in second- and fifth-grade classrooms. Although there were wide variations in levels of performance across classes, many were found to have under 100 cumulative hours of engaged time across a 150-day school year. Frederick et al. (1979) examined engaged time and scores on the Iowa Tests of Basic Skills among 175 classrooms in Chicago, and found engagement rates and achievement scores to be moderately correlated (r = .54). The importance of engaged time has led to a number of studies examining the levels of student engagement across special education classrooms. Leinhardt, Zigmond, and Cooley (1981) examined engagement rates within reading instruction periods of self-contained classrooms for students with learning disabilities. Results of their investigation noted that reading behavior was significantly predicted by pretest scores, teacher instructional variables, and teacher contact. However, students were found to spend only 10% of their academic day in oral or silent reading activities with teachers, averaging 16 minutes daily of actual direct instruction. Haynes and Jenkins (1986), examining resource rooms for students with learning disabilities (LD), found similar results, with students engaged in silent or oral reading activities only 44% of the time scheduled for reading within the resource room. Similarly, most student time (54%) in these settings was spent in individual seatwork.

In a review of the engaged time literature, Gettinger (1986) noted that there appears to be "a positive association between academic engaged time and learning" (p. 9). She did offer substantial caution, however, that the literature is far from definitive and that other factors that may interact with engaged time (e.g., time needed for learning) need continued investigation. Despite these cautions, it appears that academic engaged time may be a critical variable for the assessment of academic skills. Indeed, Gettinger offers a significant number of excellent recommendations for increasing engaged time that have been supported by research.

Continues...


Excerpted from Academic Skills Problems by Edward S. Shapiro Copyright © 2004 by The Guilford Press. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.