In-training evaluation (ITE) is the process of observing and systematically documenting the ongoing performance of a learner in real clinical settings during a specific period of training. Clinical performance (what the learner does) is in part a consequence of competency (what the learner knows) but is also influenced by other variables.¹³ In-training evaluation may include the assessment of competencies, such as knowledge or the acquisition and development of skills. It also includes the assessment of performance behaviors resulting from these competencies.⁴⁸ Behaviors that should develop and mature during clinical training, such as professionalism, are most appropriately evaluated during clinical interactions.⁹

Objective competency-based assessment methods should be considered as part of any system of in-training evaluation and include written examinations, objective structured clinical examinations (OSCEs) using standardized patients, formal or structured oral examinations, and chart-stimulated recall assessment.¹⁰ Performance-based methods of in-training assessment include the introduction of standardized patients into daily clinical activity, videotaped patient encounters, chart review, log books, and rating forms used to document observed behaviors. Most techniques used in evaluating residents are quantitative in nature, but qualitative evaluations (such as field notes) are also of value.¹⁰

Although objective evaluation methods must be a part of the overall ITE, we focus on issues of performance assessment with special emphasis on the residency years. The ongoing assessment of residents (ITE) as they participate in patient care is the best opportunity to evaluate many essential practice behaviors and to provide meaningful formative and summative information. Present forms of assessment often fail to provide sufficient information to facilitate resident learning and assist training programs to make promotion decisions.

Current approaches used in resident performance assessment rely heavily on input from attending staff to a program director who integrates all of this material and creates a final document.^{11, 12} Increasingly, however, the use of other observers, particularly nurses,^{1, 13, 14} has been suggested to be of value, particularly in the assessment of communication skills, ethical behaviors, reliability, and integrity.¹⁵¹⁷ Peer and resident assessments remain primarily research tools at the present but will likely be useful in future evaluation.^{3, 18} Patients have also been shown to provide valuable information pertaining to resident performance.¹⁶ Self-evaluation has not been extensively studied, but the development of such skills has been proposed as a form of guidance and support to residents.¹⁹

Go to:

THE ROLES OF IN-TRAINING EVALUATION

The purpose of ITE is to provide an accurate measurement of the learner's performance abilities. Program directors and ultimately the public must know that competent physicians are entering practice. Effective programs of ITE must also provide feedback to the trainee and to the program to enhance further learning and modifications in training.²⁰²² Systems of ITE must be based on clear learning objectives as these objectives will motivate students and direct learning. Programs evaluate what they perceive to be important. Although learners may have unique learning approaches, the learning style they ultimately utilize is a function of multiple variables. Prominent among these is the evaluation process.²³

Systems of ITE must be perceived as having validity and reliabilitythat is, as evaluating essential competencies necessary for future practice in a reproducible fashion utilizing methods that are feasible, defensible, and accountable.^{24, 25} This information can be used for purposes such as licensure, the acquisition and maintenance of hospital privileges, annual promotion, and certification or recertification.

Go to:

PROBLEMS WITH EXISTING SYSTEMS

As a method of evaluation, ITE has been widely criticized. Traditional ITEs lack validity in that they assess a restricted range of competencies and do not appear to consider many of the essential skills necessary for future practice. Tools, such as resident rating scales, have been found to have limited interrater and intrarater reliability and thus do not discriminate between residents or performances (i.e., they lack reproducibility).^{7, 20, 25, 26} Such weaknesses arise from two major factors. First, faculty must serve as both teacher and evaluator but receive little training to assume this dual role. The lack of training may result in observations that are not fully accountable or legally defensible.²⁷ Rater errors include those of distribution such as the leniency/severity error (doves/hawks) and the central tendency error (failure to use the entire rating scale),²⁰ and those of correlation(commonly called the halo effect) in which assessments reflect the tendency of a rater to be influenced by some information or impression about a resident and to allow that impression to spill over into the judgment of other components of the resident's performance.²⁰ Faculty training can help to reduce these errors.

Second, even when practice behaviors are observed on a daily basis, they frequently are not documented. Consequently, evaluations are not done in a timely fashion, and the opportunity for the systematic objective evaluation of performance behaviors is lost, and with it, the opportunity for meaningful summative assessment and feedback to the learner. Observations that are necessary for meaningful evaluations do require considerable faculty time. Evaluations that take place long after the event, and are based on faculty recall of rotational performance, are not sufficiently detailed to be helpful for decision making.

Go to:

ADVANTAGES OF RECORDING PERFORMANCE ASSESSMENT

Daily or frequent resident performance can and should be observed, evaluated, and documented.²⁰ This form of evaluation captures information not available by other measurement techniques. For the resident, ongoing evaluation and feedback on an individual patient basis, pertaining to essential external performance criteria, encourages the internalization of performance standards and is a prerequisite for learning.^{21, 28} For the faculty, individual patient-based evaluation permits multiple observations of actual behaviors in the clinical setting.

Important educational initiatives over the last decade based on the evolving needs and expectations of society have led to the identification of new roles and responsibilities for physicians. Roles such as the communicator, scholar, advocate, collaborator, manager, and professional and their associated competencies are most appropriately assessed in the context of ITE.^{3, 23, 29, 30} Frequent in-training assessment permits the opportunity for the assessment of history taking and physical examination skills, as well as communication skills as trainees relate to colleagues and patients in a variety of practice milieus.³¹ Newer definitions of competency include essential roles such as diagnostic reasoning skills, as well as patient and practice management skills including health promotion, advocacy, collaboration, counseling, and psychomotor skills, which can be best assessed in usual clinical settings. Critical appraisal, continued learning, and teaching skills are also evaluated most appropriately in this setting. Finally, professional and interpersonal behaviors can be best evaluated through in-training assessment as residents relate to patients, allied health professionals, and colleagues.^{1, 1317}

The development and implementation of improved strategies for in-training assessment will not require new evaluation methodologies, but rather the implementation of existing methods in a more consistent and objective fashion. To demonstrate efficacy, new programs of in-training assessment must be both reliable and feasible. A generalizability study utilizing objective evaluations by multiple observers at the time of the behavior, such as in our pilot project, will be an important adjunct to the literature. Standard setting is also a problem for in-training assessment as trainees of different levels and backgrounds perform similar tasks with differing expectations. Faculty must also consider not only an individual's performance, but also whether improvement is occurring in keeping with the expectations of the program. Future research must also consider the relative contribution of different evaluators and how many evaluations are required to provide a reliable estimate of resident performance and whether ITE correlates with future practice performance as a measure of predictive validity.

Go to:

SUMMARY

For faculties of medicine and training colleges to develop reliable, valid, and accountable systems of in-training assessment (Table 1), there must be a major change in the way we view this process. New systems of residency performance assessment that accurately document the characteristics necessary for the future practice of medicine must make evaluations an ongoing part of day-to-day practice. The solution to the problem of ineffective in-training assessment is not the further revision of existing forms, but the restructuring of how we evaluate our trainees and reorient our faculty development and reward systems to recognize this essential part of our practice. An improved system for in-training assessment will not only provide an accountable system for the identification of those individuals who are in difficulty, but also encourage and facilitate learning for all residents in accordance with the objectives of the training program.

1) What stood out to you and why

2) Limitations of the article

3) Key takeaways and conclusion