|
Title:
APA Educator Portfolio Analysis Tool
|
|
MedEdPORTAL ID#:
1659
|
|
Version:
1
|
|
Resource Type:
Faculty Development Materials
|
Description:
This unique analysis tool for the evaluation of educator portfolios (EPs) is based on measurable outcomes designed to allow reproducible analysis of the quality and impact of educational activities. The categories of evaluation include educational philosophy, five year goals, teaching, assessment of learners, evaluation of teaching, curriculum development, mentoring /advising, educational leadership/administration, other information (awards, reviewing and moderating activities), scholarly approach to education and products of educational scholarship. The analysis tool is organized in tabular format, and combines 18 quantitative, 25 qualitative, and 5 holistic ratings. We sought a balance between qualitative and quantitative items in a tool that captured important nuances of educational performance, but was not too long to preclude practical use by a promotions committee.
A unique feature of this tool is the inclusion of 7 index scores that combine several quantitative measures to measure depth and impact of an educational activity. For example, the curriculum impact index, which measures the impact of all the curricula developed by a faculty member, combines for each curriculum the number of learners taught, a geographic impact score, and an implementation score. Qualitative items are scored on a three point rating scale (1=novice, 2=intermediate 3=expert); verbal specifications for an intermediate rating are provided. Educational scholarship is measured in two dimensions: qualitative measures of a scholarly approach to education using the six criteria established by Glassick, and traditional quantitative measures of scholarship such as publications, presentations and grants. The accompanying instructions for use provide step by step guidance on how to score both quantitative and qualitative items with specific examples.
The analysis tool was developed by expert educators with extensive testing using actual EPs from educational scholars who used our EP template, previously peer reviewed and approved by MedEdPORTAL. The EP template was revised in parallel with the analysis tool to ensure that EP data enabled valid and reliable evaluation. Use of this comprehensive, yet practical tool is offered as a method to enhance opportunities for the promotion and advancement of educators, based on well defined and documented educational outcome measures. It is relevant for clinical educators across disciplines and across institutions.
|
|
|
Author Institution:
The School of Medicine at Stony Brook University Medical Ctr
|
|
Primary Author:
|
Latha Chandran, MD,MPH
The School of Medicine at Stony Brook University Medical Ctr
HSC T11-020 Department of Pediatrics
SUNY at Stony Brook
Stony Brook,
NY
11794-8111
USA
(631) 444-2730
lchandran@notes.cc.sunysb.edu
|
|
Other Authors:
|
Maryellen
Gusic, MD
Pennsylvania State University College of Medicine
mgusic@psu.edu
|
Constance
Baldwin, PhD
University of Rochester School of Medicine and Dentistry
constance_baldwin@urmc.rochester.edu
|
Teri
Turner
Baylor College of Medicine
tturner@bcm.edu
|
Elisa
Zenni
University of Florida College of Medicine
elisa.zenni@jax.ufl.edu
|
J Lindsey
Lane
Jefferson Medical College
jllane@nemours.org
|
Dorene
Balmer
Columbia University Medical Center
balmer@email.chop.edu
|
Miriam
Bar-on
University of Nevada School of Medicine
mbar-on@medicine.nevada.edu
|
Daniel
Rauch
New York University School of Medicine
daniel.rauch@med.nyu.edu
|
Diane
Indyk
Albert Einstein College of Medicine of Yeshiva University
indyk@aecom.yu.edu
|
|
Resource File(s):
|
-
Download File: MEP_1659_1 / 124.0 KB
- This resource is comprised of multiple files that have been zipped into a single file for quick and easy download.
- Click on the above link to download the resource as a ".zip" file.
- To extract the ".zip" file you must have decompressing software installed on your computer which is available for free from any of the following sites:
|
|
Additional Resource File Information:
|
1 .zip file containing the following resources:
1. EP_Analysis_Instructions.doc
2. EP_Analysis_Tool.doc
3. Copyright_License.doc
The MedEdPORTAL approved EP template associated with the analysis tool:
Gusic ME, Chandran L, Balmer DF, D'Alessandro DM, Baldwin CD, Educator Portfolio Template of the Pediatric Academic Societies' Educational Scholars Program. MedEdPORTAL; 2007. Available from: http://services.aamc.org/jsp/mededportal/retrieveSubmissionDetailById.do?subId=626
|
|
AAMC Hot Topics:
|
|
|
|
Content Last Updated:
|
| 04/30/2008 |
|
Specialty/Discipline:
|
|
|
|
Educational Objectives:
|
To systematically analyze the quality and impact of educational activities documented in an educator portfolio.
|
|
Resource Keyword/Symptom:
|
|
Teaching (MeSH)
American Psychological Association (MeSH)
Analysis Tool
Portfolio
Scholarship
Academic Promotions
Educator Portfolio
Educational Scholarship
Educator Evaluation
Promotion
Fellowships and Scholarships (MeSH)
Promotion & Tenure
Tenure
Mentors (MeSH)
Curriculum (MeSH)
Educational Measurement (MeSH)
|
|
Accreditation Council for Graduate Medical Education (ACGME) Competencies Addressed:
|
|
|
|
Intended Learner Audience:
|
|
Continuing Medical/Dental Education (CME) for Faculty Self Learning
|
|
Intended Faculty Audience:
|
|
Basic Science Faculty
Clinical Science Faculty
Medical/Dental School Administrators (broadly defined)
|
|
Effectiveness and Significance of Publication:
|
Traditional promotion standards rely heavily on quantification of research grants and publications in the CV. Promotion and retention of educators, however is challenged by the lack of accepted standards to evaluate the depth, breadth, quality and impact of educational activities. Several institutions have adopted versions of educator portfolios in an attempt to assess scholarly activities among educators. Standardized analysis of the EP is challenging to promotion committees and this factor may diminish the chances of successful approval of faculty under consideration by the committees. Studies have shown that educators are more likely to hold lower academic ranks than their research counterparts and that promotions committees still heavily rely on student satisfaction ratings as a key source of evaluation of educational quality.
Our 48 item tool is comprehensive, yet practical in its approach, with a blend of quantitative items to measure concrete information (e.g., teaching hours, number of learners and mentees, geographic dissemination) and qualitative items to measure important but more subjective information requiring careful judgment (e.g., variety of teaching strategies, appropriateness and balance of evaluation methods, quality of needs assessment). Index scores are included to reduce the number of quantitative measures required for a comprehensive evaluation of the EP. The tool also allows qualitative evaluation of narrative sections of an EP in a relatively standardized manner that could be used across different institutions to better evaluate a faculty member's scholarly approach to the work presented in the EP. The holistic rating scales allow for a qualitative summative assessment of an educator's educational philosophy and performance, career planning and scholarship. A scholarly iterative consensus building process was used by ten expert educators (faculty of the Educational Scholars Program of the Academic Pediatric Association) and an expert evaluator to create and refine the tool, using actual EPs of scholars currently in the program.
|
|
Special Implementation Requirements or Guidelines:
|
| The EP analysis tool is used in conjunction with the standard MedEdPORTAL approved EP template. A set of instructions for reviewers is provided clarifying each item in the analysis tool and how it is derived. |
|
Lessons Learned:
|
Our lessons learned as a consensus development team may prove useful to other raters who use or adapt this tool.
1. Evaluation of quantitative items taught us (repeatedly) to "Keep it Simple!" Proliferation of items is tempting, but a focused selection of essential items makes the tool practical.
2. Good quantitative items are based on judgments of quality; numbers are only useful if their educational value is explicit. Clear instructions help the rater who interprets a quantitative item to measure quality, not just quantity.
3. Evaluation of qualitative items needs to be grounded in sound principles that are clear to the rater. Specification of ratings for each item was critical to achieve concordance in our qualitative ratings.
4. Qualitative items need to be recorded in a numerical (e.g., ranked category) scoring system to give them "competitive equivalence" with quantitative measures. A simple three-level scoring system (1=novice, 2= intermediate, 3=expert) helped us to achieve concordance.
5. The quality of our analysis tool was highly dependent on the quality of our data source: creation of a structured analysis tool without a similarly structured EP proved futile. The corollary of this lesson is that educators need to be taught how to document their work meticulously, so it can be rated by a valid and reliable process.
6. We found it essential to use actual EP examples in both the development of the tool and the training of raters. Constant reference to the "real world" of EPs was indispensable to testing and achieving consensus.
7. To judge excellence in educational performance, we needed a combination of quantitative items that judge quality, and qualitative items that can be expressed numerically, so all ratings are amenable to parallel review and potentially, holistic summation. |
|
Publications, Presentations, and/or Citations For This Publication:
|
| The EP analysis tool has been presented at the Leadership Conference of the Academic Pediatric Association in March 2008. The description of the scholarly iterative process of developing the EP analysis tool and the lessons learned have been submitted as an article to Academic Medicine. |
|
Sponsorship (Funding Source):
|
| None. |
|
Citation Formats:
|
|
NLM:
|
|
Chandran L , Gusic M, Baldwin C, Turner T, Zenni E, Lane J, et al.
APA Educator Portfolio Analysis Tool. MedEdPORTAL;
2009.
Available from: http://services.aamc.org/30/mededportal/servlet/s/segment/mededportal/?subid=1659
|
|
APA:
|
|
Chandran, L., Gusic, M., Baldwin, C., Turner, T., Zenni, E., Lane, J., et al.
(2009).
APA Educator Portfolio Analysis Tool. MedEdPORTAL:
http://services.aamc.org/30/mededportal/servlet/s/segment/mededportal/?subid=1659
|
|
|
|
