The Wizard of OT

Competency Based Assessment in OT Education

Heather Kuhaneck 12-11-24

As occupational therapy educational practices continue to evolve, so too must the methods by which we assess future practitioners. One of the most promising shifts in OT education is the move towards competency-based assessment (CBA). Unlike traditional grading systems, CBA focuses on the demonstration of specific skills and knowledge necessary for effective practice. This blog post explores the reasons for adopting competency-based assessment in OT education, the challenges it faces in the United States, and potential steps to facilitate this transition.

What is it?

Competency-based assessment emphasizes the mastery of essential skills and knowledge rather than the accumulation of grades. In programs that utilize CBA, rather than developing curricular assessments based on traditionally defined subject content, student assessments are developed to ensure that students acquire the knowledge, skills, and attitudes to perform in their chosen fields. Assessment often involves observation of simulation or real-world performance. This approach ensures that students are fully prepared to meet the demands of professional practice. In OT, where practical application of skills is crucial, CBA provides a more accurate measure of a student’s readiness to enter the workforce. With CBA, the criteria for success are clear and transparent. Students know exactly what is expected of them and can focus on meeting specific goals. This clarity fosters accountability and motivation. CBA can align with the standards set by ACOTE and by focusing on competencies required for professional practice, educational programs can ensure that their graduates are meeting the standards expected in the field. OT educators who teach labs and use performance-based lab exams will find CBA feels familiar and these types of courses may be the best ones to start with.  However, newly developing programs may choose to focus on CBA throughout their entire curriculum as it is initially designed.  

What is the difference between CBA and Competency Based education (CBE)?

CBE is a curricular approach across an entire program or university system, whereas CBA is a tool that can be used in a single course. CBE allows for a personalized and flexible learning experience for students, as they progress at their own pace, moving forward only when they have demonstrated competence in a particular area. This approach accommodates different learning styles and paces, and allows students the time they need to make early mistakes, learn from them, and gain mastery over time. Traditional programs have relied on the Carnegie unit or credit hour as the measure of “learning.” Because CBE  requires that students move through the curriculum at their own pace, this approach requires a large system-wide change in how universities bill students, manage credit hours, how they equate learning with time in class, etc.  Many programs and systems are unable or unable to tackle this large of a change. CBA however, is a tool that measures students’ performance against specific competencies to ensure students graduate with necessary skills.  Methods of CBA often include direct faculty observation, or portfolios, and self-assessments.

Why use CBA?

As stated well in a recent article “competency-based medical education (CBME) is becoming a pervasive framework for the design and implementation of educational programs worldwide, yet the research and theory that underlie its principles and practices are still very limited (Frank et al. 2010a; Swing et al. 2010). As has occurred in other large-scale educational innovations (e.g. problem based learning), the initial adoption of CBME has been based largely on educational principles and logical expectations with respect to potential initiatives and desired results. Only later, as evidence accumulates, can these expectations be substantiated – or discredited. However, such evidence is often accumulated haphazardly, through opportunistic implementation studies and incidental observation.” (Gruppen et al, 2017, pp 1).

The current state of the evidence is somewhat poor. There is somewhat mixed evidence that suggests that using CBA promotes student engagement and improves workplace readiness although much of the work has been completed in medical education, with little evidence specific to OT. Studies have shown in other fields that CBA and a focus on learning outcomes may be linked to improved student confidence and self-assessed competence. However, it may be important to approach the use of CBA with multiple low stakes assessments as opposed to singular high stakes summative assessments as these may negatively impact learning. 

So, why use CBA in OT?  It has logical appeal in the current times, where basic content is available in an instant from Google and elsewhere, but skills take time to develop and learn. CBA provides authentic assessment of skills, may motivate students to become more engaged in their learning and growth, and may promote student confidence.  It may ultimately lead to better prepared practitioners.

How to use it? (Steps Towards Implementing CBA in OT Education)

First, as with any big change, programs must engage faculty, students, and even fieldwork educators / employers for successful implementation. Open communication and collaboration can help address concerns, build consensus, and foster a shared vision for CBA.  Reviewing what other programs have done, or how CBA is being used in other countries can provide solid examples to draw from. Next, it can be helpful to implement a pilot program, try CBA out in one course or course series, to provide valuable insights and lessons prior to broader adoption. Faculty development may be essential for the success of CBA. Faculty will likely need training in CBA principles and techniques, and curriculum may need revision. Technology can play a significant role in facilitating CBA. Online platforms and tools can streamline the assessment process, provide real-time feedback, and track student progress. Leveraging technology can make CBA more efficient and accessible for both students and faculty. Ongoing professional development ensures that faculty are equipped to support students effectively throughout the change in grading processes and the adoption of new technology. Faculty will also be essential in developing the clear and measurable competencies that are specific, observable, and aligned with ACOTE standards. 

Since many OT programs already have lab courses that use lab exams, these are natural places to begin.  However, CBA can be used in all OT courses as long as content is applied to some form of OT related skill that can be assessed directly.

One related topic to consider in implementing CBA, is the Entrustable Professional Activity (EPA). Competencies may be considered individual qualities.  EPAs are specific work tasks.  Almost all work tasks in health care and in OT specifically may require multiple competencies in the “real world.”  For example, in order to complete an evaluation the provider must have competencies in communication, collaboration, specific content knowledge, and clinical decision making. When EPAs are used with students, they provide a way to make decisions about the level of the student’s ability to perform, and with what level of supervision required.  EPAs are often divided into levels such as 

-allowed to observe

– allowed to perform under direct supervision

– allowed to perform under indirect supervision 

-allowed to perform the activity unsupervised

– allowed to supervise the activity performed by more junior learners)

Challenges of Implementing CBA in the U.S.

One of the significant barriers to implementing CBA in the United States is resistance to change. Educational institutions, faculty, and students are accustomed to traditional grading systems. Transitioning to a new model requires a cultural shift, which can be met with skepticism and resistance. Developing and implementing CBA initially is resource-intensive. It requires significant time and effort to create competency-based curricula, develop assessment tools, and train faculty. Programs may struggle with the financial and logistical demands of such a transition. Additionally, ensuring consistency and fairness in CBA can be a challenge. Standardizing assessments across different courses requires careful planning and coordination. Faculty must be adequately trained to assess competencies reliably and fairly. Accreditation requirements can pose hurdles as well. OT programs must ensure that in the transition to CBA they continue to meet the ACOTE standards which can be a complex and time-consuming process.

References

Brydges, R., Boyd, V. A., Tavares, W., Ginsburg, S., Kuper, A., Anderson, M., & Stroud, L. (2021). Assumptions about competency-based medical education and the state of the underlying evidence: a critical narrative review. Academic Medicine, 96(2), 296-306.

Cate, O. T. (2019). An updated primer on entrustable professional activities (EPAs). Revista Brasileira de Educação Médica, 43(1 suppl 1), 712-720.

Chen, H. C., ten Cate, O., & Molloy, E. (2018). Assessment through entrustable professional activities. Learning and teaching in clinical contexts: a practical guide. 2nd ed. Chatswood: Elsevier, 286-304.

Chun, I., Taff, S. D., Mehta, A., & Chang, C. H. (2020). Identifying essential competency areas for occupational therapy education: A scoping review. Journal of Occupational Therapy Education, 4(4), 2.

Dauphinee, W. D., Boulet, J. R., & Norcini, J. J. (2019). Considerations that will determine if competency-based assessment is a sustainable innovation. Advances in Health Sciences Education, 24, 413-421.

Fu, C. P., Huang, C. K., Yang, Y. C., Liao, W. S., Huang, S. M., Chang, W. D., … & Chang, W. Y. (2024). Developing an entrustable professional activity for providing health education and consultation in occupational therapy and examining its validity. BMC Medical Education, 24(1), 705.

Gallardo, K. (2020). Competency-based assessment and the use of performance-based evaluation rubrics in higher education: Challenges towards the next decade. Problems of Education in the 21st Century, 78(1), 61-79.

Gruppen L, Frank JR, Lockyer J, Ross S, Bould MD, Harris P, Bhanji F, Hodges BD, Snell L, Ten Cate O; ICBME Collaborators. (2017). Toward a research agenda for competency-based medical education. Med Teach, 39(6):623-630. doi: 10.1080/0142159X.2017.1315065. PMID: 28598741.

Hamed, R., Banks, T. M., Mahoney, D., Simon, P., Timmerberg, J. F., & Nilsen, D. M. (2023). A Call to Shift to Competency-Based Education. The American Journal of Occupational Therapy, 77(6), 7706347010.

Harris, P., Bhanji, F., Topps, M., Ross, S., Lieberman, S., Frank, J. R., … & ICBME Collaborators. (2017). Evolving concepts of assessment in a competency-based world. Medical teacher, 39(6), 603-608.

Idrissi, M. K., Hnida, M., & Bennani, S. (2017). Competency-based assessment: from conceptual model to operational tool. In Innovative practices for higher education assessment and measurement (pp. 57-78). IGI Global.

La Chimea, T., Kanji, Z., & Schmitz, S. (2020). Assessment of clinical competence in competency-based education. Canadian Journal of Dental Hygiene, 54(2), 83.

Lurie, S. J. (2012). History and practice of competency‐based assessment. Medical education, 46(1), 49-57.

Meyer, E. G., Chen, H. C., Uijtdehaage, S., Durning, S. J., & Maggio, L. A. (2019). Scoping review of entrustable professional activities in undergraduate medical education. Academic Medicine, 94(7), 1040-1049.

Miller, L. T., Bossers, A. M., Polatajko, H. J., & Hartley, M. (2001). Development of the competency based fieldwork evaluation (CBFE). Occupational Therapy International, 8(4), 244-262.

Myers, C. T., Schaefer, N., & Coudron, A. (2017). Continuing competence assessment and maintenance in occupational therapy: Scoping review with stakeholder consultation. Australian Occupational Therapy Journal, 64(6), 486-500.

Rekman, J., Gofton, W., Dudek, N., Gofton, T., & Hamstra, S. J. (2016). Entrustability scales: outlining their usefulness for competency-based clinical assessment. Academic Medicine, 91(2), 186-190.

Rushanan, S. G., Nilsen, D. M., & Grajo, L. (2021). Determining the Initial Content Validity of the Clinical Competency Assessment Tool (CAT) for Occupational Therapists Treating Patients with Neurodegenerative Disease. The Open Journal of Occupational Therapy, 9(4), 1-11.

Shin, J. Y., Kim, J. R., & Lee, J. S. (2018). Development of Occupational Therapist Competency Indicators for Occupation-Based Intervention: Delphi Study. Therapeutic Science for Rehabilitation, 7(1), 37-50.