Evaluation of Clinical and Surgical Proficiency at a Residency of Child Orthopedics and Traumatology Use of the Mini-CEX (Mini-Clinical Evaluation Exercise) and DOPS (Direct Observation of Procedural Skills)
Article Sidebar
Published:
Jun 15, 2021
Keywords:
Residency, assessment, competencies, Mini-CEX, DOPS
Main Article Content
Abstract
Purpose: To assess professional competencies in a Pediatric Orthopedic and Traumatology Residency Program by the implementation of two performance-assessment instruments: Mini-Clinical Evaluation Exercise (Mini-CEX) and Direct Observation of Procedural Skills (DOPS) for clinical and surgical skills. Both tools aim to assess the top of Miller’s pyramid, for its reliability and validity.
Materials and Methods: Prospective observational cohort study of six medical trainees in the first, second and third year of their residency program (R1-R2-R3) who were randomly assessed by six examiners during their daily training at outpatient clinics, emergency room, inpatients unit, operating room, and plaster room. The statistical analysis was carried out with the Chi-Square and Wilcoxon-Rank paired test for univariate variables. The residents’ relationship cohorts were compared using the Kruskal-Wallis test. The reliability of the methodological tool was determined by the psychometric test of Crombach. Alfa was set at ≤ 0.05. Diagnostic study: level IV.
Results: We performed 65 assessments. Each resident was evaluated 10 times on average by 3 to 6 examiners. The oldest residents had better performances in overall clinical competencies. However, the R1 group achieved satisfactory results whereas the R2-R3 groups had the most outstanding scores. There were no statistical differences in general surgical competencies, but the R3 group was outstanding in cases of unforeseen surgical situations. The Alfa Crombach coefficient was over 0.90.
Conclusion: The Mini-CEX, DOPS, and interactive feedback were powerful tools to provide high-quality assessment and were widely accepted by residents and examiners. The statistical analysis allowed us to identify the weaknesses and strengths of the trainees. The Crombach coefficient had a high psychometric impact.
Materials and Methods: Prospective observational cohort study of six medical trainees in the first, second and third year of their residency program (R1-R2-R3) who were randomly assessed by six examiners during their daily training at outpatient clinics, emergency room, inpatients unit, operating room, and plaster room. The statistical analysis was carried out with the Chi-Square and Wilcoxon-Rank paired test for univariate variables. The residents’ relationship cohorts were compared using the Kruskal-Wallis test. The reliability of the methodological tool was determined by the psychometric test of Crombach. Alfa was set at ≤ 0.05. Diagnostic study: level IV.
Results: We performed 65 assessments. Each resident was evaluated 10 times on average by 3 to 6 examiners. The oldest residents had better performances in overall clinical competencies. However, the R1 group achieved satisfactory results whereas the R2-R3 groups had the most outstanding scores. There were no statistical differences in general surgical competencies, but the R3 group was outstanding in cases of unforeseen surgical situations. The Alfa Crombach coefficient was over 0.90.
Conclusion: The Mini-CEX, DOPS, and interactive feedback were powerful tools to provide high-quality assessment and were widely accepted by residents and examiners. The statistical analysis allowed us to identify the weaknesses and strengths of the trainees. The Crombach coefficient had a high psychometric impact.
Downloads
Download data is not yet available.
Metrics
Metrics Loading ...
Article Details
How to Cite
Fernández, C. A., & Miranda, M. G. (2021). Evaluation of Clinical and Surgical Proficiency at a Residency of Child Orthopedics and Traumatology. Revista De La Asociación Argentina De Ortopedia Y Traumatología, 86(3), 417-427. https://doi.org/10.15417/issn.1852-7434.2021.86.3.1188
Issue
Section
Artículo Especial
Copyright (c) 2021 Revista de la Asociación Argentina de Ortopedia y Traumatología
Manuscript acceptance by the Journal implies the simultaneous non-submission to any other journal or publishing house. The RAAOT is under the Licencia Creative Commnos Atribución-NoComercial-Compartir Obras Derivadas Igual 4.0 Internacional (CC-BY-NC.SA 4.0) (http://creativecommons.org/licences/by-nc-sa/4.0/deed.es). Articles can be shared, copied, distributed, modified, altered, transformed into a derivative work, executed and publicly communicated, provided a) the authors and the original publication (Journal, Publisher and URL) are mentioned, b) they are not used for commercial purposes, c) the same terms of the license are maintained.
In the event that the manuscript is approved for its next publication, the authors retain the copyright and will assign to the journal the rights of publication, edition, reproduction, distribution, exhibition and communication at a national and international level in the different databases. data, repositories and portals.
It is hereby stated that the mentioned manuscript has not been published and that it is not being printed in any other national or foreign journal.
The authors hereby accept the necessary modifications, suggested by the reviewers, in order to adapt the manuscript to the style and publication rules of this Journal.
References
1. Giacomantone O, Suárez I. Profesionalismo médico, su relación con la educación médica del siglo XXI. Educación Médica Permanente. Revista de la Sociedad de Educación Médica de La Plata 2009;I(1):4-18. Disponible en: http://sedici.unlp.edu.ar/handle/10915/8517
2. Peinado Herreros JM. Competencias médicas. Educación Médica 2005;8(Supl. 2):S4-S6. Disponible en:
http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1575-18132005000600002&lng=es
3. Reta-De Rosas AM, López MJ, Vargas AL, Montbrun MC. Evaluación de competencias médicas en un currículo de grado no diseñado por competencias. Educación Médica 2008;11(4):207-18. https://doi.org/10.4321/S1575-18132008000400005
4. Tello C. Comité de Residencias. Rev Asoc Argent Ortop Traumatol 2015;80(3):143-4. https://doi.org/10.15417/550
5. Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011;33(3):206-14. https://doi.org/10.3109/0142159X.2011.551559
6. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65(Suppl):S63-70.
https://doi.org/10.1097/00001888-199009000-00045
7. Miller A, Archer J. Impact of workplace-based assessment on doctors´ education and performance: a systematic
review. BMJ 2010;341:50-64. https://doi.org/10.1136/bmj.c5064
8. Fornells-Vallés JM. El ABC del Mini-CEX. Educación Médica 2009;12(2):83-9. Disponible en: http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1575-18132009000300004&lng=es
9. Schuwirth LWT, van der Vleuten CPM. Conceptualising surgical education assessment. En: Fry H, Kneebone R
(eds.). Surgical education: Theorising an emerging domain. Springer Science; 2011:81-9. https://doi.org/10.1007/978-94-007-1682-7
10. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The Mini-CEX (Clinical Evaluation Exercise): a preliminary
investigation. Ann Intern Med 1995;123:795-9. https://doi.org/10.7326/0003-4819-123-10-199511150-00008
11. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins R. Construct validity of the miniclinical evaluation exercise
(miniCEX). Acad Med 2003;78(8):826-30. https://doi.org/10.1097/00001888-200308000-00018
12. Amini A, Shirzad F, Mohseni M, Sadeghpour A, Elmi A. Designing Direct Observation of Procedural Skills (DOPS) test for selective skills of orthopedic residents and evaluating its effects from their points of view. Res Dev Med Educ 2015;4(2):147-52. https://doi.org/10.15171/rdme.2015.026
13. Beard J, Strachan A, Davies H, Patterson F, Stark P, Ball S, et al. Developing an education and assessment framework for the Foundation Programme. Med Educ 2005;39:841-51. https://doi.org/10.1111/j.1365-2929.2005.02236.x
14. Khan AAM, Gorman M, Gwozdziewicz L, Sobani A, Gibson C. Direct Observation of Procedural Skills (DOPS)
as an assessment tool for surgical trainees. J Pak Med Stud 2013;3(3):137-40. Disponible en: https://www.jpmsPAGES137-140-PA.pdfonline.com/wp-content/uploads/2019/07/JPMS-VOL3-ISSUE3
15. Durante E. Algunos métodos de evaluación de las competencias: Escalando la pirámide de Miller. Rev Hosp Ital Buenos Aires 2006;26(2):55-61. Disponible en: http://revista.hospitalitaliano.org.ar
16. Nishan FN, Cleland J, McKenzie H, Cassar K. Identifying the factors that determine feedback given to
undergraduate medical students following formative Mini-CEX assessments. Med Educ 2008;42:89-95.
https://doi.org/10.1111/j.1365-2923.2007.02939.x
17. Dawson-Saunders B, Trapp R. Bioestadística médica, 2ª ed. México, DF: El Manual Moderno; 1997.
18. Sánchez Mendiola M, González MA. Evaluación del y para el aprendizaje: instrumentos y estrategias. Ciudad de México: UNAM; 2020:348. https://creativecommons.org/licenses/by-nc-sa/4.0/deed.es
19. Borges JL. Funes el memorioso. Artificios (1944). En: Ficciones, 7a ed. Buenos Aires: Debolsillo; 2014:125-35.
20. Bunge M. Filosofía para médicos. España: Editorial Gedisa; 2012:13-7.
21. Tello C, Vedoya S, Autorino C, Varaona J, Pelaez R, Fazio A, et al. Encuesta dirigida a los residentes sobre aspectos de la Residencia en Ortopedia y Traumatología. Rev Asoc Argent Ortop Traumatol 2017;82(2):150-6.
https://doi.org/10.15417/694
22. Alves de Lima A, Henquin R, Thierer J. A qualitative study of the impact on learning of the mini-clinical evaluation exercise in postgraduate training. Med Teach 2005;27(1):46-52. https://doi.org/10.1080/01421590400013529
23. Fernández Gálvez GM. Evaluación de las competencias clínicas en una residencia de pediatría con el Mini-CEX
(Mini-Clinical Evaluation Exercise). Arch Argent Pediatr 2011;109(4):314-20. Disponible en: http://www.scielo.org.ar/scielo.php?script=sci_arttext&pid=S0325-00752011000400006&lng=es
24. Alves de Lima A. Devolución constructiva una estrategia para mejorar el aprendizaje. Medicina (Buenos Aires)
2008;68:88-92. Disponible en: http://www.scielo.org.ar/scielo.php?script=sci_arttext&pid=S0025-76802008000100014
25. Kant I. Filosofía de la historia. Buenos Aires: Terramar Ediciones; 2004:33.
2. Peinado Herreros JM. Competencias médicas. Educación Médica 2005;8(Supl. 2):S4-S6. Disponible en:
http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1575-18132005000600002&lng=es
3. Reta-De Rosas AM, López MJ, Vargas AL, Montbrun MC. Evaluación de competencias médicas en un currículo de grado no diseñado por competencias. Educación Médica 2008;11(4):207-18. https://doi.org/10.4321/S1575-18132008000400005
4. Tello C. Comité de Residencias. Rev Asoc Argent Ortop Traumatol 2015;80(3):143-4. https://doi.org/10.15417/550
5. Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011;33(3):206-14. https://doi.org/10.3109/0142159X.2011.551559
6. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65(Suppl):S63-70.
https://doi.org/10.1097/00001888-199009000-00045
7. Miller A, Archer J. Impact of workplace-based assessment on doctors´ education and performance: a systematic
review. BMJ 2010;341:50-64. https://doi.org/10.1136/bmj.c5064
8. Fornells-Vallés JM. El ABC del Mini-CEX. Educación Médica 2009;12(2):83-9. Disponible en: http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1575-18132009000300004&lng=es
9. Schuwirth LWT, van der Vleuten CPM. Conceptualising surgical education assessment. En: Fry H, Kneebone R
(eds.). Surgical education: Theorising an emerging domain. Springer Science; 2011:81-9. https://doi.org/10.1007/978-94-007-1682-7
10. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The Mini-CEX (Clinical Evaluation Exercise): a preliminary
investigation. Ann Intern Med 1995;123:795-9. https://doi.org/10.7326/0003-4819-123-10-199511150-00008
11. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins R. Construct validity of the miniclinical evaluation exercise
(miniCEX). Acad Med 2003;78(8):826-30. https://doi.org/10.1097/00001888-200308000-00018
12. Amini A, Shirzad F, Mohseni M, Sadeghpour A, Elmi A. Designing Direct Observation of Procedural Skills (DOPS) test for selective skills of orthopedic residents and evaluating its effects from their points of view. Res Dev Med Educ 2015;4(2):147-52. https://doi.org/10.15171/rdme.2015.026
13. Beard J, Strachan A, Davies H, Patterson F, Stark P, Ball S, et al. Developing an education and assessment framework for the Foundation Programme. Med Educ 2005;39:841-51. https://doi.org/10.1111/j.1365-2929.2005.02236.x
14. Khan AAM, Gorman M, Gwozdziewicz L, Sobani A, Gibson C. Direct Observation of Procedural Skills (DOPS)
as an assessment tool for surgical trainees. J Pak Med Stud 2013;3(3):137-40. Disponible en: https://www.jpmsPAGES137-140-PA.pdfonline.com/wp-content/uploads/2019/07/JPMS-VOL3-ISSUE3
15. Durante E. Algunos métodos de evaluación de las competencias: Escalando la pirámide de Miller. Rev Hosp Ital Buenos Aires 2006;26(2):55-61. Disponible en: http://revista.hospitalitaliano.org.ar
16. Nishan FN, Cleland J, McKenzie H, Cassar K. Identifying the factors that determine feedback given to
undergraduate medical students following formative Mini-CEX assessments. Med Educ 2008;42:89-95.
https://doi.org/10.1111/j.1365-2923.2007.02939.x
17. Dawson-Saunders B, Trapp R. Bioestadística médica, 2ª ed. México, DF: El Manual Moderno; 1997.
18. Sánchez Mendiola M, González MA. Evaluación del y para el aprendizaje: instrumentos y estrategias. Ciudad de México: UNAM; 2020:348. https://creativecommons.org/licenses/by-nc-sa/4.0/deed.es
19. Borges JL. Funes el memorioso. Artificios (1944). En: Ficciones, 7a ed. Buenos Aires: Debolsillo; 2014:125-35.
20. Bunge M. Filosofía para médicos. España: Editorial Gedisa; 2012:13-7.
21. Tello C, Vedoya S, Autorino C, Varaona J, Pelaez R, Fazio A, et al. Encuesta dirigida a los residentes sobre aspectos de la Residencia en Ortopedia y Traumatología. Rev Asoc Argent Ortop Traumatol 2017;82(2):150-6.
https://doi.org/10.15417/694
22. Alves de Lima A, Henquin R, Thierer J. A qualitative study of the impact on learning of the mini-clinical evaluation exercise in postgraduate training. Med Teach 2005;27(1):46-52. https://doi.org/10.1080/01421590400013529
23. Fernández Gálvez GM. Evaluación de las competencias clínicas en una residencia de pediatría con el Mini-CEX
(Mini-Clinical Evaluation Exercise). Arch Argent Pediatr 2011;109(4):314-20. Disponible en: http://www.scielo.org.ar/scielo.php?script=sci_arttext&pid=S0325-00752011000400006&lng=es
24. Alves de Lima A. Devolución constructiva una estrategia para mejorar el aprendizaje. Medicina (Buenos Aires)
2008;68:88-92. Disponible en: http://www.scielo.org.ar/scielo.php?script=sci_arttext&pid=S0025-76802008000100014
25. Kant I. Filosofía de la historia. Buenos Aires: Terramar Ediciones; 2004:33.