ABSTRACT

Introduction

The purpose of medical specialty training is to provide the physician with sufficient knowledge, skills, attitudes, and qualifications in the chosen field of medicine. In medical training, communication and social opportunities are crucial as well as appropriate goals, programs, and learning environments. The purpose of this paper is to develop a valid and reliable measurement tool to evaluate the compliance of this training with the standards.

Methods

In this survey-type research, in line with the literature, a draft scale to evaluate the training in both the department and the institution consisting of 49 items, was created.

Results

The study involved 497 residents from family medicine residency programs across Türkiye. The exploratory factor analysis revealed that the Scale for Department Evaluation of Medical Specialty Training had four factors whereas the Scale for Institution Evaluation of Medical Specialty Training had three factors. The factorial structures of both scales were confirmed by confirmatory factor analysis. The Cronbach’s alpha values were 0.941 and 0.928, respectively. Moreover, both scales explain a high amount of total variance.

Conclusion

The Department Evaluation Scale for Medical Specialty Training, consisting of 25 items and four factors, and the Institution Evaluation Scale for Medical Specialty Training consisting of 20 items and three factors are valid and reliable tools to study medical specialty training.

Introduction

Lately, there have been various studies in the literature, indicating that medical specialty training should be a process structured within certain standards, honor and ethical principles of medicine, shaped by the needs and expectations of society, and certified by objective-based measurements and evaluations (1-3).

The World Federation for Medical Education (WFME) reports three main categories, termed the “trilogy,” for quality improvement: basic medical education, postgraduate medical education, and continuing professional development (2). For post-graduate medical education WFME had eight areas to improve: which are “1. mission and values, 2. curriculum, 3. assessment, 4. postgraduate doctors, 5. teachers and clinical supervisors, 6. education and training resources, 7. quality improvement and 8. governance and administration” (2). In medical education, it is necessary to create awareness of the educational environment in the institution for both trainers and trainees, along with the content of the curriculum. How the educational environment is perceived by the trainees plays a key role in determining the quality of learning processes (4). WFME highlights the educational environment as one of the main areas for the evaluation of medical education programs (2, 4-10). The Türkiye Board of Medical Specialization Curriculum Development and Standardization also defines educational environments in detail, and highlights the importance of the matter (1). However, the number of publications on how it is evaluated by the trainees is quite small (8, 10-13). 

The purpose of this paper is to develop a valid and reliable tool that can evaluate and measure the compliance of the education received by family medicine specialty students with the standards, the educational and social opportunities, the administrative processes, and the training environment.

Methods

Type of research: Since the purpose of this research is to develop a measurement tool to evaluate the training received by family medicine specialty students, it qualifies as survey research.

The population and sample: the study population consists of family medicine residents who use the internet and social media and are receiving family medicine specialty training in Türkiye on account of the widespread use of social media, the large number of groups involving residents and the chance to reach more participants, the researchers decided to collect the data through the internet.

As the literature recommends that the sample size be at least 10 times the number of questions in the scale, the researchers aimed to reach 490 participants (14). The researchers adopted the convenience sampling method and shared the link to the questionnaire, which briefly contained the research purpose and was created using “Google Docs,” on “WhatsApp, Gmail, and Yahoo” groups. Those who followed the link and answered the questions were accepted as voluntary participants.

Ethics statement: Ethical approval was granted by Necmettin Erbakan University, Non-Interventional Clinical Research Ethics Committee (approval number: 2023/4720, date: 15/12/2023). This research was carried out in line with the Declaration of Helsinki. All participants were informed about the research and confirmed that they voluntarily participated in the research.

Data collection tools: The researchers created an item pool of 49 statements inspired by a practitioner tool used in a published thesis, main topics of national and international recommendations, and similar studies (1-3,8,9,11-13,15-17). The internal consistency coefficient of the questionnaire used in the abovementioned thesis was found to be α=0.940 for the departmental evaluation, and α=0.952 for the institutional evaluation in the specialty training; and it was considered to have high reliability (13).

The questionnaire used in this research consists of three sections. The first one collects the socio-demographic information of the participants, the second evaluates training in the department, and the third evaluates training in the institution as noted below in more detail:

Medical specialty training-department evaluation scale: This section has 25 items about the department in which the resident is receiving specialty training, including the conformity of the training to the standards, training opportunities, social relations in the department, qualifications of the trainers, training environment and academic development. The five-point Likert scale has “1- never, 2- rarely, 3- sometimes, 4- often, 5- always” response options indicating the level of agreement with the statements, has no reverse-coded items.

Medical specialty training-institution evaluation scale: This section has 24 items related to interpersonal relations, social facilities, and the educational environment of the institution in which the resident receives specialty training. The five-point Likert scale has “1- never 2- rarely, 3- sometimes, 4- often, 5- always” response options indicating the level of agreement with the statements, and has no reverse-coded items.

Statistical Analysis

The researchers used IBM SPSS AMOS (Version 23) in the data analysis. Exploratory and confirmatory factor analyses for construct validity, and Cronbach’s Alpha coefficient to check internal consistency were used and explained in detail in the results section.

Results

With an average age of 29.08±4.1 years, 497 specialty students participated in this research. Among the participants, 70.21% (n=349) were females, 57.33% (n=285) were married, and 30.2% (n=150) had children. 71.4% (n=355) of the participants were satisfied with their city of residence, and 65.24% (n=324) were satisfied with their chosen specialty. The median order of preference for family medicine in the medical specialty exam was 2 (1-57), and 60.91% (n=303) of the participants stated that they would choose family medicine again if they had a second chance.

Medical Specialty Training-Analysis of Department Evaluation Scale

Construct Validity

The researchers employed exploratory and confirmatory factor analyses for construct validity. Kaiser-Meyer-Olkin (KMO) and Bartlett’s tests were used to assess whether the data gathered during the pilot study were suitable for factor analysis. The results indicated that the KMO value (0.943) is acceptable (KMO >0.50), which means that the data is suitable for factor analysis (18). The results of Bartlett’s test showed that there are significant relationships between the variables and the data are suitable for factor analysis [χ²: 6587.550; standard deviation (SD): 300; p<0.001].

The Exploratory Factor Analysis (EFA)

In the analysis, the researchers used varimax rotation. As a consequence, 25 items with eigenvalues greater than 1.0 were grouped under four factors, explaining 58.68% of the total variance. This value is acceptable according to the literature (19). The factor loadings above 0.30 are acceptable, as the findings show that they exceed this threshold (20, 21).  Eigenvalues, factor loadings, and the total variance explained are shown in Table 1. As a result of the analysis, the researchers identified the factors related to the departmental evaluation of medical specialty training, taking the items and the relevant literature into account. Accordingly, they called the factors “training” items (1-9), “communication” items (10-18), “scientific activities” items (19-23), and “assessment and evaluation” items (24, 25).

The Confirmatory Factor Analysis (CFA)

To confirm the factorial structure, the researchers performed a confirmatory factor analysis. The correlation matrix in Table 2 indicates that the highest correlation is between the “training” and “communication” factors (r=0.781). The path diagram of the scale is given in Figure 1. Moreover, fit indices indicate a good fit to the factorial structure. [χ²=756.436, df=260, χ²/df=2.909 root mean square error of approximation (RMSEA)=0.062, comparative fit index (CFI)=0.923, goodness of fit index (GFI)=0.893, tucker-lewis index (TLI)=0.911 adjusted GFI (AGFI)=0.866] (21-23).

Internal Consistency

The researchers adopted Cronbach’s Alpha coefficient to check internal consistency. The coefficients for the factors of the scale are 0.895 for the “training” factor; 0.890 for the “communication” factor; 0.871 for the “scientific activities” factor; and 0.705 for the “assessment and evaluation” factor. As for the whole scale, it is 0.941 (Table 1). Accordingly, the scale is highly reliable (24).

Medical Specialty Training-Analysis of Institution Evaluation Scale

Construct Validity

The construct validity of the scale was tested using exploratory and confirmatory factor analyses. To check whether the data are suitable for factor analysis, the researchers employed KMO and Bartlett’s tests. The results indicated that the KMO value (0.935) is acceptable (KMO >0.50), which means that the data are suitable for factor analysis (18). The results of the Bartlett’s test also showed that there are significant relationships between the variables and the data are suitable for factor analysis (χ²: 5185.997; SD: 190; p<0.001).

The Exploratory Factor Analysis (EFA)

In EFA, the researchers used the varimax rotation method. They included 24 items in the analysis, although some of them (items 26, 32, 33, 46) were excluded from the analysis since they had high factor loadings on more than one factor with loading differences below 0.10 (25, 26). As a consequence, 20 items, with eigenvalues greater than 1.00, were grouped into three factors explaining 57.94% of the total variance. Eigenvalues, factor loadings, and the total variance explained are shown in Table 3.

As a result of the analysis, the researchers identified factors related to the institutional evaluation of medical specialty training, taking the items and the relevant literature into account (2). Consequently, they called the factors “environment-relationships” items (1-9), “social facilities” items (10-15), and “training environment” items (16-20).

The Confirmatory Factor Analysis (CFA)

To confirm the factorial structure, the researchers did a confirmatory factor analysis. The correlation matrix in Table 4 indicates that the highest correlation is between the “training environment” and “social facilities” factors (r=0.572). The path diagram of the scale is given in Figure 2. In addition, the fit indices to the factorial structure indicate that there is a good fit. (χ²=539.211; df=161; χ²/df=3.349; RMSEA=0.069; CFI=0.925; GFI=0.903; TLI=0.912; AGFI=0.873) (21-23).

The Internal Consistency

The alpha coefficients for the factors of the scale are 0.905 for the “environment-relationships” factor, 0.786 for the “social facilities” factor, and 0.837 for the “training environment” factor. Regarding the whole scale, the reliability coefficient is 0.928 (Table 3). In brief, the scale is highly reliable (24).

Discussion

The effects of training processes and training environments on learning are of growing importance. The training environment can exert a positive or negative influence on student motivation. It has been asserted that a comprehensive understanding of the components of the educational environment, held by both students and trainers, can serve as a foundation for establishing arrangements that are conducive to learning objectives (4-10). Thus, the literature review indicates that several papers have studied the quality of training (7-9,11-13,15-17). However, the fact that almost none of those studies used a structured and tested scale has affected the results and generalizability of the studies, and prevented the formation of a common standard. That’s why the present study is of high importance, both nationally and internationally, as it addresses the lack of a measurement tool. In this paper, the researchers developed two scales that can be used separately or together to evaluate medical specialty training. The Medical Specialty Training Department Evaluation Scale, used to evaluate specialty training in the department, consists of 25 items and four factors. Similarly, the “Medical Specialty Training Institution Evaluation Scale” is a valid and reliable tool meant to evaluate the reflections of specialty training in the institution, consisting of 20 items and three factors.

The study involved 497 family medicine residents from educational institutions across Türkiye The data collected from family medicine residents, who are crucial in specialty training and have the opportunity to observe the training process from different perspectives due to their training rotations, are the strength of this paper. Moreover, some expressions, specific to particular medical disciplines, were omitted from the first study, thereby making the scale useful for all medical specialties. Alpha values of the measured scales are 0.941 and 0.928, respectively, which indicates high reliability.

The WFME emphasizes that in postgraduate medical education it is essential to define the requirements of education and to follow a systematic educational programme that defines the general and discipline-specific components of education. However, in a limited number of studies conducted in Türkiye or in specific hospitals, residents are observed to complain about the limitations of educational and research opportunities, and that service delivery is prioritized over their education (11, 12). However, one of the most basic standards set by the WFME and the Board of Specialization in Medicine Curriculum Creation and Standard Setting System (TUKMOS in Turkish), is to prioritize the education of residents and to ensure that the integration of health services is structured according to education (1, 2). In fact, one study showed that residents’ satisfaction with their education increased with their participation in scientific activities (17). A much larger study in Japan also found that participation in scientific activities increased satisfaction with education (27). We believe that the present study will make an important contribution to this field by providing a scale related to the quality of education, which is emphasized by both the WFME and TUKMOS, but has not yet been measured in our country with a structured data collection tool.

Study Limitations

The voluntary participation, data collection via internet, and study population being limited to only family medicine residents may introduce bias. The scale and items developed in this study cannot be discussed in detail. This is due to the fact that there is no other scale related to the subject for comparison.

Conclusion

In conclusion, the scales are valid and reliable, and can be used to evaluate the training received by the residents. In addition, when the scales are used together, (department and institution) they provide a thorough evaluation. Moreover, the scales can also be used independently to eliminate the positive or negative effects of the institution on the department or vice versa. It is thought that both scales will become more effective and efficient as they are used in different fields of expertise and institutions. The scales, as used in new research, will be able to evaluate the educational processes more objectively.