Inter-rater and Intra-rater Reliability of Novice Clinician Users of Diagnostic Ultrasound to Assess Anterior Talofibular Ankle Ligament Length
Abstract
Purpose:
To assess the reliability of athletic trainers to use diagnostic ultrasound to measure anterior talofibular ligament length.
Methods:
Seven clinician raters who were athletic trainers measured the length of the anterior talofibular ligament in 16 healthy ankles over two different data collection periods. Inter-class correlational coefficients (ICCs) were used to determine reliability across clinicians.
Results:
On average, clinician raters demonstrated poor inter-rater reliability in ankle neutral (ICC2,k = 0.46, P = .04) and in inversion (ICC2,k = 0.24, P = .14), but both improved across days (day 2 neutral ICC2,k = 0.51, P = .002; day 2 inversion ICC2,k = 0.37, P = .100). Individual clinicians ranged from poor to good overall intra-rater reliability for both neutral (range ICC2,k = 0.22 to 0.81) and inversion (range ICC2,k = 0.14 to 0.75).
Conclusions:
For novice raters, ankle ligament length may be unreliable. An improved level of reliability needs to be established prior to widespread application and implementation of the ultrasound as a clinical diagnostic tool for assessing ankle injuries.
[Athletic Training & Sports Health Care. 2021;13(6):e425–e431.]
Introduction
Ankle sprains are the most common injuries in sports.1 The National Collegiate Athletic Association reported more than 16,000 annual ankle sprains in a 6-year study spanning 25 sports.2 However, it has been noted that many people who have sustained an ankle sprain do not report it, leading to inadequate healing and outcomes.3 Improper treatment increases the risk of re-injury and the potential for future misdiagnosis of subsequent injuries.4
The gold-standard clinical assessment for ankle injuries involves a physical examination,4,5 days after injury, that assesses both sensorimotor and mechanical impairments.5 However, research suggests that clinicians have limited knowledge of the full range of impairments associated with a lateral ankle sprain.5 Due to the volume of swelling and possible hematoma, many diagnostic examinations and assessments can be unreliable. The anterior drawer test, one of the primary selective tissue tests used to diagnose ankle instability, may be inaccurate due to the immense amount of swelling and pain caused by the injury.6 Stress radiologic imaging, another common diagnostic tool, may also be inaccurate due to increased pain levels.7 Stress radiographs can differ based on the position of the patient's foot, the amount of laxity of the talocrural ligament, the amount of force applied, and the patient's ability to resist the force applied.7 Disadvantages to stress radiography include a high cost and high amount of radiation exposure, and it is unable to visualize soft tissue.7 Magnetic resonance imaging (MRI) may be a better diagnostic tool because it does not use ionizing radiation and allows visualization of soft tissue (eg, ligaments).7 MRI demonstrates an increased sensitivity when compared to other diagnostic tools; however, with increased sensitivity comes an increase in cost, procedure time, and patient discomfort.7
Diagnostic ultrasound (DUS) is an alternative option to other imaging modalities and may improve diagnostic capabilities for ankle injury at the point of care. Using DUS allows for rapid and enhanced viewing of the injured structures at low cost to the patient with no radiation transmission.8 The ability to provide early diagnosis can optimize care and reduce time lost to injury, decrease the rate of complications from injury, and allow clinicians to provide optimal care to their patients.8 DUS allows the clinician to receive real time images of internal structures and can potentially reduce diagnosis delays.8 Liu et al9 found 90% agreement between the DUS and MRI at detecting soft tissue injuries. Although MRIs are valuable for diagnosing bone and joint injuries, one limitation is that they cannot indicate stability of the ankle due to the static testing position. In addition, the sensitivity of DUS to detect ankle injury is comparable to MRI.10 Thus, DUS may be more advantageous compared to MRI based on its cost effectiveness, efficiency, and portability.
An ankle sprain may be able to be detected by the increased measurement of the length of the anterior talofibular ligament (ATFL).11 However, DUS as an effective option for ankle imaging and diagnosis has not been recommended due to the lack of research aimed at establishing its diagnostic utility and reliability for musculoskeletal conditions.4,8,12 Previous studies have found that those who have sustained an ATFL injury have greater ATFL length (1.45 ± 0.23 cm) compared to normative values (1.19 ± 0.25 cm).13 Thus, measuring the length may provide clinicians with enough information to detect that a ligamentous injury has occurred.
Despite research on the accuracy of DUS, there is a dearth of evidence to support the reliability of measuring ligaments with this diagnostic modality. This gap in the literature may lead to a lack of use in practice.14 Therefore, the purpose of this study was to assess the reliability among novice athletic trainers using DUS to measure the length of the ATFL in healthy individuals.
Methods
Participants
Participants were recruited from the local population as a sample of convenience. Two groups of participants were selected: clinician raters and ankle participants. Clinician raters (Table 1) were included in the study if they were licensed athletic trainers in the state of Nebraska and willing to participate in the study. Ankle participants (Table 2) were included in the study if they were adults aged 19 years and older. Ankle participants were excluded if there was a previous ankle injury or any injury that required them to be non–weight-bearing for more than 3 days, had any lower extremity surgery, and any current ankle injury. Thus, ankle participants included in this study were eligible with a single ankle or both ankles that were not injured previously. This study was approved by the local human subjects' review board and all clinician raters and ankle participants were provided consent prior to collection.
| Clinician ID | Gender | Age (y) | Years of Clinical Experience | Previous Experience With DUS |
|---|---|---|---|---|
| 001-000 | F | 24 | 2 | N |
| 002-000 | M | 34 | 13 | Y |
| 003-000 | M | 28 | 8 | Y |
| 004-000 | F | 35 | 11 | N |
| 005-000 | M | 24 | 2 | N |
| 006-000 | M | 31 | 9 | Y |
| 007-000 | F | 34 | 10 | N |
| Mean ± SD | 30.0 ± 4.7 | 7.9 ± 4.3 |
| Participant ID | Mass (kg) | Height (cm) | Age (y) | Gender |
|---|---|---|---|---|
| 000–001 | 80.7 | 162.6 | 24 | F |
| 000–002 | 69.0 | 180.3 | 25 | M |
| 000–003 | 79.4 | 165.1 | 24 | F |
| 000–004 | 54.4 | 157.5 | 21 | F |
| 000–005 | 104.3 | 175.3 | 24 | M |
| 000–006 | 69.9 | 170.2 | 23 | F |
| 000–007 | 59.0 | 160.0 | 24 | F |
| 000–008 | 113.4 | 188.0 | 26 | M |
| 000–009 | 83.91 | 180.3 | 31 | M |
| 000–010 | 22.8 | 188.0 | 21 | M |
| 000–011 | 77.1 | 175.3 | 21 | M |
| Mean ± SD | 69.9 ± 24.4 | 173.0 ± 10.2 | 24.0 ± 2.7 |
Instrumentation
The Lumify Ultrasound Unit (Phillips, Inc) and proprietary application, Lumify (v1.9), was used. The machine uses an L12-4 transducer wand (Koninklijke Philips 2018) that plugs into an Android-based tablet device (Samsung Galaxy Tab S3; Samsung Electronics Co).15,16 This transducer allowed the clinician to visualize musculoskeletal structures with high detail.15 The Lumify device uses broadband technology to produce a linear array frequency range from 12 to 4 mHz and an aperture of 38.4 mm.15 The Lumify application has a range of different modes such as regional anesthesia, vascular, superficial musculoskeletal, lung, and traumatic implications.15 The specific mode used for this study was the musculoskeletal mode.
Protocol
The primary investigator (JG) received in-depth training from a musculoskeletal DUS clinical expert on how to use the device. The primary investigator's training session included exploring the different modes, how to properly use the measurement tools within the application, and locating the ATFL. Multiple practice sessions that ranged from 1 to 2 hours/week for 2 months were completed to ensure proficiency with the DUS and measures. This was accomplished so that the primary investigator could properly demonstrate and teach the clinician raters how to use the device. After the primary investigator was proficient at identifying the ATFL, each clinician received a brief familiarization session that lasted anywhere from 30 to 60 minutes or until the clinician rater felt adequate at using the device. During this session, clinician raters learned to properly use the DUS and to identify the ATFL. To simulate real-world application for an athletic training staff where a DUS is often purchased with an initial training session, no additional instruction or assistance was provided to the clinician raters once data collection began. Before measurements were taken, musculoskeletal settings (brightness and zoom) were standardized and consistent across participants. During data collection, clinician raters placed the device in a position that they could view the image properly and only touched the DUS to freeze images and to use the measurement tool to measure the ATFL. These procedures were implemented to simulate how the device may be used in a clinical setting such as an athletic training facility.
Ankle participants laid prone with their ankle off the edge of a treatment table. Each healthy ankle was placed in two different positions, neutral and inversion, for measurements. Neutral was defined by how the participant's ankle was positioned at rest, resulting in slight plantarflexion. This position was chosen instead of using zero degrees of dorsiflexion/plantarflexion and zero degrees of inversion/eversion because using the patient's neutral position in reference to their body is more clinically relevant and comfortable for the participant than placing each participant in zero degrees. Inversion was measured with a goniometer and the position was maintained by the primary investigator holding the ankle in inversion during the measurements.17 Based on pilot testing and previous work, 15 degrees of inversion, relative to zero degrees of inversion, was chosen for consistency and is a standardized movement that each participant was able to meet.18 Length measurements were taken three separate times and means scores calculated. Mean scores were used for statistical analysis. The clinicians were instructed to measure the length of the ATFL from the origin on the lateral malleolus to the insertion point on the talus. Once the image was frozen on the tablet screen, the clinician used the distance measurement tool within the DUS application to measure the length. Between each measurement, the DUS probe was removed from the participant's skin and each time the clinician had to relocate the ATFL for a total of nine measurements per ankle. Measurements were collected in two different data collection sessions that were at least 7 days apart and the same ankle participants were used in both days. Ankle participants were asked at the second day if they had suffered any injuries between sessions, but none reported any injuries.
Statistical Analysis
Intraclass correlation coefficients (ICC2,k), Cronbach's alpha, and standard error of the measurement (SEM) were used to assess the inter-rater and intra-rater reliability of clinician raters to assess the ATFL length via DUS. Alpha levels were set a priori to a P value of less than .05. ICCs were considered poor if they were less than 0.5, moderate between 0.50 and 0.75, good between 0.75 and 0.90, and excellent at greater than 0.90.19 The SEM represents how much variability each clinician had when performing the different measurements, with lower values indicating improved accuracy.
Results
Table 3 displays the summary statistics for inter-rater reliability in neutral and inversion for the clinician raters. One clinician's data were excluded due to being an outlier and having missing data points, leaving 7 of the 8 clinician raters data to be evaluated. Overall, the clinician's inter-rater reliability improved from day 1 to day 2. One clinician was deemed to be an outlier and was removed from the data collection due to significantly longer ligament measurements (> 3 standard deviations). For day 1, clinician raters had poor intra-rater reliability while measuring in the neutral position, but they improved to what was considered good reliability for day 2.
| Parameter | ICC | P | 95% CI | Cronbach's Alpha | Mean Ligament Length (cm) | SD Ligament Length | SEM |
|---|---|---|---|---|---|---|---|
| Neutral | |||||||
| Day 1 | 0.463 | .041 | −0.082 to 0.785 | 0.463 | 1.611 | 0.199 | 0.381 |
| Day 2 | 0.510 | .002 | 0.190 to 0.732 | 0.510 | 1.682 | 0.183 | 0.392 |
| Inversion | |||||||
| Day 1 | 0.244 | .137 | −0.249 to 0.587 | 0.244 | 1.8599 | 0.194 | 0.466 |
| Day 2 | 0.367 | .100 | −0.275 to 0.747 | 0.367 | 1.863 | 0.240 | 0.502 |
Individual intra-rater reliability and SEM for clinician raters were calculated and are displayed in Tables 4–5. Individual raters for both neutral and inversion ranged from poor to good for both neutral and inversion. The clinicians with a lower SEM had more consistent measurements when repeatedly measuring the ATFL. Clinician raters overall had lower SEM values in neutral compared to those in inversion. This indicates that there was less variability of the clinician raters at measuring the ATFL in the neutral position compared to inversion.
| Clinician | ICC | 95% CI | Cronbach's Alpha | Mean Ligament Length (cm) | SD Ligament Length | SEM |
|---|---|---|---|---|---|---|
| 001 | 0.480 | −0.487 to 0.818 | 0.476 | 1.902 | 0.253 | 0.300 |
| 002 | 0.567 | −0.307 to 0.852 | 0.552 | 1.108 | 0.116 | 0.129 |
| 003 | 0.497 | −0.295 to 0.817 | 0.522 | 3.269 | 0.344 | 0.412 |
| 005 | 0.217 | −1.440 to 0.734 | 0.207 | 1.287 | 0.112 | 0.149 |
| 006 | 0.807 | 0.468 to 0.932 | 0.819 | 1.191 | 0.165 | 0.134 |
| 007 | 0.619 | −0.116 to 0.868 | 0.609 | 1.821 | 0.203 | 0.213 |
| Clinician | ICC | 95% CI | Cronbach's Alpha | Mean Ligament Length (cm) | SD Ligament Length | SEM |
|---|---|---|---|---|---|---|
| 001 | 0.356 | −0.711 to 0.769 | 0.367 | 2.094 | 0.232 | 0.300 |
| 002 | 0.597 | −0.197 to 0.861 | 0.584 | 1.103 | 0.160 | 0.172 |
| 003 | 0.139 | −1.750 to 0.710 | 0.132 | 3.325 | 0.350 | 0.480 |
| 005 | 0.575 | −0.078 to 0.845 | 0.637 | 1.466 | 0.106 | 0.119 |
| 006 | 0.752 | 0.268 to 0.914 | 0.740 | 1.199 | 0.166 | 0.1478 |
| 007 | 0.617 | −0.147 to 0.686 | 0.602 | 1.981 | 0.286 | 0.303 |
Discussion
Overall, the results of this study indicated that there was poor reliability across clinician raters for all days except neutral day 2, which increased to moderate for all clinicians. These improvements may be attributed to a training effect that occurred as the clinicians became more familiar with the testing protocol and with the device and potentially due to the lack of in-depth training sessions. Intra-rater reliability for clinicians demonstrated good to excellent reliability, but when analyzed individually, clinicians showed mixed results. This may have significant implications on how clinicians integrate DUS at the point of care.
Clinician raters demonstrated moderate inter-rater reliability in neutral and poor reliability in inversion. This decrease could be due to the inability to consistently maintain the proper patient positioning.20,21 In the current investigation, we standardized ankle position at 15 degrees of inversion. However, the consistency at maintaining the proper amount of inversion may be adversely affected by human error due to the fact that inversion was maintained by the primary investigator manually holding this position. In the future, a foot-ankle arthrosis or an arthrometer may be necessary to improve consistency between patients.21 A study that used a foot arthrosis has shown that there was an excellent reliability in measuring ligaments in healthy ankles and a moderate to excellent reliability at measuring ligaments in participants with injured ankles.21 However, Abdeen et al21 used sonographers who had many years of experience using the DUS, whereas the current study used clinicians with minimal DUS experience. Therefore, it is difficult to determine which was more effective in improving the reliability, the sonographer or the arthrosis.
Based on the results, it appears that the clinicians demonstrated an improved reliability from day 1 to day 2 in the neutral position. This improvement from poor to moderate suggests that as clinicians became more familiar with and had additional practice using the DUS, the more reliable they became. This increase in reliability demonstrates the need for longer training periods for the use of this device.22 Similarly, clinician raters with more clinical experience demonstrated better individual intra-rater reliability than those with less than 5 years of clinical experience. These data suggest that the more practice clinicians have, the more consistent they will be at using this device. Other research also indicates that the more practice clinicians have, whether it be clinical practice or repeated use of the DUS, the more consistent they will become.21,22 An experienced examiner is a common practice in studies that have assessed the reliability of clinicians at using DUS.21–23 For DUS to become more mainstream, clinicians need more exposure and experience.8,22 On the contrary, other research suggests that a minimum of 2 years' experience is necessary to reproduce excellent reliability.12,20 The primary investigator received multiple training and familiarization sessions to be able to explain and show clinicians the proper use and location of the ATFL measurement sites to the clinician raters. The clinicians were offered a familiarization and training period for as long as they felt they needed to become confident in using the DUS. Clinicians often overestimate their ability to assess and diagnose, even with the addition of diagnostic imaging.24 Based on the results of this study, novice clinician raters in particular should be overly cautious of interpreting DUS without extensive training.
Limitations
There are several potential limitations to this study. A single familiarization session on how to use DUS may not be enough training for someone to properly use and identify an injured structure. Another limitation is the small sample size. Only eight clinicians were able to participate in this study and one clinician was excluded due to being an outlier. This small sample size provides a limited amount of knowledge that might be better represented with a larger sample size. Along with a small sample size, there was a wide range of clinical experience (average = 7.9 years), and specific experience or training sessions with DUS data pertaining to clinician raters were not collected. Previous research used a minimum of 10 years of experience.21,22 Thus, to improve the generalizability a larger, diverse sample is suggested. Similarly, another limitation of this study is the number of ankle participants. The mean age of the ankle participants was 24.0 ± 2.7 years, making this study more clinically applicable to the young adult population. To provide normative values for ankle ligament characteristics, a more diverse sample of younger and older participants is necessary. Another clinical consideration is that injured ankles may have significant differences in visualization compared to healthy ankles. Therefore, this study should be repeated in individuals who have sustained an ankle injury to determine the accuracy of DUS at detecting ankle structures in different anatomical positions.
Future studies should emphasize consistency between testing position by using a foot-ankle arthrosis or an arthrometer should be used to ensure consistency between patients. An in-depth and repetitive training session should also be used to help determine how long or how many sessions a person needs to demonstrate proper accuracy and consistency when using the device. An increased level of reliability needs to be determined before this device can be properly implemented into clinical practice.
Implications for Clinical Practice
When used properly, DUS is a useful tool that can visualize soft tissue structures and can easily be integrated into routine point-of-care settings as a diagnostic tool for lateral ankle sprains. For instance, a sprained ligament would be reflected by an increased length of the ATFL relative to the length prior to injury.11 By producing real-time images of structures at a low cost to patients, DUS can help provide accurate diagnosis, treatment, and rehabilitation guidance for injured athletes.6
Currently, a lack of a standardized approach in training, imaging protocol, and patient positioning with acute ankle injuries limits the full implementation of this device. The goal of this study was to assess the reliability of clinicians at using this device on healthy ankles to provide a baseline of reliability to inform and assist in clinical decision-making. This study showed that reliability is poor for novice DUS users. Clinicians seeking to integrate DUS into clinical practice should proceed with caution and rely on expert clinicians and those with DUS experience to accurately use these devices. Clinicians looking to learn the proper use of this device should seek out a formal training course from a trained individual or attend continuing education events and workshops to improve their skills.
Conclusion
Because ankle injuries are one of the most common injuries within athletics, the addition of the DUS along with proper physical examination may help reduce the cost and time needed to reach an accurate diagnosis. This study demonstrated that reliability can increase with increased exposure to the diagnostic tool as shown from the data between day 1 and 2 measurements. Clinical use of the DUS may not become viable unless higher reliability is achieved via a widespread application and integration. Clinicians with a greater amount of clinical experience and previous exposure the DUS demonstrated a moderate reliability when compared to those who have fewer years of clinical experience and no previous DUS exposure. However, to provide improved DUS reliability, protocols need to be standardized and an in-depth training session needs to be completed before the DUS can be fully implemented. Although this device is easy to use, this study demonstrates that more than minimal training is necessary to become proficient to correctly use this device.
- 1.. An updated model of chronic ankle instability. J Athl Train. 2019; 54(6):572–588. doi:
10.4085/1062-6050-344-18 31162943 > CrossrefGoogle Scholar - 2.. The epidemiology of lateral ligament complex ankle sprains in national collegiate athletic association sports. Am J Sports Med. 2017; 45(1):201–209. doi:
10.1177/0363546516660980 > CrossrefGoogle Scholar - 3.. Lack of medical treatment from a medical professional after an ankle sprain. J Athl Train. 2019; 54(6):671–675. doi:
10.4085/1062-6050-428-17 31116568 > CrossrefGoogle Scholar - 4.. Functional anatomy, pathomechanics, and pathophysiology of lateral ankle instability. J Athl Train. 2002; 37(4):364–375. > Google Scholar
- 5.. Clinical assessment of acute lateral ankle sprain injuries (ROST): 2019 Consensus Statement and Recommendations of International Ankle Contsoritum. Br J Sports Med. 2018; 52:1304–1310. doi:
10.1136/bjsports-2017-098885 29886432 > CrossrefGoogle Scholar - 6.. Management of acute lateral ankle ligament injury in the athlete. Knee Surg Sports Traumatol Arthrosc. 2013; 21(6):1390–1395. doi:
10.1007/s00167-012-2252-7 > CrossrefGoogle Scholar - 7.. The efficacy of ultrasound in the evaluation of dynamic scapholunate ligamentous instability. J Bone Joint Surg Am. 2004; 86(7):1473–1478. doi:
10.2106/00004623-200407000-00016 15252095 > CrossrefGoogle Scholar - 8.. A review and proposed rational for the use of ultrasonography as a diagnostic modality in the identification of bone stress injuries. J Ultrasound Med. 2018; 37(10):2297–2307. doi:
10.1002/jum.14588 29655254 > CrossrefGoogle Scholar - 9.. Increased ligament thickness in previously sprained ankles as measured by musculoskeletal ultrasound. J Athl Train. 2015; 50(2):193–198. doi:
10.4085/1062-6050-49.3.77 > CrossrefGoogle Scholar - 10.. Value of stress ultrasound for the diagnosis of chronic ankle instability compared to manual anterior drawer test, stress radiography, magnetic resonance imaging, and arthroscopy. Knee Surg Sports Traumatol Arthrosc. 2016; 24(4):1022–1028. doi:
10.1007/s00167-015-3828-9 > CrossrefGoogle Scholar - 11.. Differences in lateral ankle laxity measured via stress ultrasonography in individuals with chronic ankle instability, ankle sprain copers, and healthy individuals. J Orthop Sports Phys Ther. 2012; 42(7):593–600.
10.2519/jospt.2012.3923 22446334 > CrossrefGoogle Scholar - 12.. Validity of musculoskeletal ultrasound for identification of humerora-dial joint chondral lesions: a preliminary investigation. J Athl Train. 2014; 49(1):7–14. doi:
10.4085/1062-6050-49.1.03 24377960 > CrossrefGoogle Scholar - 13.. The anterior talofibular and calcaneofibular ligaments: an anatomic study. Surg Radiol Anat. 2013; 35(6):511–516. doi:
10.1007/s00276-012-1071-3 23292089 > CrossrefGoogle Scholar - 14.. Comparative study of clinical and ultrasonographic evaluation of lateral collateral ligament sprains of the ankle. J Sports Med Phys Fitness. 2009; 49(3):285–291.
19861934 > Google Scholar - 15.. Lumify Phililips Transducer (L12-4). Published 2018. https://www.lumify.philips.com/web/clinical-solutions/msk > Google Scholar
- 16.Samsung Galaxy Tab S3. Samsung Electronics Co., Seoul, South Korea. > Google Scholar
- 17.. Evaluation of anterior talofibular ligament injury with stress radiography, ultrasonography and MR imaging. Skeletal Radiol. 2010; 39(1):41–47. doi:
10.1007/s00256-009-0767-x > CrossrefGoogle Scholar - 18.. Comparison of ankle arthrometry to stress ultrasound imaging in the assessment of ankle laxity in healthy adults. Int J Sports Phys Ther. 2011; 6(4):297–305.
22163091 > Google Scholar - 19.. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016; 15(2):155–163. doi:
10.1016/j.jcm.2016.02.012 27330520 > CrossrefGoogle Scholar - 20.. A dynamic ultrasound examination for the diagnosis of ankle syndesmotic injury in professional athletes: a preliminary study. Am J Sports Med. 2009; 37(5):1009–1016. doi:
10.1177/0363546508331202 19336613 > CrossrefGoogle Scholar - 21.. Ultrasound characteristics of foot and ankle structures in healthy, coper, and chronically unstable ankles. J Ultrasound Med. 2019; 38(4):917–926. doi:
10.1002/jum.14770 > CrossrefGoogle Scholar - 22.. Ultrasonographic assessment of the anterolateral ligament of the knee in healthy participants. Muscles Ligaments Tendons J. 2017;( 3):485–490.
10.11138/mltj/2017.7.3.485 > CrossrefGoogle Scholar - 23.. Ultrasound assessment for grading structural tendon changes in supraspinatus tendinopathy: an inter-rater reliability study. BMJ Open. 2016; 6(5):e011746
10.1136/bmjopen-2016-011746 27221128 > CrossrefGoogle Scholar - 24.. Overconfidence as a cause of diagnostic error in medicine. Am J Med. 2008; 121(5)(suppl):S2–S23.
10.1016/j.amjmed.2008.01.001 18440350 > CrossrefGoogle Scholar
