The American Society of Anesthesiologists physical status (ASA-PS) is a grading system first developed in 1941 to “devise a system for the collection and tabulation of statistical data in anesthesia,” allowing anesthesiologists to record the overall health status of patients before surgery and to properly stratify outcomes [1]. The ASA-PS was profoundly modified in 1962 [2] when changes proposed by Dripps et al. [3] were adopted. The document has since been modified, with the latest amended version was released in 2014 [4].

Although definitions and clinical examples of each class are provided by ASA, doubts remain on the individual factors such as personal experience and workplace habits influencing assignment to an ASA-PS class. Several studies have investigated the inter-rater reliability of the ASA-PS in adult [5–8] and pediatric [9] populations, but only weak/moderate inter-rater agreements were found (k-values between 0.21 and 0.53). Nevertheless, possible associations between the ASA-PS scores and specific intraoperative and postoperative complications have been widely reported in the literature [10].

In fact, the ASA-PS has an important role for anesthesiologists: it is routinely used in clinical practice and in policy-making, performance evaluation, resource allocation, and reimbursement of anesthesia services. It is also frequently adopted as a reference parameter in clinical research. The ASA-PS score consists of six classes [4] (I to VI), with the addition of an “E” to any ASA-PS class that denotes patients needing emergency surgery.

To date, previous studies [11–13] have focused on inter-rater reliability, both within and between anesthesiologists and non-anesthesiologists, but no data are available on the effect of the level of anesthesiology experience and knowledge on assigning ASA-PS classes.

The primary aims of our study were to understand whether ASA-PS scores are properly assigned by anesthesiologists, in simulated, real-life clinical cases, and to investigate a possible link between the assigned ASA-PS score and the clinical experience/knowledge of the assigning anesthesiologists.

The secondary aim of our study was to investigate the inter-rater reliability between participants in relation to different clinical experiences.

An online questionnaire was created using SurveyMonkey internet-polling software (http://www.surveymonkey.com). The survey consisted of nine multiple-choice questions. The first question asked for the years of experience as an anesthesiologist: “anesthesiology resident,” “less than five years,” “five to nine years,” “ten to nineteen years,” “twenty or more years,” or other (emergency department physicians). The following eight questions pertained to clinical cases.

The questionnaire was created to allow respondents to remain anonymous. Electronic informed consent was obtained from each participant at the beginning of the survey and both the informed consent and questionnaire were written in Italian. The survey was administered to an Italian closed-group of anesthesiologists present on Facebook (http://www.facebook.com) and consisted of 4,901 members at the time of the survey. Admission to this group is allowed by a small group of administrators and only after each member’s identity and workplace have been identified and confirmed. Membership is permitted only to anesthesiologists, anesthesiology residents, and rarely to emergency department physicians. Questionnaires received from non-anesthesiologists were excluded because Italian emergency department physicians do not receive extensive anesthesiology training. Only one submission per computer, based on IP address, was allowed. The study was approved by the Institutional Ethical Committee of Padua (ref. AOP/2018).

The questionnaire presented eight clinical questions consisting of fictitious clinical cases. The task of each respondent was to assign the correct ASA-PS score, from I to VI, based on personal experience and knowledge. All cases were built following the ASA-PS Classification System and ASA-approved examples (Table 1) [10] to elicit an univocal correct ASA-PS score in each case.

The cases presented in the questionnaire are presented below. To further support reliance on the anesthesiologists’ own experience in completing the questionnaire, the ASA-PS classification table (Table 1) was not presented and responders were explicitly asked not access any ASA-PS guidelines before or while taking the survey.

Six hundred twenty-five questionnaires were submitted but 24 were excluded (3.9%). Specifically, 23 submissions were incomplete (3.7%) and one was submitted by an emergency medical doctor (0.2%). Thus, 601 respondents were included in the analysis: 514 were anesthesiologists, of which 150 (25%) had less than 5 years of work experience, 132 (22%) had 5 to 9 years of experience, 130 (21.6%) had 10 to 19 years of experience, and 102 (17%) had 20 or more years of experience. Eighty-seven respondents (14.7%) were anesthesiology residents.

Table 2 presents the distribution of the ASA-PS Classification System assignment for each group and each case. The correct answer is highlighted in asterisk.

The mean number of correct answers was significantly different among groups (P < 0.001). Post hoc analysis indicated that the highest mean number of correct answers was obtained by residents (3.95 ± 1.13) and was significantly higher compared to the “10 to 19 years” group (3.46 ± 1.24, P = 0.034) and “≥ 20 years” group (3.14 ± 0.96, P < 0.001). In fact, the “≥ 20 years” group recorded the lowest mean number of correct answers, performing significantly worse than the “< 5 years” group (3.76 ± 1.37, P < 0.001) and “6 to 9 years” group (3.83 ± 1.31, P < 0.001) (Fig. 1).

No other statistically significant differences were observed. Table 3 reports the Fleiss’ k for all raters and for each group.

The ASA-PS classification is used routinely by anesthesiologists worldwide. Recently, it was found to be an independent predictor of post-operative complications and mortality [11]. Thus, the ASA-PS could potentially be employed as an effective risk stratification metric for medical complications and mortality after surgery, but additional data are needed to confirmed these findings.

Nevertheless, the ASA-PS has often been criticized for its weakness in both adult and pediatric settings, and a moderate inter-observer reliability has been reported in several studies [5–9]. A recent systematic review concluded that physicians should always consider the moderate value of the inter-rater agreement of this classification (confirmed by wide variability throughout different studies), while administrative staff should rely less on the ASA-PS classification for billing procedures because of its heterogeneous reliability [12]. In fact, the current healthcare system emphasizes a pay-for-performance model instead of a fee-for-service model, which was used in the past, and hospitals have to develop performance metrics in order to receive reimbursement payments.

Nonetheless, as observed by Ranta et al. [7], the ASA-PS classification has some favorable features (such as ease of use and the simplicity at the bedside without the need for any technical device), which has also allowed its diffusion outside the field of anesthesiology.

The main finding of our study was that residents and less experienced anesthesiolgists were more accurate in assigning ASA-PS scores than those with more experience, i.e. according to our data, the more experienced anesthesiologists recorded the lowest number of correct answers.

In fact, according to our data, the more experienced anesthesiologists recorded the lowest number of correct answers. Also, the inter-rater reliability (estimated by Fleiss’ k) was weak/moderate in all experience level groups. In these less-experienced groups, better performance was observed in both the total number of correct answers and the mean difference between the assigned ASA-PS score and the correct one. The weak/moderate inter-rater reliability is consistent with previous studies [5–9].

Moreover, the inter-rater agreement varied widely among the cases. For example, almost all responders assigned the correct ASA-PS in Case 1, while their answers ranged from ASAPS I to IV in Case 7. Specifically, while the assignment of the correct ASA-PS to a healthy subject is easy (Case 1), the correct assignment to a multi-morbid patient could be potentially more difficult (Case 7). Indeed, while the ASA rigorously defines ranges and specific information for some clinical variables used to assign the ASA-PS (such as BMI or interval since myocardial infarction to operation), many other clinical variables (often related to as ‘systemic diseases’) are much more loosely defined. This leads some anesthesiologists to miss the more rigorously defined items (especially if they do not frequently refresh their knowledge), while many others are left alone with their clinical judgment when deciding whether a systemic disease should be classified as mild, severe, or a constant threat to life.

Moreover, the ASA provides a clear definition for each ASAPS class but only a few examples for each class, stating that “there is no additional information that will help you further define these categories” [4]. Hurwitz et al. [13] speculated that the ASA felt that anesthesiologists would use their training and own experience to supplement the definitions when assigning the ASAPS.

Based on this assumption, our findings showed that the more experienced anesthesiologists relied more on their clinical experience than on the strict ASA criteria and ‘official’ given examples when assigning a ASA-PS score. The ASA criteria and ‘official’ examples are free and easily available on the internet [4]. We did not insert explanations of the ASA-PS in the questionnaire, nor did we invite participants to refresh their knowledge before answering the questionnaire; on the contrary, responders were even explicitly asked not to refresh their knowledge before taking the survey. With this strategy, we tried to highlight the “experience” factor. As a final result, more experienced anesthesiologists were less accurate in classifying patients compared to less experienced colleagues.

As noted in a previous work [13], adding more clinical examples to the ASA-PS classification could possibly lead to an improvement of correct patient assignment. We speculate that a more exhaustive variety of examples of ASA-PS could improve accuracy in more experienced anesthesiologists.

Previous literature has shown that the inter-rater reliability of the ASA-PS designations is suboptimal [5–9]. In addition, our study showed that “younger” anesthesiologists performed better than “older” colleagues when use of ASA grading system was required. We hypothesize that less experienced physicians are probably more confident with clinical examples provided by the ASA compared to more experienced anesthesiologists who probably rely more on their experience.

This study has limitations. First, the survey was administered to a closed-group of Italian anesthesiologists on Facebook and not to all Italian anesthesiologists. Secondly, we only analyzed completed questionnaires and the exclusion of incomplete forms is a potential source of bias. Thirdly, we used fictitious clinical cases that may not be representative of the respondents’ clinical practice. Finally, we divided participants into five groups based on experience but the experience time period was not the same in all groups. Our investigation was conducted using a social network commonly used by the younger generation. Considering this potential source of bias, we aimed to obtain a similar sample size between the five groups. Thus we arbitrary divided the youngest population, excluding anesthesiology residents, into two groups with a shorter global experience, while the oldest population was divided into two groups with a longer global experience. We acknowledge that using the same experience period for all groups may have yielded different results.

Further studies, possibly with a larger sample size, are needed to clarify the clinical implication of our results. In particular, it would be interesting to re-evaluate the inter-rater agreement of the ASA-PS classification, even using the same sample, when participants are allowed to review the most recent guidelines before taking the survey.

In conclusion, the ASA-PS remains an important but often uncertain grading system that is routinely used by anesthesiologists despite its low inter-rater reliability and, as shown by our data, a strong reliance on work experience. Thus, a rational and careful application of the ASA-PS should be recommended, especially in clinical settings where wide differences in experience exist.