Randomized controlled trials (RCTs) and their meta-analyses provide the most reliable evidence for medical interventions. Given a sufficient number of participants, random assignment to control or intervention groups produces baseline characteristics comparable between the two groups. The difference in the outcomes of interest produced by control or treatment interventions given to the respective groups represents the causal effects of the intervention of interest on the outcomes. However, the scientific reliability of the results of RCTs is challenged by recognized or unrecognized flaws in their design or conduct, analysis, and reporting of results, which underestimate or overestimate the true intervention effect [1].

Bias is defined as systemic error or deviation from the parameter of a population that affects the true intervention effect [2]. It is completely different from imprecision which refers to random error representing the difference in effect estimates upon multiple repetition of the same trial due to sampling variation. Bias caused by inadequate concealment or report of randomization exaggerates the estimate of intervention effect compared to adequate concealment or report [3]. Even the absence of a description of double blinding causes a similar, albeit less significant, result. Hence, avoiding bias maintains the reliability of RCT results. Bias can be classified into six categories, namely, selection, performance, attrition, detection, reporting, and other biases, according to the Cochrane handbook [2]. Therefore, we determined the validity of the quasi-RCTs or RCTs reported in the Korean Journal of Anesthesiology (KJA) between 2010 and 2016 by appraising the risk of these six biases.

Of the 497 studies retrieved after excluding editorials, review articles, experimental studies using non-human species, case reports, letters to the editor, corrigenda or errata, and opinions from the 1,431 papers published in the KJA between 2010 and 2016, 296 quasi-RCTs and RCTs were finally identified following intensive discussion (Cohen's kappa = 0.929). The Cohen's kappa values were 0.927, 0.322, 0.616, 0.676, 0.630, 0.673, 0.874, and 0.472 for random sequence generation, allocation concealment, blinding of participants, blinding of personnel, blinding of outcome assessment, incomplete outcome data, selective reporting, and other bias, respectively.

The proportion of ‘Low risk’ of bias was highest in incomplete outcome data (84.8%) and lowest in selective reporting (4.7%) over the entire period (2010–2016) (Fig. 2). The proportion of ‘Unclear risk’ of bias was highest in selective reporting (94.3%) and lowest in incomplete outcome data (11.1%). The highest proportion of ‘High risk’ of bias was observed in blinding of personnel (20.6%), while the lowest proportion was observed in random sequence generation (0.7%). Compared to random sequence generation, in which the proportions of ‘Low risk’ and ‘Unclear risk’ of bias were 51.7% and 47.6%, respectively, allocation concealment had a lower proportion of ‘Low risk’ of bias (20.9%) and a higher proportion of ‘Unclear risk’ of bias (69.6%). Blinding of outcome assessment was performed better than blinding of participants and blinding of personnel (42.9% vs. 15.5% and 23.0% for the percentage of ‘Low risk’ of bias, respectively).

The risk of bias for random sequence generation significantly improved from the beginning of the analysis period and reached 100% of ‘Low risk’ of bias in 2016 (P for trend < 0.001) (Fig. 3), while that for allocation concealment has remained unimproved (P for trend = 0.943), with the percentage of ‘Low risk’ of bias < 40% and that of ‘Unclear risk’ of bias > 60% (Fig. 4). Despite the lower proportion of ‘Low risk’ of bias compared to ‘Unclear risk’ of bias, the proportion of ‘Low risk’ of bias for blinding of participants increased since the earliest year of the analysis (2010) (P for trend = 0.003) (Fig. 5). Similarly, the proportion of ‘Low risk’ of bias was lower than that of ‘Unclear risk’ of bias regarding blinding of personnel. However, no improvement in the risk of bias was detected (P for trend = 0.896) (Fig. 6). The proportion of ‘Low risk’ of bias was higher in blinding of outcome assessment than in blinding of participants and personnel, with a significant improvement (P for trend = 0.026) (Fig. 7). The percentage of ‘Low risk’ of bias for incomplete outcome data was maintained at a high level (≥ 80%) throughout the analysis period (Fig. 8). The percentage of ‘Low risk’ of bias for selective reporting was 0% until 2012 and increased to 27.3% in 2016 (P for trend < 0.001) (Fig. 9). The risk of bias for other bias improved significantly between 2011 and 2012 (P for trend < 0.001) (Fig. 10).

This analysis shows that authors who contributed to the KJA between 2010 and 2016 are aware of the importance of random sequence generation, blinding of outcome assessment, and incomplete outcome data when conducting RCTs. In contrast, they missed the significance of allocation concealment, blinding of participants and personnel, and selective reporting. The risk of bias for random sequence generation, blinding of participants, blinding of outcome assessment, selective reporting, and other bias improved during the analysis period, whereas that of allocation concealment and blinding of personnel did not. Authors have dealt with incomplete outcome data appropriately since the earliest year of the analysis.

If the next allocation, which is a new promising treatment group, is open to medical personnel or patients prior to the application of the assigned treatment in an RCT to prove that the new treatment is more effective than the control treatment, medical personnel would subconsciously try to enroll the next patient who was expected to produce a favorable result and the next patient would want to receive the new treatment. Accordingly, inadequate concealment of allocation exaggerates the estimate of the intervention effect, particularly in trials where a subjective outcome was evaluated [1]. Therefore, once an unpredictable random allocation sequence is generated to create groups comparable for any known or unknown potential confounding factors, it should be implemented without knowledge of the subjects to prevent selection bias [7]. In this analysis, although random sequence generation was performed adequately in more than 50% of the quasi-RCTs or RCTs during the analysis period with 100% performance in 2016, allocation concealment was not performed by the majority of the authors without a significant improvement.

Participants, medical personnel, and outcome assessors who are aware of group allocation during and after the assigned treatment would anticipate the beneficial effects of the new promising treatment and the negative effects of the control treatment causing different behavior between the experimental and control groups, such as differences in drop-out or administration of co-interventions across the groups, which eventually affects the actual outcomes. Actually, biased intervention estimates are observed in RCTs lacking blinding [8]. The bias is more exaggerated in trials assessing subjective outcomes [1]. Our analysis shows poorer performance of blinding of participants and personnel in comparison to blinding of outcome assessment according to the proportions of ‘Low risk’ of bias. In particular, the risk of bias for blinding of personnel did not improve significantly despite a significant yearly improvement in the risk of bias for the other blinding procedures. This might mean that most of the authors have remained unaware of the importance of blinding of personnel even until the present day. In addition, the highest proportion of ‘Unclear risk’ of bias for blinding of participants suggests that many anesthesiologists do not intentionally blind patients to the group assignment because patients participating in anesthesia studies are usually unconscious. However, the annual improvement in the risk of bias is promising. In contrast, group assignments cannot be blinded to personnel and outcome assessor by nature in many anesthesia studies according to the higher proportion of ‘High risk’ of bias for blinding of personnel and outcome assessment.

Statistically significant outcomes are more reported than insignificant outcomes [9,10,11]. One previous study showed that at least one primary outcome from 62% of published RCTs was changed, newly introduced, or omitted in no accordance with their protocol [9]. In another analysis, more than 20% of the outcomes planned to be measured in the methods section were incompletely reported in the results section of the same study [11]. Selective withholding of non-significant results from publication renders a meta-analysis include insufficient information. In May 2010, the KJA began to recommend that authors submitting clinical trials register their study in a clinical trial registry. In August 2013, the recommendation became “strong.” In accordance with the change in the strength of the recommendation, the rate of registration in a clinical trial registry, which had been 0%, started to increase.

To study an entire population is neither practical nor feasible. Hence, the characteristics of the population can be statistically inferred by studying a sample, which is a set of participants extracted from the population, representing the population [12]. Thus, sample size estimation is an essential aspect of planning a clinical study. However, criteria for judging risk of bias in the ‘Risk of bias’ assessment tool do not assess the adequacy of sample size estimation. Therefore, we assessed whether sample size estimation was performed appropriately in addition to the original risk of biases belonging to the domain. We found that many authors estimated sample size appropriately with a yearly improvement.

Several limitations should be considered in this analysis. First, although the additional guidelines built by the members of statistical round belonging to the KJA editorial board were used to compensate for the ambiguity of the original criteria for judging risk of bias in the ‘Risk of bias’ assessment tool [2], there was still inconsistency in the results of the assessments between independent authors, particularly in the domains “allocation concealment” and “other bias,” which were solved by consensus meetings. In addition, because the additional guidelines did not undergo critical review by peer professionals, they should not be regarded as mandatory for the assessment of risk of bias for studies in a meta-analysis. Second, because assessment of the 497 studies by each author was highly challenging, not all independent authors were involved in the assessment of all the studies published during the entire analysis period (7 years). Alternatively, albeit less desirably, two independent authors evaluated studies published for 2–3 years. Third, we analyzed only the yearly trend of ‘Low risk’ of bias for each domain to facilitate an intuitive understanding of the yearly improvement in the risk of bias for each domain. The trend in the other levels of risk (‘Unclear risk’ or ‘High risk’) was not analyzed. Fourth, the comparisons between the risks of bias for each domain and between the levels of the risk of bias within one domain were non-statistical (i.e., arithmetic), possibly leading to controversies over the results of this analysis. Last, it was unknown whether researchers had really conducted their study according to what they described in their published report.

In summary, authors contributing to the KJA dealt with incomplete data in an appropriate way. However, many did not conceal the group allocation of subjects after random allocation sequence generation, which was performed by more than half of the authors with a yearly improvement in its risk of bias. Outcome assessors were blinded to group assignments better than participants and personnel. The risk of bias for blinding of personnel did not improve significantly despite the yearly improvement in that for other blinding methods. Approximately 75% of the studies did not have study protocols available from clinical trial registries or, if available, the study protocols were inconsistent with those published in the journal. In conclusion, it is expected that comprehensive understanding of the current status of the risk of bias in the quasi-RCTs or RCTs published in the KJA will promote awareness of the reviewers and potential authors about the effects of the risk of bias on the quality of the manuscript they review or submit, thereby making the journal one of the best in its field.