The authors should clearly indicate whether the study was conducted prospectively or retrospectively. Additionally, whether the study goal was to evaluate the feasibility of the AI system or its superiority or non-inferiority to the reference test or model (current standard methodologies or models), to conduct an exploratory analysis, or to build a predictive model should be clearly stated. Details on the study goal should also be described (e.g., screening, diagnosis, or staging of a disease; prediction of disease development; anticipation of the prognosis). A reference (e.g., current clinical standards or model), which is to be compared with the AI system to assess its performance, should be clearly described.

The authors should provide details on the setting where the study was conducted. This includes the type and size of the study environment (e.g., tertiary university hospital, private clinics, public health centers); its sublocation (e.g., operating room, examination room, immunization unit); and the availability of supporting facilities, services, or technologies relevant to the study (e.g., robotic surgical system, otoscope, COVID-19 vaccines). How the study setting and cohorts represent real-world clinical conditions should also be stated. Technical requirements and configurations specific to each study site should also be described in detail (e.g., software, hardware, specialized computing devices, vendor-specific equipment, site-specific modification of the AI algorithm). Because AI models may perform well in the environment where they were developed, providing this information allows for these limitations in the generalizability to be better understood [10].

The process used to recruit the participants should be stated, along with the inclusion and exclusion criteria. A flow diagram indicating the number of participants included at each stage should be included if possible [4]. Alternatively, a flow chart can be presented and explained in the Results section.

The characteristics and quality of the input data significantly affect the performance of AI systems [11]. In addition, a detailed description of input data handling allows for the replication of AI system use outside the study setting and can be used to determine whether data handling is standardized across study sites. Therefore, the original data source (e.g., electronic medical records, public data registry, prospective data collection) and the process used to obtain it from the study population should be stated along with the time period when the data were obtained. If the data were acquired using a specific device/software (e.g., electrocardiographic waveform from a patient monitor), the product information (e.g., device/software name and model, manufacturer information [name, city, country of origin]) and data acquisition protocols should be described in detail (e.g., the frequency at which the waveform was recorded, the type of filter [low-pass or band-pass filter], and the ranges of the filtered frequency). If the data underwent reformatting, the process should be fully reported with its relevant parameters (e.g., the frequency at which the obtained waveforms were resampled or downsampled). If data collection depended on the investigator’s subjective expertise, the number of investigators, their qualifications, and the introductions and training materials used should be described. Whether the measurements and/or observations were independent among the investigators and how inter- and intra-investigator variability was detected and handled should be specified as well. Additionally, the inclusion and exclusion criteria for the AI system input data should be described.

The authors should also specify whether the data were structured. Structured data have clearly defined features (e.g., name of diagnosis, medical procedure, medication, laboratory test result values, population characteristic variables [age, sex, height, weight]), whereas unstructured data lack features that can be explicitly defined (e.g., images, videos, audio recordings, text data, time-series data).

Data pre-processing converts raw data with different formats from various sources into a uniform and consistent format that can be read and used as input by the AI system. The data format compatible with the intended use of the AI model varies according to the type of AI system used (e.g., radiographic images, hemodynamic parameters, laboratory results). Minimum requirements should be set that determine the eligibility of the data before input to the AI system (e.g., image resolution ranges, number of complete or missing data per participant). The authors should also describe how data that did not meet the minimum requirements were handled and how this impacted the clinical pathway that includes the AI system.

In particular, the definition of missing and poor-quality data (e.g., electrocautery artifacts in the electroencephalogram) and outliers, their quantity, and how they were detected and handled should be described, as they diminish AI system performance [12]. If these data were imputed using specific techniques (e.g., last observation carried forward), the resulting biases should be described. The same information should be provided for the comparator (control or reference intervention).

Details on data transformation (e.g., normalization, standardization, rescaling, natural-log transformation, encoding categorical variables), feature engineering, and feature selection should be provided such that other researchers can reproduce the process. In particular, the data should be de-identified, protected health information should be completely removed, and facial images should also be rendered unidentifiable [13]. Accordingly, the processes used to de-identify and protect personal information should be fully described.

Whether the data were processed or unprocessed before the analysis should be specified, along with whether the data were acquired before the application of the AI system or were generated with the use of the AI system.

The ground truth, which is used as a reference for comparison in supervised learning and can be clinically measured using the gold standard (e.g., histopathologic diagnosis and consensus agreement from a panel of experts), should be annotated with a precise definition. For example, hypotension is defined as a clinical condition in which each beat-to-beat systolic blood pressure measured from a catheter placed in the lumen of the right radial artery with the transducer diaphragm placed at the level of the mid-axillary line intersecting the fourth intercostal space [14] is maintained below 80 mmHg for more than 1 min between anesthesia induction and the end of surgery. Using unclear definitions with insufficient information, such as blood pressure < 80 mmHg, should be avoided.

The output of the AI system should be specified. In the context of the clinical problem that the current study intends to address, this can include disease diagnosis, grading of disease severity, probability of a clinical event or disease occurrence, disease treatment options, and prediction of clinical parameter values. A uniform format should be used for both the output and ground truth.

If the AI output is used to determine subsequent clinical management that ultimately affects clinical outcomes, this should be described in detail. If clinical management involves medical practice performed by the researchers, this should be standardized. The output of the AI system should also be fully understood and interpreted by the researchers involved in clinical management guided by the output. For example, an intraoperative hypotension prediction algorithm that calculates the probability of a hypotensive episode requires researchers to interpret the probability and take standardized actions based on the probability threshold (e.g., administering weight-based doses of a vasoactive agent when the probability of a hypotensive episode is > 70%).

If the study design compares the performance of the AI system to that of a reference clinical protocol, the process used in the reference protocol to determine subsequent clinical management should be explained in the same manner that the AI system is used to make clinical decisions. Accordingly, the rationale for using the reference standard and its inherent limitations, including errors and biases, should be described.

The process used to split up the full dataset at the beginning of the study should be described. The dataset can be split into training and test sets, or into training, validation (tuning), and test sets. The proportion of each set to the full dataset should be reported with the rationale. Using an external test set from an independent study site to test the trained model (external validation) is the ideal standard. Otherwise (in the case of internal validation), explicitly reporting and justifying the decision not to take the test set data from a data source external to that of the training data is essential. If the data structure between the training and test sets differs, the measures used to accommodate the difference should be explained.

To prevent bias, the test set needs to represent the target population. Certain methods (e.g., stratified sampling) can be used to maintain the distribution of the clinical outcome variables in the test set such that it is similar to that of the target population. Accordingly, the distribution of variables (including demographics and clinical parameters) in the training, validation, and test sets should be reported and statistically compared to show that the distributions are similar across sets. If any systematic differences are found, the factors involved should be investigated.

To prevent overfitting of the trained model, which shows good model performance on the training set but poor model performance on the validation set, internal k-fold cross-validation can be conducted. This type of validation involves splitting the dataset into k-subsets and training and validating the model k times using each subset as the validation set and the remaining k-1 subsets as the training set. When splitting a dataset, information leakage should be prevented. The information leakage occurs when the model is trained using the training set contaminated with information from the validation or test sets that should be exclusive to the training set. To ensure that each set is divided at the patient level or higher, each set should be split from the study population at the beginning of the study before data pre-processing and model training. Details on the steps taken to prevent overfitting and information leakage, which result in poor generalization of the model, should be provided.

Scientific rationales should be used to determine the type of model to train. The model task (e.g., classification and regression [numerical prediction]) and its beneficiaries (if any) should be specified. The mathematical algorithm of the AI system; hardware environment; and supporting software, library, or package required for its operation, including the versions, should be described. Relevant information includes the name of the developer and/or manufacturer, their location, and specific configuration settings. If previous development/validation studies of the AI system used are available, they should be cited in the manuscript and presented in the same manner that information about the AI system used in the current study is presented. The provision of AI system information from development/validation studies allows for changes in the performance of the AI system to be assessed as the current study population differs from that of the development/validation studies. Using a standardized reporting system also allows for concise information about the AI model to be provided [15]. If a new unpublished mathematical model is developed and used in the study, a full description should be provided as an appendix or as supplementary material at the end of the manuscript or should be published in an established public database along with accession details, which either does not allow for the model development processes to be arbitrarily revised once they are registered in the database or mandates retaining a complete history of revisions.

Several sequential stages, which require considerable time and computing power, are required to construct the final AI model. Throughout this process, several versions of AI models might be created. If the AI system was used in a clinical trial [4] or its ability to make appropriate or optimal clinical decisions is being assessed [5], the version should be clearly indicated with a regulatory marker, such as a unique device identifier. If the version of the AI model was modified, this should be justified by scientific rationale, and the changes made to the original version should be described.

The architecture of the AI model should be fully described such that it can be reconstructed by other researchers. This includes the inputs, outputs, and components specific to the type of AI model. Scientific rationale for the selection of each component should be provided. The architecture of the AI model can be provided in code as supplementary data.

As an example, a convolutional neural network model consists of 1) an input layer (e.g., one-dimensional 5-min electrocardiogram waveform collected at 300 Hz) characterized by the number of nodes and number and size of batches; 2) convolutional and pooling layers, which are characterized by the number and order of the layers, number and size of the kernel(s) in each layer, type of activation function (e.g., rectifier activation function, softplus), type of pooling operations (e.g., max, min, average, hybrid), type of normalization of output from a previous layer (e.g., batch or layer normalization), and use of dropout layers and their dropout rates; 3) fully connected layers, which are characterized by the number of hidden layers and number of nodes from each layer, and the type of activation function, similar to that of the previous convolutional and pooling layers; 4) output layers (e.g., whether hypotension develops); 5) loss (objective) function (e.g., mean squared error, mean absolute error for regression models, binary or categorical cross-entropy loss for classification models) and model optimization algorithms (e.g., Adam optimizer, gradient descent) with their hyperparameters (e.g., learning rates [degree of error reduction], exponential decay rates, epsilon); 6) regularization algorithms (e.g., L1 regularization [Lasso regularization], L2 regularization [Ridge regularization], elastic net regularization that combines the two regularization techniques); 7) hyperparameter tuning strategy (e.g., grid search, random search); 8) stopping criteria for training (e.g., maximum number of epochs after which training processes stop regardless of whether the trained model converges, patience parameters [number of epochs during which validation performance is allowed to improve]); and 9) criteria used to select the model with the best performance.

Details on all the model training processes used should be provided such that other researchers could reproduce them. Providing them in code is strongly encouraged.

If training data augmentation is required (for images, text, audio, etc.), the techniques used should be described (e.g., geometric transformations, paraphrasing, introducing noise).

The initialization of the parameters in the AI model (to prevent issues such as vanishing or exploding gradients and to enhance the convergence speed and model performance) should be described. If the initial parameters are randomly drawn from a specific distribution (e.g., uniform or normal distribution), the distribution should be described with its key parameters (e.g., lower and upper bounds for a uniform distribution and mean and standard deviation for a normal distribution). If the initial parameters are obtained from a model previously trained on a different large dataset (transfer learning), the source of the initial parameters from the pre-trained model should be provided. When using both random initialization and transfer learning, the initialized parameters and the modality used to initialize them should be specified. If some parameters are obtained from transfer learning and cannot be modified, they should be indicated as frozen or restricted. In addition, details on the specific restrictions applied and the portion of training affected by the restrictions should be specified.

The convergence of the model should be monitored by checking whether the pre-defined stopping criteria for the training are satisfied by the best hyperparameter combinations. If convergence of the model is not achieved, reviewing the data quality, feature scaling, learning rate, batch size, model architecture, parameter initialization, regularization (if any), and gradient descent issues, such as vanishing or exploding gradient problems, optimization, and hyperparameters, is mandatory. For example, for a neural network model that fails to converge, the researcher can attempt to vary the number of hidden layers and their nodes, apply different network activation functions, or adjust the learning rate.

The metrics used for model validation should also be described in detail (e.g., sensitivity, specificity, positive predictive value [precision], negative predictive value, area under the receiver operating characteristic curve, mean squared error, root mean squared error, accuracy).

If more than one AI model is trained and planned to be evaluated using a test set that is independent of both the training and validation sets, there must be a modality and parameters for assessing model performance, which are used to select the most relevant model that typically demonstrates the best performance (e.g., area under the receiver operating characteristic curve, accuracy, precision for classification, and mean squared error for regression [numeric prediction]). The metrics used to measure model performance should be presented with statistical uncertainty (e.g., 95% CI) and compared between models using appropriate statistical tests that determine the statistical significance of the metric differences, thereby addressing the clinical problems that the current study is meant to address. If CIs cannot be directly calculated owing to unknown error distributions, they can be non-parametrically estimated using bootstrapping.

As no single methodology is perfect for evaluating a model, using more than one technique is strongly recommended. However, authors should have the flexibility to choose the most appropriate evaluation method and should provide a rationale for the selection. Strictly adhering to a pre-defined or standardized evaluation protocol is discouraged.

If multiple models are determined to be the best-performing models, the final model selection should be justified. If the goal of the study is to construct an ensemble of models, descriptions of the three components of the ensemble method should be provided: 1) the allocation function that assigns training data (e.g., via bootstrapping sampling) to each model; 2) the combination function that reconciles the prediction disagreements among models (e.g., the final prediction is made by a majority vote from models, weighting votes from each model based on their performance, or learning various combinations of each model’s prediction [stacking]); and 3) a full description of each model in the ensemble, as mentioned above. If models have previously been published that address the same clinical problem being addressed in the current study, comparisons of the final model to those models should be provided. To assess the robustness of the study findings, the sensitivity of the AI model should be analyzed using different assumptions or various initial conditions.

Because misinterpretations of model outcomes can lead to biases and inappropriate applications in healthcare settings [8], the intended manner of interpreting or explaining the results of the AI model should be described (e.g., an AI model developed to predict intraoperative hypotension is used to predict the development of hypotension in intensive care unit settings).

If possible, using common data elements that provide standardized, uniform, and consistent names, definitions, formats, and coding of variables across studies that are compatible with different study settings is strongly recommended [16].

Whenever possible, the sample size required for the study should be calculated from the results of a pilot or previous study to achieve the predetermined statistical power at an acceptable type I error rate.

Metrics with statistical uncertainty and the significance of model performance on the training, validation, and test sets should be reported as planned in the Methods section. The types of metrics reported are dependent on the data type and models used in the study (e.g., F-score, Dice-Sørensen coefficient).

In addition to the performance of the model itself, evaluating how its predictive performance translates into clinical outcomes with specific metrics such as sensitivity, specificity, positive predictive value, negative predictive value, area under the receiver operating characteristic curve, and numbers needed to treat is essential. Accordingly, a justification of the metrics selected should be presented with the scientific rationale. The performance of the final model can be statistically compared with that of the standard technique or baseline model.

The contribution of each feature to the predictive performance of the AI model should be clearly described. For example, a SHapley Additive exPlanations (SHAP) plot is useful for showing the impact of every feature from each sample on the output predicted by the model.

If a subgroup analysis was performed, the subgroups for which the AI model performed best and worst should be indicated. The performance of any important subgroup should also be reported, as mentioned above. To demonstrate the performance and limitations of a classification model, providing a confusion matrix that shows whether the predicted classification matches the actual classification can be helpful.

For a sensitivity analysis of the classification models, descriptions of cases that present the highest model confidence with correct and incorrect prediction and the lowest confidence regardless of prediction correctness can be provided. For example, in an 84-year-old male patient with hypertension and congestive heart failure who underwent emergent pneumonectomy, a classification model predicting reintubation in the post-anesthetic care unit calculated a model confidence of 95%, and his trachea was actually reintubated according to the ground truth. This case shows high model confidence and correct prediction, implying that the model correctly identified high-risk patients for reintubation with definite risk factors such as old age, multiple comorbidities, and surgery involving the respiratory tract.

Similarly, for regression models, a sensitivity analysis can be performed by describing cases with the largest difference (error) between a lower predicted value and a higher actual value, cases with the largest difference between a higher predicted value and a lower actual value, and cases with the smallest difference between the predicted and actual values. For example, a case in which the systolic blood pressure predicted by a regression model is 300 mmHg and the actual value is 150 mmHg may be identified as the case with the largest difference between a higher predicted value and a lower actual value.

The results from unsupervised learning can be assessed by field experts for accuracy and relevance by comparing them with typical patterns. Thus, sensitivity analysis enhances the understanding of the model behavior, fosters transparency, and guides future improvements.