A pilot study is performed reflecting all the procedures of the main study and validates the feasibility of the study by assessing the inclusion and exclusion criteria of the participants, preparation of the drugs and intervention, storage and testing of the instruments used for measurements in the study, as well as training of researchers and research assistants [3]. The researcher, as well as the research assistants, must fully understand the purpose, method, and procedures of the study [3,4,5]. In addition, the suitability of the method for data collection must be tested. Let us review the study by Youn and Hsu [6], where the authors compared the methods for pain reduction with a propofol injection. In the pilot study, the following four methods were tested in order to find the most appropriate method for the main trial: Group T received an injection of 0.5 mg/kg of 1% lidocaine after applying a tourniquet; Group H received an injection of 0.5 mg/kg of 1% lidocaine after the injection of 40–41℃ saline solution (200 ml); Group L received an injection of 0.5 mg/kg of 1% lidocaine after an injection of room temperature (23–24℃) saline solution (200 ml); and Group HL received an injection of 0.5 mg/kg of 1% lidocaine at 41℃.

Based on the verbal pain scores assessed after the injection of 0.5 mg/kg of propofol in each group, the final injection method was selected. In addition, changes in body temperature, based on the saline injection, and differences depending on body part used for the measurement of body temperature were also assessed.

A pilot study assesses if the randomization and blinding are appropriately executed [7,8,9]. For example, to appropriately use the sealed opaque envelopes method, which is often used in the clinical trials, the detailed procedures for preparation, storage, and delivery are assessed [10]. Papers with randomized numbers are put into an opaque envelope and organized based on the order of the assignments. These envelopes are stored in the pharmacy department and provided by the department when needed. At the time of the provision, signatures from both the recipient and the provider are required. The study drugs are provided in an equal volume in identical syringes. Neither the patient nor the researcher has information on the study drug to which they are assigned. The appropriate drug that has been prepared in advance is used based on the randomized number in the envelope obtained from the pharmacy department.

Among the randomized groups, uniformity in the demographic characteristics and appropriate blinding, based on the researcher's plans, and the participants' understanding of randomization can be assessed. In addition, the most appropriate method used to explain randomization and obtain consent from the patients can be assessed. In the Benger et al. [3] study that compared three different methods to secure the airway of patients with out-of-hospital cardiac arrest (i-gel and Laryngeal Mask Airway Supreme versus current tracheal intubation), a cluster randomized design was used1), not based on the patient, but based on the paramedic. If randomization is performed based on the patient, the few following limitations need to be considered: the paramedic has to prepare and use all devices; it is difficult to perform randomization in an emergency situation; and there is a great chance that the device assigned to the patient would not actually be used.

The researcher recruits the subjects and obtains consent for participation. Adequate information and time should be provided for the participants to make their decision and provide written consent. Thereafter, participants should be screened in order to confirm their suitability for the trial. The appropriateness of the consent form, recruitment rates, length of time to receive written consent, and the required number of researchers and research assistants is determined. In particular, the recruitment rate is directly related to the study period (duration) and success or failure. An insufficient number of participants results in lower statistical power, which can eventually lead to early termination of the trial in the worst-case scenario. Therefore, it is crucial to accurately identify the recruitment rate through a pilot study. Another solution to this problem is to set the statistical power at a higher level. Instead of using 80%, the minimum power needed in clinical studies, 90% power can result in a higher statistical power despite a low recruitment rate [11].

Recruitment rate can also be increased through modification of the experimental methods. In their study of urological pediatric patients, Vemulakonda and Jones [12] concluded that an observational cohort study, where caregivers were involved in deciding the treatment method, had higher recruitment rates than that of randomized clinical trials. In the study that utilized video clips with comic characters in order to reduce anxiety in pediatric patients, the interview method was changed, which that increased the retention rate from 20% to 72.5% [13].

Although the study drug or intervention may be significant and would be worth a try, whether the participants can accept the study drug or intervention is a separate issue [14]. It would be easier if the approved drugs or intervention were accepted for use, but difficult or new approaches or known side-effects of the drug or intervention can make it difficult for the patients to accept the treatment. Chow et al. [13] used either 1) the usual care providing information on anesthesia and operation room procedures to both the pediatric patients and their families on the day of the procedure or 2) story-telling medicine through a video clip that was approximately 20 minutes long with animation characters played on a tablet PC. Through this pilot study, feasibility and acceptability of the general experimental plan, as well as the potential for positive effects of story-telling medicine, have been assessed. The researchers suggested that story-telling medicine can be a possible and acceptable method for effectively reducing anxiety in pediatric patients. The results were used as the foundation to proceed with a full-scale randomized controlled study.

It is not easy to select the primary outcome that best reflects the intentions of the researchers. Furthermore, the primary outcome is directly related to the sample size calculation. If several primary outcomes are required, a sample size for each outcome is needed. Mouton et al. [5] performed a pilot study to assess if remote ischemic preconditioning can prevent organ damage in the patients undergoing abdominal aortic aneurysm repair and suggested cardiac events and renal injuries as primary outcomes.

One of the key reasons why a pilot study is needed is to obtain the required preliminary data for the calculation of a sample size for the primary outcome. For continuous outcomes, preliminary data such as the mean and standard deviations for the control group are needed. For categorical outcomes, preliminary data such as the success rate of the standard treatment are required [14]. When selecting more than one primary outcome, the preliminary data for each outcome needs to be obtained in order to calculate the sample size. If the sample size based on preliminary data varies, the largest sample size required is used as the sample size of the main trial. This is to maintain the statistical power for the primary outcome that requires the largest sample size while increasing the power for the remaining primary outcomes.

Many studies have a preceding pilot study in order to calculate the sample size. However, estimation of sample size required for the main trial needs to be performed with caution [2,14,16]. In order to determine the sample size of the main trial, the standardized effect size (i.e., Cohen's d) is required. Unfortunately, the standardized effect size calculated in the pilot study is an estimated value calculated from the sample and has a confidence interval. Due to an insufficient sample size, the confidence interval of the standardized effect size is extremely wide, and the corresponding sample size also has a wide range [16]. As a result, this could lead to errors in calculating the sample size or statistical power to be used in the main trial [17,18]. One of the ways to overcome this issue is to utilize the clinically meaningful difference. For this, the experience of the researcher is critical. For instance, there is a study comparing the mean arterial pressure after intubation. Assuming that the standard deviation of the mean arterial pressure measured after intubation in the treatment and control groups is 20 mmHg, the sample size varies depending on the set difference in the average arterial pressure between the two groups. If the difference is set at 5, 10, or 20 mmHg, the corresponding sample sizes are 253, 64, and 17, respectively. The value to be considered as the clinically meaningful value depends on the experience of the researcher.

Researchers have a strong desire to include the data collected from the pilot study into the main study because this allows the researchers to reduce both the number of participants required for the study and the duration of the study. However, this is only allowed in an internal pilot study that is not discussed in this text [1]. To perform an internal pilot study, it must be thoroughly planned at the study design stage of the main study and included in the study procedure. Furthermore, the researchers must consider the fact that changes in other categories associated with the main study, aside from calculating the sample size, cannot be made [1]. In addition, a slightly increased chance of a type 1 error due to the hypothesis stating that pilot study and main study are independent of each other is an important aspect to consider.

There is the question of whether the hypothesis can be tested in the analysis of a pilot study. Considering that the appropriate power and sample size were not calculated for the pilot study, the researchers must recognize the fact that pilot studies are not for testing the hypothesis testing [7,19]. Therefore, they must be cautious about reporting the results of a pilot study. Furthermore, statistical significance in a pilot study does not mean that the main study or trial is not required.

The primary purpose of pilot studies is not hypothesis testing and therefore sample size is often not calculated. Some studies recommend over 30 samples per group [20] while some suggest 12 per group [21]. An appropriate sample size needs to be determined, not for providing appropriate power for hypothesis testing, but to understand the feasibility of participant recruitment or study design. For instance, in the previously mentioned study of securing the airway in patients with out-of-hospital cardiac arrest, 30 paramedics who performed airway securing procedures at least twice a year were included in each group. Not all paramedics experience events where they need to secure the airway and applying the Poisson distribution predicted that roughly 17% of the paramedics experience 0 to 1 case. Therefore, this was taken into consideration when calculating the number of samples to be included. Furthermore, after considering around 25% dropout rate throughout the study, a total of 50 subjects per group was recruited [3]. An important point is that a sample in the pilot study needs to be identical to that of the main study; therefore, the inclusion and exclusion criteria should be identical [2].