Randomized controlled trials of Chinese herbal medicine published in English from 2010 to 2019: a bibliometrics study

Introduction

Traditional Chinese medicine (TCM) originated in ancient China and has been developed for thousands of years (1). Chinese herbal medicine (CHM) is an important part of TCM, not only widely used in China, but also used in other Asian countries and many western countries. In 2016, CHM is estimated to be about 1.025 billion US dollars in the international market (2). In most western countries, CHM has been commonly used as a form of complementary and alternative medicine (3,4).

Randomized controlled trials (RCTs) have been considered as the gold standard for evaluating the effect of interventions (5). Since the first RCT of CHM was published in China in 1982 (6), RCTs have been widely used to evaluate the clinical effect of CHM. As the number of CHM RCTs continues to grow, some researchers have analyzed the research status and methodological quality of CHM RCTs published in Chinese. Yu et al. evaluated the methodological and reporting quality of CHM RCTs published in China Journal of Chinese Materia Medica, and found that the quality of both was generally low, and there was still much room for improvement (7). Zhang et al. evaluated the reporting quality of CHM RCTs published in Chinese Journal of Integrated Traditional and Western Medicine, and found that the uses of trial design, outcome selection, estimation of sample size, randomization, blinding, trial registration and flow chart are still needed to be further improved (8).

As the use of CHM in international market increases, the number of clinical researches including RCTs of CHM which published in international journals has also grown rapidly. In order to enable international clinical researchers to better understand CHM, and make CHM play its own role and advantage in the treatment of diseases, it is important to promote the publication of high-quality English papers of CHM in international journals. While, at present, there is no previous study has systematically analyzed the overall research status of CHM RCTs published in English. It is critical to have a comprehensive literature review so that CHM researchers can understand current research progress and identify possible research directions for their future work. Therefore, we did the bibliometrics analysis to analyze those RCTs published during the past decade, to help the new researchers to better understand the research progress and seize the research frontier in the clinical trials field of CHM.

Methods

Literature search

Using “randomized controlled trial”, “controlled clinical trial”, “randomized”, “randomised”, “clinical trial”, “Traditional Chinese medicine”, “Chinese herbal drugs”, “oriental traditional medicine”, “medicinal plants”, “Chinese herbal medicine”, “complementary therapies” and “alternative medicine” as search words, we searched three electronic databases including MEDLINE, EMBASE, and Cochrane Library databases, all these searches restricted RCTs of CHM published in English between January 2010 and December 2019. The detailed MEDLINE search strategy is available in Figure S1.

Inclusion and exclusion criteria

We included studies that (I) were RCTs focusing on oral CHM, no matter single-used or combination with other interventions, (II) CHM for any disease.

We excluded the following: (I) other TCM interventions, like acupuncture, moxibustion, massage, guasha, cupping, tai chi, qi gong, and so forth, (II) other routes of CHM, like spray, washing, ointment, iontophoretic injection, and so forth, (III) phase I or pharmacokinetics trials, (IV) use for healthy subjects, (V) self-described preliminary or pilot studies, (VI) follow-up or secondary analysis of data, (VII) protocols or conference abstracts.

We defined CHM using the criteria from our previous study (9). CHM are preparations derived from plants or parts of plants (e.g., leaves, stems, buds, flowers, roots or tubers) that grow in China and have been widely used for medical purpose. CHM include single herbs (or extracts from single herbs) and compound formulas of several herbs in all forms of preparation formulation (e.g., oral liquid, tablet, capsule, pill, granule and decoction). Studies focused on Japanese herbal medicine, Korean herbal medicine, or other countries were excluded. Plant-derived chemicals or synthetic chemicals which contain constituents of plants were also excluded.

Selection of studies

Because of the large number of CHM RCTs published in English from 2010 to 2019, we used the random sampling method and proportion used in other studies (10-12). SAS for Windows (version 9.4; order number: 9C1XJD) was used to generate a 20% random sampling number table. We numbered and sorted all the retrieved articles; the numbers on the number table was corresponded to the number of total articles. The RCTs selected for our study were divided into two teams (two reviewers in each team). All reviewers (HYX, WXJ, ZR and WCY) individually and independently screened the titles and abstracts of 20% sample to determine those potentially related to our study. Based on this first assessment, we then obtained the full text of these articles, the same two reviewers independently reviewed all these potentially eligible studies to find studies that fulfilled the inclusion criteria. Any disagreement in study selection was resolved by consensus or by discussion with a third reviewer (HJ).

Data extraction and analysis

We established a database (using Microsoft Excel 2007) to extract data. The database included several components: (I) general information, including publication year, journal name and impact factor, and publication country. We identified the country where the first author’s unit is located as the publication country of the study. (II) Characteristics of the study participants, including diseases and sample size, diseases concerned were classified according to the International Classification of Disease revision 10 (ICD-10). (III) Interventions, including interventions in treatment and control groups. (IV) Outcomes, including number of outcomes, if RCTs reported primary outcome, adverse event, TCM-specific outcome and quality of life. (V) Risk of bias assessment, the four domains of Cochrane Collaboration’s tool were used to evaluate the risk of bias of included RCTs, including sequence generation, allocation concealment, blinding of participants, personnel and outcome assessors, and incomplete outcome data (13).

Two teams (two reviewers in each team, HYX, WXJ, ZR and WCY) independently extracted data and assessed risk of bias, and thereafter, data were compared. Any discrepancy was sort out by arbitration with other review author (HJ). We performed a descriptive statistical analysis, for continuous variables, medians with interquartile range (IQR) were calculated, categorical variables were presented as frequencies and percentages.

Results

Flow of included studies

A total of 39,116 articles were identified, we randomly selected 20% (n=7,824) from the eligible articles. After screened the titles and abstracts, 475 articles were eligible. Full texts of these 475 articles were retrieved and 227 RCTs were included in our study. Details of the study screening process can be seen in Figure 1.

Figure 1 Flow chart of study selection. TCM, traditional Chinese medicine; RCT, randomized controlled trial.

General information of CHM RCTs

The number of CHM RCTs published in English from 2010 to 2019 did not show an obvious growth trend with the years (Figure 2).

Figure 2 Number of CHM RCTs published from 2010 to 2019. CHM, Chinese herbal medicine; RCT, randomized controlled trial.

Of 227 CHM RCTs, mainland China published the highest number (n=197, 86.8%), followed by Hong Kong (n=11, 4.8%) and Taiwan (n=8, 3.5%). Singapore, America and Australia had 2 RCTs, respectively. Brazil, India, Iran, Netherlands and Korea each published 1 RCT.

Two hundred and twenty-seven CHM RCTs were published in 88 journals. Chinese Journal of Integrative Medicine was the journal which published most of the relevant papers (22.0%), followed by Journal of Traditional Chinese Medicine (10.6%) and Evidence-Based Complementary and Alternative Medicine (6.6%). The top 10 journals with the most publications covered a total of 139 (61.2%) CHM RCTs, of these journals, Journal of Ethnopharmacology has the highest impact factor of 3.414 (Table 1).

Characteristics of the study participants

Two hundred and twenty-seven RCTs included a total of 45,774 participants. Sample size ranged from 12 to 3,143 participants (median: 115, IQR: 72–228). Ten RCTs addressed infant, children or adolescents and 7 addressed the elderly.

Included RCTs covered a broad range of diseases (Figure 3). The most common classification (ICD-10) addressed was diseases of the circulatory system (n=36, 15.9%), followed by diseases of the genitourinary system (n=28, 12.3%) and digestive system (n=27, 11.9%).

Figure 3 Classification (ICD-10) of included RCTs. ICD-10, International Classification of Disease revision 10; RCT, randomized controlled trial.

Interventions and comparators

Of the 227 RCTs, 193 (85.0%) had two arms, 28 (12.3%) had three, and 6 (2.6%) had four arms. There were 7 dosage forms of CHM, decoction was the most form (28.2%), followed by granule (25.1%) and capsule (23.3%). 7 types of control group were included in the 227 RCTs, CHM vs. placebo was the most type (36.1%), followed by CHM plus conventional treatment vs. conventional treatment (22.0%) and CHM vs. western medicine (21.1%) (Table 2).

Outcomes

Of the 227 included RCTs, the median of the total number of outcomes was 4 (25^th percentile: 3, 75^th percentile: 6, range, 1–14 outcomes) (Figure 4).

Figure 4 Number of outcomes in included RCTs. RCT, randomized controlled trial.

Ninety-two (40.5%) of the 227 RCTs did not clearly specify any primary outcome, 57 (25.1%) RCTs did not report any adverse event. Reporting primary outcome has an upward trend with publication year between 2010 and 2019, there were 22.2% of RCTs reported primary outcome in 2010, while in 2019, the percentage was 92.0%. Reporting adverse event shows a slow growth trend with publication year (Figure 5).

Figure 5 Percentage of RCTs that reported primary outcome and adverse event by publication year. RCT, randomized controlled trial.

Forty-one (18.1%) RCTs reported TCM-specific outcomes, 1 RCT listed the outcome as primary outcome. Ten analyzed the single syndrome score, 28 analyzed the total syndrome scores, and 3 transferred TCM symptom as ordinal outcomes.

Sixty-eight (30.0%) RCTs reported quality of life, in which, 45 reported primary outcome(s), 10 RCTs listed quality of life as primary outcome, and the other 35 listed as secondary outcome.

Risk of bias assessment

Of the 227 RCTs, 142 (62.6%) used the adequate random sequence generation methods, in which; computer random number generator (48.9%) and random number table (8.7%) were the most. Eighty-five (37.4%) did not report sufficient information about the random sequence generation process.

One hundred (44.1%) RCTs used the adequate allocation concealment, with central allocation (21.7%) and opaque, sealed envelope method (11.8%) being the most.

Ninety-two (40.5%) RCTs blinded participants and key study personnel, 57 (25.1%) did not report sufficient information to judge how to implement the blindness, 78 (34.4%) did not blind the participants or key study personnel. Twenty-four (10.6%) RCTs blinded outcome assessors.

Incomplete outcome data in 149 (65.6%) RCTs were adequately addressed, 28 (12.3%) RCTs were not adequately addressed, and 50 (22.1%) were judged as unclear (Figure 6).

Figure 6 Risk of bias graph of included RCTs. RCT, randomized controlled trial.

Discussion

In this study, we did a bibliometrics study to analyze the research status of CHM RCTs published in English from 2010 to 2019. 227 included RCTs were widely distributed, they were published in 88 journals, Chinese Journal of Integrative Medicine was the journal which published most of the relevant papers, followed by Journal of Traditional Chinese Medicine and Evidence-Based Complementary and Alternative Medicine. During the past decade, few studies published in high impact factor journals. There maybe two reasons. First, the composition of CHM is always complex, and the theory of TCM is difficult to be accepted by international medical editors. Second, although the overall research quality of CHM RCTs has improved in the past 10 years, serious methodology and reporting flaws still exist (7,8,14), which also makes CHM studies difficult to publish in high impact factor journals.

Our results indicated that some methodology flaws still exist in CHM RCTs published in English. Over one-third of the RCTs in our analysis lacked sufficient detail on how the random sequence was generated. Only 44.1% RCTs used the adequate allocation concealment, 40.5% RCTs blinded participants and personnel, and only 10.6% RCTs blinded outcome assessors, if the study did not use the blinding, the potential for performance and detection bias would be increased, especially for the assessment of subjective outcomes (15). In some CHM RCTs (especially for the interventions are CHM decoctions added and subtracted with the syndrome types), it is often difficult to blind participants and personnel, but blinding the outcome assessors is often achievable. So, we recommend that for these RCTs, especially when subjective outcomes such as TCM syndrome score or quality of life are selected, the blind is used for outcome assessors.

There were seven types of control group involved in this study and the top one was CHM vs. placebo (36.1%). For RCTs, a placebo group is designed to control for several factors, including placebo effects and spontaneous remission (16). During the past two decades, an increasing number of CHM-placebo controlled clinical trials have been registered and implemented (17). CONSORT (Consolidated Standards of Reporting Trials) extension for CHM formulas was published in 2017, it included five reporting items for placebo control information, which are essential for readers to assess the validity of study results and to be able to duplicate the study protocol (18).

CONSORT statement indicated that from both the scientific and the practical point of view, the most important outcomes of RCTs are primary outcomes, outcomes in RCTs should be pre-specified and clearly reported as either primary or secondary (5). If primary outcomes are not reported clearly, it may result in outcome reporting bias and the results of the trial may be jeopardized (19,20); 40.5% of our included CHM RCTs did not clearly specify any primary outcome, although the reporting percentage has an upward trend with publication year between 2010 and 2019, the percentage was generally lower than some previous studies in other disease areas (11,21-23). So clearly specify the primary outcome in clinical trials of CHM needs to be improved.

Nearly a quarter of included RCTs did not report any adverse event. Inadequate reporting of adverse events is not unique to CHM area, some other studies have also shown that the reporting of harms in RCTs paid less attention than reporting of efficacy and effectiveness and was often inadequate (24-26). CONSORT extension of harms standardizes adverse event reporting by creating a 10-item checklist of essential AE information for trial publication, it indicated that RCT should clarify how harms-related information was collected, present the absolute risk of each adverse event (specifying type, grade, and seriousness per arm) and appropriate metrics for recurrent events (27). We recommend researchers adhere to the CONSORT extension of harms when reporting adverse events associated with CHM RCTs.

For TCM, the human body is ideally understood as an interconnected dynamical network of mental, physical, and spiritual processes, each of which is constantly affected by the other, and disease is understood to be a manifestation of imbalance of these multiple processes, which is known as holism (28). The theory makes TCM for diseases are multi-dimensional; multiple laboratory test, patient-reported, clinician-rated, TCM syndrome outcomes are often used in evaluations of treatment impact of TCM.

For the outcomes that should be included in the clinical effect evaluation indexes of TCM, some researchers suggested that the following three domains should be included (29,30): Western medicine (WM)-specific outcomes (e.g., physiological and biochemical indicators), TCM-specific outcomes (e.g., TCM syndrome), and quality of life. In our study, only 18.1% and 30.0% of CHM RCTs included TCM-specific outcomes and quality of life, respectively. We suggest that CHM researchers pay attention to these two outcomes, to comprehensively and objectively reflect the true effect of CHM.

Our study has certain limitations. First, we used the random sampling method used in other studies and randomly selected 20% of the total studies for analysis, which may lead to the omission of some important studies. While, because the purpose of our study was to analyze the research status of CHM RCTs, the omission of a single study will not cause important changes for the study results. Second, we did not compare CHM RCTs published in Chinese and English, and we felt that there was no clear reason to consider that the quality of RCTs published in Chinese would have been better (14).

To the best of our knowledge, this is the first bibliometrics study of CHM RCTs published in English during the period of 2010 to 2019. Our results indicated that the included RCTs were widely distributed, while few studies published in high impact factor journals, some methodology flaws still exist in included studies. Some RCTs did not clearly report any primary outcome and adverse event, the using percentage of TCM-specific outcomes and quality of life was generally low. We hope to help the new researchers to better understand the research progress and seize the research trends in the clinical trials field of CHM.

Acknowledgments

Funding: The study was supported by the National Natural Science Foundation of China (No. 81973694; No. 81774146), “13th Five-Year” National Science and Technology Major Project for New Drugs (No. 2019ZX09734001). The funders had no role in study design, decision to publish, or preparation of the manuscript.

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Annals of Palliative Medicine for the series “Narrative & Evidence-based Medicine for Traditional Medicine: from basic research to clinical practice and trial”. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/apm-20-1033). The series “Narrative & Evidence-based Medicine for Traditional Medicine: from basic research to clinical practice and trial” was commissioned by the editorial office without any funding or sponsorship. BL served as the unpaid Guest Editor of the series. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Wang J, Guo Y, Li GL. Current status of standardization of traditional Chinese medicine in China. Evid Based Complement Alternat Med 2016;2016:9123103. [Crossref] [PubMed]
2. Lin AX, Chan G, Hu Y, et al. Internationalization of traditional Chinese medicine: Current international market, internationalization challenges and prospective suggestions. Chin Med 2018;13:9. [Crossref] [PubMed]
3. Rashrash M, Schommer JC, Brown LM. Prevalence and predictors of herbal medicine use among adults in the United States. J Patient Exp 2017;4:108-13. [Crossref] [PubMed]
4. Wang WJ, Zhang T. Integration of traditional Chinese medicine and Western medicine in the era of precision medicine. J Integr Med 2017;15:1-7. [Crossref] [PubMed]
5. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 2010;152:726-32. [Crossref] [PubMed]
6. Chen KJ, Qian ZH, Zhang WQ, et al. Analysis of the effect of refined Guanxin tablet on 112 cases of coronary heart disease and angina pectoris. Journal of Medical Research 1982;11:24-5.
7. Yu DD, Xie YM, Liao X, et al. Methodological quality and reporting quality evaluation of randomized controlled trials published in China Journal of Chinese Materia Medica. Zhongguo Zhong Yao Za Zhi 2018;43:833-9. [PubMed]
8. Zhang MY, Yang FW, Li Y, et al. Quality evaluation of randomized controlled clinical trials in Chinese Journal of Integrated Traditional and Western Medicine. Chinese Journal of Evidence-based Medicine 2017;17:357-63.
9. Hu J, Zhang J, Zhao W, et al. Cochrane systematic reviews of Chinese herbal medicines: an overview. PLoS One 2011;6:e28696. [Crossref] [PubMed]
10. Bhaloo Z, Adams D, Liu Y, et al. Primary Outcomes Reporting in Trials (PORTal): a systematic review of inadequate reporting in pediatric randomized controlled trials. J Clin Epidemiol 2017;81:33-41. [Crossref] [PubMed]
11. Chevan J, Haskvitz EM. Reported characteristics of participants in physical therapy-related clinical trials. Phys Ther 2015;95:884-90. [Crossref] [PubMed]
12. Pinto RZ, Elkins MR, Moseley AM, et al. Many randomized trials of physical therapy interventions are not adequately registered: a survey of 200 published trials. Phys Ther 2013;93:299-309. [Crossref] [PubMed]
13. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. [Crossref] [PubMed]
14. Pan X, Lopez-Olivo MA, Song J, et al. Systematic review of the methodological quality of controlled trials evaluating Chinese herbal medicine in patients with rheumatoid arthritis. BMJ Open 2017;7:e013242. [Crossref] [PubMed]
15. Yu DL, Ding BG, Wang BS, et al. Thinking on some problems of traditional Chinese medicine curative effect evaluation. China Journal of Traditional Chinese Medicine and Pharmacy 2017;32:2372-5.
16. Meissner K, Bingel U, Colloca L, et al. The placebo effect: advances from different methodological approaches. J Neurosci 2011;31:16117-24. [Crossref] [PubMed]
17. Zhang X, Tian R, Zhao C, et al. Placebo design in WHO-registered trials of Chinese herbal medicine need improvements. BMC Complement Altern Med 2019;19:299. [Crossref] [PubMed]
18. Cheng CW, Wu TX, Shang HC, et al. CONSORT extension for Chinese herbal medicine formulas 2017: recommendations, explanation, and elaboration. Ann Intern Med 2017;167:112-21. [Crossref] [PubMed]
19. WHO. International standards for clinical trial registries. 2012. Available online: http://www.who.int/iris/bitstream/10665/76705/1/9789241504294_eng.pdf?ua=1
20. Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365. [Crossref] [PubMed]
21. Vickerstaff V, Ambler G, King M, et al. Are multiple primary outcomes analysed appropriately in randomised controlled trials? A review. Contemp Clin Trials 2015;45:8-12. [Crossref] [PubMed]
22. Khanpour Ardestani S, Karkhaneh M, Yu HC, et al. Primary outcomes reporting in trials of paediatric type 1 diabetes mellitus: a systematic review. BMJ Open 2017;7:e014610. [Crossref] [PubMed]
23. Tyler KM, Normand SL, Horton NJ. The use and abuse of multiple outcomes in randomized controlled depression trials. Contemp Clin Trials 2011;32:299-304. [Crossref] [PubMed]
24. Gorrell LM, Engel RM, Brown B, et al. The reporting of adverse events following spinal manipulation in randomized clinical trials-a systematic review. Spine J 2016;16:1143-51. [Crossref] [PubMed]
25. Hoffer D, Smith SM, Parlow J, et al. Adverse event assessment and reporting in trials of newer treatments for post-operative pain. Acta Anaesthesiol Scand 2016;60:842-51. [Crossref] [PubMed]
26. Sivendran S, Latif A, Mcbride RB, et al. Adverse event reporting in cancer clinical trial publications. J Clin Oncol 2014;32:83-9. [Crossref] [PubMed]
27. Ioannidis JP, Evans SJ, Gøtzsche PC, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141:781-8. [Crossref] [PubMed]
28. Hui KK, Zhang WJ. Innovative clinical scientific researches promotes the chinese medicine development grounding on globalization. Zhongguo Zhong Xi Yi Jie He Za Zhi 2010;30:789-92. [PubMed]
29. Lin XJ, Zhu L, Yang M, et al. Chinese medicine therapeutic effect evaluation parameters and their classification. Journal of Traditional Chinese Medicine 2016;57:91-5.
30. Wood L, Egger M, Gluud LL, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008;336:601-5. [Crossref] [PubMed]