Abstract
Reporting guidelines are tools to help improve the transparency, completeness, and clarity of published articles in health research. Specifically, the CONSORT (Consolidated Standards of Reporting Trials) and SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) statements provide evidence-based guidance on what to include in randomised trial articles and protocols to guarantee the efficacy of interventions. These guidelines are subsequently described and discussed in journal articles and used to produce checklists. Determining the online impact (i.e., number and type of links received) of these articles can provide insights into the dissemination of reporting guidelines in broader environments (web-at-large) than simply that of the scientific publications that cite them. To address the technical limitations of link analysis, here the Debug-Validate-Access-Find (DVAF) method is designed and implemented to measure different facets of the guidelines’ online impact. A total of 65 articles related to 38 reporting guidelines are taken as a baseline, providing 240,128 URL citations, which are then refined, analysed, and categorised using the DVAF method. A total of 15,582 links to journal articles related to the CONSORT and SPIRIT initiatives were identified. CONSORT 2010 and SPIRIT 2013 were the reporting guidelines that received most links (URL citations) from other online objects (5328 and 2190, respectively). Overall, the online impact obtained is scattered (URL citations are received by different article URL IDs, mainly from link-based DOIs), narrow (limited number of linking domain names, half of articles are linked from fewer than 29 domain names), concentrated (links come from just a few academic publishers, around 60% from publishers), non-reputed (84% of links come from dubious websites and fake domain names) and highly decayed (89% of linking domain names were not accessible at the time of the analysis). In light of these results, it is concluded that the online impact of these guidelines could be improved, and a set of recommendations are proposed to this end.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
Scientific publications are essential output of any research work, constituting the primary channel for the dissemination of methods and results with the immediate research community, but also, ultimately, with the whole of society. The reliability and relevance of this research is, moreover, essentially guaranteed by the publication of the corresponding article.
In health and medical research, there is evidence, however, that published articles are often poorly reported (Glasziou et al., 2014; Vinkers et al., 2021). Inadequate reporting can be problematic as it can mean that misleading results and biased conclusions are used by healthcare providers, patients, and their families. In response, reporting guidelines have been developed to help ensure the transparency, completeness, and clarity of published articles. Most such guidelines consist of a checklist or explicit recommendations for authors (though also for peer reviewers and journal editors) as to what information should be included when reporting a specific type of study (Moher et al., 2010a, 2014). Most of these reporting guidelines are published in scientific peer-reviewed journals to better promote and disseminate the guideline recommendations to the scientific community. In some occasions, the same reporting guideline can be published by different journals at the same time.
The number of randomised trial related articles published in prominent medical journals has increased considerably in recent decades (Catalá-López et al., 2020). Randomised trials, when appropriately conducted and reported, can provide the most reliable information on the efficacy of interventions for informing healthcare decision-making. In this regard, the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statements provide evidence-based guidance to report essential methodological components on what should be included by those preparing and reviewing articles and protocols of randomised trials, respectively. The CONSORT statement was first published in 1996 and updated twice (Begg et al., 1996; Moher et al., 2001; Schulz et al., 2010a). It includes a 25-item checklist (and flow diagram) for reporting how a parallel-group randomised trial was designed, conducted, and interpreted. Multiple extensions of the CONSORT statement have been published to provide additional guidance for randomised trials with more specific designs (e.g., adaptive trials, cluster trials, pilot, and feasibility studies), data (e.g., equity, harms, and patient reported outcomes), and interventions (e.g., non-pharmacological, social and psychological, and interventions involving artificial intelligence). Similarly, the SPIRIT statement was published in 2013 and provides a 33-item checklist for clinical trial protocols (Chan et al., 2013a). Important details for each checklist item can be found in the explanatory (“Explanation and Elaboration”) papers, which outline the principles underpinning the guidelines and provide published examples of complete and transparent reporting (Chan et al., 2013b; Moher et al., 2010b). The CONSORT and SPIRIT statements form part of a broader international initiative to improve the reporting of health research, the so-called “Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network” (Altman & Simera, 2016).
Reporting guidelines in health research from meta-research approach
The CONSORT statement is perhaps the most important reporting guideline for health research, the development of which many other reporting standards are based. It has been recognised among the major milestones in health research methods of the twentieth century (Gabriel & Normand, 2012). The impact and uptake of CONSORT has been measured using several metrics. For example, since its original publication in 1996, the main CONSORT publications have received more than 12,000 citations by other research articles (Shamseer et al., 2016; Caulley et al., 2020a). It is currently known that the CONSORT statement has been endorsed by over 600 health and medical journals, but also by major editorial organizations including the International Committee of Medical Journal Editors (ICMJE) and the World Association of Medical Editors (WAME). Journal endorsement of CONSORT, SPIRIT and other reporting guidelines typically occurs in the form of a supportive statement in a journal’s Instructions to Authors.
Previous meta-research studies have analysed the reporting guidelines, covering the scientific collaboration between their developers (Catalá-López et al., 2019), the guidelines’ citation metrics (Caulley et al., 2020a), their appropriateness of use (Caulley et al., 2020b), and their endorsement by high Impact Factor medical journals (Altman, 2005; Hopewell et al., 2008; Shamseer et al., 2016).
The relation between journals’ endorsement of reporting guidelines and the completeness of reporting (i.e., how published health research actually uses the reporting guidelines) has also been studied (Stevens et al., 2014; Turner et al., 2012). For example, a systematic review (Turner et al., 2012) assessed the effect of journal’s endorsement of CONSORT on the reporting of randomised trials they publish. The authors found that journal endorsement of CONSORT statement was associated with more completely reported trials, based on assessments of more than 16,000 trials. A more recently published citation analysis (Caulley et al., 2020a) showed that few authors cited the CONSORT statement when reporting the methods and results of randomised trials published in high-impact medical journals, even though there is evidence of its effectiveness.
Potential knowledge-to-practice gaps in the uptake of reporting guidelines of health research may offer opportunities to explore alternative approaches to examine impact, increase visibility and promote the use of available reporting guidelines. Traditionally, bibliometrics has relied on citation analysis to measure impact of reporting guidelines, health research, and clinical practice guidelines (Kryl et al., 2012; Thelwall & Kousha, 2016; Thelwall & Maflahi, 2016; Thelwall et al., 2017). Link analysis provides researchers with supplementary analytical methods for the study of quantitative aspects and impact of electronic (online) resources (Thelwall, 2004). However, no link-based studies have been conducted to date to examine the online impact of major reporting guidelines of health research (such as the CONSORT and SPIRIT statements). The application of these methods should, we contend, provide insights into the use of reporting guidelines in broader environments than that specifically of the scientific literature which cites them and thus shed light on how they are disseminated.
Link analysis of online research objects
Hyperlinks are connections from one online object (e.g., HTML documents, entire websites, images, media, textual files, and software) to another. They have two ends or anchors and a direction,Footnote 1 that is, from object A to object B. Each hyperlink originates at the “source” object and points to the “target” object by embedding the target object’s URL in the source object.Footnote 2 This way, users can navigate to these hyperlinked objects (and visit them in a browser or download them, for example), thus improving their navigational experience, facilitating technical interoperability, and enhancing the visibility and findability of these online objects. When these objects are academic publications, each link received by the target object is known as a URL citation (Kousha, 2019).
The statistical analysis of hyperlinks (i.e., link analysis) is an essential method for monitoring online activity. Search engines, for example, use links to build algorithms to rank online objects for a particular user query. In the field of the Information Sciences, this method is used to understand the types of links generated in academic web environments (Bar-Ilan, 2005), to create networks of actors and information resources (Thelwall & Kousha, 2015; Thelwall, 2004), and to determine the impact of online objects (i.e., the number of links received). This work focuses on determining the impact of online research objects, which in this case correspond to articles describing reporting guidelines.
While raw links counts are informative, these counts can be misleading due to the dynamic nature of the Internet (Thelwall, 2006). By including different impact facets, the accuracy and understanding of link counts can be improved. The most important impact facets are summarized as follows.
Scattering
This facet reflects the fact that an online object can be published on different locations, and therefore using different URL IDs (e.g., journal website, institutional repository, online academic profile, personal website, DOI, etc.) and different formats (e.g., PDF version and HTML version). This way, the total impact (number of links received by a particular object) is scattered through all the different existing URLs referring to the same object (Orduna-Malea & Alonso-Arroyo, 2017).Footnote 3 Therefore, we cannot measure the impact of the object accurately only by considering one of the existing URLs.
Degree of similarity
This facet reflects the fact that links counts might be correlated with other metrics (e.g., the number of links received by an online object might be correlated with the number of citations received by that object). When a significant positive/negative correlation is found, the strength of the relationship between the metrics measured is high, and consequently, we can use the value of one variable to predict the value of the other variable.
Broadness
This facet measures the number of different domain names linking to one specific online object. This way, an object increases its broadness as the linking domain names counts increase. A webpage can generate links towards one online object massively, distorting the value of links counts. However, by counting linking domain names instead of linking webpages we can figure out the impact more accurately (Orduna-Malea & Alonso-Arroyo, 2017).
Diversity (entity level)
Considering the entities responsible of linking domain names, this facet measures how many different types of entities generate links to one object. For example, we can break down entities into universities, media, companies, etc. Attaining links from a low number of entity types means specialization, whereas the opposite reflects diversity. This concept is referred to in this work for the first time.
Diversity (genre level)
This facet measures how many different object genres generate links to one online object. For example, we can break down object genres into journal articles, book chapters, news posts, encyclopaedia entries, tweets, etc. While attaining links from a low number of object genres means specialization, the opposite reflects diversity. This concept is referred to in this work for the first time.
Reputation
This facet complements the impact broadness by measuring the number of links received from reputable domain names. In this work, reputation has been operationalized by the Trust Flow indicator, provided by the Majestic link intelligence tool. A website’s Trust Flow value increases when it receives links from reputable websites, which in turn are those with higher Trust Flow values. This recursive algorithm starts operating through a curated list of websites manually categorized as reputable by a community of experts (Jones, 2012). As fake or low-quality domain names can be created to link online objects to inflate their links counts, by measuring links from reputed domain names we can measure impact more accurately. This measure has been previously successfully applied in other link-based studies (Orduña-Malea, 2021; Orduña-Malea & Costas, 2021).
URL decay
This facet measures the loss of links (and consequently the loss of impact) pointing to an online object over time. This is a consequence of the change or disappearance of the linking online objects (Koehler, 1999; Oguz & Koehler, 2016; Payne & Thelwall, 2007). This issue produces remarkable effects on the academic web (Kumar & Kumar, 2012; Spinellis, 2003; Yang et al., 2012). Consequently, measuring only current links can show misleading results when total online impact is required.
Link analysis has been applied to the study of a range of different agents engaged in the field of science and technology via the measurement of specific aggregate online objects, including academic journal websites (Vaughan & Thelwall, 2003), scientific software websites (Orduña-Malea & Costas, 2021), authors’ personal websites (Barjak et al., 2007), patents (Orduña‐Malea et al., 2017; Font-Julián et al., 2022), open access repositories (Aguillo et al., 2010), research groups (Barjak & Thelwall, 2008), research projects (Dudek et al., 2021), university websites (Ortega & Aguillo, 2009; Thelwall & Zuccala, 2008), and public health entities (Ontalba-Ruipérez et al., 2016).
However, the application of link analysis at the article-level remains scarce for two main reasons. First, commercial search engines – Google included – eliminated their search facilities (Thelwall, 2021), thus limiting the use of massive link data, and requiring the employment of alternative link data sources that are not designed to analyse research online objects, such as scientific publications, quantitatively. Second, although hyperlinks can be used to obtain supplementary evidence of the wider impact of academic research, “link spam is widespread, and hyperlinks can be generated automatically in large numbers for legitimate reasons” (Thelwall & Kousha, 2015). This means that link counts fail to capture the wider impact of research unless a data cleansing process is employed.
Against this backdrop, this study seeks to determine the impact of a set of reporting guidelines for randomised trial related articles by means of link analysis. To do so, a tailored data process (the DVAF method) is developed to address the limitations of raw link analysis, based on the use of the link intelligence tool Majestic as a data source.
Objectives
The purpose of this study is two-fold: (1) to determine the online impact and dissemination of CONSORT and SPIRIT statement related articles; and (2) to develop a link analysis method that can increase the accuracy of online impact studies. To this end, the following research questions are posed:
- RQ1a.:
-
How many URL citations do the reporting guidelines receive?
- RQ1b.:
-
Through which URL IDs are they most linked?
- RQ2.:
-
Do URL citations to reporting guidelines correlate with citation and alternative metrics indicators?
- RQ3.:
-
Do URL citations to reporting guidelines come from a wide number of websites?
- RQ4.:
-
Do URL citations to reporting guidelines come from a wide number of entity types and object genres?
- RQ5.:
-
Do URL citations to reporting guidelines come from reputed websites?
- RQ6.:
-
Do URL citations to reporting guidelines decay over time?
Methods
We conducted a cross-sectional analysis of link-based data for the CONSORT and SPIRIT initiatives. All journal articles related to the CONSORT and SPIRIT statements were identified by a senior scientist (FC-L). To do so, the CONSORT,Footnote 4 SPIRITFootnote 5 and EQUATORFootnote 6 Network websites were inspected manually (last search dated 2 October 2021), as these resources report the journal articles in which the main reporting guidelines and their extensions have been officially published. For the present study, we included articles published in English concerning the main CONSORT and SPIRIT statements, and their related extensions for trial reports and protocols. Translations of any of the articles into other non-English languages (e.g., CONSORT translations into Chinese or Spanish) were excluded. Editorials, book chapters, and corrections were likewise excluded. This process yielded a total of 38 reporting guidelines described in 65 articles. The supplementary material (Appendix A) lists the reporting guidelines collected and reports the articles in which each guideline is described.
Webometric analysis
The URLs related to each of the articles included (henceforth, referred to as the target URLs) were located. To do so, the DOI, the journal website of the article, the PubMed ID, and the PMC ID (when available) were considered for each article. While other URL IDs may exist, these four were deemed sufficient to comply with the objectives of this study. All the URLs located are listed in the raw data supplementary file (Target URLs tab). This process yielded 222 URLs.
Having identified the target URLs, the source URLs (i.e., the URLs of those online objects linking to at least one target object) were then located. The source URLs were identified using the historic index of Majestic,Footnote 7 a professional link intelligence tool.
The Majestic tool has been previously used and tested for link-based studies (Orduña-Malea, 2021; Orduña-Malea & Aguillo, 2022; Orduña-Malea & Costas, 2021). This tool is specifically dedicated to carry out professional link-based analyses, offering the following advantages: (a) availability of a wide range of basic (e.g., number of links received) and composed (e.g., Trust Flow) indicators; (b) availability of quantitative (e.g., number of linking domain names) and qualitative (e.g., language of the linking source object) indicators; (c) availability of four different analysis levels (URL, subfolder, subdomain and domain); and d) availability of two complementary databases (the fresh index, covering links crawled during the last four months; and the historic index, covering all links generated since 2006, including both active and deleted links). Specifically, the use of the Trust Flow indicator has been proved to be an effective method to filter out those websites attaining a huge quantity of links from dubious sites, improving the accuracy of link analysis (Orduña-Malea, 2021; Orduña-Malea & Aguillo, 2022).
Majestic includes a self-made search engine through which crawls the entire Web, indexing all URLs found. For each URL indexed, Majestic automatically calculates a wide range of link-based metrics.Footnote 8 The historic index includes all URLs found since June 2006, covering more than 3800 billion unique URLs as of July 2022.
For this study, the Majestic’s internal online search feature was used, setting up the historic index and the URL-level analysis. Each target URL was inserted as a search term, and the database returned all the available metrics related to those URLs, including all linking source URLs. As the dataset is limited (65 articles and 222 URLs), this process was carried out manually by a senior researcher (EO-M).
All source URLs were directly downloaded in CSV file format. In addition, for each source URL, a pair of online impact-related flow metrics (Citation Flow and Trust Flow) was gathered (see Table 1 for description), along with other technical and descriptive link-based indicators (see the Source URLs tab in the raw data supplementary file). This procedure yielded a total of 240,128 hyperlinks from 204,993 different objects.
The hyperlink connections established between the source and target objects fall into one of four categories (see Fig. 1 for a summary): The single case (Fig. 1A) exemplifies the most common situation, where an article is linked from a single source object; in the multiple source case (Fig. 1B) different source objects link to the same article; in the multilink source case (Fig. 1C) the same source object provides several links to the same article; and, finally, in the multiple target case (Fig. 1D) different articles are linked by the same source object.
Multilink source cases artificially inflate the number of links received by one specific target object. Likewise, multiple source cases can also inflate link counts when source objects are duplicates (e.g., the http and https versions of one object). These limitations are due both to the web dynamics and the way Majestic offers raw link data. For this reason, a filtering process is required. The 204,993 different source URLs were subsequently filtered following a four-step process – debugging, validating, accessing, and finding – or the DVAF method, proposed in this work, and detailed below.
The DVAF (debugging, validity, accessibility, findability) cleansing process
In the first step, the debugging process, URL protocols (e.g., http, https, and www) and URL query parameters (e.g., ‘?ijkey = 221ca3a’ and ‘?utm_source = hs_email’) were extracted from the source URLs (see Fig. 2, steps A1 and A2). Each duplicated pair ‘Source URL’ – ‘Target URL’ was then removed, as were all duplicated links embedded in the same source object (see Fig. 2, step A3).
In the second step, the validation process, each website referred to by each source URL was accessed manually. Here, all forwarded URLs (those URLs automatically redirecting to other URLs), obsolete URLs (those URLs taking the user to inaccessible websites), and dubious URLs (e.g., URLs related to websites offering illegal products or sexual content) were excluded (see Fig. 2, step B).
Entities responsible for the websites were subsequently typified, such as higher education institutions (e.g., ox.ac.uk, northwestern.edu), companies (e.g., nursingresearchwriters.net, orange.com), organizations (e.g., ons.org, rand.org), publishers (e.g., frontiersin.org, biomedcentral.com), personal websites (e.g., arasharya.de, callinanllc.com), etc.
To do this, a bottom-up process (see Orduña-Malea, 2021) was conducted by a senior researcher (EO-M), who manually accessed each website, browsed through the website contents and “about” sections, and determined the entity type. After a first iteration, a draft of categories was set, which was discussed and agreed by the research team, generating a final categorization scheme of 31 entity types. Then, a second iteration was carried out to fix errors. The Appendix D (source entity types) includes the list of all categories along with a brief description.
The third step involved carrying out an accessibility task. Because a source online object can be deleted while the domain name’s URL remains active, a python script was written to obtain the HTML response status codeFootnote 9 of each of the source URLs from the validated domain names in order to confirm their accessibility. All online objects with no access were deleted (see Fig. 2, step C). Data related to all HTML responses can be consulted in the raw data supplementary file (html response code tab).
The fourth step involved determining the genre of the specific objects within each source domain name linking to reporting guidelines. First, each source object was manually checked to verify whether the link was still active, and if so, to typify the object genre. This process was manually processed by two senior information specialists (AA-A and J-AO-R). Second, if the link was found on the source online object, the genre was noted down (e.g., academic article, encyclopaedia entry, personal webpage, etc.). To typify each object, a bottom-up process was followed. A first iteration created a basic set of genres, which was subsequently agreed by the authors by merging and polishing a final scheme of 21 genres. Then, a second iteration was carried out to assign each source object with a specific genre. Third, each link was also associated with a potential purpose. Links embedded in publications (journal articles, books, theses, etc.) were associated to scientific purposes; links embedded in news, informative posts or encyclopaedia entries were associated to informational purposes; links embedded on author personal websites or list of references created by research groups, research centres or libraries were associated to informative purposes. Finally, links embedded in bibliographic records (such as the automatic page created by a repository which describes an article and includes a link to that paper) were associated to a functional purpose. The Appendix F includes all the genres created, a brief scope, and an illustrative example of each of the genres found.
After debugging and validating, the results include all URL citations that at one time existed, regardless of whether they remain active today. These data are used to show the online impact achieved by the reporting guidelines. Data obtained after carrying out the accessibility task are used to check the URL decay of these source URLs.
GephiFootnote 10 v. 0.9.1 was used to generate a link analysis map connecting the source URLs to the target URLs.
Bibliometric analysis
For each article, the following reporting guideline descriptive fields were considered: reporting guideline name, parent guideline (e.g., CONSORT, SPIRIT, and CONSORT/SPIRIT), extension (yes/no), study design (e.g., clinical trials, experimental studies, study protocols), and application (e.g., intervention, outcomes, whole report, protocol, and abstract). All data can be consulted in the raw data supplementary file (Guidelines tab). Scopus was then used to gather the following bibliographic metadata for each publication: journal name, year of publication, DOI, document type (article, review, editorial), authorship, affiliation, keywords, citations received, references cited, and funding information (see the Publications tab in the raw data supplementary file). Articles G6-P11 and G10-P02 were not found in Scopus and were, therefore, eliminated from the analysis.
Altmetric analysis
The PlumX analytics data providerFootnote 11 was selected on the grounds that it is linked to the Scopus database. Plum Analytics is a subscription-based platform founded in 2012 which provides a wide range of Altmetrics (categorized into citations, usage, captures, mentions, and social media metrics) for each publication (Williams, 2019). The literature has pointed out PlumX as one of the major altmetrics data providers, especially to capture Mendeley readership (Ortega, 2018; Zahedi & Costas, 2018; Ortega, 2020; Karmakar et al., 2021).
Additionally, PlumX captures both clinical and policy citations (Kryl et al., 2012), of interest to capture broader impact of the reporting guidelines analyzed. PlumX covers the clinical guidelines indexed in PubMed, Dynamed Plus topics, and NICE (National Institute for Health and Care Excellence), and those policy documents indexed in the Overton database (Fang et al., 2020; Szomszor & Adie, 2022). The user can access the PlumX clinical citations cardsFootnote 12 and policy citations cardFootnote 13 to identify the specific citing documents and check whether the citations have been located correctly.
To obtain the alternative metrics related to each of the reporting guidelines under study, the DOI of each article was used as a data seed, thus furnishing a wide range of altmetrics for each reporting guideline (see raw data supplementary file, PlumX tab) through the API service.Footnote 14
Data prevalence (i.e., publications with data collected for one specific metric) was scarce for a number of metrics (e.g., 73.4% of the articles received no mentions from blogs). To avoid this issue, only those metrics with a data prevalence of at least 50% (i.e., half the publications presented data) were included. As a result, eventually citations from clinical guidelines and policy documents, Mendeley readers, Tweet counts, Facebook counts, abstract views, and export saves were the metrics considered.
Data related to all three analytic techniques (link analysis, Bibliometrics and Altmetrics) were collected and statistically analysed as of 14 October 2021. The main metrics collected and measured in this study are summarized and described in Table 1.
To analyse the degree of similarity between article-level metrics, the rho-Spearman correlation (alpha-value of 0.01) was applied because of the skewed distribution of the metrics. All statistical analyses were performed with the free version of XLStat.
The whole data collection process is summarized in Fig. 3.
Results
URL citations to reporting guidelines
A total of 15,582 links to journal articles related to the CONSORT and SPIRIT statements were identified. CONSORT 2010 (described in ten articles) and SPIRIT 2013 (described in two articles) were the reporting guidelines that received most links (URL citations) from other online objects (5328 and 2190, respectively).
If we consider the URL ID linked by source objects, we detect an initial web scattering effect. Source objects mainly linked the DOI URL version of the articles (82.9% of all links created refer to this URL ID), with this URL ID being the most linked in 29 of the 38 guidelines studied (Table 2). The journal URL ID was also commonly used, concentrating 9.6% of all links received, and was the most linked URL ID for a few specific guidelines (including, for example, the CONSORT and SPIRIT extensions for interventional trials involving artificial intelligence, i.e., CONSORT-AI, SPIRIT-AI). The remaining URL IDs presented very few links.
If we disaggregate the data at the article-level (supplementary material, Appendix B), we observe that the use of journal URL versions was higher for more recent publications (i.e., those published in 2020 and 2021). For example, article G32-P1 (CONSORT-AI published in Nature Medicine in 2020) (Liu et al., 2020) obtained seven citations, 28 links to its DOI URL ID, and 246 links to its journal URL ID. Similarly, article G38-P1 (the joint extension for CONSORT and SPIRIT, the so-called CONSERVE statement for clinical trials modified due to the COVID-19 pandemic and extenuating circumstances, published in JAMA in 2021) (Orkin et al., 2021) obtained one citation, one DOI URL link, and 40 journal URL links.
The analysis conducted at the article-level also revealed a second web scattering effect for those guidelines described in more than one article. For example, the CONSORT 2010 statement receives 5328 links from 30 different URLs, related to the ten articles in which this guideline is described (Fig. 4).
Figure 4 presents a skewed distribution in which DOI URL IDs obtained the greatest number of links. This distribution reflects the unequal impact of each of the articles describing the CONSORT 2010 statement. Article G06-P3 (CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials, published in the BMJ) received 1581 URL citations (when including all four URL IDs analysed in this study) and article G06-P9 (CONSORT 2010 explanation and elaboration document published in the BMJ) received 1211. However, the remaining articles obtained a smaller number of links.
Correlation of URL citations to reporting guidelines with citation and alternative metrics
The unequal impact of the articles describing the same guideline was evident not only as regards links received but also when measuring citations and alternative metrics (Table 3). On occasions, the article with the highest impact was the one with the highest Journal Impact Factor (JIF) (e.g., CONSORT extension for Herbal Medicine: G19-P1; STRICTA: G22-P1). However, in other instances, the JIF was not a key driver of impact (e.g., CONSORT 2010: G06-P8; CONSORT extension for Abstracts: G17-P1).
The number of URL citations received per article is strongly correlated with the remaining impact-based indicators (Fig. 5), especially with the number of citations (Spearman R = 0.82; p-value < 0.0001; alpha > 0.01) and the number of Mendeley readers (Spearman R = 0.83; p-value < 0.0001; alpha > 0.01). However, no correlation was found with Twitter (Spearman R = −0.04; p-value = 0.753; alpha > 0.01) or Facebook (Spearman R = 0.19; p-value: 0.138; alpha > 0.01), outcomes that present a statistically significant correlation with each other (Spearman R = 0.63; p-value < 0.0001; alpha > 0.01). All the alternative metrics collected can be consulted in the supplementary material (Appendix C).
Domains linking to reporting guidelines
The number of different domain names linking to the articles (i.e., the impact breath of each article) was limited. The average value is 34.64 linking domain names (median value of 28.5; 90th percentile value of 58). The distribution of the impact breath was skewed, with only three articles exceeding 100 different linking domain names (CONSORT 2010: G6-P3, G6-P9, and CONSORT-AI: G32-P1).
The number of URL citations generated by each of the linking domain names differed significantly according to each article (Fig. 6, left). Here, the anomalous behaviour of a few articles was evident, the case of CONSORT-AI (G32-P1 had 2.5 URL citations per domain name; very low percentage) and SPIRIT 2013 (G7-P1 had 23.8 URL citations per domain name; very high percentage). Other than these exceptions, the number of URL citations received per domain name was low (average of 5.73).
On the other hand, the linking domain names only linked to a few articles each (average of 2.5; median value of 1), generating a raked visual effect in the article-to-domain network generated (Fig. 6, right). A number of publishers, most notably BioMed Central (with URL citations to 58 articles), Springer (48), PLoS (44), and Frontiers in (41), constituted exceptions, hosting numerous citing publications (for more details, see Table 5).
Entities and objects linking to reporting guidelines
A total of 38 entity types (diversity breadth) were identified (Table 4), ranging from academic-oriented websites (e.g., publishers, universities, databases, research groups, research centres, academic personal websites, bibliographic databases) to general information-oriented websites (e.g., news, encyclopaedias, information portals), health information-oriented websites (e.g., health organizations, health information portals, health official bodies, hospitals), and commercial-oriented websites (e.g., private companies, shopping). A brief description of the scope of each entity type can be consulted in the supplementary material (Appendix D).
A massive concentration of URL citations from academic websites was detected, in particular, those of publishers (21.8% of all source domain names, generating 64.2% of all URL citations) and higher education institutions (8.4% of domain names, generating 14.6% of all URL citations). While the presence of organizations and companies is also strong (10 and 9.3% of domain names, respectively), the number of URL citations provided by these sources was only moderate (2.7 and 3.6%, respectively). The number of URL citations from encyclopaedias is also notable (4.9% of all URL citations received).
While the publishers’ websites were the origin of most URL citations, the distribution of links by entity type differed from article to article. For example, in the case of CONSORT 2010, a remarkable number of URL citations received by G06-P3 and G06-P9 originated from encyclopaedias, a phenomenon not observed in the remaining publications describing the same reporting guidelines (Fig. 7). The distribution of URL citations by entity category for all publications can be consulted in the supplementary material (Appendix E).
BioMed Central and Springer (both part of the Springer Nature Group) are the source domain names that provided the highest number of URL citations to the set of articles considered (39.9 and 13.3% of all URL citations, respectively) (Table 5). These two publishers also linked to the highest number of different articles (e.g., BioMed Central provided links to 58 of the 64 articles analysed). Other major domain names in the Springer Nature Group include Nature (150 URL citations), BMC Medicine (88 URL citations),Footnote 15 and SpringerOpen (75 URL citations). However, data offered in Table 5 show domain names. Consequently, they are not related to specific journals necessarily. For example, nature.com includes publications from different journals, all hosted under the same domain name.
Wikipedia also has a notable presence, generating 318 links to 13 different articles, as does the EQUATOR network (208 links); yet, the number of articles linked from this source (22) was lower than expected. Other publishers (e.g., Hindawi, Frontiers, JAMA Network, BMJ, and PLoS) were among the top source domain names generating URL citations to reporting guideline articles (Table 5).
The analysis identified a number of other special cases. For example, the University of Bedfordshire (beds.ac.uk) generated 10.2% of all URL citations; however, a fine-grained analysis revealed that the links were generated by journal mirrors,Footnote 16 specifically a mirror of BMC Medicine hosted on the University of Bedfordshire website.Footnote 17
Other domain names were found to link massively to the same guideline article– for example Bitbybitbook.com (a book openly accessible), which generated 208 URL citations to CONSORT-2010 article G6-P06 (Schulz et al., 2010b).
At the object level, 18 object genres and six potential purpose categories were identified (Table 6). A brief description of the scope of each genre can be consulted in the supplementary material (Appendix F). Object genres were detected for 9527 online objects of the 15,585 obtained following the debugging process (for further details, see Fig. 9).
URL citations were primarily created for scientific-related reasons (87.27% of all links). Most of these links were citations from other scientific publications, either from the publishers’ websites or from journal mirrors. The remaining source object genres were residual, with a notable presence of informational (e.g., links in news, posts and encyclopaedias) and promotional (e.g., links in curated references lists and personal websites) links.
Finally, the presence of links in bibliographic records (e.g., repositories) or data reports (e.g., Altmetrics data providers or search engine optimization – SEO – tools) was also noted. These links are created automatically as they respond to other functional tasks.
Reputation of websites linking to reporting guidelines
The breadth of reputation of the 64 articles related to the CONSORT and SPIRIT statements reached a figure of 91 domain names, generating 9637 URL citations (that is, 61.8% of all links received by these articles). These domain names constitute the most reputed domains from which the reporting guidelines received URL citations (Fig. 8). Publishers and higher education institutions recorded high Trust Flow values (see Tables 4 and 5 for values at the entity type and domain name, respectively). These values can be considered significant given the high number of source domain names tagged in each of these two categories.
The Trust Flow and Citation Flow values recorded were strongly correlated (Spearman R = 0.77; p-value: < 0.0001) (i.e., they present a balanced Flow Metrics profile). However, the values of these two flow metrics were unbalanced in several domain names. More specifically, the Citation Flow value of 124 domain names was twenty times that of their Trust Flow value, suggesting that the real reputation of these source domain names is dubious.
Decay of URL citations to reporting guidelines
The application of the DVAF method dramatically reduced both the number of URL citations to articles (96%; from 240,128 links to 9527) and the number of linking domain names (95%; from 8064 to 421).
While the first step (Debug) mainly eliminated duplicate links, the remaining steps highlighted the large-scale obsolescence of link data. The second step (Validate) primarily detected domain names that had once existed, but which are no longer active (6987 domain names providing 83,847 URL citations). The third step (Access) largely identified source objects that had existed at some time, but which had subsequently disappeared. More specifically, 192 objects (providing 308 URL citations) returned a 404 HTML code (Not Found), while 902 objects (providing 1307 URL citations) could not be accessed. Finally, the fourth step (Find) chiefly detected links that had once existed, but which had since disappeared. Thus, no URL citations were found in 415 objects. This gradual process of reduction with the application of successive steps is illustrated in Fig. 9.
The reduction in the number of URLs is not limited to poor quality or non-reputable domain names. The DVAF method, for example, reduced the number of URL citations from BioMed Central by 26.8% (from 5279 to 3860) and those from Springer by 27.9%.
The URL decay effect presented by the top ten linking domain names is shown in Table 7, where we can observe that for these domain names the outcomes remained largely unchanged from steps 3 to 4. In addition, a remarkable presence of publishers is found (BioMed Central, Springer, Hindawi, Frontiers in, and JAMA Network). The disappearance of informative webpages or huge changes on the domain name structure might explain the URL decay on these websites. The loss of links on Wikipedia is attributed to the dynamic and social nature of the site.
Discussion
In this study, we have presented a cross-sectional analysis of web-based data for articles addressing the CONSORT and SPIRIT statements. The online impact of these articles has been determined using link analysis, for which a tailored cleansing data process (DVAF method) has been designed, developed, and applied. Thanks to the application of this method, it has been possible to identify different facets of the online impact of these reporting guidelines for randomised trials and associated protocols.
About the online impact of reporting guidelines
We found that reporting guidelines in this study received 15,582 URL citations (RQ1a). The CONSORT 2010 for parallel-group randomised controlled trials described in 10 articles were the most cited reporting guidelines (with 5328 URL citations or links) from other online objects, particularly in comparison with other guidelines or extensions included in the analyses. In our opinion, these results can be explained by the fact that a large number of CONSORT/SPIRIT extensions or adaptations seem less well known among authors, reviewers and editors, since many of them are not being systematically incorporated into the journals’ instructions to authors. For example, a previous study (Shamseer et al., 2016) examined the online “Instructions to Authors” of 168 high Impact Factor medical journals between July and December 2014. Sixty-three percent (106/168) of the included journals mentioned CONSORT in their “Instructions to Authors" and only 22 of the journals (13%) mentioned any of the CONSORT extensions published at the time of searching.
The results evidence a scattered web impact, reflected by the existence of different URL IDs referring to the same article (RQ1b). While use of the DOI URL ID is generalized, 17% of links (2660) would have been missed if DOI were the sole URL ID employed. In fact, for a few recent articles (those published in 2020 and 2021), the journal URL ID was the one with the most links. A plausible explanation for this is that these articles would not have had sufficient time to obtain bibliographic citations, a significant source of DOI links. Moreover, the scattered effect of the online impact is accentuated in the case of those guidelines described in more than one article, where the existence of different distributions of links according to each URL ID is apparent. Consequently, the article-level link analysis should not ignore URL IDs, other than that corresponding to the DOI.
The online impact of the articles (considering all links received regardless of the URL ID linked) shows a strong and significantly positive correlation both with the number of citations received (scientific impact) and the set of Altmetrics analysed (wider impact), especially the number of Mendeley readers (RQ2). Consequently, counting the total number of links received per article (after cleansing the data in accordance with the DVAF method) might be informative of the scientific impact of articles, and indirectly, of the guidelines described. For recent articles, links can also be used as potential early predictors of their future scientific impact.
An absence of any correlation with Facebook and Twitter counts has been detected. A plausible explanation for this performance would appear to be related to the year of publication, as most of the articles were published long before the launch of these networking sites, and even before PlumX started collecting data. More recent articles (e.g., CONSORT extension for reporting randomised trials of social and psychological interventions [CONSORT-SPI]: G23 in 2018, and CONSORT extension for reporting Adaptive designs [ACE]: G30 in 2020) record considerably higher Facebook and Twitter counts, reinforcing this hypothesis.
The average number of single domain names linking to guideline articles is low (34.6), with 50% of all articles receiving links from fewer than 29 different domain names. This result points to the concentration of source domain names linking to articles describing reporting guidelines (RQ3).
While the diversity of linking entities is quite massive (38 different entity types were found, both academic and non-academic), the online impact is heavily concentrated (RQ4). In general, a strong clustering of URL citations was detected from academic websites, particularly from publishers (64% of all URL citations, and 22% of all source domain names), and higher academic institutions (15% of all URL citations, and 8% of domain names), in contrast to other institutions (e.g., government bodies and research institutes represented less than 2% of URL citations, and 1% of domain names). We interpret the different URL patterns between websites might potentially reflect editorial policies and practices, and those institutional websites incorporating reporting guidelines could be considered enablers high-quality reporting standards for randomised trial reports and protocols. Perhaps the most striking results the low representation of research funders, for their important role in the promotion and development of clinical research, with the sole exception of the U.S. National Institutes of Health (NIH), placed on the list of top source domain names.
Most URL citations (85.4%) come from articles, with a scientific purpose (citation) accounting for the creation of the link. This result might explain the high online impact of the DOI URL ID as well as the strong correlation between URL citations and citations (i.e., the DOI is incorporated as part of the reference that cites the article describing the reporting guideline). The existence of URL citations from news sources, encyclopaedias, research centres, personal websites and health information portals testifies to their wider impact, driven in this instance by motives of an informational and promotional nature.
These results are obviously limited by the categorization process carried out. Despite the large number of entity types detected (38), most websites belong to publishers (publishing groups and academic journals) that are easily identifiable. However, the presence of publishers is probably underrepresented, since all the journals hosted by universities have been counted as higher education institutions instead of publishers, since the general domain name belongs to the whole institution, being this issue the main limitation of the entity categorization performed. The subsequent classification of source object genres regardless the entity type minimizes this effect, adding accurateness to the link analysis carried out.
The reputation of the linking websites is highly suspecting (RQ5). If we consider the original 240,128 URL citations before debugging, 84.2% of these links come from dubious websites and fake domain names with a Trust Flow value of 0. After debugging, only 10.3% of all source domain names (n = 91) achieve Trust Flow and Citation Flow values of at least 50. These reputed domain names generate a significant percentage of all the URL citations (61.8%) received by the reporting guidelines. This means that the actual incidence of low-quality source domain names is low. A few source domains, however, exhibit an unbalanced Flow Metric profile, which might be because they were built for promotional or SEO purposes, as linking academic publications enhances their credibility in the eyes of readers and search engines. This issue clearly suggests that dubious websites use research objects as part of their commercial link strategies, an issue already identified in previous studies (Orduña-Malea, 2021).
The results also highlight a large-scale URL decay (RQ6). If we consider steps 2–4 of the DVAF method (those involved in the URL decay), 88.6% of linking domain names and 94.8% of URL citations could not be accessed at the time of the analysis. Non-accessible domain names were mainly associated with fake websites created for the purposes of SEO and whose lifespan is ephemeral, while most non-accessible objects were associated with changes made to reputable websites. Finally, the presence of URL citations in webpages that could not be found can be attributed to the appearance of links in ephemeral locations (e.g., sidebars, comments, and blog feeds).
About the dissemination of reporting guidelines
This study provides considerable information that can be used to very diverse purposes, such as activities related to promote health research conduct, reporting, and scientific writing and peer-review. For example, journals and publishers have an important role to play in the dissemination and implementation of the reporting guidelines. The inclusion of hyperlinks and full citations to articles (e.g., DOI and journal URL IDs) in the journals’ publication instructions is a recommended course of action in this regard. Similarly, those responsible for writing and promoting the guidelines need to explore and improve different implementation strategies aimed at increasing the adoption of the recommendations by authors, reviewers, and journal editors. In addition, enlisting the support of other relevant actors, including professional societies and organizations and funding agencies, would also appear to be crucial, while the publication of supporting articles (e.g., documents, letters, comments, editorials, and translations of “Explanation and Elaboration” papers) and the use of web-based dissemination channels (journal websites, guideline groups, and international initiatives such as EQUATOR) would do much to help in their dissemination.
While the results reported in this study are limited to the CONSORT and SPIRIT initiatives, the analytical methods described here can be applied to the measurement of other leading reporting guidelines in health research, including, for example, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement (Moher et al., 2009; Page et al., 2021). The enhanced dissemination and implementation of reporting guidelines would undoubtedly serve to improve the quality and transparency of articles reporting randomised trials, which, in turn, would have an impact on future research.
About the DVAF method
The DVAF (Debug, Validate, Access, and Find) method described here has been used to analyse the online impact of articles describing trial reporting guidelines. However, it should be borne in mind that this method relies on the measurement of links, so that all guidelines mentioned by other procedures (e.g., title textual mention) are not captured (Thelwall, 2011).
The method does, nevertheless, facilitate the cleaning of link data. The ‘Debug’ step eliminated duplicate links while the ‘Validate’ step filtered out all dubious websites, and given the huge percentage of links cleaned, these steps must be considered essential for academic link analysis. Once the link data had been debugged, the results captured primarily scientific impact (i.e., links from articles) and, to a lesser extent, wider academic impact (i.e., informational and promotional links). The URL citations received by the articles exhibited a strong correlation both with the number of citations received and a set of alternative metrics, thus reinforcing their value as supplementary sources of impact, especially for recent articles (with URLs being created faster than citations).
On the downside, the DVAF method is time-consuming (above all steps 2 and 4) and while some steps could be automated, the validation and access stages require human intervention. Here, a set of just 64 articles has been analysed, but the analysis of large sets of documents can be considerably more complex. In particular, the categorization of entities is difficult due to the existence of complex websites and this step is not, therefore, readily automated. Additionally, the presence of fraudulent websites introduces a series of inherent errors in the link analysis process, and while the proposed method considerably reduces their number, it does not eliminate them entirely. In this regard, the Trust Flow has shown itself to be a useful metric for filtering out dubious websites and enhancing academic websites, as reported previously in the literature (Orduña-Malea, 2021).
The obsolescence of link data (URL decay) constitutes an additional limitation, as the ‘Access’ and ‘Find’ steps have both highlighted. Entire websites, particular online objects or even links can disappear over time. The DVAF method allows missing links to be identified, but the researcher must decide whether to consider only currently active links or all the links that existed at one time or another. The results reported here indicate that even links from academic websites can disappear, evidencing results from classic studies (e.g., Klein et al., 2014; Koehler, 1999; Payne & Thelwall, 2007). This problem is a major limitation of link-based impact analysis, which is partially solved by using the historic index offered by Majestic.
Another limitation of the method described here is its dependence on one specific link data source, in this case Majestic. Each data source has its strengths and weaknesses, but ultimately each defines its own specific method of data collection (as is equally true of bibliographic databases). Although this issue can be addressed in part by designing one’s own crawler, this solution is not readily implemented, as Majestic's worldwide coverage (and system maintenance) is technically and economically difficult to achieve. Considering these potential limitations, the DVAF method has been designed and applied to measure the online impact of the CONSORT and SPIRIT reporting guidelines, breaking down the impact into six facets, being the results obtained highly satisfactory. In addition, due to the systemic characteristics of the DVAF method, it can be extrapolated to be used in any link analysis, regardless the nature of the source and target objects.
Conclusions
Our analysis represents the first attempt to systematically evaluate the impact of reporting guidelines for randomised trial reports and protocols using methods and tools from link analysis. In light of our results, it is concluded that the online impact of CONSORT and SPIRIT could be improved. The study has served to identify reporting guidelines for randomised trial reports and protocols, key actors disseminating them (domain names, websites, source objects), and impact (citations, URL citations, Altmetrics).
In our opinion, these findings could be used to strengthen reporting guidelines uptake to increase value and reduce waste from incomplete or unusable randomised trial reports and protocols.
Finally, a new link analysis method (DVAF) has been designed and tested, aimed at cleaning link data. The method has been shown to be efficient to decompose the online impact into different facets (scattering, degree of similarity, broadness, diversity, and URL decay), increasing the accuracy of link analysis.
Notes
Each online object is referred to by means of its URL.
For example, the URLs “https://www.bmj.com/content/361/bmj.k1079” and “https://doi.org/10.1136/bmj.k1079” point to the same research object (a journal article), and each of them receive links from different sources.
Status codes are issued by a server in response to a client request made to the server.
Clinical guidelines to the CONSORT 2010 statement published in BMJ (Schulz, 2010a): https://plu.mx/a/1jr0-Zals0nbC0qAsG9mV4myUpvyCq-xkm59jlAe1to/clinical_citation.
Policy guidelines to the CONSORT 2010 statement published in BMJ (Schulz, 2010a): https://plu.mx/a/1jr0-Zals0nbC0qAsG9mV4myUpvyCq-xkm59jlAe1to/policy_citation.
biomedcentral.eu is a mirror of https://bmcmedicine.biomedcentral.com.
A mirror website is a replica of another website with a different URL to that of the original site, but which hosts identical or near-identical content.
References
Aguillo, I. F., Ortega, J. L., Fernández, M., & Utrilla, A. (2010). Indicators for a webometric ranking of open access repositories. Scientometrics, 82(3), 477–486. https://doi.org/10.1007/s11192-010-0183-y
Altman, D. G. (2005). Endorsement of the CONSORT statement by high impact medical journals: Survey of instructions for authors. BMJ (Clinical Research Ed.), 330(7499), 1056–1057. https://doi.org/10.1136/bmj.330.7499.1056
Altman, D. G., & Simera, I. (2016). A history of the evolution of guidelines for reporting medical research: The long road to the EQUATOR Network. Journal of the Royal Society of Medicine, 109(2), 67–77. https://doi.org/10.1177/0141076815625599
Bar-Ilan, J. (2005). What do we know about links and linking? A framework for studying links in academic environments. Information Processing & Management, 41(3), 973–986. https://doi.org/10.1016/j.ipm.2004.02.005
Barjak, F., & Thelwall, M. (2008). A statistical analysis of the web presences of European life sciences research teams. Journal of the American Society for Information Science and Technology, 59(4), 628–643. https://doi.org/10.1002/asi.20776
Barjak, F., Li, X., & Thelwall, M. (2007). Which factors explain the web impact of scientists’ personal homepages? Journal of the American Society for Information Science and Technology, 58(2), 200–211. https://doi.org/10.1002/asi.20476
Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D., Olkin, I., Pitkin, R., Rennie, D., Schulz, K. F., Simel, D., & Stroup, D. F. (1996). Improving the quality of reporting of randomized controlled trials: The CONSORT statement. JAMA: The Journal of the American Medical Association, 276(8), 637–639. https://doi.org/10.1001/jama.276.8.637
Catalá-López, F., Alonso-Arroyo, A., Page, M. J., Hutton, B., Ridao, M., Tabarés-Seisdedos, R., Aleixandre-Benavent, R., & Moher, D. (2019). Reporting guidelines for health research: Protocol for a cross-sectional analysis of the EQUATOR Network Library. British Medical Journal Open, 9(3), e022769. https://doi.org/10.1136/bmjopen-2018-022769
Catalá-López, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabarés-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020). Global mapping of randomised trials related articles published in high-impact-factor medical journals: A cross-sectional analysis. Trials, 21(1), 34. https://doi.org/10.1186/s13063-019-3944-9
Caulley, L., Cheng, W., Catalá-López, F., Whelan, J., Khoury, M., Ferraro, J., Husereau, D., Altman, D. G., & Moher, D. (2020a). Citation impact was highly variable for reporting guidelines of health research: A citation analysis. Journal of Clinical Epidemiology, 127, 96–104. https://doi.org/10.1016/j.jclinepi.2020.07.013
Caulley, L., Catalá-López, F., Whelan, J., Khoury, M., Ferraro, J., Cheng, W., Husereau, D., Altman, D. G., & Moher, D. (2020b). Reporting guidelines of health research studies are frequently used inappropriately. Journal of Clinical Epidemiology, 122, 87–94. https://doi.org/10.1016/j.jclinepi.2020.03.006
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., Doré, C. J., Parulekar, W. R., Summerskill, W. S. M., Groves, T., Schulz, K. F., Sox, H. C., Rockhold, F. W., Rennie, D., & Moher, D. (2013a). SPIRIT 2013 statement: Defining standard protocol items for clinical trials. Annals of Internal Medicine, 158(3), 200–207. https://doi.org/10.7326/0003-4819-158-3-201302050-00583
Chan, A.-W., Tetzlaff, J. M., Gøtzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hróbjartsson, A., Schulz, K. F., Parulekar, W. R., Krleza-Jeric, K., Laupacis, A., & Moher, D. (2013). SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ (Clinical Research Ed.), 346, e7586. https://doi.org/10.1136/bmj.e7586
Dudek, J., Pina, D.G., & Costas, R. (2021). Co-link analysis as a monitoring tool: A webometric use case to map the web relationships of research projects. In W. Glanzel, S. Heeffer, P-S. Chi, & R. Rousseau (Eds.) 18th International Conference on Scientometrics and Informetrics, ISSI 2021 (pp. 339–344). International Society for Scientometrics and Informetrics. https://research.tudelft.nl/files/100319468/2110.04251.pdf
Fang, Z., Dudek, J., Noyons, E., & Costas, R. (2020). Science cited in policy documents: Evidence from the Overton database. In Altmetrics conference. http://altmetrics.org/wp-content/uploads/2020/11/02_submission_Fang_Dudek_Noyons_Costas-altmetrics20.pdf
Farrokhzdeh, F., Shekofteh, M., Kazerani, M., Bakhshandeh, H., & Ghadrdoost, B. (2021). CONSORT Abstract item reporting quality and altmetrics in the cardiovascular core clinical journals. Journal of Scientometric Research, 10(2), 160–165.
Font-Julián, C. I., Ontalba-Ruipérez, J.-A., Orduña-Malea, E., & Thelwall, M. (2022). Which types of online resource support US patent claims? Journal of Informetrics., 16(1), 101247. https://doi.org/10.1016/j.joi.2021.101247
Gabriel, S. E., & Normand, S.-L.T. (2012). Getting the methods right–the foundation of patient-centered outcomes research. The New England Journal of Medicine, 367(9), 787–790. https://doi.org/10.1056/NEJMp1207437
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014). Reducing waste from incomplete or unusable reports of biomedical research. Lancet, 383(9913), 267–276. https://doi.org/10.1016/S0140-6736(13)62228-X
Hopewell, S., Altman, D. G., Moher, D., & Schulz, K. F. (2008). Endorsement of the CONSORT Statement by high impact factor medical journals: A survey of journal editors and journal “Instructions to Authors.” Trials, 9(1), 20. https://doi.org/10.1186/1745-6215-9-20
Jones, D. (2012). Flow Metrics™ will change the way you look at links. Majestic Blog. https://blog.majestic.com/development/flow-metrics
Karmakar, M., Banshal, S. K., & Singh, V. K. (2021). A large-scale comparison of coverage and mentions captured by the two altmetric aggregators: Altmetric.com and PlumX. Scientometrics, 126(5), 4465–4489. https://doi.org/10.1007/s11192-021-03941-y
Klein, M., Van de Sompel, H., Sanderson, R., Shankar, H., Balakireva, L., Zhou, K., & Tobin, R. (2014). Scholarly context not found: One in five articles suffers from reference rot. PLoS ONE, 9(12), e115253. https://doi.org/10.1371/journal.pone.0115253
Koehler, W. (1999). An analysis of web page and web site constancy and permanence. Journal of the American Society for Information Science, 50(2), 162–180. https://doi.org/10.1002/(SICI)1097-4571(1999)50:2%3C162::AID-ASI7%3E3.0.CO;2-B
Kousha, K. (2019). Web citation indicators for wider impact assessment of articles. In W. Glänzel, H. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 801–818). Springer.
Kryl, D., Allen, L., Dolby, K., Sherbon, B., & Viney, I. (2012). Tracking the impact of research on policy and practice: Investigating the feasibility of using citations in clinical guidelines for research evaluation. British Medical Journal Open, 2(2), e000897. https://doi.org/10.1136/bmjopen-2012-000897
Kumar, B. S., & Kumar, K. M. (2012). Persistence and half-life of URL citations cited in LIS open access journals. Aslib Proceedings, 64(4), 405–422. https://doi.org/10.1108/00012531211244752
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., SPIRIT-AI and CONSORT-AI Working Group. (2020). Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nature Medicine, 26(9), 1364–1374. https://doi.org/10.1038/s41591-020-1034-x
Moher, D., Schulz, K. F., Altman, D., CONSORT Group. (2001). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA: The Journal of the American Medical Association, 285(15), 1987–1991. https://doi.org/10.1001/jama.285.15.1987
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010a). Guidance for developers of health research reporting guidelines. PLoS Medicine, 7(2), e1000217. https://doi.org/10.1371/journal.pmed.1000217
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010). CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. BMJ (Clinical Research Ed.), 340, c869. https://doi.org/10.1136/bmj.c869
Moher, D., Altman, D. G., Schulz, K. F., Simera, I., & Wager, E. (Eds.). (2014). Guidelines for reporting health research: A user’s manual. John Wiley & Sons.
Oguz, F., & Koehler, W. (2016). URL decay at year 20: A research note. Journal of the Association for Information Science and Technology, 67(2), 477–479. https://doi.org/10.1002/asi.23561
Ontalba-Ruipérez, J.-A., Orduna-Malea, E., & Alonso-Arroyo, A. (2016). Identifying institutional relationships in a geographically distributed public health system using interlinking and co-authorship methods. Scientometrics, 106(3), 1167–1191. https://doi.org/10.1007/s11192-016-1839-z
Orduña-Malea, E. (2021). Dot-science top level domain: Academic websites or dumpsites? Scientometrics, 126(4), 3565–3591. https://doi.org/10.1007/s11192-020-03832-8
Orduña-Malea, E., & Aguillo, I. F. (2022). Are link-based and citation-based journal metrics correlated? An Open Access mega publisher case study. Quantitative Science Studies. https://doi.org/10.1162/qss_a_00199
Orduna-Malea, E., & Alonso-Arroyo, A. (2017). Cybermetric techniques to evaluate organizations using web-based data. Elsevier.
Orduña-Malea, E., & Costas, R. (2021). Link-based approach to study scientific software usage: The case of VOSviewer. Scientometrics, 126(9), 8153–8186. https://doi.org/10.1007/s11192-021-04082-y
Orduña-Malea, E., Thelwall, M., & Kousha, K. (2017). Web citations in patents: Evidence of technological impact? Journal of the Association for Information Science and Technology, 68(8), 1967–1974. https://doi.org/10.1002/asi.23821
Orkin, A. M., Gill, P. J., Ghersi, D., Campbell, L., Sugarman, J., Emsley, R., Steg, P. G., Weijer, C., Simes, J., Rombey, T., Williams, H. C., Wittes, J., Moher, D., Richards, D. P., Kasamon, Y., Getz, K., Hopewell, S., Dickersin, K., Wu, T., et al. (2021). Guidelines for reporting trial protocols and completed trials modified due to the COVID-19 pandemic and other Extenuating Circumstances: The CONSERVE 2021 statement: The CONSERVE 2021 statement. JAMA: The Journal of the American Medical Association, 326(3), 257–265. https://doi.org/10.1001/jama.2021.9941
Ortega, J. L. (2018). Reliability and accuracy of altmetric providers: a comparison among Altmetric.com, PlumX and Crossref event data. Scientometrics, 116(3), 2123–2138. https://doi.org/10.1007/s11192-018-2838-z
Ortega, J. L. (2020). Altmetrics data providers: A metaanalysis review of the coverage of metrics and publication. Profesional De La Información, 29(1), e290107. https://doi.org/10.3145/epi.2020.ene.07
Ortega, J. L., & Aguillo, I. F. (2009). Mapping world-class universities on the web. Information Processing & Management, 45(2), 272–279. https://doi.org/10.1016/j.ipm.2008.10.001
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Systematic Reviews, 10(1), 89. https://doi.org/10.1186/s13643-021-01626-4
Payne, N., & Thelwall, M. (2007). A longitudinal study of academic webs: Growth and stabilisation. Scientometrics, 71(3), 523–539. https://doi.org/10.1007/s11192-007-1695-y
Schulz, K. F., Altman, D. G., Moher, D., CONSORT Group. (2010a). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMJ (Clinical Research Ed.), 340, c332. https://doi.org/10.1136/bmj.c332
Schulz, K. F., Altman, D. G., Moher, D., CONSORT Group. (2010b). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. PLoS Medicine, 7(3), e1000251. https://doi.org/10.1371/journal.pmed.1000251
Shamseer, L., Hopewell, S., Altman, D. G., Moher, D., & Schulz, K. F. (2016). Update on the endorsement of CONSORT by high impact factor journals: A survey of journal “Instructions to Authors” in 2014. Trials, 17(1), 301. https://doi.org/10.1186/s13063-016-1408-z
Spinellis, D. (2003). The decay and failures of web references. Communications of the ACM, 46(1), 71–77.
Stevens, A., Shamseer, L., Weinstein, E., Yazdi, F., Turner, L., Thielman, J., Altman, D. G., Hirst, A., Hoey, J., Palepu, A., Schulz, K. F., & Moher, D. (2014). Relation of completeness of reporting of health research to journals’ endorsement of reporting guidelines: Systematic review. BMJ (Clinical Research Ed.), 348, g3804. https://doi.org/10.1136/bmj.g3804
Szomszor, M., & Adie, E. (2022). Overton—A bibliometric database of policy document citations. ArXiv preprint. https://arxiv.org/pdf/2201.07643.pdf
Thelwall, M. (2004). Link analysis: An information science approach. Elsevier.
Thelwall, M. (2006). Interpreting social science link analysis research: A theoretical framework. Journal of the American Society for Information Science and Technology, 57(1), 60–68. https://doi.org/10.1002/asi.20253
Thelwall, M., & Zuccala, A. (2008). A university-centred European Union link analysis. Scientometrics, 75(3), 407–420. https://doi.org/10.1007/s11192-007-1831-8
Thelwall, M. (2011). A comparison of link and URL citation counting. Aslib Proceedings, 63(4), 419–425. https://doi.org/10.1108/00012531111148985
Thelwall, M. (2021). The story of a colony: Theory development in webometric research. In D. H. Sonnenwald (Ed.), Theory development in the information sciences (pp. 164–182). University of Texas Press.
Thelwall, M., & Kousha, K. (2015). Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web. Profesional de la Información, 24(5), 587–606. https://doi.org/10.3145/epi.2015.sep.08
Thelwall, M., & Kousha, K. (2016). Are citations from clinical trials evidence of higher impact research? An analysis of Clinicaltrialsgov. Scientometrics, 109(2), 1341–1351. https://doi.org/10.1007/s11192-016-2112-1
Thelwall, M., Kousha, K., & Abdoli, M. (2017). Is medical research informing professional practice more highly cited? Evidence from AHFS DI Essentials in drugs.com. Scientometrics, 112(1), 509–527. https://doi.org/10.1007/s11192-017-2292-3
Thelwall, M., & Maflahi, N. (2016). Guideline references and academic citations as evidence of the clinical value of health research. Journal of the Association for Information Science and Technology, 67(4), 960–966. https://doi.org/10.1002/asi.23432
Turner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C., & Moher, D. (2012). Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database of Systematic Reviews, 11(11), MR000030. https://doi.org/10.1002/14651858.MR000030.pub2
Vaughan, L., & Thelwall, M. (2003). Scholarly use of the web: What are the key inducers of links to journal Web sites? Journal of the American Society for Information Science and Technology, 54(1), 29–38. https://doi.org/10.1002/asi.10184
Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. PLoS Biology, 19(4), e3001162. https://doi.org/10.1371/journal.pbio.3001162
Williams, A. E. (2019). Exploring the utility of an emerging altmetric platform: A SWOT analysis of plum analytics. Digital Library Perspectives, 35(3–4), 193–204. https://doi.org/10.1108/DLP-08-2019-0031
Yang, S., Han, R., Ding, J., & Song, Y. (2012). The distribution of web citations. Information Processing & Management, 48(4), 779–790. https://doi.org/10.1016/j.ipm.2011.10.002
Zahedi, Z., & Costas, R. (2018). General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators. PLoS ONE, 13(5), e0197326. https://doi.org/10.1371/journal.pone.0197326
Funding
Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Orduña-Malea, E., Alonso-Arroyo, A., Ontalba-Ruipérez, JA. et al. Evaluating the online impact of reporting guidelines for randomised trial reports and protocols: a cross-sectional web-based data analysis of CONSORT and SPIRIT initiatives. Scientometrics 128, 407–440 (2023). https://doi.org/10.1007/s11192-022-04542-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-022-04542-z