Validity as a measure of importance

Impact factor

The impact factor (IF) of an academic journal is a measure reflecting the average number of citations to recent articles published in the journal. It is frequently used as a proxy for the relative importance of a journal within its field, with journals with higher impact factors deemed to be more important than those with lower ones. The impact factor was devised by Eugene Garfield, the founder of the Institute for Scientific Information. Impact factors are calculated yearly starting from 1975 for those journals that are indexed in the Journal Citation Reports.

Calculation

In any given year, the impact factor of a journal is the average number of citations received per paper published in that journal during the two preceding years. For example, if a journal has an impact factor of 3 in 2008, then its papers published in 2006 and 2007 received 3 citations each on average in 2008. The 2008 impact factor of a journal would be calculated as follows:

A = the number of times that articles published in that journal in 2006 and 2007, were cited by articles in indexed journals during 2008.

B = the total number of "citable items" published by that journal in 2006 and 2007. ("Citable items" are usually articles, reviews, proceedings, or notes; not editorials or letters to the editor.)

2008 impact factor = A/B.

(Note that 2008 impact factors are actually published in 2009; they cannot be calculated until all of the 2008 publications have been processed by the indexing agency.)

New journals, which are indexed from their first published issue, will receive an impact factor after two years of indexing; in this case, the citations to the year prior to Volume 1, and the number of articles published in the year prior to Volume 1 are known zero values. Journals that are indexed starting with a volume other than the first volume will not get an impact factor until they have been indexed for three years. Annuals and other irregular publications sometimes publish no items in a particular year, affecting the count. The impact factor relates to a specific time period; it is possible to calculate it for any desired period, and the Journal Citation Reports (JCR) also includes a five-year impact factor. The JCR shows rankings of journals by impact factor, if desired by discipline, such as organic chemistry or psychiatry.

Use

The impact factor is used to compare different journals within a certain field. The ISI Web of Knowledge indexes more than 11,000 science and social science journals.

Criticisms

Numerous criticisms have been made of the use of an impact factor. For one thing, the impact factor might not be consistently reproduced in an independent audit There is a more general debate on the validity of the impact factor as a measure of journal importance and the effect of policies that editors may adopt to boost their impact factor (perhaps to the detriment of readers and writers). In short, there is some controversy about the appropriate use of impact factors.

Validity as a measure of importance

The impact factor is highly dependent on the academic discipline, possibly on the speed with which papers get cited in a field. The percentage of total citations occurring in the first two years after publication varies highly among disciplines from 1–3% in the mathematical and physical sciences to 5–8% in the biological sciences. Thus impact factors cannot be used to compare journals across disciplines.

The impact factor is based on the arithmetic mean number of citations per paper, yet citation counts follow a Bradford distribution (i.e., a power law distribution) and therefore the arithmetic mean is a statistically inappropriate measure For example, about 90% of Nature's 2004 impact factor was based on only a quarter of its publications, and thus the importance of any one publication will be different from, and in most cases less than, the overall number. Furthermore, the strength of the relationship between impact factors of journals and the citation rates of the papers therein has been steadily decreasing since articles began to be available digitally

This problem was exacerbated when the use of impact factors is extended to evaluate not only the journals, but the papers therein. The Higher Education Funding Council for England was urged by the House of Commons Science and Technology Select Committee to remind Research Assessment Exercise panels that they are obliged to assess the quality of the content of individual articles, not the reputation of the journal in which they are published. The effect of outliers can be seen in the case of the article "A short history of SHELX", which included this sentence: "This paper could serve as a general literature citation when one or more of the open-source SHELX programs (and the Bruker AXS version SHELXTL) are employed in the course of a crystal-structure determination". This article received more than 6,600 citations. As a consequence, the impact factor of the journal Acta Crystallographica Section A rose from 2.051 in 2008 to 49.926 in 2009, more than Nature (at 31.434) and Science (at 28.103). The second-most cited article in Acta Crystallographica Section A in 2008 only had 28 citations.

Finally, journal rankings constructed based solely on impact factors only moderately correlate with those compiled from the results of expert surveys.

It is important to note that impact factor is a journal metric and should not be used to assess individual researchers or institutions.