Links between information construction and information gain. Entropy and bibliometric distributions.

Thierry Lafouge

**Christine
Michel**^{[1]}

Laboratoire Recodoc

Université Claude Bernard Lyon 1, Bat 721

43 Bd du 11 novembre 1918

69622 Villeurbanne Cedex, France.

__Christine.Michel@montaigne.u-bordeaux.fr__

__ __

__Keywords__ :
bibliometric distribution / entropy / least effort law /
information theory

The study of the statistical regularities observed in the field of the information production and use has confirmed the existence of important similarities. Thus, the existence of regularities and measurable ratios allow the prevision and the concept of laws. In the fifties, C. Shannon (Shannon C., Weaver W. 1975 : Théorie mathématique de la communication, Bibliothèque du CELP, 1975) modeled the information circulation theory. The entropy hypothesis of this theory is: the more ranked a system is, the less information it produces. Theoretical studies have tried to formalize the connection between the bibliometric distribution and the entropy. In this paper we try to extend previous results linked with "the least effort principle" and the analytical slope of a bibliometric distribution. In the first and second parts we present some recalls about entropy and bibliometric distributions, and after that, we describe different links between them.

Let a
source of information produce *n* random events of
respective probabilities *p _{1}, p_{2}...., p_{n}*where

_{}_{}

As
well as in the systems studied in physical science, the higher
the entropy source is, the less organized the system is
(foreseeable here). Shannon supposes that the more ranked a
system is and the less information it produces. Thus, the
entropy *H* is at a maximum if all the events are
equiprobable, that is to say if we have _{} and in
this case *H = Log _{2}(n).*

The logarithm function of base two is often used because it is coherent with the electronic information binary digit code (the bit being defined as the maximum entropy of the random binary source).

It is possible to spread and generalize the entropy's definition for a continuous probability's law.

In this
case, we will not talk about distribution but density function
(noted *v*) of a random phenomenon. We will define its
entropy by the function:

_{}

In bibliometry, the events normally studied are: the papers or keywords' output, the books borrowing, the author quoting ... The sources taken into account will then be the authors, the bibliographical references and the books. These events are noticeable because they are featured by statistical regularities. Thus, it is interesting to observe how, by the entropy calculus, the quantity of information changes from these various sources according to the random process that lead them.

The classical bibliometric distributions are: the Lotka's law related to the publications' output number, the Zipf's law related to how often words appear in a text and the Bradford's one related to the articles' dispersion in journals. According to the studies phenomena, these laws will be put clearly either in a frequential or in a ranked way.

The
frequential approach is the oldest one (Lotka 1926). The
probability that an event appears is calculated according to
its apparition frequency. An example of a frequential
distribution law is the law that analyses the scientific
article's output by searchers. Lotka suggests to write the
distribution of the number of scientists who have written
*i* publications:

_{} where _{} is the
maximum productivity of a scientist. This law is generalized in
the formula: _{} where
*K* and *a*are constants
depending on the studied field.

The statement of a law according to the rank implies that the information source has been previously ranked according to its output. These distributions per rank are used when the source's production ranking is inevitable to point out the apparition of regularity. The most characteristic example of a distribution per rank is Zipf's law. It observes how often words appear in English texts. By ranking these numbers in a decreasing way, he observes that there was an inversely proportional connection between the presentation rank of a word and its apparition frequency. Zipf expresses this regularity with the following equation:

_{}
where g(r) represents the frequency of the rank r.

Numerous
works have shown equivalencies between the distributions per
rank and the frequential distributions (Egghe 1988). The choice
between the one or the other of the presentations depends on
the study one wants to carry out. In the study of keyword's
distributions, the distribution per rank will be chosen because
it is the most significant (Quoniam 1992). In a recent
publication (Lhen and al. 1995), in collaboration with the
«CRRM»^{[2]}, we have shown the relevance of the generalized entropy's
theory (Reyni 1960) to handle distributions.

Pareto's distribution for the continuous case has the same role as Lotka's distribution in the discrete case . It is written as follows:

_{}

to
instead of to _{}

Haitun
(Haitun 1982) defines a Zipfian distribution with the following
hyperbolic density function: _{} where t belongs to the interval
_{} and where a and *C*
are positive constants. If _{}, we are in the well-known case of Lokta's law.
All the mathematical properties of such distributions have been
widely studied. S.D. Haitun opposes this type of distribution
to the Gaussian ones.

Very often the geometrical distribution is used to quantify some regularities observed in bibliometry, especially in the field of documentary uses in libraries (Lafouge et al. 1997).

A geometrical distribution is written as follows:

_{}

If the
continuous equivalent of this distribution is written the
following exponential is
obtained: _{}_{}_{}*.*

Another distribution called the negative binomial distribution is often used (Lafouge 1999) to model the use distributions.

It is written as follows:

_{}

_{}

If
_{} we have
the equation of a geometric distribution. As far as we know
there is not any law strictly equivalent to the negative
binomial one in the continuous case.

Let (F,I) be a bibliometric distribution where :

**F** represents the set of all the patterns
referring to an identified bibliographic source as for example
the authors, publications.... The pattern may also be a word,
defined by sequences of characters surrounded with separators,
or several words.

**I** represents the
set of all the items: each item is a positive number indicating
the pattern occurrence (the number of appearances).

All
these values form the source output. We have previously seen
that several representations of a distribution are possible. We
suppose that this source is ruled by a stable random process
that can be observed and it is characterized by a function
conveying the effort to produce all the different items. Thus
the studied question is as follows: **for a given quantity of
effort, what is the connection between the random distribution
of the source and the effort function when the quantity of
information (considered on the Shannon sense) produced by the
source is maximized**? The technique used is called MEP,
Maximum Entropy Principle, and has already been used in other
studies.

Kantor
(Kantor 1998) presents an application where MEP is used to
improve information retrieval. Let us consider the K index
terms * _{}* of a
query, Kantor considers the 2

We make
the assumption that every document is either relevant or not.
In an atom, the relevant documents take the value 1, the others
the value 0. An atom A(i) has the probability _{} to be relevant, and _{} to be non relevant. By knowing
* _{}* (k=1, ..., K) "the probability of
relevance for documents indexed by terms T

Find a
positive (condition 1) distribution _{} respecting (condition 2)
_{},

and
which maximizes the entropy _{}

subjected to (condition 3')_{}
where _{} for
each k.

and
(condition 3'')_{}

The evaluation study results show that the MEP method is useful in IR for small collections but not for big ones. Final discussions argue that this method may be improved by taking into account finest criterion than «presence or absence of terms in a document» to estimate the document's relevance.

In
the case of continuous distributions, Yablonsky (Yablonsky
1980) used the MEP in order to find the distribution of the
«least effort principle» in the case of scientific article
production. The necessary effort to produce an article is
modeled by the function *E*, defined by
_{}, where
*k* is a positive constant.

The
effort made to produce the first article is _{}, it is called the "minimum state of the
scientist". The effort made to produce the second one
_{} requires
less effort from the scientist, and so on.

The
aim is to find a positive (condition 1) density function
*v(t)* on theinterval [a**,** ¥[ _{},respecting (condition 2)
_{}
(*v*
density function)

and
which maximizes the entropy :
_{}_{ }

subjected to (condition 3) _{} (Constraint of an effort)

This model is known to be the «least effort law»: the density function which results from the question of the entropy maximization, under an effort constraint, is the Zipfian function.

Indeed,
the entropy maximization for the effort function
* _{}* is
obtained for a density function whose analytical shape
is:

The
calculation of the entropy according to a gives: _{}

a being positive by definition, it is easy to show that the entropy is a decreasing function of a. The classical interpretation of Lotka's law is found, that is to say the higher a is, the bigger the gap between the number of scientists who produce a little is. (Knowing that there are few scientists who produce a lot compared to the number of scientists who produce a little).

Our aim is to spread these results to other bibliometric distributions: the geometric distribution and the binomial negative distribution.

We will consider the continuous case, so we will keep Yablonsky`s notations:

§ Let *v(t)* be the bibliometric
distribution density function, *t* being defined on
[a**,** ¥[, such as :
*v(t)**³
0*on [a**,** ¥[_{} (condition 1).

§ Let *E(t)* be the
effort production function (*E* is a positive constant
corresponding to the average effort),

§ And *H*_{ } be the
entropy.

We
have found couples (*E(t), v(t*)) where :

(condition 2)
_{}

(condition 3) _{}
(Constraint of an
effort)

(condition 4) _{}_{ is
maximized}

Let us remember the density function of a geometrical distribution :

_{}_{} *(
1- q = p )*

We have
chosen to take the linear effort function
* _{}*.

The
case *t*
=1 as previously seen corresponds
to the minimal state of the scientist who has produced one
publication.

If we put
ourselves as previously in the case of the scientific output,
in the case of a linear function_{}, we want to
show that when the MEP is applied with constraint of an effort,
the resulting function is a geometric distribution with a
density function such as :

_{}

Remark :
in the case of *t =1*, we have * _{}* the average of

^{ }

^{ }

^{Demonstration}

^{ }

These results are shown using variational calculation techniques.

Let us demonstrate that the function:

_{}** ^{ }**
where

checks the conditions

*v* *³**0* on [1**,** ¥[** **
(1)

* *
_{}
(2)

** **
_{}
(3)

and
maximizes the function: _{}

We can
easily show that the function *w*checks
the conditions (1) (2) and (3). Let us show that
*w*maximizes the entropy. We will prove
that *H*
reaches its maximum for the
function *w*.

Let F be the following function:

_{}* *where *l** *is a
constant whose value is:_{}

We
have: _{}

We can
easily show that this derivative cancels for
*w*.

So for t
fixed, we have: _{} and _{}

*F* being convex
and _{}canceling for
*w* with any value of t determined we can
write:

_{},_{}

So_{},_{}

And
so _{},_{}

Let n be any function checking the normalization (conditions 2) and (condition 3)

_{}** ***³*_{} hence
the result is proved.

* _{}*.

We have
the same result as previously for the variation of *H* in
function of _{}. Moreover
we can notice that the entropy calculated with a linear effort
law is always inferior to an entropy calculated with a
logarithmic effort law and that the difference varies in an
inversely proportional way. The dispersion, and then the
entropy, is stronger in the Zipfian case. This result justifies
the choice of the entropy of order 1 to feature the diversity
of a Zipfian distribution (Lhen et al.1995).

Note

We can
show Yablonsky's previous result using the same technique with
the function denoted *F*:

_{}

We have said that we do not know the density function of the negative binomial law in the continuous case. So we will use convolution techniques to build a new distribution that we will call here " pseudo negative binomial".

The convolution technique is defined by :

If
*X _{1}* and

_{}

This definition is generalized for a convolution product of order j.

Indeed,
if *X _{j}* is
a finite series of independent identically distributed random
variables of density

*F _{1}* =

_{}

_{}

The convolution techniques have been used to give a new interpretation of Lotka's law (Egghe 1994).

We know
(Calot 1984) that in the discrete case the sum of *j*
independent variables of a geometric law *G(q)* is a
negative binomial law _{}

The
exponential distribution is the geometric distribution density
function: _{}. If we
build a density function *v _{j}* from the
convolution of

A simple
calculation shows that the convolution product of order j
of the *v* function is :

_{}

Egghe
has shown (Egghe 1994) the stability properties for the
geometric distribution using this convolution product. If the
original distribution is an exponential one, we can interpret
this distribution *v _{j}* in different
ways:

-In a
context of articles output*,* _{} is the
proportion of authors who have written *i* articles,
each article having exactly *j* authors.

- In a
context of bibliographic references keywords
distributions, _{} is the
proportion of words used *i* times, each reference having
exactly *j* keywords

Then the
question that we want to solve is: if we set the effort
quantity (noted *E*_{j}), what is the nature
(linear, logarithmic...) of the effort distribution (noted
*EF*) linked to a random process of a negative
binomial pseudo type, when the information quantity is maximum?
This question is mathematically written:

Let's
consider the following
distribution :_{}

What
is the nature of the effort distribution *EF* that
checks the following conditions:

_{}
(Constraint of an effort)

and
maximizes the entropy *H*: -_{}
for any function
n checking the conditions:

* *
*v(t) *
*³**0* on
[0**,** ¥[

_{}

We will
see if the effort function *EF*
,* _{}*is valid
to solve the problem.

__ __

__ __

__ __

__Demonstration__:

We can easily check
by recurrence that _{}.

It's more difficult
to know the value of the effort_{}.

Let us put
_{}

_{}

_{}

however we
have _{} and we have seen
that _{}

So
_{}

So we have shown
that: _{}

Let's now calculated
the value of _{}

_{}

_{}

With
_{}

Let's
calculate _{}

We will show
recurrently that _{}

_{} and
_{}

Now, let suppose
that _{} ,
_{}.

It's easy to show
that _{}

So
_{}

And So
_{}

In Annex 1 we have the result :

_{}

In Annex 2 we have the result :

_{}

So
_{}

We have
_{}

So
_{}

_{}

By using complete Gamma function it is possible to say that :

_{} (g
is Euler's
constant :0.5772....)

So
_{}

With initial
notations : _{}

So
_{}

So
_{}

Now let us show that EF maximizes the entropy.

We will
show that the entropy *H* reaches its minimum for the
function _{}.

Let
*F* be the following function:

_{}* *where l
is the constant
_{}* *

We
have: _{}

Let us
put : _{}

_{}

_{}

If we
replace l
by its value then this derivative
cancels for _{}.

For t
fixed we have: _{}

*F*being convex and
_{}canceling
in _{}* *for any
value of *t* fixed we can write:

_{}

that is to say:

_{}

_{}

Let n be any function checking the normalization (conditions 2) and (condition3):

** ** _{}

So the entropy is
maximum for *n _{j}(t)*.

Calculation of the
entropy** ** _{}

_{}

_{}

_{}We have:
_{} ,
_{}and

_{}

So

_{}

We can
remark that for j=1 we have the same results as for the
geometrical low, i.e. _{}

We can
remark that_{}

We want
now to analyze the characteristics of the two functions
_{} and _{}.

In all
cases of a,
_{} and _{} are
increasing functions of j, the author's number. It means that
publishing with many authors required more effort but induce,
in all cases, a gain of information. We analyze now the
difference _{}-_{}.

_{} j
=1,2.......

_{} is
depending on *j* and *a*. The 3
following curves give the results for a=1 (figure
1), a =2 (figure 2) and a =3 (figure
3).

Figure 1 : case of a =1

Figure 2 : case of a =3 Figure 3 : case of a = 5

As it's
represented in figure 1, in the case of a=1 we
have always_{}, which
means that the gain of information is always higher than the
effort need to produce it. On the other case, if
a>1 (figure 2 and 3), the curves of
_{} is
first negative, cut the x-axe on a particular point
*m*_{a} and stay
positive after than. The value *m*_{a} define the minimum number of authors required for having a
gain of information relative to a paper, biggest than the
effort due to produce it.

We have
observed that, in the Zipfian distribution law case, the
corresponding effort function was logarithmic, in the geometric
law case, it was linear. In each case the entropy is decreasing
with a.
In the pseudo negative
binomial case, the effort function is composed by two
functions: the fist one is linear (effort constant) and the
other logarithmic (the least effort law). For *j* fixed
the corresponding entropy is decreasing with a.

Let us now vary
j. Let us remember a possible
interpretation of the pseudo negative binomial law: in a context of articles
output*,* _{} is the proportion
of authors who have written *i* articles,
each article having exactly *j* authors. There is a link
between the effort and the entropy variation. We have shown
that for each value of a, there is a number
m_{a}
representing the minimum number
of authors required to write a paper having more information
than the effort due to produce it. In the case of
*a=1*, we have
*m _{1}=1*, i.e. whatever the number of author,
each new article produce more information that the effort
developed to write it. That needs to be
verify.

EGGHE
L.(1988): *On the classification of the classical
bibliometric laws.* __Jounal of Documentation__, Vol
44(1), 1988, p. 53-62.

EGGHE L.
(1994): *Special features of the author publication
relationship and a new explanation of Lotka's law based on
convolution theory.* __Journal of the American Society for
Information Science__, Vol 45(6),1994, p. 422-427.

CALOT G.
(1984): __Cours de calcul de probabilité__. Chapitre 12.
Dunod décision 1984, 476 pages.

CAUMEL Y.
(1988): __Probabilités, théorie et applications.__ Chapitre
8. Eyrolles, 1988, 290 pages.

HAITUN SD.
Stationary Scientometrics Distributions. __Scientometrics
N°4__, 1982, Part I p.5-25, Part II p.89-104, Part III
p.181-194.

KANTOR
Paul B., JUNG Jin Lee: *Testing the Maximum Entropy Principle
for information Retrieval.* __Journal of the American
Society for Information Science__, Vol 49 (6), 1998, p
557-556.

LAFOUGE
T**.,** LAINE CRUZEL S. (1997): *A new explanation of the
geometrical law in the case of library circulation data .*
__Information Processing and Management__, Vol 33 (4), 1997,
p. 523-527.

LAFOUGE
T.,GUINET E. (1999): *A new explanation of the negative
binomial law and the Poisson law with regard to library
circulation data.* __Journal of Information Science__ Vol 25 (1), 1999, p 89-93.

LHEN J.,
LAFOUGE T., ELSKENS Y., QUONIAM L., DOU H. (1995): *La
« statistique des lois de Zipf.* __Revue Française de
Bibliométrie__ N°14, 1995, p. 135-146.

LOTKA
A. (1926) : *The frequency distribution of scientific
productivity*. __Journal of the Washington Academy of
Sciences__, 16, 1926, p.317-323.

QUONIAM L.
(1992): *Bibliométrie sur les références bibliographiques:
méthodologie*, p. 244, 261.__La Veille technologique;
l'information scientifique, technique et industrielle.__
Dunod, 1992, 436 pages.

REYNI A.
(1961): *On measures of entropy and information.* In
NEYMAN J. ed Berkeley symposium on mathematical statistic and
probability .

SHANNON
C.,WEAVER W.(1975): __Théorie mathématique de la
communication.__ Bibliothèque du CEPL, 1975, 188
pages.

YABLONSKY A.L (1980): *On fundamental regularities of
the distribution of scientific productivity.*
__Scientometrics__, Vol 2,(1), 1980, p. 3-34.

_{}

_{}

_{}

_{}

We know
that _{} And _{}

So
_{}

Moreover _{}

_{}

_{}

So
_{}

_{}

_{}

_{}

And
_{}

So_{}

_{}

_{}

_{}

_{}