Maximum Entropy

The principle of maximum entropy originated in the last century with Gibbs and was re-constructed in its modern, broader form by Jaynes. Because of their interests it was natural that the first major application was to statistical mechanics and the derivation of classical thermodynamics. The original papers in the modern sequence are

31.``Information Theory and Statistical Mechanics," E.T. Jaynes, Phys. Rev. 106, 620-630 (1957). (A)

32.``Information Theory and Statistical Mechanics. II," E.T. Jaynes, Phys. Rev. 108, 171-190 (1957). (A)

33.``Foundations of Probability Theory and Statistical Mechanics," E.T. Jaynes, in Delaware Seminar in the Foundations of Physics, edited by M.Bunge (Springer-Verlag, Berlin, 1967), pp.77-101. (A)

34.``Gibbs vs. Boltzmann Entropies," E.T. Jaynes, Am. J. Phys. 33, 391-398 (1965). (I)

These last four papers are reprinted, along with others, in

35.E.T. Jaynes: Papers on Probability, Statistics and Statistical Physics, edited by R.D. Rosenkrantz (Reidel, Dordrecht, Holland, 1983). (A)

A twentieth-anniversary conference was held in 1978:

36.The Maximum Entropy Formalism, edited by R.D. Levine and M. Tribus (The MIT Press, Cambridge, Massachsetts, 1979). (A)

37.``Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross-Entropy," J.E. Shore and R.W. Johnson, IEEE Trans. IT-26 26-37, (1980). A consistency proof of the PME. (A)

A number of textbooks and monographs developing statistical mechanics based on Jaynes's ideas have appeared since 1957, the following group being somewhat comprehensive.

38.Thermostatics and Thermodynamics, M. Tribus (Van Nostrand, Princeton, 1961). (I)

39.Concepts in Statistical Mechanics, A. Hobson (Gordon and Breach, New York, 1971). (I)

40.Atoms and Information Theory, R. Baierlein (Freeman, San Francisco, 1971). (I)

41.Foundations of Statistical Mechanics, Volume I: Equilibrium Theory, W.T. Grandy, Jr. (Reidel, Dordrecht, Holland, 1987). (A)

42.Foundations of Statistical Mechanics, Volume II: Nonequilibrium Phenomena, W.T. Grandy, Jr. (Reidel, Dordrecht, Holland, 1988). (A)

Representative applications to more specific problems in statistical mechanics are given in the following articles.

43.``Dissipative evolution, initial conditions, and information theory," A.N. Proto, J. Aliaga, D.R. Napoli, D. Otero, and A. Plastino, Phys. Rev. A 39, 4223-4229 (1989). (A)

44. ``Maximum-entropy approach to classical hard-sphere and hard-disk equations of state," D. Wang and L.R. Mead, J. Math. Phys. 32, 2258-2262 (1991). (A)

In his 1975 Ph.D. thesis John Burg introduced the first application of maximum-entropy techniques into data analysis, in the context of geophysical time series (Ref.45), though he had reported the idea about eight years earlier. This opened an entirely new and rich area for the use of information-theoretic methods applied to physical problems. A short time later there followed an imaginative adaptation to image reconstruction by Gull and Daniell (Ref.46), and various authors began applying these methods to general spectral analysis.

45.``Maximum Entropy Spectral Analysis," J.P. Burg, Ph.D. thesis, Stanford University, 1975. (A)

46.``Image reconstruction from incomplete and noisy data," S.F. Gull and G.J. Daniell, Nature 272, 686-690 (1978). (A)

47.``On the Rationale of Maximum-Entropy Methods," E.T. Jaynes, Proc. IEEE 70, 939-952, (1982). This contains a lucid explication of Burg's method for applying the PME to time series analysis. (A)

48.Nonlinear Maximum Entropy Spectral Analysis Methods for Signal Recognition, C.H. Chen (Research Studies Press, Chichester, England, 1982). (A)

49.Nonlinear Methods of Spectral Analysis, edited by S. Haykin (Springer-Verlag, New York, 1983). (A)

For the past 15 years annual international workshops have been conducted on maximum-entropy methods, primarily but not exclusively in data analysis, and the proceedings volumes constitute an excellent source for these and numerous other applications.

50.Maximum Entropy and Bayesian Methods in Inverse Problems, edited by C.R. Smith and W.T. Grandy, Jr. (Reidel, Dordrecht, Holland, 1985). (A)

51.Maximum Entropy and Bayesian Methods in Applied Statistics, edited by J.H. Justice (Cambridge University Press, Cambridge, 1986). (A)

52.Maximum Entropy and Bayesian Spectral Analysis and Estimation, edited by C.R. Smith and G.J. Erickson (Reidel, Dordrecht, Holland, 1987). (A)

53.Maximum Entropy and Bayesian Methods in Science and Engineering, Volume 1: Foundations, edited by G.J. Erickson and C.R. Smith (Reidel, Dordrecht, Holland, 1988). (A)

54.Maximum Entropy and Bayesian Methods in Science and Engineering, Volume 2: Applications, edited by G.J. Erickson and C.R. Smith (Reidel, Dordrecht, Holland, 1988). (A)

55.Maximum Entropy and Bayesian Methods, Cambridge, England, 1988, edited by J. Skilling (Kluwer, Dordrecht, Holland, 1989). (A)

56. Maximum Entropy and Bayesian Methods, Dartmouth, U.S.A., 1989, edited by P.F. Fougere (Kluwer, Dordrecht, Holland, 1990). (A)

57.Maximum Entropy and Bayesian Methods, Laramie, Wyoming, 1990, edited by W.T. Grandy, Jr. and L.H. Schick (Kluwer, Dordrecht, Holland, 1991). (A)

58.Maximum Entropy and Bayesian Methods, Seattle, 1991, edited by C.R. Smith, G.J. Erickson and P.O. Neudorfer (Kluwer, Dordrecht, Holland, 1992). (A)

59.Maximum Entropy and Bayesian Methods, Paris, France, 1992, edited by Ali Mohammad-Djafari and G. Demoment (Kluwer, Dordrecht, Holland, 1993). (A)

60.Maximum Entropy and Bayesian Methods, Santa Barbara, California, 1993, edited by G.R. Heidbreder (Kluwer, Dordrecht, Holland, 1996). (A)

61.Maximum Entropy and Bayesian Methods, Cambridge, England, 1994, edited by J. Skilling and S. Sibisi (Kluwer, Dordrecht, Holland, 1996). (A)

In addition, an excellent tutorial volume in the application of maximum entropy methods is

62.Maximum Entropy in Action, edited by B. Buck and V.A. Macauley (Clarendon Press, Oxford, 1991.). (A)