:: wikimiki.org ::

Multivariate Normal Distribution

Multivariate normal distribution

In probability theory and statistics, a multivariate normal distribution, also sometimes called a multivariate Gaussian distribution, is a specific probability distribution, which can be thought of as a generalization to higher dimensions of the one-dimensional normal distribution (also called a Gaussian distribution).

General case

A random vector

X = [X_1, \cdots, X_N]

follows a multivariate normal distribution if it satisfies the following equivalent conditions:
- every linear combination

Y = a_1 X_1 + \cdots + a_N X_N

is normally distributed
- there is a random vector

Z = [Z_1, \cdots, Z_M]

, whose components are independent standard normal random variables, a vector

\mu = [\mu_1, \cdots, \mu_N]

and an

N \times M

matrix

A

such that

X = A Z + \mu

.
- there is a vector

\mu

and a symmetric, positive semi-definite matrix

\Gamma

such that the characteristic function of

X

is ::

\phi_X(u)
=
\exp
\left(
 i \mu^\top u - \frac u^\top \Gamma u
\right)
.

The following is not quite equivalent to the conditions above, since it fails to allow for a singular matrix as the variance:
- there is a vector

\mu = [\mu_1, \cdots, \mu_N]

and a symmetric, positive definite covariance matrix

\Sigma

(

N \times N

matrix) such that

X

has density :

f_X(x_1, \cdots, x_N)
=
\frac
 
 
\exp
\left(
 -\frac
 ( x - \mu)^\top \Sigma^ (x - \mu)
\right)

where

\left| \Sigma \right|

is the determinant of

\Sigma

. Note how the equation above reduces to that of the univariate normal distribution if

\Sigma

is a scalar (i.e., a real number). The vector

\mu

in these conditions is the expected value of

X

and the matrix

\Sigma = A A^T

is the covariance matrix of the components

X_i

. It is important to realize that the covariance matrix must be allowed to be singular. That case arises frequently in statistics; for example, in the distribution of the vector of residuals in ordinary linear regression problems. Note also that the

X_i

are in general not independent; they can be seen as the result of applying the linear transformation

A

to a collection of independent Gaussian variables

Z

. The multivariate normal can be written in the following notation: :

X \sim N(\mu, \Sigma)

or to make it explicitly known

X

is N-dimensional :

X \sim N_N(\mu, \Sigma)

Cumulative distribution function

The cumulative distribution function (cdf)

F(x)

is defined as the probability that all values in a random vector

X

are less than or equal to the corresponding values in vector

x

. Though there is no closed form for

F(x)

, there are a number of algorithms that estimate it numerically. For one such example, see [http://alex.strashny.org/a/Multivariate-normal-cumulative-distribution-function-(cdf)-in-MATLAB.html] (includes MATLAB code).

A counterexample

The fact that two random variables X and Y are normally distributed does not imply that the pair (X, Y) has a bivariate normal distribution. A simple example is one in which Y = X if |X| > 1 and Y = −X if |X| < 1. If X and Y are normally distributed and independent, then they are "jointly normally distributed", i.e., the pair (X, Y) does have a bivariate normal distribution. There are of course also many bivariate normal distributions in which the components are correlated.

Bivariate case

In the 2-dimensional nonsingular case, the probability density function (with mean (0,0)) is :

f(x,y)
=
\frac
\exp
\left(
 -\frac
 \left(
  \frac +
  \frac -
  \frac
 \right)
\right)

where

\rho

is the correlation between

X

and

Y

Linear transformation

Y = B X

is a linear transformation of

X

where

B

is an

m \times p

matrix then

Y

has a multivariate normal distribution with expected value

B \mu

and variance

B \Sigma B^T

(i.e.,

Y \sim N \left(B \mu, B \Sigma B^T\right)

. Corollary: any subset of the

X_i

has a marginal distribution that is also multivariate normal. To see this consider the following example: to extract the subset

(X_1, X_2, X_4)^T

, use :

B
=
\begin
 1 & 0 & 0 & 0 & 0 & \ldots & 0 \\
 0 & 1 & 0 & 0 & 0 & \ldots & 0 \\
 0 & 0 & 0 & 1 & 0 & \ldots & 0
\end

which extracts the desired elements directly.

Correlations and independence

In general, random variables may be uncorrelated but highly dependent. But if a random vector has a multivariate normal distribution then any two or more of its components that are uncorrelated are independent. This implies that any two or more of its components that are pairwise independent are independent. But it is not true that two random variables that are (separately, marginally) normally distributed and uncorrelated are independent. Two random variables that are normally distributed may fail to be jointly normally distributed, i.e., the vector whose components they are may fail to have a multivariate normal distribution. For an example of two normally distributed random variables that are uncorrelated but not independent, see normally distributed and uncorrelated does not imply independent.

Conditional distributions

Then if

\mu

and

\Sigma

are partitioned as follows :

\mu
=
\begin
 \mu_1 \\
 \mu_2
\end
\quad

with sizes

\begin q \times 1 \\ N-q \times 1 \end

\Sigma
=
\begin
 \Sigma_ & \Sigma_ \\
 \Sigma_ & \Sigma_
\end
\quad

with sizes

\begin q \times q & q \times N-q \\ N-q \times q & N-q \times N-q \end

then the distribution of

x_1

conditional on

x_2=a

is multivariate normal

X_1|X_2=a \sim N(\bar, \overline)

where :

\bar
=
\mu_1 + \Sigma_ \Sigma_^
\left(
 a - \mu_2
\right)

and covariance matrix :

\overline
=
\Sigma_ - \Sigma_ \Sigma_^ \Sigma_.

This matrix is the Schur complement of

. Note that knowing the value of

x_2

to be

a

alters the variance; perhaps more surprisingly, the mean is shifted by

\Sigma_ \Sigma_^ \left(a - \mu_2 \right)

; compare this with the situation of not knowing the value of

a

, in which case

x_1

would have distribution

N_q \left(\mu_1, \Sigma_ \right)

. The matrix

\Sigma_ \Sigma_^

is known as the matrix of regression coefficients.

Fisher information matrix

The Fisher information matrix (FIM) for a normal distribution takes a special formulation. The

(m,n)

element of the FIM for

X \sim N(\mu(\theta), \Sigma(\theta))

is :

\mathcal_
=
\frac
\Sigma^
\frac
+
\frac
\mathrm
\left(
 \Sigma^
 \frac
 \Sigma^
 \frac
\right)

where
-

\frac
=
\begin
 \frac &
 \frac &
 \cdots &
 \frac &
\end

\frac
=
\left(
 \frac
\right)^\top
=
\begin
 \frac \\  \\
 \frac \\  \\
 \vdots \\  \\
 \frac \\  \\
\end

\frac
=
\begin
 \frac &
 \frac &
 \cdots &
 \frac \\  \\
 \frac &
 \frac &
 \cdots &
 \frac \\  \\
 \vdots & \vdots & \ddots & \vdots \\  \\
 \frac &
 \frac &
 \cdots &
 \frac
\end

\mathrm

is the trace function

Kullback-Leibler divergence

The Kullback-Leibler divergence from

N0_N(\mu_0, \Sigma_0)

N1_N(\mu_1, \Sigma_1)

is: :

KL(N0, N1) =  ( \log (  ) + \mathrm ( \Sigma_0^ \Sigma_1 ) + ( \mu_1 - \mu_0)^\top \Sigma_0^ ( \mu_1 - \mu_0 ) - N)

Estimation of parameters

The derivation of the maximum-likelihood estimator of the covariance matrix of a multivariate normal distribution is perhaps surprisingly subtle and elegant. See estimation of covariance matrices. In short, the pdf is :

f(x)=(2 \pi)^ \det(\Sigma)^ \exp\left( (x-\mu)^T \Sigma^ (x-\mu)\right)

and the ML estimator of the covariance matrix is :

\Sigma = \sum_^n (X_i-\overline)(X_i-\overline)^T

which is simply the sample covariance matrix.

Generating values drawn from the distribution

To generate values from a multivariate normal distribution given μ and A such that X = AZ + μ as detailed above, simply generate a suitable vector of independent standard normal values Z using for example the Box-Muller transform, and apply the foregoing equation. In order to draw a random vector

X

from the

n

-dimensional multivariate normal distribution with mean vector

\mu

and covariance matrix

\Sigma

(required to be symmetric and positive definite), one proceeds as follows: First, compute the Cholesky decomposition (matrix square root) of

\Sigma

, that is, find the unique lower triangular matrix

A

such that

A\,A^T = \Sigma

. Second, let

Z=(z_1,\dots,z_n)

be a vector whose components are

n

independent standard normal variates. Then compute

X

\mu + A\,Z

. Category:Continuous distributions

Statistics

Statistics is a broad mathematical discipline which studies ways to collect, summarize and draw conclusions from data. It is applicable to a wide variety of academic disciplines from the physical and social sciences to the humanities, as well as to business, government, and industry. Once data is collected, either through a formal sampling procedure or by recording responses to treatments in an experimental setting (cf experimental design), or by repeatedly observing a process over time (time series), graphical and numerical summaries may be obtained using descriptive statistics. Patterns in the data are modeled to draw inferences about the larger population, using inferential statistics to account for randomness and uncertainty in the observations. These inferences may take the form of answers to essentially yes/no questions (hypothesis testing), estimates of numerical characteristics (estimation), prediction of future observations, descriptions of association (correlation), or modeling of relationships (regression). The framework described above is sometimes referred to as applied statistics. In contrast, mathematical statistics (or simply statistical theory) is the subdiscipline of applied mathematics which uses probability theory and analysis to place statistical practice on a firm theoretical basis. The word statistics is also the plural of statistic (singular), which refers to the result of applying a statistical algorithm to a set of data.

Origin

The word statistics ultimately derives from the modern Latin term statisticum collegium ("council of state") and the Italian word statista ("statesman" or "politician"). The German Statistik, first introduced by Gottfried Achenwall (1749), originally designated the analysis of data about the state. It acquired the meaning of the collection and classification of data generally in the early nineteenth century. It was introduced into English by Sir John Sinclair. Thus, the original principal purpose of statistics was data to be used by governmental and (often centralized) administrative bodies. The collection of data about states and localities continues, largely through national and international statistical services; in particular, censuses provide regular information about the population. Today, however, the use of statistics has broadened far beyond the service of a state or government, to include such areas as business, natural and social sciences, and medicine, among others.

Statistical methods

Experimental and observational studies

A common goal for a statistical research project is to investigate causality, and in particular to draw a conclusion on the effect of changes in the values of predictors or independent variables on a response or dependent variable. There are two major types of causal statistical studies, experimental studies and observational studies. In both types of studies, the effect of changes of an independent variable (or variables) on the behavior of the dependent variable are observed. The difference between the two types is in how the study is actually conducted. An experimental study involves taking measurements of the system under study, manipulating the system, and then taking additional measurements using the same procedure to determine if the manipulation may have modified the values of the measurements. In contrast, an observational study does not involve experimental manipulation. Instead data is gathered and correlations between predictors and the response are investigated. An example of an experimental study is the famous Hawthorne studies which attempted to test changes to the working environment at the Hawthorne plant of the Western Electric Company. The researchers were interested in whether increased illumination would increase the productivity of the assembly line workers. The researchers first measured productivity in the plant then modified the illumination in an area of the plant to see if changes in illumination would affect productivity. Due to errors in experimental procedures, specifically the lack of a control group, the researchers while unable to do what they planned were able to provide the world with the Hawthorne effect. An example of an observational study is a study which explores the correlation between smoking and lung cancer. This type of study typically uses a survey to collect observations about the area of interest and then perform statistical analysis. In this case, the researchers would collect observations of both smokers and non-smokers and then look at the number of cases of lung cancer in each group. The basic steps for an experiment are to: # plan the research including determining information sources, research subject selection, and ethical considerations for the proposed research and method, # design the experiment concentrating on the system model and the interaction of independent and dependent variables, # summarize a collection of observations to feature their commonality by suppressing details (descriptive statistics), # reach consensus about what the observations tell us about the world we observe (statistical inference), # document and present the results of the study.

Levels of measurement

There are four types of measurements or measurement scales used in statistics. The four types or levels of measurement (ordinal, nominal, interval, and ratio) have different degrees of usefulness in statistical research. Ratio measurements, where both a zero value and distances between different measurements are defined, provide the greatest flexibility in statistical methods that can be used for analysing the data. Interval measurements, with meaningful distances between measurements but no meaningful zero value (such as IQ measurements or temperature measurements in degrees Celsius). Ordinal measurements have imprecise differences between consecutive values but a meaningful order to those values. Nominal measurements have no meaningful rank order among values.

Statistical techniques

Some well known statistical tests and procedures for research observations are:
- Student's t-test
- chi-square
- analysis of variance (ANOVA)
- Mann-Whitney U
- regression analysis
- correlation
- Fischer's Least Significant Difference test
- Pearson product-moment correlation coefficient
- Spearman's rank correlation coefficient

Probability

The probability of an event is often defined as a number between one and zero. In reality however there is virtually nothing that has a probability of 1 or 0. You could say that the sun will certainly rise in the morning, but what if an extremely unlikely event destroys the sun? What if there is a nuclear war and the sky is covered in ash and smoke? We often round the probability of such things up or down because they are so likely or unlikely to occur, that it's easier to recognize them as a probability of one or zero. However, this can often lead to misunderstandings and dangerous behaviour, because people are unable to distinguish between, e.g., a probability of 10⁻⁴ and a probability of 10⁻⁹, despite the very practical difference between them. If you expect to cross the road about 10⁵ or 10⁶ times in your life, then reducing your risk of being run over per road crossing to 10⁻⁹ will make it unlikely that you will be run over while crossing the road for your whole life, while a risk per road crossing of 10⁻⁴ will make it very likely that you will have an accident, despite the intuitive feeling that 0.01% is a very small risk. Use of prior probabilities of 0 (or 1) causes problems in Bayesian statistics, since the posterior distribution is then forced to be 0 (or 1) as well. In other words, the data is not taken into account at all! As Lindley puts it, if a coherent Bayesian attaches a prior probability of zero to the hypothesis that the Moon is made of green cheese, then even whole armies of astronauts coming back bearing green cheese cannot convince him. Lindley advocates never using prior probabilities of 0 or 1. He calls it Cromwell's rule, from a letter Oliver Cromwell wrote to the synod of the Church of Scotland on August 5th, 1650 in which he said "I beseech you, in the bowels of Christ, consider it possible that you are mistaken."

Important contributors to statistics

- Carl Gauss
- Blaise Pascal
- Sir Francis Galton
- William Sealey Gosset (known as "Student")
- Karl Pearson
- Sir Ronald Fisher
- Gertrude Cox
- Charles Spearman
- Pafnuty Chebyshev
- Aleksandr Lyapunov
- Isaac Newton
- Abraham De Moivre
- Adolph Quetelet
- Florence Nightingale
- John Tukey
- George Dantzig See also list of statisticians.

Specialized disciplines

Some sciences use applied statistics so extensively that they have specialized terminology. These disciplines include:
- Biostatistics
- Business statistics
- Data mining (applying statistics and pattern recognition to discover knowledge from data)
- Economic statistics (Econometrics)
- Engineering statistics
- Statistical physics
- Demography
- Psychological statistics
- Social statistics (for all the social sciences)
- Statistical literacy
- Process analysis and chemometrics (for analysis of data from analytical chemistry and chemical engineering)
- Reliability engineering
- Statistics in various sports, particularly baseball and cricket Statistics form a key basis tool in business and manufacturing as well. It is used to understand measurement systems variability, control processes (as in statistical process control or SPC), for summarizing data, and to make data-driven decisions. In these roles it is a key tool, and perhaps the only reliable tool.

Software

Modern statistics is supported by computers to perform some of the very large and complex calculations required. Whole branches of statistics have been made possible by computing, for example neural networks. The computer revolution has implications for the future of statistics, with a new emphasis on 'experimental' and 'empirical' statistics. Statistical packages in common use include:

External links

- [http://www.hkshum.net/stats/ Clear explanation of the three Statistical Distributions studied throughout secondary school] great for younger students.

General sites and organizations

- [http://lib.stat.cmu.edu/ Statlib: Data, Software and News from the Statistics Community (Carnegie Mellon)]
- [http://www.cbs.nl/isi/ International Statistical Institute]
- [http://www.mathcs.carleton.edu/probweb/probweb.html The Probability Web]

Link collections

- [http://www.cbs.nl/isi/FreeTools.htm Free Statistical Tools on the WEB (at ISI)]
- [http://www.york.ac.uk/depts/maths/histstat Materials for the History of Statistics (Univ. of York)]
- [http://www.xycoon.com/ Statistics resources and calculators (Xycoon)]
- [http://members.aol.com/johnp71/javastat.html StatPages.net (statistical calculations, free software, etc.)]
- [http://www.nih.gov/sigs/bioethics/casestudies.html Bioethics Resources on the Web from the U.S. National Institute of Health (links to tutorials, case studies, and on-line courses)]

Online courses and textbooks

- [http://www.statsoft.com/textbook/stathome.html Electronic Statistics Textbook (StatSoft,Inc.)]
- [http://www.vias.org/tmdatanaleng/ Teach/Me Data Analysis (a Springer-Verlag book)]
- [http://www.richland.cc.il.us/james/lecture/m170/ Statistics: Lecture Notes (from a professor at Richland Community College)]
- [http://statistics.cyberk.com/splash/ CyberStats: Electronic Statistics Textbook (CyberGnostics, Inc)]
- [http://www.stat.ucla.edu/%7Edinov/courses_students.html A variety of class notes and educational materials on probability and statistics]

Statistical software

- [http://www.r-project.org/ R Project for Statistical Computing (free software)]
- [http://www.socr.ucla.edu/ Statistics Online Computational Resource (UCLA)]
- [http://root.cern.ch/ Root Analysis Framework (CERN)]
- [http://www.newmdsx.com/ Multidimensional Scaling Software]
- [http://www.rosuda.org/Software/ Software for interactive graphical analyses]
- [http://www.rank1st.com/website_monitoring/index.html Website Analytics and Monitoring]
- [http://www.csdassn.org/software_reports.cfm Software Reports] by Statistical Software Newsletter
- [http://chirouble.univ-lyon2.fr/~ricco/tanagra/ Tanagra (free software)], including machine learning and data mining techniques

Other resources

- [http://www.sixsigmafirst.com/anova.htm ANOVA]
- [http://www.math.uah.edu/stat/index.html Virtual Laboratories in Probability and Statistics (Univ. of Alabama)] (requires MathML and Java 2 Runtime Environment)
- [http://www.ericdigests.org/2000-2/resources.htm Resources for Teaching and Learning about Probability and Statistics (ERIC Digests)]
- [http://www.ericdigests.org/1993/marriage.htm Resampling: A Marriage of Computers and Statistics (ERIC Digests)]
- [http://www.execpc.com/~helberg/statframes.html Statistical Resources on the Web]
- [http://www.conceptstew.co.uk/PAGES/s4t_glossary_A.html Statistics glossary]
- [http://www.statistics.com/content/glossary/index.php3 Statistics Glossary at statistics.com]
- [http://jobs.strategy-blogs.com/Statisticians.html Statistician Job Outlook - Analysis of wages and working environment for the occupation]
- [http://www.amstat.org/sections/sis/ Statistics in Sports (Section of the ASA)]
- [http://meta.wikimedia.org/wiki/Statistics Statistics - Meta], statistics of Wikimedia projects

Additional references

Lindley, D. Making Decisions. John Wiley. Second Edition 1985. ISBN 0471908088 Category:Mathematical science occupations
- Category:Applied mathematics Category:Academic disciplines ms:Statistik ja:統計学 simple:Statistics th:สถิติศาสตร์ fiu-vro:Statistiga

Random vector

A multivariate random variable or random vector is a vector X = (X₁, ..., X_n) whose components are scalar-valued random variables on the same probability space (Ω, P). Every such random vector gives rise to a probability measure on Rⁿ with the Borel algebra as underlying sigma-algebra. This measure is also known as the joint distribution of the random vector. The distributions of each of the component random variables X_i are called marginal distributions. Category:Probability theory

Matrix (math)

:For the square matrix section, see square matrix. In mathematics, a matrix (plural matrices) is a rectangular table of numbers or, more generally, of elements of a ring-like algebraic structure. In this article, the entries of a matrix are real or complex numbers unless otherwise noted. Matrices are useful to record data that depend on two categories, and to keep track of the coefficients of systems of linear equations and linear transformations. For the development and applications of matrices, see matrix theory.

Definitions and notations

The horizontal lines in a matrix are called rows and the vertical lines are called columns. A matrix with m rows and n columns is called an m-by-n matrix (or m×n matrix) and m and n are called its dimensions. The entry of a matrix A that lies in the i -th row and the j-th column is called the i,j entry or (i,j)-th entry of A. This is written as A_i,j or A[i,j]. We often write

A:=(a_)_

to define an m × n matrix A with each entry in the matrix A[i,j] called a_ij for all 1 ≤ i ≤ m and 1 ≤ j ≤ n.

Example

The matrix :

\begin
1 & 2 & 3 \\
1 & 2 & 7 \\
4&9&2 \\
6&0&5\end

is a 4×3 matrix. The element A[2,3] or a_2,3 is 7.

Adding and multiplying matrices

Sum

If two m-by-n matrices A and B are given, we may define their sum A + B as the m-by-n matrix computed by adding corresponding elements, i.e., (A + B)[i, j] = A[i, j] + B[i, j]. For example :

\begin
    1 & 3 & 2 \\
    1 & 0 & 0 \\
    1 & 2 & 2
  \end
+
  \begin
    0 & 0 & 5 \\
    7 & 5 & 0 \\
    2 & 1 & 1
  \end
=
  \begin
    1+0 & 3+0 & 2+5 \\
    1+7 & 0+5 & 0+0 \\
    1+2 & 2+1 & 2+1
  \end
=
  \begin
    1 & 3 & 7 \\
    8 & 5 & 0 \\
    3 & 3 & 3
  \end

Another, much less often used notion of matrix addition is the direct sum.

Scalar multiplication

If a matrix A and a number c are given, we may define the scalar multiplication cA by (cA)[i, j] = cA[i, j]. For example :

2
  \begin
    1 & 8 & -3 \\
    4 & -2 & 5
  \end
=
  \begin
    2\times 1 & 2\times 8 & 2\times -3 \\
    2\times 4 & 2\times -2 & 2\times 5
  \end
=
  \begin
    2 & 16 & -6 \\
    8 & -4 & 10
  \end

These two operations turn the set M(m, n, R) of all m-by-n matrices with real entries into a real vector space of dimension mn.

Multiplication

Multiplication of two matrices is well-defined only if the number of columns of the first matrix is the same as the number of rows of the second matrix. If A is an m-by-n matrix (m rows, n columns) and B is an n-by-p matrix (n rows, p columns), then their product AB is the m-by-p matrix (m rows, p columns) given by :(AB)[i, j] = A[i, 1]
- B[1, j] + A[i, 2]
- B[2, j] + ... + A[i, n]
- B[n, j] for each pair i and j. It is easy to remember how to do this by imagining the matrix as a vector of vectors: :Let

A = 
  \begin
    a_1 & a_2 & a_3\\
    a_4 & a_5 & a_5 \\
  \end = 

  \begin
    V_1 \\
    V_2 \\
  \end

and

B = 
  \begin
    b_1 & b_2 \\
    b_3 & b_4 \\
    b_5 & b_6 \\
  \end = 

  \begin
    V_3 & V_4 \\
  \end

:Then

A \times B = 

  \begin
    V_1 \cdot V_3  &  V_1 \cdot V_4 \\
    V_2 \cdot V_3  &  V_2 \cdot V_4 \\
  \end

For instance: :

\begin
    1 & 0 & 2 \\
    -1 & 3 & 1 \\
  \end
\times
  \begin
    3 & 1 \\
    2 & 1 \\
    1 & 0
  \end
=
  \begin
     (1 \times 3  +  0 \times 2  +  2 \times 1) & (1 \times 1   +   0 \times 1   +   2 \times 0) \\
    (-1 \times 3  +  3 \times 2  +  1 \times 1) & (-1 \times 1   +   3 \times 1   +   1 \times 0) \\
  \end
=
  \begin
    5 & 1 \\
    4 & 2 \\
  \end

This multiplication has the following properties:
- (AB)C = A(BC) for all k-by-m matrices A, m-by-n matrices B and n-by-p matrices C ("associativity").
- (A + B)C = AC + BC for all m-by-n matrices A and B and n-by-k matrices C ("right distributivity").
- C(A + B) = CA + CB for all m-by-n matrices A and B and k-by-m matrices C ("left distributivity"). It is important to note that commutativity does not generally hold; that is, given matrices A and B and their product defined, then generally AB ≠ BA. Matrices are said to anticommute if AB = -BA. Such matrices are very important in representations of Lie algebras and in Representations of Clifford algebras

Linear transformations, ranks and transpose

Matrices can conveniently represent linear transformations because matrix multiplication neatly corresponds to the composition of maps, as will be described next. This same property makes them powerful data structures in high-level programming languages. Here and in the sequel we identify Rⁿ with the set of "rows" or n-by-1 matrices. For every linear map f : Rⁿ -> R^m there exists a unique m-by-n matrix A such that f(x) = Ax for all x in Rⁿ. We say that the matrix A "represents" the linear map f. Now if the k-by-m matrix B represents another linear map g : R^m -> R^k, then the linear map g o f is represented by BA. This follows from the above-mentioned associativity of matrix multiplication. More generally, a linear map from an n-dimensional vector space to an m-dimensional vector space is represented by an m-by-n matrix, provided that bases have been chosen for each. The rank of a matrix A is the dimension of the image of the linear map represented by A; this is the same as the dimension of the space generated by the rows of A, and also the same as the dimension of the space generated by the columns of A. The transpose of an m-by-n matrix A is the n-by-m matrix A^tr (also sometimes written as A^T or ^tA) formed by turning rows into columns and columns into rows, i.e. A^tr[i, j] = A[j, i] for all indices i and j. If A describes a linear map with respect to two bases, then the matrix A^tr describes the transpose of the linear map with respect to the dual bases, see dual space. We have (A + B)^tr = A^tr + B^tr and (AB)^tr = B^tr
- A^tr.

Square matrices and related definitions

A square matrix is a matrix which has the same number of rows as columns. The set of all square n-by-n matrices, together with matrix addition and matrix multiplication is a ring. Unless n = 1, this ring is not commutative. M(n, R), the ring of real square matrices, is a real unitary associative algebra. M(n, C), the ring of complex square matrices, is a complex associative algebra. The unit matrix or identity matrix I_n, with elements on the main diagonal set to 1 and all other elements set to 0, satisfies MI_n=M and I_nN=N for any m-by-n matrix M and n-by-k matrix N. For example, if n = 3: :

I_3 =
  \begin
    1 & 0 & 0 \\
    0 & 1 & 0 \\
    0 & 0 & 1
  \end

The identity matrix is the identity element in the ring of square matrices. Invertible elements in this ring are called invertible matrices or non-singular matrices. An n by n matrix A is invertible if and only if there exists a matrix B such that :AB = I_n ( = BA). In this case, B is the inverse matrix of A, denoted by A⁻¹. The set of all invertible n-by-n matrices forms a group (specifically a Lie group) under matrix multiplication, the general linear group. If λ is a number and v is a non-zero vector such that Av = λv, then we call v an eigenvector of A and λ the associated eigenvalue. (Eigen means "own" in German.) The number λ is an eigenvalue of A if and only if A−λI_n is not invertible, which happens if and only if p_A(λ) = 0. Here p_A(x) is the characteristic polynomial of A. This is a polynomial of degree n and has therefore n complex roots (counting multiple roots according to their multiplicity). In this sense, every square matrix has n complex eigenvalues. The determinant of a square matrix A is the product of its n eigenvalues, but it can also be defined by the Leibniz formula. Invertible matrices are precisely those matrices with nonzero determinant. The Gauss-Jordan elimination algorithm is of central importance: it can be used to compute determinants, ranks and inverses of matrices and to solve systems of linear equations. The trace of a square matrix is the sum of its diagonal entries, which equals the sum of its n eigenvalues. Matrix exponential is defined for square matrices, using power series.

Special types of matrices

In many areas in mathematics, matrices with certain structure arise. A few important examples are
- Symmetric matrices are such that elements symmetric to the main diagonal (from the upper left to the lower right) are equal, that is, a_i,j=a_j,i.
- Skew-symmetric matrices are such that elements symmetric to the main diagonal are the negative of each other, that is, a_i,j= - a_j,i. In a skew-symmetric matrix, all diagonal elements are zero, that is, a_i,i=0.
- Hermitian (or self-adjoint) matrices are such that elements symmetric to the diagonal are each others complex conjugates, that is, a_i,j=a^-_j,i, where the superscript '
- ' signifies complex conjugation.
- Toeplitz matrices have common elements on their diagonals, that is, a_i,j=a_i+1,j+1.
- Stochastic matrices are square matrices whose columns are probability vectors; they are used to define Markov chains. For a more extensive list see list of matrices.

Matrices in abstract algebra

If we start with a ring R, we can consider the set M(m,n, R) of all m by n matrices with entries in R. Addition and multiplication of these matrices can be defined as in the case of real or complex matrices (see above). The set M(n, R) of all square n by n matrices over R is a ring in its own right, isomorphic to the endomorphism ring of the left R-module Rⁿ. Similarly, if the entries are taken from a semiring S, matrix addition and multiplication can still be defined as usual. The set of all square n×n matrices over S is itself a semiring. Note that fast matrix multiplication algorithms such as the Strassen algorithm generally only apply to matrices over rings and will not work for matrices over semirings that are not rings. If R is a commutative ring, then M(n, R) is a unitary associative algebra over R. It is then also meaningful to define the determinant of square matrices using the Leibniz formula; a matrix is invertible if and only if its determinant is invertible in R. All statements mentioned in this articles for real or complex matrices remain correct for matrices over an arbitrary field. Matrices over a polynomial ring are important in the study of control theory.

History

The study of matrices is quite old. Latin squares and magic squares have been studied since prehistoric times. Matrices have a long history of application in solving linear equations. Leibniz, one of the two founders of calculus, developed the theory of determinants in 1693. Cramer developed the theory further, presenting Cramer's rule in 1750. Carl Friedrich Gauss and Wilhelm Jordan developed Gauss-Jordan elimination in the 1800s. The term "matrix" was first coined in 1848 by J. J. Sylvester. Cayley, Hamilton, Grassmann, Frobenius and von Neumann are among the famous mathematicians who have worked on matrix theory. Olga Taussky Todd (1906-1995) started to use matrix theory when investigating an aerodynamic phenomenon called flutter, during WWII.

External links

- [http://www.easycalculation.com/matrix/index.php Matrix Calculators: dynamic online calculators]
- [http://www.ualr.edu/~lasmoller/matrices.html Matrix name and history: very brief overview]
- [http://wims.unice.fr/wims/wims.cgi?module=tool/linear/matrix.en WIMS Matrix Calculator] computes determinant, rank, inverse etc. online.
- [http://www.xycoon.com/matrix_algebra.htm Introduction to Matrix Algebra: definitions and properties]
- [http://digilander.libero.it/foxes/ Excel add-ins for Matrix Algebra and Extended Precision functions] These are freeware, open source. Category:Abstract algebra Category:Linear algebra ko:행렬 ja:行列 simple:Matrix (mathematics) th:เมทริกซ์ (คณิตศาสตร์)

Positive-definite

In mathematics, a definite bilinear form B is one for which :B(v, v) has a fixed sign (positive or negative) when it is not 0. To give a formal definition, let K be one of the fields R (real numbers) or C (complex numbers). Suppose that V is a vector space over K, and :B : V × V → K is a bilinear map which is Hermitian in the sense that B(x, y) is always the complex conjugate of B(y, x). Then B is positive-definite if :B(x, x) > 0 for every nonzero x in V. If it is greater than or equal to zero, we say B is positive semidefinite. Similarly for negative definite and negative semidefinite. If it is otherwise unconstrained, we say B is indefinite. A self-adjoint operator A on an inner product space is positive-definite if :(x, Ax) > 0 for every nonzero vector x. See in particular positive-definite matrix.

Covariance matrix

In statistics and probability theory, the covariance matrix is a matrix of covariances between elements of a vector. It is the natural generalization to higher dimensions, of the concept of the variance of a scalar-valued random variable. If

X

is a column vector with

n

scalar random variable components, and

\mu_k

is the expected value of the k^th element of

X

, i.e.,

\mu_k = \mathrm(X_k)

, then the covariance matrix is defined as: :

\Sigma=\mathrm
\left[
 \left(
  \textbf - \mathrm[\textbf]
 \right)
 \left(
  \textbf - \mathrm[\textbf]
 \right)^\top
\right]

=
\begin
 \mathrm[(X_1 - \mu_1)(X_1 - \mu_1)] & \mathrm[(X_1 - \mu_1)(X_2 - \mu_2)] & \cdots & \mathrm[(X_1 - \mu_1)(X_n - \mu_n)] \\  \\
 \mathrm[(X_2 - \mu_2)(X_1 - \mu_1)] & \mathrm[(X_2 - \mu_2)(X_2 - \mu_2)] & \cdots & \mathrm[(X_2 - \mu_2)(X_n - \mu_n)] \\  \\
 \vdots & \vdots & \ddots & \vdots \\  \\
 \mathrm[(X_n - \mu_n)(X_1 - \mu_1)] & \mathrm[(X_n - \mu_n)(X_2 - \mu_2)] & \cdots & \mathrm[(X_n - \mu_n)(X_n - \mu_n)]
\end

The

(i,j)

element is the covariance between

X_i

and

X_j

. This concept generalizes to higher dimensions the concept of variance of a scalar-valued random variable

X

, defined as :

\sigma^2 = \mathrm(X)
= \mathrm[(X-\mu)^2] \,

where

\mu = \mathrm(X)

Conflicting nomenclatures and notations

Nomenclatures differ. Some statisticians, following the probabilist William Feller, call this matrix the variance of the random vector

X

, because it is the natural generalization to higher dimensions of the 1-dimensional variance. Others call it the covariance matrix, because it is the matrix of covariances between the scalar components of the vector

X

. Unfortunately, several different conventions jar to some degree with each other: Standard notation: :

\operatorname(\textbf)
=
\mathrm
\left[
 (\textbf - \mathrm [\textbf])
 (\textbf - \mathrm [\textbf])^\top
\right]

Also standard notation (unfortunately conflicting with the above): :

\operatorname(\textbf)
=
\mathrm
\left[
 (\textbf - \mathrm[\textbf])
 (\textbf - \mathrm[\textbf])^\top
\right]

Also standard notation: :

\operatorname(\textbf,\textbf)
=
\mathrm
\left[
 (\textbf - \mathrm[\textbf])
 (\textbf - \mathrm[\textbf])^\top
\right]

(the "cross-covariance" between two random vectors) The first two of these usages conflict with each other. The first and third are in perfect harmony. The first notation is found in William Feller's universally admired two-volume book on probability.

Properties

For

\Sigma=\mathrm \left[  \left(   \textbf - \mathrm[\textbf]  \right)  \left( \textbf - \mathrm[\textbf] \right)^\top \right]

and

\mu = \mathrm(\textbf)

the following basic properties apply: #

\Sigma = \mathrm(\mathbf) - \mathbf\mathbf

\operatorname(\mathbf\mathbf) = \mathbf \operatorname(\mathbf) \mathbf

\mathbf \geq 0

\operatorname(\mathbf + \mathbf) = \mathbf \operatorname(\mathbf) \mathbf

\operatorname(\mathbf,\mathbf) = \operatorname(\mathbf,\mathbf)^\top

\operatorname(\mathbf + \mathbf,\mathbf) = \operatorname(\mathbf,\mathbf) + \operatorname(\mathbf, \mathbf)

# If

p = q

\operatorname(\mathbf + \mathbf) = \operatorname(\mathbf) + \operatorname(\mathbf,\mathbf) + \operatorname(\mathbf, \mathbf) + \operatorname(\mathbf)

\operatorname(\mathbf, \mathbf) = \mathbf \operatorname(\mathbf, \mathbf) \mathbf^\top

# If

\mathbf

and

\mathbf

are independent, then

\operatorname(\mathbf, \mathbf) = 0

where

\mathbf, \mathbf

and

\mathbf

are a random

\mathbf

vectors,

\mathbf

is a random

\mathbf

vector,

\mathbf

\mathbf

vector,

\mathbf

and

\mathbf

are

\mathbf

matrices. This covariance matrix (though very simple) is a very useful tool in many very different areas. From it a transformation matrix can be derived that allows one to completely decorrelate the data or, from a different point of view, to find an optimal basis for representing the data in a compact way. This is called principal components analysis (PCA) in statistics and Karhunen-Loève transform (KL-transform) in image processing.

Complex random vectors

The variance of a complex scalar-valued random variable with expected value μ is conventionally defined using complex conjugation: :

\operatorname(z)
=
\operatorname
\left[
 (z-\mu)(z-\mu)^
\right]

where the complex conjugate of a complex number

z

is denoted

z^

. If

Z

is a column-vector of complex-valued random variables, then we take the conjugate transpose by both transposing and conjugating, getting a square matrix: :

\operatorname
\left[
 (Z-\mu)(Z-\mu)^
\right]

where

Z^

denotes the conjugate transpose, which is applicable to the scalar case since the transpose of a scalar is still a scalar.

Estimation

The derivation of the maximum-likelihood estimator of the covariance matrix of a multivariate normal distribution is perhaps surprisingly subtle. It involves the spectral theorem and the reason why it can be better to view a scalar as the trace of a

1 \times 1

matrix than as a mere scalar. See estimation of covariance matrices.

External references

- [http://mathworld.wolfram.com/CovarianceMatrix.html Covariance Matrix] at Mathworld Category:Statistics

Determinant

In linear algebra, a determinant is a function depending on n that associates a scalar det(A) to every n×n square matrix A. The fundamental geometric meaning of a determinant is as the scale factor for volume when A is regarded as a linear transformation. Determinants are important both in calculus, where they enter the substitution rule for several variables, and in multilinear algebra. For a fixed positive integer n, there is a unique determinant function for the n×n matrices over any commutative ring R. R is the field of real or complex numbers. A determinant of A is also sometimes denoted by |A|, but this notation is ambiguous: it is also used to for certain matrix norms, and for the square root of

^ -

Determinants of 2-by-2 matrices

The 2×2 matrix :

A=\begina&b\\
c&d\end

has determinant :

\det(A)=ad-bc \,

. The interpretation when the matrix has real number entries is that this gives the area of the parallelogram with vertices at (0,0), (a,c), (b,d), and (a + b, c + d), with a sign factor (which is −1 if A as a transformation matrix flips the unit square over). A formula for larger matrices will be given below .

Applications

Determinants are used to characterize invertible matrices (namely as those matrices, and only those matrices, with non-zero determinants), and to explicitly describe the solution to a system of linear equations with Cramer's rule. They can be used to find the eigenvalues of the matrix

A

through the characteristic polynomial :

p(x) = \det(xI - A) \,

where I is the identity matrix of the same format as A. One often thinks of the determinant as assigning a number to every sequence of

n

vectors in

\Bbb^n

, by using the square matrix whose columns are the given vectors. With this understanding, the sign of the determinant of a basis can be used to define the notion of orientation in Euclidean spaces. The determinant of a set of vectors is positive if the vectors form a right-handed coordinate system, and negative if left-handed. Determinants are used to calculate volumes in vector calculus: the absolute value of the determinant of real vectors is equal to the volume of the parallelepiped spanned by those vectors. As a consequence, if the linear map

f: \Bbb^n \rightarrow \Bbb^n

is represented by the matrix

A

, and

S

is any measurable subset of

\Bbb^n

, then the volume of

f(S)

is given by

\left| \det(A) \right| \times \operatorname(S)

. More generally, if the linear map

f: \Bbb^n \rightarrow \Bbb^m

is represented by the

m

-by-

n

matrix

A

, and

S

is any measurable subset of

\Bbb^

, then the

n

-dimensional volume of

f(S)

is given by

\sqrt \times \operatorname(S)

. By calculating the volume of the tetrahedron bounded by four points, they can be used to identify skew lines. The volume of any tetrahedron, given its vertices a, b, c, and d, is (1/6)·|det(a−b, b−c, c−d)|, or any other combination of pairs of vertices that form a simply connected graph.

General definition and computation

Suppose

A = (A_) \,

is a square matrix. If

A

is a 1-by-1 matrix, then

\det(A) = A_ \,

A

is a 2-by-2 matrix, then

\det(A) = A_A_ - A_A_ \,

For a 3-by-3 matrix

A

, the formula is more complicated: :

\begin
\det(A) & = & A_A_A_ + A_A_A_ + A_A_A_\\
& & - A_A_A_ - A_A_A_ - A_A_A_.
\end\,

For a general

n

-by-

n

matrix, the determinant was defined by Gottfried Leibniz with what is now known as the Leibniz formula: :

\det(A) = \sum_ 
\sgn(\sigma) \prod_^n A_

The sum is computed over all permutations

\sigma

of the numbers and

\sgn(\sigma)

denotes the signature of the permutation

\sigma

: +1 if

\sigma

is an even permutation and −1 if it is odd (see even and odd permutations). This formula contains

n!

(factorial) summands and is therefore impractical to use it to calculate determinants for large

n

. In general, determinants can be computed with the Gauss algorithm using the following rules:
- If

A

is a triangular matrix, i.e.

A_ = 0 \,

whenever

i > j

, then

\det(A) =  A_ A_ \cdots A_ \,

- If

B

results from

A

by exchanging two rows or columns, then

\det(B) = -\det(A) \,

- If

B

results from

A

by multiplying one row or column with the number

c

, then

\det(B) = c\,\det(A) \,

- If

B

results from

A

by adding a multiple of one row to another row, or a multiple of one column to another column, then

\det(B) = \det(A) \,

Explicitly, starting out with some matrix, use the last three rules to convert it into a triangular matrix, then use the first rule to compute its determinant. It is also possible to expand a determinant along a row or column using Laplace's formula, which is efficient for relatively small matrices. To do this along row

i

, say, we write :

\det(A) = \sum_^n A_C_ = \sum_^n A_ (-1)^ M_

where the

C_

represent the matrix cofactors, i.e.

C_

(-1)^

times the minor

M_

, which is the determinant of the matrix that results from

A

by removing the

i

-th row and the

j

-th column.

Example

Suppose we want to compute the determinant of :

A = \begin-2&2&-3\\
-1& 1& 3\\
2 &0 &-1\end

We can go ahead and use the Leibniz formula directly: : Alternatively, we can use Laplace's formula to expand the determinant along a row or column. It is best to choose a row or column with many zeros, so we will expand along the second column: : A third way (and the method of choice for larger matrices) would involve the Gauss algorithm. When doing computations by hand, one can often shorten things dramatically by smartly adding multiples of columns or rows to other columns or rows; this doesn't change the value of the determinant, but may create zero entries which simplifies the subsequent calculations. In our example, adding the second column to the first one is especially useful: :

\begin0&2&-3\\
0 &1 &3\\
2 &0 &-1\end

and this determinant can be quickly expanded along the first column: :

Properties

The determinant is a multiplicative map in the sense that :

\det(AB) = \det(A)\det(B) \,

for all n-by-n matrices

A

and

B

. This is generalized by the Cauchy-Binet formula to products of non-square matrices. It is easy to see that

\det(rI_n) = r^n \,

and thus :

\det(rA) = \det(rI_n \cdot A) = r^n \det(A) \,

for all

n

-by-

n

matrices

A

and all scalars

r

. The matrix

A

(over the real or complex numbers, or some other field) is invertible if and only if det(A)≠0; in this case we have :

\det(A^) = \det(A)^ \,

Expressed differently: the vectors v₁,...,v_n in Rⁿ form a basis if and only if det(v₁,...,v_n) is non-zero. A real matrix and its transpose have the same determinant: :

\det(A^\top) = \det(A) \,

. The determinants of a complex matrix and of its conjugate transpose are conjugate: :

\det(A^ - ) = \det(A)^ -  \,

. (Note the conjugate transpose is identical to the transpose for a real matrix) If

A

and

B

are similar, i.e., if there exists an invertible matrix

X

such that

A

X^ B X

, then by the multiplicative property, :

\det(A) = \det(B) \,

This means that the determinant is a similarity invariant. Because of this, the determinant of some linear transformation T : V → V for some finite dimensional vector space V is independent of the basis for V. The relationship is one-way, however: there exist matrices which have the same determinant but are not similar. If

A

is a square

n

-by-

n

matrix with real or complex entries and if λ₁,...,λ_n are the (complex) eigenvalues of

A

listed according to their algebraic multiplicities, then :

\det(A) = \lambda_\lambda_ \cdots \lambda_

This follows from the fact that

A

is always similar to its Jordan normal form, an upper triangular matrix with the eigenvalues on the main diagonal. From this connection between the determinant and the eigenvalues, one can derive a connection between the trace function, the exponential function, and the determinant: :

\det(\exp(A)) = \exp(\operatorname(A))

. Performing the substitution

A \mapsto \ln A

in the above equation yields :

\det(A) = e^. \

Derivative

The determinant of real square matrices is a polynomial function from

\Bbb^

\Bbb

, and as such is everywhere differentiable. Its derivative can be expressed using Jacobi's formula: :

d \,\det(A) = \operatorname(\operatorname(A) \,dA)

where adj(A) denotes the adjugate of A. In particular, if A is invertible, we have :

d \,\det(A) = \det(A) \,\operatorname(A^ \,dA)

or, more colloquially, :

\det(A + X) - \det(A) \approx \det(A) \,\operatorname(A^ X)

if the entries in the matrix

X

are sufficiently small. The special case where

A

is equal to the identity matrix

I

yields :

\det(I + X) \approx 1 + \operatorname(X)

Generalizations and related functions

As was pointed out above, it is possible to unambiguously define the determinant of any linear map f : V → V, if V is a finite-dimensional vector space. It makes sense to define the determinant for matrices whose entries come from any commutative ring. The computation rules, the Leibniz formula and the compatibility with matrix multiplication remain valid, except that now a matrix

A

is invertible if and only if

\det(A)

is an invertible element of the ground ring. Abstractly, one may define the determinant as a certain anti-symmetric multilinear map as follows: if

R

is a commutative ring and

M = R^n

denotes the free R-module with

n

generators, then :

\det: M^n \rightarrow R

is the unique map with the following properties:
- det is

R

-linear in each of the

n

arguments.
- det is anti-symmetric, meaning that if two of the

n

arguments are equal, then the determinant is zero.
-

\det(e_1,\ldots,e_n) = 1

, where

e_i

is that element of

M

which has a 1 in the

i

-th coordinate and zeros elsewhere. Linear algebraists prefer to use the multilinear map approach to define determinant, whereas combinatorialists may prefer the Leibniz formula. (Of course, even when using the above abstract approach, one has to use the Leibniz formula to show that such a multilinear map actually exists.) The Pfaffian is an analog of the determinant for

2n\times 2n

antisymmetric matrices. It is a polynomial of degree

n

, and its square is equal to the determinant of the matrix. There is no direct generalisation of determinants, or of the notion of volume, to spaces of infinite dimension. There are various approaches possible, including the use of the extension of the trace of a matrix, and functional determinants.

History

Historically, determinants were considered before matrices. Originally, a determinant was defined as a property of a system of linear equations. The determinant "determines" whether the system has a unique solution (which occurs precisely if the determinant is non-zero). In this sense, two-by-two determinants were considered by Cardano at the end of the 16th century and larger ones by Leibniz about 100 years later. Following him Cramer (1750) added to the theory, treating the subject in relation to sets of equations. The recurrent law was first announced by Bezout (1764). It was Vandermonde (1771) who first recognized determinants as independent functions. Laplace (1772) gave the general method of expanding a determinant in terms of its complementary minors: Vandermonde had already given a special case. Immediately following, Lagrange (1773) treated determinants of the second and third order. Lagrange was the first to apply determinants to questions outside elimination theory; he proved many special cases of general identities. Gauss (1801) made the next advance. Like Lagrange, he made much use of determinants in the theory of numbers. He introduced the word determinants (Laplace had used resultant), though not in the present signification, but rather as applied to the discriminant of a quantic. Gauss also arrived at the notion of reciprocal (inverse) determinants, and came very near the multiplication theorem. The next contributor of importance is Binet (1811, 1812), who formally stated the theorem relating to the product of two matrices of

m

columns and

n

rows, which for the special case of

m = n

reduces to the multiplication theorem. On the same day (Nov. 30, 1812) that Binet presented his paper to the Academy, Cauchy also presented one on the subject. (See Cauchy-Binet formula.) In this he used the word determinant in its present sense, summarized and simplified what was then known on the subject, improved the notation, and gave the multiplication theorem with a proof more satisfactory than Binet's. With him begins the theory in its generality. The next important figure was Jacobi (from 1827). He early used the functional determinant which Sylvester later called the Jacobian, and in his memoirs in Crelle for 1841 he specially treats this subject, as well as the class of alternating functions which Sylvester has called alternants. About the time of Jacobi's last memoirs, Sylvester (1839) and Cayley began their work. The study of special forms of determinants has been the natural result of the completion of the general theory. Axisymmetric determinants have been studied by Lebesgue, Hesse, and Sylvester; persymmetric determinants by Sylvester and Hankel; circulants by Catalan, Spottiswoode, Glaisher, and Scott; skew determinants and Pfaffians, in connection with the theory of orthogonal transformation, by Cayley; continuants by Sylvester; Wronskians (so called by Muir) by Christoffel and Frobenius; compound determinants by Sylvester, Reiss, and Picquet; Jacobians and Hessians by Sylvester; and symmetric gauche determinants by Trudi. Of the text-books on the subject Spottiswoode's was the first. In America, Hanus (1886) and Weld (1893) published treatises. Category:Matrix theory Category:Linear algebra Category:Abstract algebra Category:Algebra ko:행렬식 ja:行列式

Expected value

In probability theory (and especially gambling), the expected value (or mathematical expectation) of a random variable is the sum of the probability of each possible outcome of the experiment multiplied by its payoff ("value"). Thus, it represents the average amount one "expects" to win per bet if bets with identical odds are repeated many times. Note that the value itself may not be expected in the general sense; it may be unlikely or even impossible. For example, an American roulette wheel has 38 equally possible outcomes. A bet placed on a single number pays 35-to-1 (this means that you are paid 35 times your bet and your bet is returned, so you get 36 times your bet). So the expected value of the profit resulting from a $1 bet on a single number is, considering all 38 possible outcomes: :

\left( -$1 \times \frac \right) + \left( $35 \times \frac \right),

which is about -$0.0526. Therefore one expects, on average, to lose over five cents for every dollar bet.

Mathematical definition

In general, if

X

is a random variable defined on a probability space

(\Omega, P)

, then the expected value of

X

(denoted

\mathrm(X)

or sometimes

\langle X \rangle

\mathbbX

) is defined as :

\mathrm(X) = \int_\Omega X\, dP

where the Lebesgue integral is employed. Note that not all random variables have an expected value, since the integral may not exist (e.g., Cauchy distribution). Two variables with the same probability distribution will have the same expected value, if it is defined. If

X

is a discrete random variable with values

x_1

x_2

, ... and corresponding probabilities

p_1

p_2

, ... which add up to 1, then

\mathrm(X)

can be computed as the sum or series :

\mathrm(X) = \sum_i p_i x_i\,

as in the gambling example mentioned above. If the probability distribution of

X

admits a probability density function

f(x)

, then the expected value can be computed as :

\mathrm(X) = \int_^\infty x f(x)\, \mathrm d x.

It follows directly from the discrete case definition that if

X

is a constant random variable, i.e.

X = b

for some fixed real number

b

, then the expected value of

X

is also

b

. The expected value of an arbitrary function of x, g(x), with respect to the probability density function f(x) is given by :

\mathrm(g(X)) = \int_^\infty g(x) f(x)\, \mathrm d x.

Properties

Linearity

The expected value operator (or expectation operator)

\mathrm

is linear in the sense that :

\mathrm(a X + b Y) = a \mathrm(X) + b \mathrm(Y)\,

for any two random variables

X

and

Y

(which need to be defined on the same probability space) and any real numbers

a

and

b

Functional non-invariance

In general, the expectation operator and functions of random variables do not commute; that is :

\mathrm(g(X)) = \int_ g(X)\, \mathrm d P \neq g(\operatornameX),

except as noted above.

Non-multiplicativity

In general, the expected value operator is not multiplicative, i.e.

\mathrm(X Y)

is not necessarily equal to

\mathrm(X) \mathrm(Y)

, except if

X

and

Y

are independent or uncorrelated. This lack of multiplicativity gives rise to study of covariance and correlation.

Iterated expectation

For any two random variables

X,Y

one may define the conditional expectation: :

\mathrm[X|Y](y) = \mathrm[X|Y=y] = \sum_x x \cdot \mathrm(X=x|Y=y).

Then the expectation of

X

satisfies :

\begin 
\mathrm \left( \mathrm[X|Y] \right) & = & \sum_y \mathrm[X|Y=y] \cdot \mathrm(Y=y) \\
                                          & = & \sum_y \left( \sum_x x \cdot \mathrm(X=x|Y=y) \right) \cdot \mathrm(Y=y) \\ 
                                          & = & \sum_y \sum_x x \cdot \mathrm(X=x|Y=y) \cdot \mathrm(Y=y) \\
                                          & = & \sum_y \sum_x x \cdot \mathrm(Y=y|X=x) \cdot \mathrm(X=x) \\
                                          & = & \sum_x x \cdot \mathrm(X=x) \cdot \left( \sum_y \mathrm(Y=y|X=x) \right) \\
                                          & = & \sum_x x \cdot \mathrm(X=x) \\
                                          & = &  \mathrm[X]. \end

Hence, the following equations holds: :

\mathrm[X] = \mathrm \left( \mathrm[X|Y] \right).

The right hand side of this equation is referred to as the iterated expectation. This proposition is treated in law of total expectation.

Inequality

If a random variable X is always less than or equal to another random variable Y, the expectation of X is less than or equal to that of Y: If

X \leq Y

, then

\mathrm[X] \leq \mathrm[Y]

. In particular, since

X \leq |X|

and

-X \leq |X|

, the absolute value of expectation of a random variable is less or equal to the expectation of its absolute value: :

|\mathrm[X]| \leq \mathrm[|X|]

Representation

It is easily seen that the following formula holds for any nonnegative real--valued random variable

X

(such that

\mathrm[X] < \infty

), and positive real number

\alpha

: :

\mathrm[X^\alpha] = \alpha \int_^ t^\mathrm(X>t) \mathrm d t.

Uses and applications of the expected value

The expected values of the powers of

X

are called the moments of

X

; the moments about the mean of

X

are expected values of powers of

X - \mathrm(X)

. The moments of some random variables can be used to specify their distributions, via their moment generating functions. To empirically estimate the expected value of a random variable, one repeatedly measures observations of the variable and computes the arithmetic mean of the results. This estimates the true expected value in an unbiased manner and has the property of minimizing the sum of the squares of the residuals (the sum of the squared differences between the observations and the estimate). The law of large numbers demonstrates that (under fairly mild conditions) as the size of the sample gets larger, the variance of this estimate gets smaller. In classical mechanics, the center of mass is an analogous concept to expectation. For example, suppose

X

is a discrete random variable with values

x_i

and corresponding probabilities

p_i

. Now consider a weightless rod on which are placed weights, at locations

x_i

along the rod and having masses

p_i

(whose sum is one). The point at which the rod balances (its center of gravity) is

\mathrm(X)

. (Note however, that the center of mass is not the same as the center of gravity.)

Expectation of matrices

X

is an

m \times n

matrix, then the expected value of the matrix is a matrix of expected values: :

\mathrm[X]
=
\mathrm
\begin
 x_ & x_ & \cdots & x_ \\
 x_ & x_ & \cdots & x_ \\
 \vdots \\
 x_ & x_ & \cdots & x_
\end
=
\begin
 \mathrm(x_) & \mathrm(x_) & \cdots & \mathrm(x_) \\
 \mathrm(x_) & \mathrm(x_) & \cdots & \mathrm(x_) \\
 \vdots \\
 \mathrm(x_) & \mathrm(x_) & \cdots & \mathrm(x_)
\end

This property is utilized in covariance matrices.

External links

- Category:Probability theory ja:期待値

Covariance matrix

X

is a column vector with

n

scalar random variable components, and

\mu_k

is the expected value of the k^th element of

X

, i.e.,

\mu_k = \mathrm(X_k)

, then the covariance matrix is defined as: :

\Sigma=\mathrm
\left[
 \left(
  \textbf - \mathrm[\textbf]
 \right)
 \left(
  \textbf - \mathrm[\textbf]
 \right)^\top
\right]

=
\begin
 \mathrm[(X_1 - \mu_1)(X_1 - \mu_1)] & \mathrm[(X_1 - \mu_1)(X_2 - \mu_2)] & \cdots & \mathrm[(X_1 - \mu_1)(X_n - \mu_n)] \\  \\
 \mathrm[(X_2 - \mu_2)(X_1 - \mu_1)] & \mathrm[(X_2 - \mu_2)(X_2 - \mu_2)] & \cdots & \mathrm[(X_2 - \mu_2)(X_n - \mu_n)] \\  \\
 \vdots & \vdots & \ddots & \vdots \\  \\
 \mathrm[(X_n - \mu_n)(X_1 - \mu_1)] & \mathrm[(X_n - \mu_n)(X_2 - \mu_2)] & \cdots & \mathrm[(X_n - \mu_n)(X_n - \mu_n)]
\end

The

(i,j)

element is the covariance between

X_i

and

X_j

. This concept generalizes to higher dimensions the concept of variance of a scalar-valued random variable

X

, defined as :

\sigma^2 = \mathrm(X)
= \mathrm[(X-\mu)^2] \,

where

\mu = \mathrm(X)

Conflicting nomenclatures and notations

Nomenclatures differ. Some statisticians, following the probabilist William Feller, call this matrix the variance of the random vector

X

X

. Unfortunately, several different conventions jar to some degree with each other: Standard notation: :

\operatorname(\textbf)
=
\mathrm
\left[
 (\textbf - \mathrm [\textbf])
 (\textbf - \mathrm [\textbf])^\top
\right]

Also standard notation (unfortunately conflicting with the above): :

\operatorname(\textbf)
=
\mathrm
\left[
 (\textbf - \mathrm[\textbf])
 (\textbf - \mathrm[\textbf])^\top
\right]

Also standard notation: :

\operatorname(\textbf,\textbf)
=
\mathrm
\left[
 (\textbf - \mathrm[\textbf])
 (\textbf - \mathrm[\textbf])^\top
\right]

Properties

For

\Sigma=\mathrm \left[  \left(   \textbf - \mathrm[\textbf]  \right)  \left( \textbf - \mathrm[\textbf] \right)^\top \right]

and

\mu = \mathrm(\textbf)

the following basic properties apply: #

\Sigma = \mathrm(\mathbf) - \mathbf\mathbf

\operatorname(\mathbf\mathbf) = \mathbf \operatorname(\mathbf) \mathbf

\mathbf \geq 0

\operatorname(\mathbf + \mathbf) = \mathbf \operatorname(\mathbf) \mathbf

\operatorname(\mathbf,\mathbf) = \operatorname(\mathbf,\mathbf)^\top

\operatorname(\mathbf + \mathbf,\mathbf) = \operatorname(\mathbf,\mathbf) + \operatorname(\mathbf, \mathbf)

# If

p = q

\operatorname(\mathbf + \mathbf) = \operatorname(\mathbf) + \operatorname(\mathbf,\mathbf) + \operatorname(\mathbf, \mathbf) + \operatorname(\mathbf)

\operatorname(\mathbf, \mathbf) = \mathbf \operatorname(\mathbf, \mathbf) \mathbf^\top

# If

\mathbf

and

\mathbf

are independent, then

\operatorname(\mathbf, \mathbf) = 0

where

\mathbf, \mathbf

and

\mathbf

are a random

\mathbf

vectors,

\mathbf

is a random

\mathbf

vector,

\mathbf

\mathbf

vector,

\mathbf

and

\mathbf

are

\mathbf

Complex random vectors

The variance of a complex scalar-valued random variable with expected value μ is conventionally defined using complex conjugation: :

\operatorname(z)
=
\operatorname
\left[
 (z-\mu)(z-\mu)^
\right]

where the complex conjugate of a complex number

z

is denoted

z^

. If

Z

is a column-vector of complex-valued random variables, then we take the conjugate transpose by both transposing and conjugating, getting a square matrix: :

\operatorname
\left[
 (Z-\mu)(Z-\mu)^
\right]

where

Z^

denotes the conjugate transpose, which is applicable to the scalar case since the transpose of a scalar is still a scalar.

Estimation

1 \times 1

matrix than as a mere scalar. See estimation of covariance matrices.

External references

- [http://mathworld.wolfram.com/CovarianceMatrix.html Covariance Matrix] at Mathworld Category:Statistics

MATLAB

MATLAB refers to a numerical computing environment and its core programming language. Created by The MathWorks, MATLAB allows easy matrix manipulation, plotting of functions and data, implementation of algorithms, creation of user interfaces, and interfacing with programs in other languages. Although it specializes in numerical computing, an optional toolbox interfaces with the Maple symbolic engine, making it a full computer algebra system. It is used by more than one million people in industry and academia and runs on most modern operating systems, including Windows, Mac OS, Linux and Unix. The current version is MATLAB 7.1 Service Pack 3. It is available for commercial use for approximately US$2000 and US$100 for an academic license with a limited set of Toolboxes.

History

Short for "MATrix LABoratory", MATLAB was invented in the late 1970s by Cleve Moler, then chairman of the computer science department at the University of New Mexico. He designed it to give his students access to LINPACK and EISPACK without having to learn Fortran. It soon spread to other universities and found a strong audience within the applied mathematics community. Jack Little, an engineer, was exposed to it during a visit Moler made to Stanford University in 1983. Recognizing its commercial potential, he joined with Moler and Steve Bangert. They rewrote MATLAB in C and founded The MathWorks in 1984 to continue its development. These rewritten libraries were lovingly known as JACKPAC. MATLAB was first adopted by control design engineers, Little's specialty, but quickly spread to many other domains. It is now also used in education, in particular the teaching of linear algebra and numerical analysis.

Example MATLAB code

This code, excerpted from the function magic.m, creates a magic square M for odd values of n. [J,I] = meshgrid(1:n); A = mod(I+J-(n+3)/2,n); B = mod(I+2
- J-2,n); M = n
- A + B + 1; Note that this code performs operations on vectors and matrices without the use of "for" loops. Idiomatic MATLAB programs usually operate on whole arrays at a time. The MESHGRID utility function above creates arrays like these: I = 1 1 1 J = 1 2 3 2 2 2 1 2 3 3 3 3 1 2 3 Most scalar functions can also be used on arrays, and will apply themselves in parallel to each element. Thus mod(2
- J,n) will (scalar) multiply the entire J array with 2, before reducing each element modulo n. MATLAB does include standard "for" and "while" loops, but it is almost always faster to write and execute code that expresses vector operations. There are many other programs that do similar tasks to MATLAB; for a list, see List of numerical analysis software.

Criticism

MATLAB itself is a proprietary product of The MathWorks. Unlike common programming languages such as C or FORTRAN, the MATLAB language is not managed or specified by a 3rd party standards committee such as ANSI. Obtaining a fully compatible and up to date MATLAB platform requires purchasing the product. Some programs are available that implement significant subsets of the MATLAB programming language (notably the free software GNU Octave project), but these are not 100% compatible and do not include various domain-specific tools. Consequently, MATLAB customers may be subject to vendor lock-in. MATLAB was originally written in FORTRAN and later re-written in C. The language shows this mixed heritage with a sometimes erratic syntax: neither C nor FORTRAN, but a combination of both. This mixed syntax can lead to interpretation problems. For example, the expression: y = f(x) could either refer to function f with argument x or the x value of matrix f. This ambiguity is difficult to resolve without closely examining the code. Similar difficulties surround the
- and ' operators. One of the basic datatypes in MATLAB is a matrix, an array of numbers devoid of important attributes required by real world data such as engineering units, sampling rates and time/date markers. Especially the lack of sample rate information is a serious shortcoming for signal processing applications, where data is typically sampled at a constant interval. These attributes must be managed by the user with custom programming, which is error-prone and time-consuming. MATLAB is a procedural programming language, so it cannot automatically update variables in response to input changes as one might want for simulations or exploratory data analysis. Consider, for example, the following fragment: t = 1:100; y = log(t); If variable t changes, e.g. t = 100:1000, the user must manually re-evaluate y to obtain the updated result. The MathWorks offers a supplementary package, Simulink, that partially automates these tasks for systems modeling and simulation applications. Despite these shortcomings, MATLAB continues to be employed in many technical analysis applications, though several viable competitors are emerging.

External links

- [http://www.mathworks.com/products/matlab/ The MATLAB product page at The MathWorks]
- [http://www.mathworks.com/matlabcentral/ MATLAB Central the MATLAB user community]
- [http://dmoz.org/Science/Math/Software/MATLAB/ The MATLAB category at the Open Directory Project]
- [http://www.mathworks.com/company/newsletters/news_notes/clevescorner/dec04.html Additional information about the history of and inspiration for MATLAB, written by Cleve Moler]
- [news://comp.soft-sys.matlab comp.soft-sys.matlab] Category:Domain-specific programming languages Category:Numerical programming languages Category:Array programming languages ja:MATLAB

Probability density function

In mathematics, a probability density function (pdf) serves to represent a probability distribution in terms of integrals. A probability density function is everywhere non-negative and its integral from −∞ to +∞ is equal to 1. If a probability distribution has density f(x), then intuitively the infinitesimal interval [x, x + dx] has probability f(x) dx. Informally, a probability density function can be seen as a "smoothed out" version of a histogram: if one empirically measures values of a continuous random variable repeatedly and produces a histogram depicting relative frequencies of output ranges, then this histogram will resemble the random variable's probability density (assuming that the variable is sampled sufficiently often and the output ranges are sufficiently narrow). Formally, a probability distribution has density f(x) if f(x) is a non-negative Lebesgue-integrable function R → R such that the probability of the interval [a, b] is given by :

\int_a^b f(x)\,dx

for any two numbers a and b. This implies that the total integral of f must be 1. Conversely, any non-negative Lebesgue-integrable function with total integral 1 is the probability density of a suitably defined probability distribution.

Simplified explanation

A probability density function is any function f(x) that describes the probability density in terms of the input variable x in a manner described below.
- f(x) is greater than or equal to zero for all values of x
- The total area under the graph is 1: ::

\int_^\infty \,f(x)\,dx = 1

The actual probability can then be calculated by taking the integral of the function f(x) by the integration interval of the input variable x. For example: the variable x being within the interval 4.3 < x < 7.8 would have the actual probability of :

\Pr(4.3

Further details

For example, the continuous uniform distribution on the interval [0,1] has probability density f(x) = 1 for 0 ≤ x ≤ 1 and zero elsewhere. The standard normal distribution has probability density :

f(x)=.

If a random variable X is given and its distribution admits a probability density function f(x), then the expected value of X (if it exists) can be calculated as :

\operatorname(X)=\int_^\infty x\,f(x)\,dx.

Not every probability distribution has a density function: the distributions of discrete random variables do not; nor does the Cantor distribution, even though it has no discrete component, i.e., does not assign positive probability to any individual point. A distribution has a density function if and only if its cumulative distribution function F(x) is absolutely continuous. In this case, F is almost everywhere differentiable, and its derivative can be used as probability density: :

\fracF(x) = f(x).

If a probability distribution admits a density, then the probability of every one-point set is zero. It is a common mistake to think of f(a) as the probability of , but this is incorrect; in fact, f(a) will often be bigger than 1 - consider a random variable with a uniform distribution between 0 and 1/2. Two probability densities f and g represent the same probability distribution precisely if they differ only on a set of Lebesgue measure zero. In the field of statistical physics, a non-formal reformulation of the relation above between the derivative of the cumulative distribution function and the probability density function is generally used as the definition of the probability density function. This alternate definition is the following: If dt is an infinitely small number, the probability that

X

is included within the interval [t, t + dt] is equal to

f(t)\,dt

, or: :

\Pr(t

Link between discrete and continuous distributions

The definition of a probability density function at the start of this page makes it possible to describe the variable associated with a continuous distribution using a set of binary discrete variables associated with the intervals [a; b] (for example, a variable being worth 1 if X is in [a; b], and 0 if not). It is also possible to represent certain discrete random variables using a density of probability, via the Dirac delta function. For example, let us consider a binary discrete random variable taking −1 or 1 for values, with for probability 1/2 each. The density of probability associated with this variable is: :

f(t) = \frac(\delta(t+1)+\delta(t-1)).

More generally, if a discrete variable can take 'n' different values among real numbers, then the associated probability density function is: :

f(t) = \frac\sum_^nP_i\, \delta(t-x_i),

where

x_1, \ldots, x_n

are the discrete values accessible to the variable and

P_1, \ldots, P_n

are the probabilities associated with these values. This expression allows for determining statistical characteristics of such a discrete variable (such as its mean, its variance and its kurtosis), starting from the formulas given for a continuous distribution. In physics, this description is also useful in order to characterize mathematically the initial configuration of a Brownian movement.

Probability function associated to multiple variables

For a set of continuous random variables

X_1,\ldots,X_N

, it is also possible to define a probability density function associated to the set as a whole. This density function is defined as a function of the N variables, such that, for any domain D in the N-dimension space of the values of the variables

X_1,\ldots,X_N

, the probability that a realisation of the set variables falls inside the domain D is: :

P \left( X_1,\ldots,X_N \,\, \epsilon \,\, D \right) 
 = \int_D dX_1 \ldots dX_N \,\, P(X_1,\ldots,X_N)

For i in [1;N], let us call

p_i(X_i)

the probability density function associated to variable

X_i

alone. This probability can be deduced from the probability associated to the set

X_1,\ldots,X_N

by integrating on all values of the N-1 other variables in the set: :

p_i(X_i) = \int dX_1 \ldots dX_\,dX_\ldots dX_N \,\, P(X_1,\ldots,X_N)

Independence theorem

The variables

X_1,\ldots,X_N

are all independent from each other if and only if: :

P(X_1,\ldots,X_N) = p_1(X_1)\ldots p_n(X_n)

Corollary

if the probability density function associate to a set of N random variables can be separated into a product of N functions of one random variable: :

P(X_1,\ldots,X_N) = f_1(X_1)\ldots f_n(X_n)

, then the N variables in the set are all independent from each other, and the probability density function associated to each of them is: :

p_i(X_i,\ldots,X_N) = \frac

Example

This elementary example illustrates the above definition of multidimensional probability density functions in the simple case of a function of a set of two variables. Let us call

\vec R

a 2D random vector of coordinates

(X,Y)

: the probability to obtain

\vec R

in the quarter plane of positive X and Y is: :

P \left( X > 0, Y > 0 \right) 
 = \int_0^\infty dX \int_0^\infty dY P(X,Y)

Linear transformation

In mathematics, a linear transformation (also called linear map or linear operator) is a function between two vector spaces that preserves the operations of vector addition and scalar multiplication. In other words, it "preserves linear combinations". In the language of abstract algebra, a linear transformation is a

Multivariate normal distribution

General case

Cumulative distribution function

A counterexample

Bivariate case

Linear transformation

Correlations and independence

Conditional distributions

Fisher information matrix

Kullback-Leibler divergence

Estimation of parameters

Generating values drawn from the distribution

Statistics

Origin

Statistical methods

Experimental and observational studies

Levels of measurement

Statistical techniques

Probability

Important contributors to statistics

Specialized disciplines

Software

See also

External links

General sites and organizations

Link collections

Online courses and textbooks

Statistical software

Other resources

Additional references

Random vector

Matrix (math)

Definitions and notations

Example

Adding and multiplying matrices

Sum

Scalar multiplication

Multiplication

Linear transformations, ranks and transpose

Square matrices and related definitions

Special types of matrices

Matrices in abstract algebra

History

Further reading

External links

Positive-definite

See also

Covariance matrix

Conflicting nomenclatures and notations

Properties

Complex random vectors

Estimation

External references

Determinant

Determinants of 2-by-2 matrices

Applications

General definition and computation

Example

Properties

Derivative

Generalizations and related functions

History

Expected value

Mathematical definition

Properties

Linearity

Functional non-invariance

Non-multiplicativity

Iterated expectation

Inequality

Representation

Uses and applications of the expected value

Expectation of matrices

See also

External links

Covariance matrix

Conflicting nomenclatures and notations

Properties

Complex random vectors

Estimation