Hi,
My question is:
How to compute the entropy h(AX) where X is a random vector, A is a (m
x n) deterministic matrix, m is less than or equal to n, and rank(A) =
m.
------------
The following theorems actually gives the answer to my question when m
= n, and rank(A) = m. However, I am not sure what is the form of
entropy when A is not invertible.
Theorem 1 [Theorem 5.11 in 1]:
If X is a continuous random vector and A is an invertible matrix, then
Y = AX + b has PDF (probability density function):
f_{Y}(y) = 1/|det(A)| f_{X}(A^{-1} (y-b))
Theorem 2 [2]:
Entropy h(AX) = h(X) + log |A|, where |A| is the absolute value of the
determinant of matrix A.
-------------
[1] Roy D. Yates and David J. Goodman (2004). Probability and Stochastic
Processes: A Friendly Introduction For Electrical and Computer
Engineers. Second Edition
[2] Thomas M. Cover, Joy A. Thomas (1991). Elements of Information Theory |
Request for Question Clarification by
mathtalk-ga
on
14 Sep 2004 12:26 PDT
Hi, elgoog_elgoog-ga:
A brief answer would be that you need to replace f_(X) by the
corresponding marginal distribution on the orthogonal complement of
the nullspace of A.
Do you have sufficient information about the pdf f_(X) that such a
reply would be of interest?
regards, mathtalk-ga
|
Clarification of Question by
elgoog_elgoog-ga
on
14 Sep 2004 13:10 PDT
Hi mathtalk-ga,
Thank you for the response.
Unfortunately, I don't have much more information about the pdf of X.
I want to find a general form of h(AX), or actually, find a general
form of the pdf of AX. Theorem 1 gives the answer when A is
invertible, so I don't know if it is possible to compute pdf(AX) when
A is not invertible.
thanks
|
Request for Question Clarification by
mathtalk-ga
on
14 Sep 2004 21:22 PDT
Just as Thm. 1 expresses the pdf of AX in terms of the pdf of X when A
is invertible, it is possible to express the pdf of AX in terms of the
pdf of X when A is not invertible. However the expression is more
complicated, to the extent that a marginal distribution must be
derived from the original distribution of X.
For example, if A is restricted to the orthogonal complement of the
nullspace (kernel) of A, then it becomes invertible there. The result
of Thm. 1 will then apply, but with the pdf for X replaced by the
marginal pdf for the projection of X onto the said orthogonal
complement of the nullspace of A.
It a formula, though not as nice as the case for A invertible.
regards, mathtalk-ga
|
