V_ENTROPY calculates the v_entropy of discrete and sampled continuous distributions H=(P,DIM,COND,ARG,STEP) Inputs: P is a vector or matrix of probabilities - one dimension per variable DIM lists dimensions along which to evaluate the v_entropy [default: 1st non singleton dimension] COND lists dimensions to use as conditional variables [default - none] ARG lists dimensions to use as parameters in the ouput [default - none] STEP for continuous distributions STEP gives the sample increment for each dimension of P if STEP is a scalar, the increment is assumed to be the same for each dimension Outputs: H is the v_entropy. It will have the same number of dimensions as the length of the ARG input. If the STEP argument is specified then this will be the differential v_entropy. Example: Suppose P(W,X,Y,Z) represents the joint probability of four correlated random variables (a) H(W,X,Y,Z) = v_entropy(P,[1 2 3 4]). (b) H(W) = v_entropy(P), or equivalently v_entropy(P,1) (c) H(W,Z | X,Y) = v_entropy(P,[1 4],[2 3]) (d) H(W | X, Z=z) = v_entropy(P,1,2,4); this is a function of z and will be a column vector As a special case, if the dimensions included in DIM are all singletons, the entries in P are treated as Bernoulli variable probabilities.
0001 function h=v_entropy(p,dim,cond,arg,step) 0002 %V_ENTROPY calculates the v_entropy of discrete and sampled continuous distributions H=(P,DIM,COND,ARG,STEP) 0003 % 0004 % Inputs: P is a vector or matrix of probabilities - one dimension per variable 0005 % DIM lists dimensions along which to evaluate the v_entropy [default: 1st non singleton dimension] 0006 % COND lists dimensions to use as conditional variables [default - none] 0007 % ARG lists dimensions to use as parameters in the ouput [default - none] 0008 % STEP for continuous distributions STEP gives the sample increment for each dimension of P 0009 % if STEP is a scalar, the increment is assumed to be the same for each dimension 0010 % 0011 % Outputs: H is the v_entropy. It will have the same number of dimensions as the length of the ARG input. 0012 % If the STEP argument is specified then this will be the differential v_entropy. 0013 % 0014 % Example: Suppose P(W,X,Y,Z) represents the joint probability of four correlated random variables 0015 % 0016 % (a) H(W,X,Y,Z) = v_entropy(P,[1 2 3 4]). 0017 % (b) H(W) = v_entropy(P), or equivalently v_entropy(P,1) 0018 % (c) H(W,Z | X,Y) = v_entropy(P,[1 4],[2 3]) 0019 % (d) H(W | X, Z=z) = v_entropy(P,1,2,4); this is a function of z and will be a column vector 0020 % 0021 % As a special case, if the dimensions included in DIM are all singletons, the entries in P are treated 0022 % as Bernoulli variable probabilities. 0023 0024 % Copyright (C) Mike Brookes 2006 0025 % Version: $Id: v_entropy.m 10865 2018-09-21 17:22:45Z dmb $ 0026 % 0027 % VOICEBOX is a MATLAB toolbox for speech processing. 0028 % Home page: http://www.ee.ic.ac.uk/hp/staff/dmb/voicebox/voicebox.html 0029 % 0030 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 0031 % This program is free software; you can redistribute it and/or modify 0032 % it under the terms of the GNU General Public License as published by 0033 % the Free Software Foundation; either version 2 of the License, or 0034 % (at your option) any later version. 0035 % 0036 % This program is distributed in the hope that it will be useful, 0037 % but WITHOUT ANY WARRANTY; without even the implied warranty of 0038 % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 0039 % GNU General Public License for more details. 0040 % 0041 % You can obtain a copy of the GNU General Public License from 0042 % http://www.gnu.org/copyleft/gpl.html or by writing to 0043 % Free Software Foundation, Inc.,675 Mass Ave, Cambridge, MA 02139, USA. 0044 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 0045 0046 if nargin<5 0047 stp=zeros(ndims(p),1); 0048 else 0049 stp=repmat(step(1),ndims(p),1); 0050 stp(1:length(step))=step(:); 0051 stp=log2(stp); 0052 end 0053 if nargin<4 0054 arg=[]; 0055 else 0056 arg=arg(arg>0); 0057 end 0058 if nargin<3 0059 cond=[]; 0060 else 0061 cond=cond(cond>0); 0062 end 0063 if ~length(cond) 0064 s=size(p); 0065 if nargin<2 0066 dim=find(s>=min(2,max(s))); 0067 dim=dim(1); 0068 else 0069 dim=dim(dim>0); 0070 end 0071 st=prod(s); 0072 sd=prod(s(dim)); 0073 sa=prod(s(arg)); 0074 marg=1:length(s); 0075 marg(arg)=0; 0076 marg(dim)=0; 0077 marg=marg(marg>0); 0078 sm=st/sd/sa; 0079 if sm>1 0080 ip=[arg dim(:)' marg(:)']; 0081 sp=[s([arg dim(:)']) prod(s(marg))]; 0082 q=sum(reshape(permute(p,[arg(:)' dim(:)' marg]),sa,sd,sm),3); 0083 else 0084 q=reshape(permute(p,[arg(:)' dim(:)' marg]),sa,sd); 0085 end 0086 if sd==1 0087 h=-log2(q+(q==0)).*q-log2(1-q+(q==1)).*(1-q); % special treatment for bernoulli variables 0088 else 0089 sq=sum(q,2); 0090 h=sum(-log2(q+(q==0)).*q,2)./sq+log2(sq); 0091 end 0092 if length(arg)>1 0093 h=reshape(h,s(arg)); 0094 end 0095 h=h+sum(stp(dim)); 0096 else 0097 % we could probably make this more efficient by avoiding the recursive call 0098 h=v_entropy(p,[dim(:); cond(:)],0,arg)-v_entropy(p,cond,0,arg); 0099 end