EE 503 - Statistical Signal Processing and Modeling

00:00 Introduction
01:55 Communication example (motivation)
09:45 Loss functions (communication example)
13:41 Risk/Cost (communication example)
18:38 Restricting the estimator to linear estimation (communication example)
20:10 LMMSE Problem (communication example)
26:45 Connection with practice through law of large numbers (communication example)
27:13 Law of large numbers (communication example)
33:20 Average behavior and one time events
38:57 P implies Q (Mathematical Reasoning)
39:50 Necessary, Sufficient conditions
42:00 Truth div (P implies Q)
48:40 Direct Proof Approach (P implies Q)
51:08 Proof by Contraposition (P implies Q)

Correction: @47:30 and around: "P:True and Q:False" should be "P: False and Q:True" (Ugur Berk S.) 

00:00 - Proof by Contradiction (P implies Q)
05:32 - "if and only if" statement
10:45 - Proving "if and only if " statements
13:05 - Comments: Q as an indicator in P implies Q
14:25 - Comments: Many necessary conditions leading to necessary & sufficient cond.
19:00 - Example: reasoning in a court case (Murder in Kizilay)
25:55 - Linear Algebra Review
28:30 - Matrix multiplication
30:38 - Linear combination
31:46 - Range space of A matrix
33:45 - Null space of A matrix
36:33 - Importance of null space in linear equation solutions
40:10 - Unique solution condition (if there is a solution)
41:29 - Checking the dimension of Null space
43:18 - Checking linear independence
43:55 - Column Rank of A matrix
47:41 - Projection Matrices
51:58 - Norm

Corrections:
42:30 Additional note: Ax = 0 equation system is assumed to hold true for a non-trivial solution, i.e. x \neq 0

0:00 - Projection Problem (reminder of last lecture)
3:10 - Distance Metric
4:05 - Distance Metric Axioms
6:10 - Norm Function
9:44 - Metric induced from norm function
14:00 - Projection to Range(A) (problem definition)
15:22 - Ever present engineering questions on existence, uniqueness and feasible method for solution
19:09 - Projection to plane (3D case)
26:08 - Projection to circle example
33:40 - Projection to a convex set example
36:22 - Optimality condition for projection (wide angle condition)
41:22 - Inner product
41:43 - Inner product axioms
47:29 - Norm induced by inner product
51:08 - Cauchy - Schwarz inequality (statement)

0:00 - Inner Product / Norm Axiom (reminder)
2:00 - Cauchy Schwarz Inequality (proof)
8:40 - Cauchy Schwarz Equality case
13:00 - "angle" between vectors
14:40 - Example: Cosine theorem of 2D/3D Euclidean geometry
20:00 - Orthogonality of vectors, aligned vectors
24:30 - Triangle inequality for the induced norm (proof)
27:40 - Vector spaces, normed spaces, inner product spaces
30:30 - Projection matrices (problem statement, again!)
34:27 - Orthogonality conditions for the projection point (projectors)
41:03 - Orthogonality condition in matrix-vector form (projectors)
42:59 - Solving the projection point (projectors)
43:50 - Case of invertible Gram matrix (Solving the projection point)
45:30 - Projection matrix expression (finally!)

Corrections:
11:05 x = \lambda_x y should be x = - \lambda_x y (Ugur Berk S.)
15:30 (x,y) inner product should be (x,y) = x1 \times y1 + x2 \times y2 (Ada G.)

0:00 - Case of non-invertible Gram matrix (extra!)
2:55 - Gram matrix definition and linear independence of vectors
4:40 - Range(A^T x A ) = Range (A^T) proof by SVD (extra!)
6:45 - Mini talk about SVD (extra!)
9:57 - Orthogonal Projectors (definition)
13:10 - Symmetric/Hermitian symmetric matrices (definition)
14:40 - Showing P_A is an orthogonal projector
17:23 - Transpose operation { (ABC)^T = C^T x B^T x A^T }
19:20 - Symmetric/Hermitian symmetric eigenvectors are orthogonal (statement only!)
20:00 - More general fact: Eigenvectors of normal matrices (statement only!)
21:30 - Orthogonal matrices (definition)
23:38 - Gram matrix as matrix of inner products (orthogonal matrices)
26:50 - Eigenspace of P_A
27:40 - Eigenvalues of P_A (eigenspace P_A)
31:25 - Eigenvectors of P_A (eigenspace P_A)
32:00 - Eigendecomposition expression (eigenspace P_A)
39:10 - Multiplication of matrices A and B via the columns of and rows of B
43:42 - Eigendecomposition expression (eigenspace P_A, finally!)
46:25 - On the representation basis and P_A matrix (uniqueness of P_A)
53:27 - Complementary Projector (definition)
56:40 - Orthogonality of P_A and P_A^\perp matrices
58:35 - Eigendecomposition of complementary projector
1:02:40 - Decomposition of vector b to Range(A) space and its complementary space

Corrections:
55:55 - Left side of the board, point 2: ( P_A^\perp )^T = ( P_A^\perp )^T should be ( P_A^\perp )^T = P_A^\perp

0:00 - Orthogonal Basis Representations
7:10 - Projection operator with orthonormal basis
15:52 - Example 1: Canonical Basis
17:15 - Example 2: DFT Basis
22:30 - Inner product definition for complex-valued vectors
32:30 - Gram-Schmidt Orthogonalization
44:30 - QR decomposition and Gram-Schmidt relation

Corrections:
19:30 - N'th row N'th column of F matrix should be W^{(N-1)x(N-1)} (Gulin T.)

0:00 - Introduction
1:00 - Quadratics (1 independent variable)
12:00 - Quadratics (2 independent variables)
13:00 - Level curves of J(x,y)
17:50 - Quadratic form : J(x) = x' A x + b'x + c
20:30 - Discussion on x' A x term
24:45 - Gradient of J(x) = (A + A')x + b
33:10 - Ju(u) = Jx(u+x_opt)
41:00 - Positive Definite Matrices (definition)
45:00 - Relation between positive definite matrices and eigenvalues
55:10 - Indefinite matrices (definition)
57:00 - Positive (semi) definiteness checks

Corrections:
30:00 - 2a11 + a12y + ... should be 2a11x + a12y + .... (similarly for the line below) (Gulin T.)

45:30 - A = [2 4; 4 2] should be A = [ 1 2; 2 1] ([2 4; 4 2] is the Hessian matrix of J(x,y) which is 2 x [1 2; 2 1]; This explanation is also OK.; but I would like to investigate the nature of quadratic terms u^T A u in this discussion. All explanations are correct, but possibly confusing since it is not clear why we switch to 2 \times A, instead of A = [ 1 2; 2 1]) (Gulin T.)

50:17 - The vector [1 -1] points in the opposite direction shown, no harm in this presentation since -1 x [1 -1] = [-1 1] is also an eigenvector and I am considering the values of the function on the eigenvector \times "t" where t is in (-\infty, \infty) (Onur Selim K.)

0:00 - Positive Definite Matrices (case of non-symmetric matrices)
7:10 - Over-determined equation systems
9:00 - Tall matrices / Fat and Short matrices
10:06 - Minimizing || Ax - b||^2
15:15 - Solving A^T A x = A^T b (Normal Equation)
18:17 - Positive Definiteness of A^T A
23:15 - Review of some DSP Topics
23:34 - Analog / Continuous Time SP
24:30 - LTI systems
25:30 - Example: RC circuit (Low-pass filter)
27:46 - Discrete-time processing of analog signals (block diagram)
31:02 - D/C Conversion
32:17 - D/C Conversion (zero-order hold interpolation)
35:30 - D/C Conversion (linear interpolation)
39:20 - D/C Conversion (sinc interpolation)
43:58 - sinc(x) (definition)
47:00 - sinc interpolation (further explained)
52:25 - Discrete-time processing of analog signals (block diagram)
53:50 - Discrete-time processing of analog signals (bandlimited signals)
55:00 - Some problems of analog signal processing (discussion)
56:50 - Exact discrete-time implementation of analog systems via impulse invariance
1:06:33 - Fourier Transform (definition)
1:08:20 - Fourier Transform of rect(t/T)

0:00 - Review of Probability Concepts
1:00 - Sample Space, Outcome, Event, Probability
6:20 - Kolmogorov's Probability Axioms
9:42 - De Morgan's Laws
13:50 - Brief Note on sigma algebra
18:50 - Random variable
21:03 - Random vector
26:15 - X = x, X: random variable, x: its value
27:10 - c.d.f. (cumulative density function)
30:20 - p.d.f. (probability density function)
34:02 - Probability value as "area under density"
39:43 - Units of density function and probability
41:57 - Probability mass functions
43:22 - Independence of events A and B
44:25 - Independence of random variables X and Y
47:25 - c.d.f/p.d.f for joint representation of X and Y
50:50 - Conditional Probability
59:30 - Conditional random variables
1:01:40 - Bayes' Theorem

0:00 - Conditional Probability (review)
02:30 - Bayes' Theorem (review)
03:58 - Marginalization operation (two random variables)
12:20 - Total Probability Theorem
15:54 - Example: X is 1 if Heads, X is Unif([0,2]) if Tails
33:03 - Expectation Operation
34:45 - Expectation (law of large numbers interpretation)
39:51 - E{ g(X) }
40:50 - Moments, E{ X^k }
41:30 - Central Moments, E{ (X - Xbar)^k }
44:28 - Moment generating function
52:07 - Moments as partial description of a r.v.
53:34 - Conditional Expectation
57:52 - Iterated Expectation

00:00 - "similarity" of two r.v.'s
00:45 - Inner product as an indicator of "similarity" for N-dim vectors (reminder)
04:14 - Correlation coefficient (definition for zero mean r.v.'s)
08:28 - Properties of correlation coefficient
09:31 - |r_{xy}| \leq 1 (proof)
13:35 - |r_{xy}| = 1 implies Y = aX
16:41 - Y = aX imples |r_{xy}| = 1
20:03 - r_{xy} as angle between two r.v.'s
22:01 - Orthgonal r.v.'s (definition)
23:05 - Example: r_{xy} calculation for Y = aX + N
29:52 - SNR definition (example continues)
32:55 - Interpretation of results (example continues)
37:40 - Correlation (definition)
38:22 - Correlation coefficient (definition)
42:25 - Cov(X,Y) (definition)
45:20 - Uncorrelated r.v.'s
47:35 - Example: Cov(X,Y) for r.v.'s X and Y which are indicator functions events A and B
55:03 - Interpretation of results (example continues)
57:51 - Properties of Cov(X,Y)
1:00:58 - Example: Var ( \sum_{i=1}^N x_i )
1:06:20 - Independence implies uncorrelatedness (proof)

A Matlab illustration on the correlation coefficient

.PDF Document

00:00 - Random vectors
03:04 - Correlation matrix (definition)
03:31 - Covariance matrix (definition)
11:10 - Properties of covariance matrix
12:14 - Hermitian symmetry (properties continue)
15:18 - Positive semi-definiteness (properties continue)
20:47 - Gaussian Distribution
21:16 - 1D Gaussian r.v.
28:10 - N-dimensional Gaussian vectors
33:08 - 2-dimensional Gaussian vectors
40:15 - Level curves (2D Gaussian vectors)
45:15 - Level curves (2D Gaussian vector, Cx : diagonal)
49:10 - Level curves (2D Gaussian vector, Cx \propto I )
50:30 - Facts on Gaussian vectors
50:45 - Marginalization (facts continue)
52:48 - Example on marginalization of Gaussian vectors
55:30 - Linear processing of Gaussian vectors (facts continue)
56:15 - Example: Ry matrix in terms of Rx for y = Mx
1:02:13 - Example: Var ( \sum_{i=1}^N x_i ) (redo earlier example with vector operations)

Corrections:
36:20 - 2D Gaussian case: pdf (2nd line on the left side) should have Cx^{-1} not Cx (Cagatay C.)

0:00 - Linear processing of random vectors (summary)
3:40 - Decorrelation of random vectors
5:45 - Case 1: Diagonalization by eigendecomposition
19:15 - Case 2: Diagonalization by unitary transformation and scaling
23:35 - Case 3: Diagonalization by LU decomposition
29:29 - Unit lower triangular matrix (definition)
36:27 - Causal decorrelation operation by LU decomposition

0:00 - Diagonalization of Covariance Matrices (review)
3:50 - Diagonalization by eigendecomposition (review)
06:01 - Diagonalization by unitary Transformation and followed by scaling (review)
08:35 - Diagonalization by LU decomposition (review)
11:40 - Joint Diagonalization of two covariance matrices
15:13 - Joint Diagonalization of two covariance matrices (Step 1)
18:14 - Joint Diagonalization of two covariance matrices (Step 2)
26:35 - Joint Diagonalization of two covariance matrices (with a single step)
40:50 - Random processes

0:00 - Descriptions for Random Processes
0:35 - Joint pdf description
6:42 - Example: joint pdf description of x(t) = A cos(2\pi f t + \theta), \theta: r.v. ~ Unif. [0,2\pi)
12:10 - 1st order pdf description (example)
13:22 - One function of one r.v. discussion
26:47 - 1st order pdf description (example, continued)
37:12 - 1st order pdf description (comments)
41:51 - 2nd order pdf description (example)
58:51 - 3rd order pdf description (example)

0:00 - Random Process Descriptions (cont'd)
0:58 - Gaussian Processes
2:20 - Joint pdf of samples (Gaussian processes)
6:00 - Example: Marginal pdf's are Gaussian, while joint pdf is not
14:50 - Example: x(t) = a t+ b (a,b: r.v.'s)
19:13 - 1st order pdf description (example)
27:05 - 2nd order pdf description (example)
37:40 - 3rd order pdf description (example)
42:27 - Discussion of degenerate cov. mat. (example)

0:15 - Moments Description
2:35 - 1st Order Moment Description: mean function
3:19 - 2nd Order Moment Description: Autocorrelation/Covariance functions
5:08 - Comments: on the Moments Descriptions
9:18 - Example: x(t) a r.p. with mu_x(t)=3 and R_x(t1,t2)=9+4*exp(-0.2|t1-t2|), z=x(5), w=x(8), Find E{z}, E{w}, E{z^2}, E{w^2} and E{zw}
14:50 - Example: z = x(t1) + x(t2), Find E{z^2}
17:12 - Example: s = integral_{from a to b}(x(t)dt), a) Find E{s}, b) Find E{s^2}
21:33 - Example: x(t)=Acos(wt+Q), A,Q are independent r.v.'s, Q~uniform[0,2*pi), Find mu_x(t) and R_x(t1,t2)
34:14 - Notes: about complex valued processes
37:13 - Example: x(t)=A*exp(wt+Q), A,Q are independent r.v.'s, Q~uniform[0,2*pi), Find mu_x(t) and R_x(t1,t2)

0:00 - White noise process
8:10 - Example: x_1[k] = w and x_2[k] = w_k
14:17 - Joint pdf description (example)
21:15 - Moment description (example)
30:05 - Example: w_1[n] \in {1,-1} iid; w_2[n] ~ N(0,1), iid
33:50 - Joint pdf description (example)
38:30 - Moment description (example)
space

0:00 - Linear systems with stochastic inputs
0:33 - Linear systems - continuous time (review)
2:16 - LTI systems - continuous time (review)
5:06 - Linear systems - discrete time (review)
7:40 - LTI systems - discrete time (review)
8:58 - LTI systems - discrete time, convolution matrix (review)
11:11 - Moment descriptions for linear systems with stochastic inputs
11:30 - Linear processing output mean function calculation
14:02 - Basic Assumption: Commutation of Expectation operation and Linear operations
16:29 - Linear processing output auto-correlation function calculation
17:01 - Linear processing output auto-correlation function calculation (step 1: cross correlation)
25:10 - Linear processing output auto-correlation function calculation (step 2: auto-correlation)
32:04 - Example: y(t) = L{x(t)} = d/dt x(t), Find output mean and autocorrelation functions
41:00 - Connections with earlier finite dimensional results
47:12 - Brief discussion on LTI and WSS input case
53:15 - Stationary Random Processes
58:03 - Stationarity in joint pdf description
1:02:07 - Stationarity in moment description

0:00 - Stationarity in pdf/moment descriptions (review)
10:40 - SSS implies WSS
11:48 - Gaussian process and WSS is equivalent to SSS
13:41 - Example: x(t) = a cos(wt) + b sin(wt), find conditions on a,b r.v.'s for x(t) be WSS
14:30 - Stationarity in the mean (example, cont'd)
19:51 - Stationarity in the autocorrelation (example, cont'd)
32:38 - Example: x[n] r.p with inpendent sample with x[2n] ~ Unif(-\sqrt(3),\sqrt(3) and x[2n+1] ~ N(0,1). Is x[n] WSS/SSS?
42:13 - Jointly WSS random processes
45:10 - LTI processing of WSS processes
47:05 - Mean function calculation (LTI processing of WSS processes)
space

Corrections:
39:38- left board, very bottom line should be \delta[k] instead of \delta[n-k] (Thanks to Gulin T.)

0:00 - Auto-correlation calculation, (LTI processing of WSS processes)
0:58 - Step 1: Cross-correlation calculation (auto-correlation calculation)
07:34 - Step 2: Auto-correlation calculation (auto-correlation calculation)
11:54 - Power spectral density
14:46 - Evaluation of output auto-correlation and input-output cross-correlation with Fourier transform
15:50 - DTFT of conj( h[ -n ] )
19:53 - Output power spectral density relations in Fourier-domain
22:30 - Z transform of conj( h[ -n ] )
27:00 - Output power spectral density relations in z-domain
30:28 - H(z) and H^*(1/z^*); conjugate reciprocal pole/zero pairs
40:57 - Power Spectral Density (properties)
41:00 - Property 1: Power spectral density is real valued
44:25 - Property 2: Power spectral density is non-negative (proved later)
44:48 - Property 3: Area under power spectral density is r_x[0] = E{ | x[n] |^2 }

0:00 - Processing of WSS with LTI systems (review)
8:27 - Example: x[n] = A exp(j \omega n + \phi) A, \phi : ind. r.v's. Find S_x( e^{j\omega} )
13:59 - Proof of S_x( e^{j\omega} ) is non-negative
28:30 - Conditions for valid auto-correlation function
35:55 - Q: Can we generate a process whose psd is an arbitrary non-negative valued function?
36:54 - Example (Papoulis p.324): s(t) = A exp(j \omega( t - r(t)/c ) ) (doppler spread example)
space

Corrections:
48:00 - On the right side of the board inverse Fourier transform should be 2\pi \times Inverse F.T. (not affecting any results in the upcoming calculations) (Thanks to Gulin T.)

0:00 - Example (Papoulis p.324): s(t) = A exp(j \omega( t - r(t)/c ) ) (doppler spread example, cont'd)
9:16 - Example (Hayes 3.4.1): H(z) = 1 / (1 - 0.25 z^-1), x[n]: white noise. Find r_y[k].
10:20 -Z-transform, ROC, causal/anti-causal sequences etc. (review)
17:24 - r_y[k] calculation by inverse Z-transform (example)

0:00 - A 2nd Characterization for Power Spectral Density (windowed F.T. interpretation)
7:28 - Relation between WSS and Fourier Transform (F.T. decorrelates WSS processes)

0:00 - Moving Average (MA) Processes
2:27 - Moving Average filter in time domain
8:40 - Pole-zero diagram for all-zero (MA) filters
12:40 - Interpretation of frequency response from pole-zero diagram
22:26 - Auto-correlation calculation for MA processes
27:25 - Deterministic auto-correlation
38:36 - Auto-regressive (AR) Processes
41:00 - Pole-zero diagram for all-pole (AR) filters
43:10 - Interpretation of frequency response from pole-zero diagram
48:57- Auto-correlation calculation for AR processes
1:07:30 - Yule-Walker equations in recursion form

0:00 - Yule-Walker equations
6:02 - Finding H_AR(z) s.t. a given r_AR[k] is synthesized
15:09 -Finding r_AR[k] given H_AR(z)
27:09 - Example: AR(1) process auto-correlation calculation
38:01 - ARMA (Auto-regressive Moving-Average) Processes
51:50 - Auto-correlation recursions for ARMA processes

0:00 - Periodic (Harmonic Processes)
0:51 - | r_x[k] | \leq r_x[0]
1:31 - Proof 1: | r_x[k] | \leq r_x[0] (by Rx \geq 0)
6:50 - Proof 2: | r_x[k] | \leq r_x[0] (by Cauchy Schwarz)
8:50 - Proof 3: | r_x[k] | \leq r_x[0] (by Power Spectral Density)
11:20 - Questioning the equality case, r_x[ T ] = r_x[0]
12:46 - Mean-square (MS) periodic processes (definition)
17:01 - MS-periodicity and periodicity of r_x[k]
24:10 - MS-periodicity implication on R_x matrix
33:45 - MS-periodicity and impulsive power spectrum (line spectra)
36:07 - Wold's decomposition theorem
40:22 - Example: Line spectra for complex sinusoid process with random phase and amplitude
44:30 - Spectral Factorization
44:50 - Synthesis Filter (random process synthesis)
46:30 - Whitening Filter
49:44 - Example: Spectral Factorization P(e^(j\omega)) = ( 5 - 4 cos(\omega) ) / (10 - 6 cos(\omega) )
1:08:24 - Minimum Phase Filters (causal, sdiv, causally-invertible) (example)
1:16:50 - Stochastic modeling and deterministic modeling

0:00 - Introduction
0:40 - Stochastic Signal Modeling (reminder)
2:35 - Deterministic Signal Modeling
5:00 - Example: H(z) = b0/ ( 1 - a1z^-1) to match Z{x[n]}
8:26 - Example: H(z) = (b0 + b1z^-1) / ( 1 - a1z^-1) to match Z{x[n]}
11:55 - Pade's Approximation
19:50 - Convolution Matrix
29:00 - Example: Pade's app. with H(z) = b0 / (1 + a1z^-1 +a2z^-2)
31:46 - Prony's method (matrix form)
37:10 - Prony's method (summation form)
44:30 - Minimizing Prony's cost function (summation form)

0:00 - All pole modeling (deterministic signal modeling)
3:53 - Prony's cost for all pole modeling
19:45 - Deterministic auto-correlation
21:15 - How/when deterministic and stochastic modeling overlap
26:06 - All pole modeling with finite data records
28:00 - Auto-correlation method
34:01 - Covariance method
41:10 - Comparison of auto-correlation & covariance methods
48:08 - Optimizing real-valued functions of complex variables (brief explanation)
58:46 - Link to paper: .pdf

0:00 - Estimation Problem
6:15 - Classification of Estimation Problems (non-random, random)
11:20 - Maximum likelihood estimation
17:13 - Non-random parameter estimation
17:24 - Example: x[n] = c + w[n], w[n]: AWGN, c: non-random, Find chat_ML.
23:35 - log-likelihood (example, cont'd)
30:20 - Properties of Estimators
30:35 - Bias (properties of estimators)
39:47 - Consistency (properties of estimators)
52:03 - Efficiency (properties of estimators)
1:05:20 - Example: CRB for a parameter non-linearly related with observations (illustration)
1:10:01 - Asymptotical efficiency (properties of estimators)
1:12:27 - Folk's Theorem: ML is an asymptotically unbiased and efficient estimator

0:00 - Random Parameter Estimation
1:50 - Cost function (square or absolute error)
5:10 - Risk (definition)
12:00 - Conditional mean as a minimizer of MSE (derivation)
21:48 - Regression line
23:48 - Properties of Conditional Mean Estimator
24:04 - Property 1: Conditional mean vector for multiple parameter estimation
30:36 - Property 2: Orthogonality of estimation error to non-linear processing of observations
48:45 - Example: f(x,y) uniform in 1x1 area in 1st and 3rd quadrants Find yhat(x).
52:20 - Estimator derivation - conditional mean estimator - (example)
52:20 - min MSE value calculation (example)

00:00 - Example: f(x,y) uniform in 1x1 square in 1st and 3rd quadrants (revisited, Lec.30)
05:15 - Another proof min MSE via orthogonality (example)
17:22 - Uncorrelated check for error of min.MSE estimator with x^k, k:odd!
29:10 - Example: x = y + n, y and n zero-mean jointly Gaussian, find min. MSE yhat(x).
48:50 - Conclusion #1: For jointly gaussian rv's, the posterior density is Gaussian
50:44 - Conclusion #2: Non-zero mean case is a simple adaptation of zero mean case

Document for the proof of Conclusion #1: .pdf

Corrections:
12:13 - Unif([a,b+\Delta]) should be Unif([a,a+\Delta]) (Thanks Ugur Berk S.)

18:57 - For the explanations given on the board "k: arbitrary integer" should be "k: arbitary **odd** integer" (the results are trivially valid for even valued k, since f(x,y) = f(-x,-y), i.e. symmetric wrt to origin and many integrals with even valued k is equal to zero due to this symmetry) (Thanks Ugur Berk S.)

20:23 - sgn(x) x^k = |x|^k is only valid for odd valued k (Thanks Ugur Berk S.) For even valued say k = 2m, 1. E{ sgn(x) x^{2m} } = 0. Since, sgn(x) x^k = sgn(x) x^{2m} is an odd function of "x": Hence E{ sgn(x) x^{2m} } = \int f(x) sgn(x) x^{2m} dx = 0 since f(x) is an even function (i.e. symmetric wrt to origin f(x) = f(-x)) and then the integrand is an odd function. 2. E{ yx^{2m} } = 0. Since g(x,y) = yx^{2m} is odd symmetric with respect to the origin (g(x,y) = - g(-x,-y)); hence, \int \int f(x,y) g(x,y) dx dy = 0. This equality can be verified by a simple exchange of variables x2 = -x, y2=-y, then \int \int f(x,y) g(x,y) dx dy = - \int \int f(x2,y2) g(x2,y2) dx2 dy2 which gives the result of integration as 0. Hence all the results given on the board assume "k" is taken as an odd integer. I am explaining this a bit more in a later lecture, that is in 6:35 of Lec. 35a, but I should have clarified this during this lecture, indeed sorry CC)

00:00 - Linear Minimum Mean Square Error (LMMSE) Estimators
02:00 - Example: LMMSE estimator in the form yhat = ax + b
05:04 - Deriving normal equations, by partial diff. (example)
10:59 - Solving normal equations (example)
16:40 - LMMSE calculation (example)
27:55 - Comment #1: LMMSE are parametric estimators not as good as min. MSE est. in general
29:40 - Comment #2: LMMSE est. = min. MSE est for jointly Gaussian observations and desired r.v.
30:56 - Comment #3: LMMSE is used in practice, since we only have moment estimates at our disposal
33:51 - 2nd derivation (more general) for normal Equations (gradient calc.)
42:35 - Normal equation: R_x w = r_xy
45:31 - 2nd derivation (more general) for LMMSE value
51:01 - Example: LMMSE estimator in the form yhat = ax + b (revisited)

00:00:00 - LMMSE Estimation (summary)
00:03:32 - Example: x[n] = c + w[n], c and w[n] uncorrelated zero mean r.v.'s, n={1, ... , N}
00:05:20 - Discussing on problem set-up (example)
00:09:34 - Writing normal equations (example)
00:19:03 - Introducing SNR definition (example)
00:23:00 - Solving normal equations (example)
00:31:40 - Calculating LMMSE value (example)
00:34:30 - Comments: special case of infinite SNR (example)
00:34:48 - Comments: N observations with SNR = 1 or 1 observation with SNR = N
00:37:04 - Example: x[n] = c + w[n], c (non-zero mean) and w[n] uncorrelated r.v.'s, n={1, ... , N}
00:40:14 - Introducing SNR definition (example)
00:41:50 - Solution of normal equations (example)
00:42:57 - Is LMMSE estimator for this problem biased? (example)
00:47:17 - Removing bias with an affine estimator (example)
00:53:15 - Writing normal equations for affine estimator (example)
00:59:05 - Solving normal equations for affine estimator (example)
01:00:32 - Finalizing the estimator expression (example)

Corrections:
22:33 - NxN entry of the matrix should be 1 + 1/SNR_N (not 1) (Thanks to Ugur Berk S.)

38:50 - E{ \sigma_w1^2 } should be E{ w_1^2} in E{x_1^2 } = E{c^2} + E{ \sigma_w1^2 } (Thanks to Ege. E)

00:00 - Example: xvec = pvec \times c + nvec; pvec: known vector; c and nvec r.v.'s
02:33 - Derivation of Normal Equations for LMMSE est. (complex valued case)
13:02 - Derivation of LMMSE value (complex valued case)
22:57 : Link to paper: .pdf
25:03 - Discussion on the application related with example
26:45 - Solution for Identity Noise Cov. Mat. (example)
30:38 - Matrix inversion lemma
38:11 - Expression for the estimator (example)
41:27- Solution for General Noise Cov. Mat. (example)
42:42 - Solution by whitening (example)

Matrix Inversion Lemma: Wikipedia link

00:00 - Example: x,y unif. dist. in 1x1 square in 1st and 3 quadrants (Lec.30, revisited)
01:01 - Min. MSE Linear/Affine estimator yhat(x) (example)
06:30 - E{yx^k} and E{x^k} expressions (example)
15:05 - Using powers of the observation value in the min. MSE estimator (example)
25:28 - min. Cubic MSE estimator and its MSE value (example)
29:20 - Higher Order Estimators, Matlab Results (example) link: https://www.youtube.com/watch?v=tnlHE6sVOe0
32:38 - Properties of LMMSE Estimators
32:43 - Property 1: Geometric (Vector Space) Interpretation
42:17 - Property 2: Multiple r.v. estimation (Total MSE minimization)
54:17 - Estimator expression for total MSE minimization (property 2)
55:34 - Example: xvec = Hyvec + n. Find LMMSE estimator for xvec. (property 2)

Link to supplementary video for MATLAB content: https://www.youtube.com/watch?v=tnlHE6sVOe0

00:00 - Property 3: Linear combination of observations as input to LMMSE estimation
04:28 - Recursive estimation discussion
23:37 - Recursive estimator expression
24:04 - Innovation
25:15 - Property 4: LMMSE estimation of a linear combination of desired r.v.'s from same set of observations

Corrections:
28:50 - zhat = M yhat should be zhat = N yhat (Thanks Ugur Berk S.)

00:00 - Wiener Filtering (Problem Setup)
03:08 - FIR Wiener Filtering
04:42 - Deriving Wiener-Hopf (or normal) Equations
22:06 - Min.MSE calculation
25:00 - Example: x[n] = d[n] + v[n], r_d[k] = \alpha^{|k|} , r_v[k] = \sigma_v^2 \delta[k]
32:25 - Writing Normal equations for 2-tap filter (example)
36:50 - Calculation of LMMSE value for 2-tap filter (example)
40:42 - 1-tap FIR Wiener filter derivaiton (example)
45:06 - SNR before and after Wiener filtering (example)
51:30 - SNR-after with 1-tap Wiener filter (example)
53:04 - SNR-after with 2-tap Wiener filter (example)
59:10 - What is the maximum SNR-after with 2-tap filter? (example)
59:30 - Max. SNR filter derivation (example)

00:00 - FIR Wiener filtering (review)
4:30 - Categorization of Signal Processing Operations
4:47 - Filtering (categorization)
6:15 - Smoothing (categorization)
13:20 - Prediction (categorization)
19:50 - Example: Two-tap linear predictor for x[n] with r_x[k] = \alpha^|k|
32:34 - Comments: Prediction error as uncorrelated part of the observation
40:10 - Comments: Prediction error filter
42:30 - Backward prediction
47:25 - Auto-correlation of time-reversed WSS process (backward prediction)
50:20 - Remark: Auto-correlation matrix estimation with time-reversed vectors for WSS processes
54:17- Auto-correlation matrix for the samples of WSS processes (remark)
58:47 - Expression for the auto-correlation matrix estimation for WSS process samples (remark)

00:00:00 - FIR Wiener Filtering (review)
00:02:10 - IIR Non-Causal Wiener Filtering
00:10:34 - IIR Non-Causal Wiener filter expression in Fourier domain
00:12:15 - MSE expression for IIR Non-Causal Wiener Filter
00:23:00 - Filtering application (noise removal) for IIR Non-Causal Wiener filtering
00:26:55 - Comment #1: Impulse response of IIR Non-Causal Wiener filter is an even sequence
00:28:20 - Comment #2: IIR Non-Causal Wiener filter processes each spectrum sample independent of other samples
00:30:26 - Connection between Comment #1 and the forward-backward time invariance of WSS processes (Lec. 37)
00:35:51 - Connection between Comment #2 and the fact that FT decorrelates WSS processes (Lec 27b)
00:37:55 - Pictorial interpretation of IIR Non-Causal Wiener filtering for filtering (noise removal) application
00:41:44 - MSE calculation for IIR Non-Causal Wiener filtering for filtering (noise removal) application
00:48:20 - Example: x[n] = d[n] + v[n], r_d[k] = 0.8^|k| , r_v[k] = \delta_v[k], Find IIR NC Wiener filter to estimate d[n].
1:07:05 - MSE calculation for IIR Non-Causal Wiener filter (example)

00:00 - IIR Causal Wiener Filtering
02:13 - Deriving normal equations by differentiation
06:54 - Special case: Input is white noise
09:57 - General case: Decorrelating input to generate white noise input artificially
30:03 - IIR Causal Wiener Filter in z-domain
37:06 - Example: x[n] = d[n] + v[n], r_d[k] = 0.8^|k|, r_v[k] = \sigma_v^2\delta[k]
01:09:44 - Summary of results for 1 Tap, 2 Tap, Causal IIR, Non-Causal IIR Wiener filter result for the same example

Corrections:
37:10 - r_v[k] should be r_v[k] = \sigma_v^2 \delta[k] (white noise)

00:00 - Ergodicity
03:00 - Mean ergodicity
04:02 - Sample mean estimator
08:30 - Question on the convergence issues for sample mean estimator as N goes to infinity.
12:20 - Unbiasedness of sample mean estimator
14:01 - Consistency of sample mean estimator
27:10 - Necessary and Sufficient Condition for mean ergodicity
29:04 - An equivalent necessary and sufficient condition for mean ergodicity
29:40 - Sufficient condition for mean ergodicity
33:06 - Example: Mean-ergodic and not mean-ergodic r.p. example
49:00 - Ergodicity in auto-correlation
54:05 - Biased and unbiased estimators for auto-correlation
58:10 - Biased auto-correlation estimators always gives a valid auto-correlation sequence
01:06:10 - Consistency of auto-correlation estimator

00:00:00 - Best Linear Unbiased Estimator (BLUE)
00:01:15 - Illustration of BLUE on a simple sample: x_k = c + w_k, k={1,2}, Find chat_BLUE.
00:07:11 - MSE expression for chat (illustration)
00:09:41 - Comment on MSE: Unrealizable estimator unless estimator is unbiased (illustration)
00:12:15 - Optimization problem for minimum MSE unbiased estimator (illustration)
00:19:15 - Solution for uncorrelated noise (illustration)
00:22:34 - Precision = 1/noise-variance definition (illustration)
00:28:45 - Comparison with a similar random parameter estimation problem (Lec.33 link : youtube-link)
00:31:04 - General Case for the BLUE estimator
00:33:47 - Total MSE derivation (general case)
00:36:26 - Condition for the unbiased estimator (general case)
00:41:07 - Total MSE expression (general case)
00:44:17 - Optimization problem for BLUE estimator (general case)
00:47:23 - BLUE Estimator expression (general case)
00:49:45 - Total MSE expression for BLUE estimator (general case)
00:51:37 - Special case #1: White noise (Rn = \sigma_n^2 \times Identity )
00:54:30 - Comment on case #1: LS solution is the BLUE for white noise
00:55:55 - Special case #2: Non-white noise (Rn \neq \sigma_n^2 \times Identity )
01:02:16 - Comment on case #2: Whitened LS solution is the BLUE estimator
01:03:30 - Revisiting earlier illustrative example
01:08:26 - Comparison with random parameter case, LMMSE estimation (general case)

00:00:00 - Introduction to the end of EE 503!
00:00:30 - Karhunen Loeve (KL) Transform
00:03:57 - Desired properties for the KL transformation
00:06:28 - KL Transformation for 1-dimensional approximation (1D case)
00:14:25 - MSE expression for the problem (1D case)
00:23:23 - MSE minimizing solution (1D case)
00:25:15 - Eigenvectors of R_x matrix (reminder)
00:29:16 - Solution for optimal sub-space (1D case)
00:32:00 - min. MSE expression (1D case)
00:33:48 - KL Transformation for 2-dimensional approximation (2D case)
00:38:56 - Reduction to the 1D approximation case (2D case)
00:56:14 - Solution for optimal sub-space (2D case)
00:57:27 - min. MSE expression (2D case)
01:03:30 - Comment on the uncorrelatedness of expansion coefficients
01:05:55 - KL transformation (General case)
01:07:16 - Comment: Application Example: Signal Compression
01:14:34 - Comment: Case of WSS processes (Lec. 23b)
01:32:26 - Goodbye!