Bayes optimal predictor for regression




     


What is Bayes predictor?

What  is the optimal performance, regardless of the finiteness of
the training data? In other words, what should be done if we have
a perfect knowledge of the underlying probability distribution of
the data? This is what we learn in decision theory:  loss,  risk,
optimal  predictors.  What  is the optimal prediction and perfor‐
mance given infinite data and infinite computational resources?

What is Bayes classifier?

What is Bayes risk?

What is the connection between  the  squared  loss  function  and
Bayes predictor?

What  does  this mean? "Clearly, we cannot hope that the learning
algorithm will find a hypothesis whose error is smaller than  the
minimal possible error, that of the Bayes predictor."