In this section, I want to give a high-level overview
of the models we'll build on the CAS server:
logistic regression, support vector machines, decision
trees, random forests, gradient boosting, and neural networks.
Let's start with a simple data set
consisting of two interval inputs, X-1 and X-2,
along with a binary target, blue or yellow.
Although most real data will have more than two inputs,
it is useful to visualize how our models work
in two dimensions.
The intuition we build in this two-dimensional case
can be extended to higher dimensions
when working with real data.
Each point in the plot represents
a single observation from our data
set, with its coordinates corresponding
to the values of the input variables and the color
corresponding to the value of the target variable.
Each of our predictive modeling algorithms
will create a rule that is applied to this plot
to distinguish the blue dots from the yellow dots.
The simplest model we will consider
is logistic regression, which uses
a linear combination of the input variables
to predict the target.
With our two input variables, we get three model parameters:
an intercept and a weight for each input.
Instead of predicting an unbounded number
like linear regression, we want to predict a probability,
so we use the logit link function to convert the model
output from an unbounded number to a probability bounded
between zero and 1.
The model is used to estimate values
of p-hat, which is the probability of an observation
being an event or not.
In this case, p-hat tells us the probability
of an observation being yellow (the event of interest)
versus blue.
In the context of the two-dimensional plot,
the logistic regression model draws
lines on the plot corresponding to different probabilities
of an observation being yellow.
Each line represents a different probability.
After we choose a cutoff probability (for example,
0.50), we would say every observation above the line
for that probability is yellow and every observation below
that line is blue.
The angle and location of the lines in the plot
are determined by the parameter estimates,
which are found by maximizing the log-likelihood function.
The limitation of logistic regression
is that it can draw only straight lines in this plot,
even if the data would be better separated
by a more complicated geometry.
In higher dimensions, this problem
persists, although lines are replaced
by planes (in three dimensions) or hyperplanes
(in higher dimensions).
One way to overcome the straight-line limitation
and to introduce curved decision boundaries
is to include higher-order polynomial terms in the model.
A second-order polynomial regression model
would include all quadratic terms, so in our example,
we would have a parameter for X-1-squared
a parameter for X-2-squared and a parameter for X-1 times X-2.
Adding these terms to the model equation
enables the model to draw quadratic decision
boundaries rather than just linear boundaries.
Adding third-order terms (like X-1-cubed)
enables the model to draw cubic decision boundaries, and so on,
for higher-order terms.
These extra terms increase the model flexibility, possibly
leading to overfitting.
If we know that the inputs are nonlinearly related
to the target, it can be useful to add polynomial terms.
But it is important to still evaluate
performance on validation data to ensure that we're not
overfitting the data.
