All right. Let's continue talking about finite difference approximation to numerical derivatives. Now, what's this about? The very definition of a derivative seems to suggest an obvious approximation, where I just take my independent variable, give it the slide change, see how much my function changes, and divide one by another. This looks like a natural approximation to the derivative. Since by definition the derivative is the limit of this ratio, then it seems natural to take the step size as small as possible, and it seems like we're done. Although actually not quite. Let's see. Let's take a simple example. Here I took an example of a very simple function where I can compute the derivative exactly. That is the difference between the exact result and the two-point forward approximation from the previous slide versus the step size. Notice the log-log scale, this is just for convenience. There is quite a bit going on on the sketch. So let's take it step by step. First, if I start with a reasonable large step size and decrease it, the error seems to decrease as we would never expect. But then, it reaches a minimum. After which, it stops decreasing, it actually increases in a reasonably wild manner. So lesson one. Our naive expectation that we should take the step size as small as possible doesn't hope true. So what's going on here? By the looks of it, it seems that there are two factors at play. One is increasing, one is decreasing so that their combination, their sum has a minimum. So what can be these factors? It's worse to analyze the regimes of large and small step sizes separately. Let's start with a simple one with a regime of large h. So we see a linear trend, and the linear trend in a log-log scale means that our error is a [inaudible] as a function of h. This is because if a half y, being x_Alpha, log y is Alpha times log x. So that the exponent of the power law, gives a slope on the log-log scale. In this case, the slope seems to equal to one. So the error is actually linear in the step size. You can now understand that or in fact we can. Suppose that the function is smooth enough in the neighborhood of our point x, now expand it to the Taylor series. Let's say, expand it to Taylor series as a function of h, keeping x fixed. If we do this, we can easily see that our approximation, our two-step forward finite difference is our original derivative target plus this correction which is right to the curvature, to the second derivative of function f times the step h. So indeed, the error is linear in h just because we constructed the scheme in this way. This explains this linear bit. If these were the only factor, then the error would just keep decreasing. But that's not what it actually does, so there is something else. Let's see what this something else can be. Our two-point forward difference scheme is not the only possibility. We can cook up so-called central scheme. Here I take my function f, evaluate it twice, slide it to the right and to the left of my point x. You can think of this central finite difference scheme as a combination of a forward and a backward step, and half sum of the two. Then, if we expand these two values of f around point x, and combine the terms for the Taylor series, we see that because the scheme is symmetric, the term with the second derivative disappears, and the correction is related to the third derivative, and has the order of h square. Yeah, that's just a consequence of symmetry. Let's see. So the expectation is, if we evaluate the central finite difference, the correction at larger h will be quadratic. Let's test it at the same function that we had before x_11. Indeed, we see that the central scheme it is reddish, yellow points as a larger slope than the forward scheme. It looks like the slope is indeed two meaning, there is quadratic. So lesson two. By constructing schemes with higher order approximation, we can decrease this truncation error. We will talk about this a little bit more a little later. Let's keep staring at this graph. So for the central scheme, there is still a minimum, minimum for achievable error. There is still something going on at smaller values of h. Somehow in the range of very small h, these wider oscillations are the same for both central and forward difference scheme. So there must be something which is pretty and sensitive to the specifics of the scheme. Just something very robust and crude and something about just evaluations. Let's see what it can be.