01 - Derivative

Total Derivative

Function $\Omega \subseteq {\mathbb{R}}^n\to {\mathbb{R}}^m$ is differentiable at $\underline{a}\in \Omega$ if

There exists Linear Map $L:{\mathbb{R}}^n\to {\mathbb{R}}^m$ such that

$$\underset{\underline{h}\to \underline{0}}{\lim }\frac{f(\underline{a}+\underline{h})-f(\underline{a})-\underline{L}h}{|\underline{h}|}=\underline{0}$$

$L$ is known as the (total) derivative of $f$ at $\underline{a}$ denoted as $df(\underline{a})$

Differentiable Function in $\Omega$

If $f$ is differentiable at every point $\underline{a}\in \Omega$

Recall that the matrix for linear map $L$ with respect to the usual bases of ${\mathbb{R}}^n$ and ${\mathbb{R}}^m$ is the Jacobian Matrix

Proof - Usual Definition of Derivative not enough for $n>1$

Assume that $f$ is differentiable at $\underline{a}$ then
Define

$$R(\underline{h}):=\frac{f(\underline{a}+\underline{h})-f(\underline{a})-L\underline{h}}{|\underline{h}|}\in {\mathbb{R}}^m$$

Then

$$f(\underline{a}+\underline{h})-f(\underline{a})=\underline{L}h+R(\underline{h})|\underline{h}|$$

So for $\underline{h}\ne \underline{0}$

$$\frac{f(\underline{a}+\underline{h})-f(\underline{a})}{|\underline{h}|}=L\left(\frac{\underline{h}}{|\underline{h}|}\right)+R(\underline{h})$$

Where now $R(\underline{h})\to \underline{0}$ as $\underline{h}\to \underline{0}$
Letting $\underline{h}\to \underline{0}$ and as $L$ is independent of $\underline{h}$, then for $\underline{h}\ne \underline{0}$

$$\underset{\underline{h}\to \underline{0}}{\lim }\frac{f(\underline{a}+\underline{h})-f(\underline{a})}{|\underline{h}|}=L\underset{\underline{h}\to \underline{0}}{\lim }\left(\frac{\underline{h}}{|\underline{h}|}\right)$$

Assuming that both limits exists (if one does so does the other)

Case for $n=1$ For $n=1$ then $\underline{h}=h$ so $\frac{\underline{h}}{|\underline{h}|}=1\text{ or }-1$ Regardless of the sign, the limit exists so we get

$$\underset{h\to 0}{\lim }\frac{f(a+h)-f(a)}{h}=L$$

Now $L$ is the usual derivative but just interpreted as a linear map on $\mathbb{R}$

However for $n=2$ then $\frac{\underline{h}}{|\underline{h}|}$ is a unit length vector in direction of $\underline{h}\ne 0$
As there is complete freedom in $\underline{h}\to \underline{0}$, then any such sequence for $\underline{h}$ works

Hence the usual definition of the derivative is inadequate for $n>1$

---

Directional Derivatives

Let $\underline{n}={\mathbb{R}}^n$ be a unit vector then

$f:\Omega \subseteq {\mathbb{R}}^n\to {\mathbb{R}}^m$ is differentiable at $\underline{a}\in \Omega$ in direction $\underline{n}$ if

$$\underset{\lambda \to 0}{\lim }\frac{f(\underline{a}+\lambda \underline{n})-f(\underline{a})}{\lambda }\text{ exists}$$

Proof - Relation between total derivatives and directional derivatives

Assume that $f$ is (totally) differentiable at $\underline{a}$
Restrict $\underline{h}=\lambda \underline{n}$ for $\lambda \in \mathbb{R}$ so

$$\underset{\underline{h}\to \underline{0}}{\lim }\left(\frac{\underline{h}}{|\underline{h}|}\right)=\underline{n}\text{ does now exist}$$

Hence

$$\underset{\lambda \to 0}{\lim }\frac{f(\underline{a}+\lambda \underline{n})-f(\underline{a})}{\lambda }=L\underline{n}$$

So the derivative in direction $\underline{n}$ exists and given by applying matrix $L$ to unit vector $\underline{n}$

---

Partial Derivatives $n=2$ and $m=1$ such that we have

For simplicity consider

$$f:{\mathbb{R}}^2\to \mathbb{R}$$

Then for the directional derivatives consider the Basis Vectors $(1,0)$ and $(0,1)$

Suppose the directional derivatives exist then they are denoted (respectively)

$$f_x(a,b)\text{ and }{f}_y(a,b)$$

Assuming that $f:{\mathbb{R}}^2\to \mathbb{R}$ is differentiable at $\underline{a}=(a,b)$ then we have

$$df(\underline{a}):(1,0)\mapsto f_x(a,b)df(\underline{a}):(0,a)\mapsto f_y(a,b)$$

Writing $df(\underline{a})$ as a matrix with respect to the usual basis we get

$$df(\underline{a})=(f_x(a,b),f_y(a,b))$$

Relation between ... and ...

For $n=1\text{ or }2$ and $m=1$ then we either have

$$f:\Omega \subseteq \mathbb{R}\to \mathbb{R}\text{ or }f:\Omega \subseteq {\mathbb{R}}^2\to \mathbb{R}$$

This lets us visualise the graphs for

$$G_f=\{(x,f(x)):x\in \Omega \subseteq \mathbb{R}\}\text{ or }{G}_f=\{(x,y,f(x,y)):(x,y)\in \Omega \subseteq {\mathbb{R}}^2\}$$

around the point $(\underline{a},f(\underline{a}))$

For $n=1$ then

$$f(a+h)=f(a)+Lh+R(h)|h|$$

where $L=f_x(a)$ is the derivative and $R(h)\to 0$ as $h\to 0$
So we get tangent line

$$f(a)+Lh\text{ is a good approximation for}f(x)\text{ near }x=a$$