Numerical Methods For MAT 120: Name Email
Numerical Methods For MAT 120: Name Email
Name Email
Irfan Sadat Ameen [email protected]
Syed Emad Uddin Shubha [email protected]
Fatema Tuz Zohora [email protected]
Rakibul Alam Shamim [email protected]
Krity Haque Charu [email protected]
Sarah Zabeen [email protected]
Reaz Shafqat [email protected]
Omer Tahsin [email protected]
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.3 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4.1 Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4.2 Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.3 Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.4 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2 Numerical Differentiation 15
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.1.3 Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.4 Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.1.1 Example: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.2.1 Example: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
1.1 Introduction
Roots of a function means the solutions of f ( x ) = 0. Except for few specific functions, it is
generally not feasible to obtain an exact analytical expression for the root, thus preventing the
precise determination of the solution. We know, for example, that the solution of an equation
of the form √
2 −b ± b2 − 4ac
ax + bx + c = 0 is x= . (1.1)
2a
A analytic formula like this does not exist for polynomials of all degrees. Such problem can be
solved by approximating a solution using an iterative method. We can solve such problems
numerically using few algorithms.
The bisection method is based on the Bolzano’s Theorem which states that
If a continuous function has values of opposite sign inside an interval, then it has a root in
that interval.
If a continuous function changes sign over an interval, we can narrow down the approximate
solution by using the midpoint of the interval and searching to the left and right of it (similar
to a binary search). This can be broken down into a few steps:
1
1. Confirm that a root lies between the interval by checking if f ( a) f (b) < 0. Otherwise,
choose different interval.
x0 = ( a + b)/2. (1.2)
4. Repeat the process from step 2 until the width, b − a, is less than a very small number,
for example ϵ = 0.001.
Figure 1.1: Figure: Bisection Method applied to a function f ( x ) with initial guesses as a and
b.
2
1.2.1 Example
Problem
f ( x ) = x3 + x2 + x − 8 (1.3)
within the interval [1, 2] using the bisection method with an accuracy of ε = 0.1.
Solution
We apply the bisection method step-by-step to narrow down the interval until it is smaller
than the required accuracy.
f ( 1 ) = 13 + 12 + 1 − 8 = − 5
f ( 2 ) = 23 + 22 + 2 − 8 = 6
Since f (1) = −5 and f (2) = 6 have opposite signs, there is at least one root in ([1, 2].
We will iteratively bisect the interval and evaluate f ( x ) at the midpoint until the interval width
is smaller than ε = 0.1.
Iteration 1
1+2
x0 = = 1.5
2
f (1.5) = 1.53 + 1.52 + 1.5 − 8 = −0.875
Since f (1.5) = −0.875 and f (2) = 6 have opposite signs, the root is in [1.5, 2].
3
Iteration 2
1.5 + 2
x0 = = 1.75
2
f (1.75) = 1.753 + 1.752 + 1.75 − 8 = 2.171875
Since f (1.5) = −0.875 and f (1.75) = 2.171875 have opposite signs, the root is in [1.5, 1.75].
Iteration 3
1.5 + 1.75
x0 = = 1.625
2
f (1.625) = 1.6253 + 1.6252 + 1.625 − 8 = −0.435546875
Since f (1.625) = −0.435546875 and f (1.75) = 2.171875 have opposite signs, the root is in
[1.625, 1.75].
Iteration 4
1.625 + 1.75
x0 = = 1.6875
2
f (1.6875) = 1.68753 + 1.68752 + 1.6875 − 8 = 0.340576171875
Since f (1.625) = −0.435546875 and f (1.6875) = 0.340576171875 have opposite signs, the root
is in [1.625, 1.6875].
Iteration 5
1.625 + 1.6875
x0 = = 1.65625
2
f (1.65625) = 1.656253 + 1.656252 + 1.65625 − 8 = −0.0494384765625
4
Since f (1.65625) = −0.0494384765625 and f (1.6875) = 0.340576171875 have opposite signs,
the root is in [1.65625, 1.6875].
Approximate Root
which is less than the required accuracy of 0.1. Therefore, we stop here and take the midpoint
as an approximate root:
Approximate root ≈ 1.671875.
def bisection(expr,a,b):
#Check for wrong input:
if expr.subs(x,a)*expr.subs(x,b) >= 0:
print("Invalid values.")
else:
while (b-a) >= eps:
#Find the midpoint of a and b:
x0 = (a+b)/2
#set condition to see if root lies left or right
if expr.subs(x,a)*expr.subs(x,x0) < 0:
b = x0
elif expr.subs(x,b)*expr.subs(x,x0) < 0:
a = x0
print("The root is ", x0)
return
5
1.2.3 Exercises
One can find the root of a continuous function using another method known as the Newton-
Raphson method. This method does not require that the root must lie within an interval, [ a, b],
rather an arbitrary point, x a , is chosen and a root (if there is one) that lies close to that point
will be discovered. The method starts with the point x a and a slope at point is taken. The point
where the slope crosses the x − axis is found. This point becomes the new value for x a .
Let x a be a good estimate of r and let r = x a + h. Since the true root is r, and h = r − x a , the
number h measures how far the estimate x a is from the truth. Since h is ’small’,
f ′′ ( x a )
f (r ) = f ( x a + h ) ≈ f ( x a ) + f ′ ( x a ) + h + ... = 0. (1.4)
2!
6
and therefore, as x a is an approximate and it is not equal to the root, f ( x a ) is close to 0,
f (xa )
h≈− . (1.5)
f ′ (xa )
f ( xn )
x n +1 = x n − . (1.7)
f ′ ( xn )
df
Here, f ′ = dx . This iterative process will continue until | f ( xn+1 )| < ϵ.
Figure 1.2: Figure: Demonstration of Newton-Raphson method. xn moves closer to the actual
root per iteration.
1.3.1 Example
Problem
f ( x ) = x2 − 4
starting from an initial guess of x0 = 5 using Newton’s method, with an accuracy of ε = 0.1.
7
Solution
f ( xn )
x n +1 = x n − (1.8)
f ′ ( xn )
f ( x ) = x2 − 4,
, we have
f ′ ( x ) = 2x.
Starting with x0 = 5, we will compute each successive approximation xn+1 until the difference
between consecutive approximations is less than ε = 0.1.
Iteration 1
f ( x0 )
x1 = x0 −
f ′ ( x0 )
52 − 4
= 5−
2·5
21
= 5−
10
= 2.9
Iteration 2
f ( x1 )
x2 = x1 −
f ′ ( x1 )
2.92 − 4
= 2.9 −
2 · 2.9
4.41
= 2.9 −
5.8
≈ 2.1397
8
Iteration 3
f ( x2 )
x3 = x2 −
f ′ ( x2 )
2.13972 − 4
= 2.1397 −
2 · 2.1397
≈ 2.0062
Iteration 4
f ( x3 )
x4 = x3 −
f ′ ( x3 )
2.00622 − 4
= 2.0062 −
2 · 2.0062
≈ 2.00002
Since the difference between x3 ≈ 2.0062 and x4 ≈ 2.00002 is less than ε = 0.1, we stop here.
Final Answer
9
1.3.2 Python Implementation
Here, f.subs(x, obj) will replace the symbol x in the expression f with the variable obj. This
variable can be of any type and will be replaced accordingly. If the substitute variable is an
integer or float, they will undergo the arithmetic operations in the expression and give us the
value. One can use sympy.lambdify() or define a lambda function and pass it as the argument
instead of an expression. In that case, .subs() is not required. The derivative of a symbolic
expression is taken, which is done analytically, using sympy.diff() where the first argument
is the expression and the second is the variable with respect to which the derivative is taken.
1.3.3 Exercise
10
(d) Higher Degree Polynomial (Medium-Hard)
Use the Newton-Raphson Method to approximate a root for f ( x ) = x4 − x − 10 with an
initial guess of x0 = 2. Perform four iterations and track the convergence of the solution.
The Fixed Point Iteration method is an iterative process to find the solution of equations of the
form:
x = f (x)
x n +1 = f ( x n )
This process is repeated until the sequence converges to a fixed point, i.e., when xn+1 ≈ xn .
1.4.1 Example 1
Consider the equation x3 − x + 1 = 0. We can rearrange this equation in two different ways:
√
3
• Using x = g( x ) = x − 1: Starting with an initial guess x0 , the iteration becomes:
p
3
x n +1 = g ( x n ) = xn − 1
So this seems to converge at a point. If we iterate it for a few more steps, we will reach
x15 = −1.3247179572395291, which is accurate up to 10 decimal points!
11
• Using x = f ( x ) = x3 + 1: Starting with an initial guess x0 , the iteration becomes:
xn+1 = f ( xn ) = xn3 + 1
x0 x1 x2 x3 x4 ...
-1 0 1 2 9 ...
So this is diverging! Why? We can guess that, since | x3 | > 1 for | x | > 1, at some point of
iteration, | x | exceed 1 and hence the sequence converges.
1.4.2 Example 2
Now say we want to find a real root of f ( x ) = x2 − x − 1. From our previous experience, we
√
may want to write it in the form x = x + 1. And if we start from any x0 ≥ −1, we will
√
1+ 5
approach the root x = 2 .
√
1− 5
However, there is another root x = 2 , which can never be obtained using this form (why?).
√
So we proceed with the form x = − x + 1 to find the negative root of f ( x ) and the iteration
is given by:
p
x n +1 = − xn + 1
x0 x1 x2 x3 x4 x5 ...
0 -1 0 -1 0 -1 ...
So this is neither diverging nor converging. The sequence is dancing between 0 and −1.
What if we start with x0 = 0.5? We will find x1 , but x2 , x3 , .. will be undefined.
However, starting with x0 = −0.5 yields:
x0 x1 x2 x3 x4 x5
-0.5 -0.70710678 -0.5411961 -0.67735 -0.5680223 -0.65725
√
1− 5
So the sequence seems approach x = 2 ≈ −0.618034.
Hence we conclude that xn+1 = f ( xn ) may generate a converging, diverging, or oscillating
sequence (or undefined values) depending on the initial point x0 and choice of f ( x ). We will
find out the reason soon.
12
1.4.3 Homework
Using the fixed point iteration method, we want to find the real solution of x3 + x2 − 1 = 0.
The actual solution is x ≈ 0.75487766624669276.
√
3
√
If we express it as x = f ( x ), we can have either f ( x ) = 1 − x2 or f ( x ) = 1 − x3 . Find x0
for both cases to generate sequences that converge, diverge, or oscillate.
1.4.4 Visualization
13
1.5 Banach Fixed-Point Theorem
From the previous examples, it should be apparent that f(x) plays a crucial role for the con-
√
vergence of the series. When f ( x ) = 1 + x3 , the sequence diverges, but f ( x ) = 3 x − 1 make
the sequence convergence. Why is so? The later seems to “contract” the sequence leading to a
fixed point. We call ϕ : [ a, b] → [ a, b] a contraction map if
From the definitions, it should be clear that the fixed point is a solution to x = f ( x ). Now the
Banach fixed-point theorem states that every contraction mapping on a non-empty complete
metric space has a unique fixed point, and for any x0 , the sequence xn+1 = f ( xn ) converges to
that fixed point, i.e.,
x∗ = ϕ lim xn = ϕ( x∗ )
n→∞
I won’t bore you with the proof but let’s define ϵk = xk − x∗ , where x∗ . Now,
x k +1 = ϵ k +1 + x ∗ = ϕ ( x k ) = ϕ ( ϵ k + x ∗ )
Note that ϵk is the error in each step, which will approach 0 if |ϕ′ ( x∗ )| < 1.
The Newton’s method is given by:
f ( xk )
x k +1 = x k −
f ′ ( xk )
f (x)
Let, g( x ) = x − f ′ (x)
, then xk+1 = g( xk ), which is just fixed point iteration!
14
Chapter 2
Numerical Differentiation
2.1 Introduction
• The function is known only at specific data points (e.g., experimental measurements).
1. Engineering Applications: Estimating the rate of heat transfer in a material when tem-
perature measurements are taken at discrete points.
3. Financial Modeling: Determining the rate of change of stock prices using historical data.
4. Biological Systems: Estimating population growth rates from periodic census data.
15
2.2 Key Aspects and Intuition on Taylor Expansion
The Taylor expansion is a powerful mathematical tool that expresses a function as an infinite
sum of terms calculated from the derivatives of the function at a single point. For a function
f ( x ) that is infinitely differentiable at a point x = x0 , the Taylor series around x0 is:
f ′′ ( x0 ) f ′′′ ( x0 )
f ( x ) = f ( x0 ) + f ′ ( x0 )( x − x0 ) + ( x − x0 )2 + ( x − x0 )3 + · · ·
2! 3!
• Foundation for Numerical Methods: Many numerical techniques, including finite dif-
ference methods for differentiation, are derived from the Taylor expansion.
• Error Estimation: By truncating the series after a finite number of terms, we introduce a
truncation error. The remainder term gives us an estimate of this error.
16
2.3.1 First-Order Derivative Approximations
f ′′ ( x ) 2 f ′′′ ( x ) 3
f ( x + h) = f ( x ) + f ′ ( x )h + h + h +···
2 6
f ( x + h) − f ( x ) f ′′ ( x ) f ′′′ ( x ) 2
f ′ (x) = − h− h +···
h 2 6
Approximation:
f ( x + h) − f ( x )
f ′ (x) ≈ with an error of O(h)
h
f ′′ ( x ) 2 f ′′′ ( x ) 3
f ( x + h) = f ( x ) + f ′ ( x )h + h + h +···
2 6
4 f ′′′ ( x ) 3
f ( x + 2h) = f ( x ) + 2 f ′ ( x )h + 2 f ′′ ( x )h2 + h +···
3
−4 f ( x + h) + 3 f ( x ) + f ( x + 2h) = −2 f ′ ( x )h + O(h3 )
Solve for f ′ ( x ):
−4 f ( x + h) + 3 f ( x ) + f ( x + 2h)
f ′ (x) = + O ( h2 )
2h
Approximation:
17
−4 f ( x + h) + 3 f ( x ) + f ( x + 2h)
f ′ (x) ≈ with an error of O(h2 )
2h
f ′′ ( x ) 2 f ′′′ ( x ) 3
f ( x − h) = f ( x ) − f ′ ( x )h + h − h +···
2 6
f ( x ) − f ( x − h) f ′′ ( x ) f ′′′ ( x ) 2
f ′ (x) = − h+ h +···
h 2 6
Approximation:
f ( x ) − f ( x − h)
f ′ (x) ≈ with an error of O(h)
h
f ′′ ( x ) 2 f ′′′ ( x ) 3
f ( x − h) = f ( x ) − f ′ ( x )h + h − h +···
2 6
4 f ′′′ ( x ) 3
f ( x − 2h) = f ( x ) − 2 f ′ ( x )h + 2 f ′′ ( x )h2 − h +···
3
4 f ( x − h) − 3 f ( x ) − f ( x − 2h) = 2 f ′ ( x )h + O(h3 )
Solve for f ′ ( x ):
4 f ( x − h) − 3 f ( x ) − f ( x − 2h)
f ′ (x) = + O ( h2 )
2h
Approximation:
18
4 f ( x − h) − 3 f ( x ) − f ( x − 2h)
f ′ (x) ≈ with an error of O(h2 )
2h
f ′′ ( x ) 2 f (3) ( x ) 3
f ( x + h) = f ( x ) + f ′ ( x )h + h + h +···
2 6
f ′′ ( x ) 2 f (3) ( x ) 3
f ( x − h) = f ( x ) − f ′ ( x )h + h − h +···
2 6
f (3) ( x ) 3
f ( x + h) − f ( x − h) = 2 f ′ ( x )h + h +···
3
f ( x + h) − f ( x − h) f (3) ( x ) 2
f ′ (x) = − h +···
2h 6
Approximation:
f ( x + h) − f ( x − h)
f ′ (x) ≈ with an error of O(h2 )
2h
f ′′ ( x ) 2 f (3) ( x ) 3
f ( x + h) = f ( x ) + f ′ ( x )h + h + h +···
2 6
4 f (3) ( x ) 3
f ( x + 2h) = f ( x ) + 2 f ′ ( x )h + 2 f ′′ ( x )h2 + h +···
3
19
Form a combination:
Solve for f ′′ ( x ):
f ( x + 2h) − 2 f ( x + h) + f ( x ) f (3) ( x )
f ′′ ( x ) = − h+···
h2 3
Approximation:
f ( x + 2h) − 2 f ( x + h) + f ( x )
f ′′ ( x ) ≈ with an error of O(h)
h2
f ′′ ( x ) 2 f (3) ( x ) 3
f ( x − h) = f ( x ) − f ′ ( x )h + h − h +···
2 6
4 f (3) ( x ) 3
f ( x − 2h) = f ( x ) − 2 f ′ ( x )h + 2 f ′′ ( x )h2 − h +···
3
Form a combination:
Solve for f ′′ ( x ):
f ( x ) − 2 f ( x − h) + f ( x − 2h) f (3) ( x )
f ′′ ( x ) = − h+···
h2 3
Approximation:
f ( x ) − 2 f ( x − h) + f ( x − 2h)
f ′′ ( x ) ≈ with an error of O(h)
h2
20
Central Difference Scheme
f ′′ ( x ) 2 f (3) ( x ) 3 f (4) ( x ) 4
f ( x + h) = f ( x ) + f ′ ( x )h + h + h + h +···
2 6 24
f ′′ ( x ) 2 f (3) ( x ) 3 f (4) ( x ) 4
f ( x − h) = f ( x ) − f ′ ( x )h + h − h + h −···
2 6 24
f (4) ( x ) 4
f ( x + h) + f ( x − h) = 2 f ( x ) + f ′′ ( x )h2 + h +···
12
f ( x + h) − 2 f ( x ) + f ( x − h) f (4) ( x ) 2
f ′′ ( x ) = − h +···
h2 12
Approximation:
f ( x + h) − 2 f ( x ) + f ( x − h)
f ′′ ( x ) ≈ with an error of O(h2 )
h2
21
2.5 Conclusion
Understanding the truncation errors associated with each scheme allows you to choose the
most appropriate method for your specific application. The central difference scheme gener-
ally offers higher accuracy (O(h2 )) compared to forward and backward schemes (O(h)) for the
same step size h.
By mastering these techniques, you can effectively tackle problems in engineering, physics,
finance, and other fields where analytical differentiation is impractical.
2. “Applied Numerical Methods with MATLAB for Engineers and Scientists” by Steven
C. Chapra: This text offers practical insights into implementing numerical methods using
MATLAB.
3. Online Resources:
• https://ocw.mit.edu/courses/find-by-topic/
cat=engineering&subcat=civilengineering&spec=computationalmethods
MIT OpenCourseWare: Numerical Methods
Feel free to delve into these resources to deepen your understanding of numerical differentia-
tion and its applications.
22
Chapter 3
The trapezoidal rule is a method used to approximate the definite integral of a function by
summing the areas of trapezoids under the curve. This is particularly useful when the func-
tion is complex or its integral cannot be determined analytically. In such cases, numerical
integration methods like the trapezoidal rule provide a practical solution.
Consider a continuous function f ( x ) defined over the interval [ a, b]. To approximate the inte-
´b
gral a f ( x ) dx, we divide the interval [ a, b] into n subintervals of equal width h = b− a
n .
Let the points dividing the interval be x0 = a, x1 = a + h, x2 = a + 2h, ..., xn = b. For each
subinterval, we approximate the area under the curve by the area of a trapezoid, where the
height of the trapezoid is h, and the two parallel sides are the function values f ( xi ) and f ( xi+1 ).
h
Area of trapezoid = [ f ( xi ) + f ( xi+1 )] (3.1)
2
23
Figure 3.1: Approximating the area under a curve using trapezoids.
Summing the areas of all trapezoids gives us the approximation for the integral. We calculate
the area of each trapezoid individually and sum them:
h h h
( f ( x0 ) + f ( x1 )) + ( f ( x1 ) + f ( x2 )) + · · · + ( f ( xn−1 ) + f ( xn ))
2 2 2
h
= ( f ( x0 ) + 2 f ( x1 ) + 2 f ( x2 ) + · · · + 2 f ( xn−1 ) + f ( xn ))
2
This is the general form of the trapezoidal rule formula. The factor of 2 appears for the function
values that are shared between adjacent trapezoids (i.e., f ( x1 ) to f ( xn−1 )).
ˆ b
h
f ( x ) dx ≈ ( f ( x0 ) + 2 f ( x1 ) + 2 f ( x2 ) + · · · + 2 f ( xn−1 ) + f ( xn )) (3.2)
a 2
24
ˆ b n −1
!
h
a
f ( x ) dx ≈
2
f ( x0 ) + 2 ∑ f ( xi ) + f ( x n ) (3.3)
i =1
b− a
Where h = n and xi = a + i · h. The use of smaller sub-intervals improves the accuracy of
the approximation.
1. Determine the interval: Choose the interval [ a, b] over which the integration is to be
performed.
b− a
3. Calculate the step size h: The width of each sub-interval is given by h = n .
5. Apply the composite trapezoidal formula: Plug in the values into the trapezoidal rule
formula.
3.1.3 Example 1
´1
Consider the function f ( x ) = x2 , and we want to approximate the integral 0 x2 dx using 4
sub-intervals.
• a = 0, b = 1, n = 4
1−0
• h= 4 = 0.25
• Apply formula:
ˆ 1
0.25
x2 dx ≈ (0 + 2(0.0625 + 0.25 + 0.5625) + 1) = 0.34375
0 2
25
3.1.4 Example 2
Let’s consider the function f ( x ) = sin( x ) over the interval [0, π ]. Using 6 sub-intervals, the
width h is given by:
π−0 π
h= = (3.4)
6 6
Explanation of Sub-intervals: In this example, we are dividing the interval [0, π ] into 6 equal
π
sub-intervals. The width of each sub-interval is h = 6. This means that the points where the
function is evaluated (i.e., x0 , x1 , . . . , x6 ) are:
π 2π π 3π π
x0 = 0, x1 = , x2 = = , x3 = = , ..., x6 = π
6 6 3 6 2
These points correspond to the function values at which we apply the trapezoidal rule. For
example, f ( x1 ) = sin π6 , f ( x2 ) = sin π3 , and so on.
ˆ π π
6
h π π i
sin( x ) dx ≈ sin(0) + 2(sin( ) + sin( ) + . . . ) + sin(π ) = 2 (3.5)
0 2 6 3
´π
Figure 3.2: Trapezoidal Rule Approximation of 0 sin( x ) dx with 6 sub-intervals.
In the graph above, the blue curve represents the function f ( x ) = sin( x ), and the orange
trapezoids represent the areas approximated by the trapezoidal rule. By summing the areas of
the trapezoids, we obtain a numerical approximation of the integral.
26
3.1.5 Exercises
´2
2. Approximate 0 ( x3 − 3x + 2) dx using 5 sub-intervals.
´4
3. Approximate 1 ln( x ) dx using 6 sub-intervals.
4. A car’s velocity v(t) (in m/s) is measured at several intervals over a 10-second period.
The values are provided below. Estimate the total distance traveled by the car over 10
seconds using the trapezoidal rule with 5 sub-intervals.
Time t (s) 0 2 4 6 8 10
Velocity v(t) (m/s) 0 4 8 7 3 0
5. The cross-sectional area A(d) (in m2 ) of a reservoir is recorded at different depths d. The
data is provided below. Approximate the total volume of water in the reservoir using the
trapezoidal rule with 5 sub-intervals.
Depth d (m) 0 2 4 6 8 10
Area A(d) (m2 ) 200 180 140 100 50 10
3.1.6 Conclusion
The trapezoidal rule provides a simple yet powerful tool for approximating definite integrals,
particularly when an analytical solution is difficult to obtain. By dividing the interval into
smaller sub-intervals and applying the trapezoidal formula, we can achieve higher precision
in our approximations.
Monte Carlo integration is a technique for approximating the value of definite integrals us-
ing random sampling. It is particularly useful when traditional numerical methods (like the
trapezoidal or Simpson’s rule) are difficult to apply, especially in higher dimensions.
27
Monte Carlo integration is based on the principle of simulating random variables to estimate
the area under a curve. It can be applied to a wide variety of problems, particularly in physics,
statistics, and computer science.
In this section, we will discuss two popular methods for Monte Carlo integration:
In the averaging method, the idea is to randomly sample points within the interval of interest,
evaluate the function at those points, and then take the average of these function values to
estimate the integral.
Suppose we want to approximate the integral of a function f ( x ) over the interval [ a, b]:
ˆ b
I= f ( x ) dx
a
2. Evaluate the function: Evaluate the function f ( x ) at each of the sampled points.
N
1
N ∑ f ( xi )
i =1
4. Estimate the integral: Multiply the average by the length of the interval (b − a) to esti-
mate the integral:
N
1
I ≈ (b − a) ·
N ∑ f ( xi )
i =1
28
Why It Works
The averaging method is based on the law of large numbers, which states that as the number
of random points increases, the average of the values approaches the expected value of the
function. Since the integral is essentially the expected value of the function over the interval,
this method provides an accurate estimate of the integral as the number of points, N, increases.
Let’s use the averaging method to estimate the integral of f ( x ) = e x over the interval [0, 1].
Follow the steps outlined above:
ˆ 1
I= e x dx
0
1. Generate random points: - We randomly generate N = 1000 points uniformly in the interval
[0, 1]. These points are denoted as x1 , x2 , . . . , x N .
3. Compute the average: - Compute the average value of f ( xi ) for all the randomly sampled
points:
N
1
N ∑ ex i
i =1
After computing this sum, we find that the average value of f ( x ) at these random points is
approximately 1.718.
4. Estimate the integral: - Multiply the average value by the length of the interval (b − a),
which is 1, to estimate the integral:
I ≈ (1 − 0) · 1.718 = 1.718
ˆ 1
e x dx = e − 1 = 1.718
0
The estimated value is very close to the true value, illustrating the accuracy of the Monte Carlo
29
averaging method.
Figure 3.3: Monte Carlo Integration for f ( x ) = e x using the averaging method.
Let’s estimate the integral of f ( x ) = sin( x ) over the interval [0, π ] using the same steps.
ˆ π
I= sin( x ) dx
0
Steps:
1. Generate random points: - Randomly generate N = 1000 points uniformly in the interval
[0, π ].
2. Evaluate the function: - Evaluate f ( xi ) = sin( xi ) at each of the randomly sampled points.
3. Compute the average: - Compute the average of f ( xi ) over all sampled points:
N
1
N ∑ sin(xi )
i =1
4. Estimate the integral: - Multiply the average by the length of the interval (π − 0) to estimate
30
the integral:
I ≈ π · 0.637 = 2.001
The exact value of the integral is 2, so the Monte Carlo method provides a very close approxi-
mation.
Figure 3.4: Monte Carlo Integration for f ( x ) = sin( x ) using the averaging method.
In the sampling method, we take a different approach. Instead of just averaging function
values over an interval, we randomly sample points both in the x-axis and the y-axis. We
count how many points fall under the curve to estimate the integral.
´b
Let’s say we want to approximate the same integral I = a f ( x ) dx:
2. Count how many points ( xi , yi ) fall under the curve, i.e., where yi ≤ f ( xi ).
3. The ratio of points under the curve to the total points approximates the area under the
31
curve:
Let’s consider estimating the integral of f ( x ) = sin( x ) over the interval [0, π ] using the sam-
pling method.
- Suppose we generate 1000 random points in the 2D plane where x is distributed uniformly in
[0, π ] and y is distributed in [0, 1]. - Out of these 1000 points, we find that 700 points lie below
the curve y = sin( x ), and 300 points lie above the curve.
The total number of points under the curve is 700. Thus, the ratio of points under the curve to
the total points is:
700
= 0.7
1000
The length of the interval is π, and the maximum value of sin( x ) is 1, so the integral is approx-
imated as:
I ≈ π × 1 × 0.7 = 2.199
The exact value of the integral is 2, so this hypothetical example shows that we can closely
estimate the integral using the sampling method.
Figure 3.5: Monte Carlo Integration using the sampling method. Random points are sampled,
and the fraction of points under the curve is used to estimate the integral.
32
3.2.3 Comparison of Methods
Both the averaging and sampling methods are powerful, especially in higher dimensions,
where other numerical methods struggle. However, they may converge more slowly than
other techniques, particularly when the function is highly irregular.
• Averaging method: Easy to implement and can handle smooth functions very well. It is
generally preferred when we can sample points along the x-axis efficiently.
• Sampling method: More versatile, as it can be used to approximate the area under a
curve by counting points, even if the function has complex shapes.
3.2.4 Conclusion
Monte Carlo integration provides a flexible, powerful approach to estimating definite inte-
grals, especially in higher dimensions where other methods fall short. While it may not al-
ways be the fastest converging method, its simplicity and versatility make it an essential tool
in numerical analysis.
33
Chapter 4
Many ODEs cannot be solved analytically (i.e., with an exact solution). For such cases, we
use numerical methods like Euler’s method, Runge-Kutta methods, etc., to approximate the
solution.
The Euler’s method is one of the simplest and most basic numerical methods used to solve
ordinary differential equations (ODEs). Despite its simplicity, it is a foundational tool in nu-
merical analysis. It uses a straightforward iterative process to approximate the solution of a
differential equation.
dy( x )
= f ( x, y) (4.1)
dx
34
with the initial condition: y( x0 ) = y0 . Where f ( x, y) is any given function in the problem. We
have to solve for y( x ).
dy y ( xi + h ) − y ( xi )
≈
dx h
y i +1 − y i
= (4.2)
h
y i +1 = y i + h f ( x i , y i ) (4.4)
To use this formula we start with the initial condition given y( x0 ) = y0 at x0 . We can obtain
y1 ≡ y( x1 ) using Euler formula given by, y1 = y0 + h f ( x0 , y0 ). And then use this to find
y2 = y1 + h f ( x1 , y1 ) and so on. We need to define the step size h = xi+1 − xi which should be
a small number. The starting point is x0 and the end point is xn .
4.1.1 Example:
To illustrate Euler’s method, we consider the differential equation y′ = −y with the initial
condition y(0) = 1. Equation (4) with h = 0.01 gives,
The exact solution is y = e− x and from this the value at x = 0.04 is y = 0.9608
35
Problem 1: Decay Equation
Consider a sample of radioactive material with an initial N0 number of nuclei. As time passes,
some of the nuclei will decay into counterparts. Our job is to predict the number of nuclei
remaining at a given time. The decay equation follows as
dN
= −0.1N (4.5)
dt
Let’s solve it for t = (0, 50) and with initial condition N (0) = 500. Here, f (t, N ) = −0.1N and
we choose h = 1
Ni+1 = Ni + h f (ti , Ni )
= Ni − 0.1Ni
i ti Ni Ni+1 = 0.9Ni
0 t0 = 0 N0 = 500 N1 = 0.9 × 500 = 450.00
1 1 450.00 405.00
2 2 405.00 364.50
3 3 364.50 328.05
... ... ... ...
36
4.1.2 Second-Order ODE
d2 y dy
2
+ a + by + c = 0 (4.6)
dx dx
dy
Now, let’s define dx = v. With this, we have two first-order equations in our hand
dy
=v
dx
and,
dv
= −( av + by + c)
dx
Problem 2:
Damped Harmonic Oscillator: F = −kx − 2mλv
d2 x dx
2
+ λ + ω2 x = 0
dt dt
Let ω = 2, λ = 0.5 and the initial conditions: x (0) = 0 and v(0) = ẋ (0) = 10.
d2 x dx
+ + 4x = 0
dt2 dt
Solution:
dx
=v
dt
dv
= −( av + bx + c)
dt
37
Euler’s method for these two equations gives the following
xi+1 = xi + 0.1vi
To derive the local error for the Euler method, we start with the Taylor series expansion of the
true solution y( x + h):
h2 ′′ h3
y( x + h) = y( x ) + hy′ ( x ) + y ( x ) + y′′′ ( x ) + O(h4 ) (4.9)
2 6
38
In the Euler method given by (4.4), we approximate y( x + h) using:
y( x + h) ≈ y( x ) + h f ( x, y( x )).
y( x + h) ≈ y( x ) + hy′ ( x ).
As we can see this only involves up to the second term in (4.9) which is the first order in h
leaving out the second and higher order terms. This means the error is given by,
h2 ′′ h3
τ= y ( x ) + y′′′ ( x ) + O(h4 )
2 6
On the other hand, the global error is the cumulative effect of the local errors committed in
t n − t0
each step. But the number of steps n = h , which is proportional to 1h , and the error com-
mitted in each step is proportional to h2 . Thus, it is to be expected that the global error will be
proportional to h. The following figure depicts how the accuracy of this method improves as
we make h smaller.
Algorithm:
39
• Initial values x0 , y0 , step size h and number of steps N (or the end value for x)
• Set x = x0 , y = y0
• The result will be the final ( x, y) values or the entire sequence of ( x, y) pairs.
Python Template:
The goal is to approximate the value of the unknown function y( x ) at discrete points x0 , x1 , x2 , ...,
given an initial condition y( x0 ) = y0 and the ODE:
dy( x )
= f ( x, y) (4.10)
dx
The Runge-Kutta methods do this by estimating the slope (derivative) at multiple points be-
tween xn and xn+1 , and then combining them to give a weighted average slope, which im-
proves the accuracy over simpler methods like Euler’s.
The fourth order Runge-Kutta Method (RK4) is the most widely used form of Runge-Kutta
methods, because of its higher order of accuracy (O(h4 )).
k1 = h f ( xn , yn ) (4.11)
40
h 1
k2 = h f ( xn + , yn + k1 ) (4.12)
2 2
h 1
k3 = h f ( xn + , yn + k2 ) (4.13)
2 2
k4 = h f ( xn + h, yn + k3 ) (4.14)
1
yn+1 = yn + (k1 + 2k2 + 2k3 + k4 ) (4.15)
6
Where,
k1 : The slope at the beginning of the interval (Euler’s method approximation).
k2 : The slope at the midpoint using k1 to estimate y
k3 : Another slope at the midpoint, but now using k2
k4 : The slope at the end of the interval using k3
The final estimate for yn+1 is a weighted average of these four slopes.
4.2.1 Example:
dy
Given dx = 1 + y2 , where y = 0 when x = 0. Find y(0.2), y(0.4) and y(0.6)
k1 = 0.2
k2 = 0.2(1.01) = 0.202
and,
1
y(0.2) = 0 + (k1 + 2k2 + 2k3 + k4 ) = 0.2027
6
41
Problem 3: Investment Growth with Interest
Imagine you have an investment that grows continuously with a constant interest rate. The
equation for the growth of an investment over time is:
dy
= r·y
dt
Where:
This differential equation models the exponential growth of money due to interest.
To predict the value of the investment at future points in time, we can apply the RK4 method.
Let’s say the initial amount of money, y0 , is $1000, and the interest rate r = 5% or 0.05. We
want to know how the money grows over time with a time step h (e.g., 1 year).
The RK4 method uses four slopes to calculate the next value yn+1 from the current value yn .
These slopes are:
k1 = h · f (yn ) = h · (r · y n )
k1 k1
k2 = h · f yn + = h · r · yn +
2 2
k2 k2
k3 = h · f yn + = h · r · yn +
2 2
k4 =h · f (yn + k3 ) = h · (r · (yn + k3 ))
1
yn+1 = yn + (k1 + 2k2 + 2k3 + k4 )
6
Let’s calculate the investment’s value after one year using RK4 with a time step h = 1 and an
initial investment of $1000.
42
• Initial y0 = 1000, r = 0.05
• k1 = 1 · (0.05 · 1000) = 50
Then:
1
y1 = 1000 + (50 + 2(51.25) + 2(51.28125) + 52.5640625) ≈ 1051.271
6
After one year, the investment grows from $1000 to approximately $1051.27 using the RK4
method.
This example shows how RK4 can be used in a straightforward financial problem like invest-
ment growth with interest!
43
4.2.2 Algorithm and Code Template for RK4 Method
Algorithm:
• Set x = x0 , y = y0
• The result will be the final ( x, y) values or the entire sequence of ( x, y) pairs.
Python Template:
44