Calculus of a Single Variable, Chapter 3, 3.3, Section 3.3, Problem 48

You need to determine the relative extrema of the function, hence, you need to find the solutions to the equation f'(x) = 0.
You need to evaluate the derivative of the function, using the quotient rule:
f'(x) = (sin ' x * (1 + cos^2 x) - sin x*(1 + cos^2 x)')/((1 + cos^2 x)^2)
f'(x) = (cos x* (1 + cos^2 x) + sin x*(2sin x*cos x))/((1 + cos^2 x)^2)
f'(x) = (cos x* (1 + cos^2 x) + 2sin^2 x*cos x)/((1 + cos^2 x)^2)
Replace 1 - cos^2 x for sin^2 x, such that:
f'(x) = (cos x* (1 + cos^2 x)+ 2(1 - cos^2 x)*cos x)/((1 + cos^2 x)^2)
f'(x) = (cos x + cos^3 x + 2cos x- 2cos^3 x)/((1 + cos^2 x)^2)
f'(x) = (3cos x- cos^3 x)/((1 + cos^2 x)^2)
You need to solve for x the equation f'(x) = 0:
(3cos x- cos^3 x)/((1 + cos^2 x)^2) = 0
3cos x- cos^3 x = 0
Factoring out cos x :
cos x*(3 - cos^2 x) = 0 => cos x = 0 or 3 - cos^2 x = 0
The function cosine cancels on (0,2pi) at x = pi/2 and x = 3pi/2.
Solving for x the equation 3 - cos^2 x = 0 , yields:
3 - cos^2 x = 0 => 3 = cos^2 x => cos x = +-sqrt 3 impossible since cos x in [-1,1].
The function f'(x) is negative on intervals (pi/2, pi) and (pi,3pi/2) , and positive on (0,pi/2), (3pi/2,2pi) , hence, the function increases on (0,pi/2) U(3pi/2,2pi) , and it decreases on (pi/2, pi)U (pi,3pi/2). The function has relative extrema at (pi/2,f(pi/2)) and (3pi/2,f(3pi/2)).