Calculus of a Single Variable, Chapter 3, 3.3, Section 3.3, Problem 45

You need to find the open intervals on which the function is increasing or decreasing, hence, you need to find where the derivative is positive or negative, so, you need to evaluate the first derivative of the function, using chain rule, such that:
f'(x) = (cos^2(2x))' => f'(x) = 2 cos(2x)*(- sin(2x))*(2x)'
f'(x) = -2*sin(4x)
You need to solve for x the equation f'(x) = 0:
-2*sin(4x) = 0 => sin 4x = 0 => 4x = pi => x = pi/4
For x = pi/6 < pi/4, sin (4*pi/6) = sin (2pi/3) = 2 sin (pi/3) cos(pi/3) = sqrt3/2 > 0
For x = pi/3 > pi/4 => sin (4*pi/3) = sin (2*2pi/3) = 2 sin (2pi/3) cos(2pi/3) = sqrt3*(1/4 - 3/4) = -sqrt3/2< 0
You need to notice that f'(x)<0 on intervals (pi/4,pi)U(5pi/4,2pi) and f'(x)>0 on (0,pi/4)U(pi,5pi/4).
Hence, the function increases for x in(0,pi/4)U(pi,5pi/4 ) and the function decreases for x in (pi/4,pi)U(5pi/4,2pi).