Calculus: Early Transcendentals, Chapter 3, 3.5, Section 3.5, Problem 17

Take the derivative of both sides:
(dy)/(dx) (tan^(-1)(x^2y))=(dy/dx)(x+xy^2)
compute the derivative of each side.
first the left hand side, using chain rule:
(d)/(du)(arctan(u))(d)/(dx)(x^2y)
=((1)/(1+u^2))(2xy+(dy/dx))
=((1)/(1+x^4y^2))(2xy+(dy)/(dx))
then the right hand side:
(d)/(dx)(x+xy^2)
=1+(d)/(dx)(xy^2)
=1+(1y^2+x2y(dy)/(dx))
=1+y^2+(dy)/(dx)2xy
Now equate the two sides and solve for dy/dx:
((1)/(1+x^4y^2))(2xy+(dy/dx))=1+y^2+(dy)/(dx)2xy
2xy+(dy)/(dx)=(1+y^2+2xy(dy)/(dx))(1+x^4y^2)
(dy)/(dx)=1+y^2+2xy(dy)/(dx)+x^4y^2+x^4y^4+2x^5y^3(dy)/(dx)-2xy
(dy)/(dx)(1-2xy-2x^5y^3)=1-2xy+y^2+x^4y^2+x^4y^4
(dy)/(dx)=(1-2xy+y^2+x^4y^2+x^4y^4)/(1-2xy-2x^5y^3)

as required.