Did both Tom and Father Green abuse Skippy?

Interesting question! If we follow the story, both Father Green (the French teacher) and Tom Roche (the swim coach) did abuse Skippy, but in different ways. The former abused his authority over Skippy by humiliating the young boy in class. As for Tom Roche, the story later reveals that he sexually abused Skippy after the latter passed out.
In the story, Father Green actually harbors pedophiliac tendencies. He finds himself developing feelings of attraction for Skippy, but he doesn't act on them. Instead, he is routinely plagued by feelings of extreme guilt from his days in Africa, when he did sexually abuse boys. Because of his pedophiliac intentions towards Skippy, Father Green uses explicitly sexual banter to toy with the young boy in class. The priest's unfounded allegations about Skippy's misconduct, tinged with salacious implications, cause Skippy to throw up in class.
Later, we learn about Tom Roche's confession to Howard. Accordingly, Tom admitted to his colleague that he had sexually molested Skippy after giving him some painkillers. Apparently, Skippy had complained of pain when he came in to see his coach. The painkillers that Tom gave him, however, knocked Skippy out. After Skippy passed out, Tom had his way with the young man. The next day, Tom Roche gave Skippy another pill and carried on as normal, banking on the fact that Skippy would not remember what happened.
So, the two men did abuse Skippy, but in different ways.