Why did Casey not swing at the first pitch?

"Casey at the Bat," by Ernest Lawrence Thayer, describes the performance of a baseball team, the Mudville nine after an initial outlook that "wasn't brilliant." The poet leads the reader to understand that Casey is a good player who can be relied upon: "we'd put up even money now, with Casey at the bat." However, Casey is not able to get "a whack at that" at first, because two others, Flynn and Jimmy Blake, precede him, neither of whom instilled much hope in the audience, one being described as a "hoodoo" and the other as a "cake." However, "to the wonderment of all," Flynn "let drive a single." This allows these two players to advance to second and third base, meaning that "Casey, mighty Casey" can finally proceed to the bat, to the great delight of the audience.
When he finally gets to his base, however, Casey does not swing at first pitch, instead watching the ball "in haughty grandeur" and declaring that the pitch "ain't my style." As such, Casey does not swing, allowing the ball to go by and causing uproar in the audience, which Casey stills by raising his hand. He also goes on to allow the second pitch to go past him, but after this, the audience "knows" he will not let the third pitch go by unimpeded. In the end, however, "there is no joy in Mudville," as Casey's arrogance has gotten the better of him. "Mighty Casey has struck out."