How does Feste prove his intelligence in Twelfth Night?

Feste appears in Shakespeare's Twelfth Night, or What You Will, which was written around 1600. In analyzing the character of Feste, it's important to note that Twelfth Night was written at the time that Robert Armin joined Shakespeare's acting troupe, the Lord Chamberlain's Men, replacing Will Kempe as the actor who played fools and clowns.
Will Kempe was known for his physical comedy and low-brow humor and played rustic fools and clowns—called "natural fools," like Lance in The Two Gentlemen of Verona, Bottom in A Midsummer Night's Dream, and Dogberry in Much Ado About Nothing.
When Armin joined the troupe, the fools in Shakespeare's plays were written as "wise fools," who were less clownish and more witty and worldly-wise, starting with Touchstone in As You Like It, then Feste in Twelfth Night, and then the Fool in King Lear.
Armin was a well-known actor before he joined Shakespeare's company. He had also written several books and plays and was a noted singer, something that Shakespeare incorporated into his plays, including Twelfth Night. Feste sings songs that are well-integrated into the play, not simply included as a diversion or incidental entertainment.
Shakespeare created fools for Armin to act who were considerably more verbal and sharp-tongued than previous fools, a bit more skeptical and cynical, and they were often considerably smarter and wiser than other characters in the plays. Wordplay and intelligent, witty repartee became more important in the plays than pratfalls.
Viola quite well sums up Feste and Robert Armin:

VIOLA: This fellow's wise enough to play the fool;And to do that well craves a kind of wit:He must observe their mood on whom he jests,The quality of persons, and the time . . .This is a practiseAs full of labour as a wise man's art (3.1.57–63)

Feste's intelligence arose as much from the character of the actor who played him as from Shakespeare's writing of the character.

Feste, the fool in Shakespeare's Twelfth Night, is, like many of Shakespeare's fools, more clever than he seems. The primary way that Feste proves his intelligence is through his wit. Nearly every line he speaks contains some witticism, joke, or barb that he exchanges with his interlocutor. Feste does not traffic in the kind of physical comedy that a modern audience might expect from a clown or jester; rather, he engages in world play and puns. Olivia describes him as a "dry fool," and she is right. His sense of humor is exceedingly dry. Take, for example, the following exchange:

FESTE
Good madonna, why mournest thou?OLIVIA
Good fool, for my brother's death.FESTE
I think his soul is in hell, madonna.OLIVIA
I know his soul is in heaven, fool.FESTE
The more fool, madonna, to mourn for your brother'ssoul being in heaven.
Here, Feste uses logic to drive Olivia to the conclusion he wants her to arrive at. He goads her by saying her brother must be in hell, knowing she will rebut him and that he will be able to use her rebuttal to prove his point that her mourning period should end. It's a battle of wits, and he easily wins.
Feste is in a unique position throughout the play as the only character whose status gives him the right to say basically whatever he wants without fear of reprisal. He is not a lord or lady, like Orsino and Olivia, who must conform to societal standards, nor is he a servant, like Antonio or Cesario/Viola, who must protect their employment. He is also detached enough from the main plot that he can pass judgement and impartially comment on the foolishness of the other characters and their actions. He gets away with it too, as all of his criticisms are disguised as jokes.