Who is Nestor?

In act I, scene 1, Antonio is talking to his friend Solanio. Solanio is trying to understand why Antonio seems so sad. At first, he thinks it is because his friend is fearful of losing his cargo. When Antonio says he feels safe about that, Solanio guesses that he is in love, an idea that Antonio denies. At this point, Solanio muses aloud at how strange humans are—we express our emotions in such different ways. He then alludes to Nestor, saying that some people will laugh at anything, while others will remain solemn and never smile, no matter how funny the joke. These people are

of such vinegar aspect
That they’ll not show their teeth in way of smile
Though Nestor swear the jest be laughable.

The reference to Nestor is a reference to an older figure who gave wise advice in Homer's The Iliad and The Odyssey but did so in a way that made people laugh at him because he would always preface his wisdom by bragging about his own past exploits. Therefore, a person who won't laugh at a joke that wise Nestor says is funny and who won't laugh at Nestor himself (who gives out such advice in an unintentionally comic way) is fairly immune to humor—or has an unshakeable poker face.

Nestor is not actually a character in The Merchant of Venice by William Shakespeare. Instead, he is one of the leaders of the Greeks in the Trojan War and figures prominently in Homer's Iliad. Because of the importance of classical culture in anglophone literature, most people writing in English assume their readers will be familiar with the Homeric epics, and thus allusions to figures such as Nestor are common.
Nestor was the King of Pylos. His distinguishing characteristics in the Iliad are his age and his wisdom. Unlike many of the other heroes, he is not described as being at the peak of his fighting prowess, but because of his age, brings considerable patience, experience, and expertise to the Greek army and offers sage counsel to Agamemnon and the other leaders. He is renowned for his wisdom. Unlike Odysseus, who is clever and unscrupulous, Nestor is wise in a way that includes good moral judgment.