How do Rosencrantz and Guildenstern react to Hamlet’s questions about why they are there? How does this give him a ‘heads-up’ to their purpose?

In act 2, scene 2, Rosencrantz and Guildenstern approach Hamlet and attempt to find out why he has been acting oddly in order to inform Claudius. Initially, Rosencrantz and Guildenstern attempt to have a casual conversation with Hamlet, who verbally jousts with them by disagreeing with their perspective on nearly everything. When Rosencrantz mentions that the world has become honest, Hamlet responds by saying, "Then is doomsday near. But your news is not true" (Shakespeare, 2.2.227).
Hamlet then refers to Denmark as a prison, and the two men question their friend on his assessment of the nation. Rosencrantz and Guildenstern then suggest that Hamlet's view regarding Denmark are a sign of his ambition. The fact that the two men mention ambition and are acting sheepishly indicates their true intentions, which Hamlet recognizes.
When Hamlet initially questions if they were sent by someone, Guildenstern uncomfortably says, "What should we say, my lord?" (Shakespeare, 2.2.260). His response is sheepish and indirect, which immediately informs Hamlet that Claudius sent them. Hamlet responds by saying that "there is a kind of confession in your looks," which Guildenstern is too honest to disguise. Essentially, Hamlet can tell by the look on their faces and their indirect mannerisms that Rosencrantz and Guildenstern have been sent by the king. Interestingly, Rosencrantz and Guildenstern do not immediately admit that they have been sent and Hamlet has to remind them of their long friendship and love for one another. The fact that Rosencrantz and Guildenstern have attempted to deceive Hamlet and collude with King Claudius upsets Hamlet, and he no longer trusts his old friends.

Rosencrantz and Guildenstern behave with decided shiftiness when Hamlet begins to question them. Even a simple question such as "how do ye both?" is met with attempts from Rosencrantz and Guildenstern to amuse and distract Hamlet, as if to prevent him from thinking too much about their sudden reappearance. When he asks them what news there is and Rosencrantz says there is "none . . . but that the world's grown honest," Hamlet becomes suspicious, stating that this news is "not true."
Given what we have seen of Rosencrantz and Guildenstern, we can interpret that it is unusual for them not to be forthcoming with Hamlet. Appropriately, Hamlet at this point suspects that his mother has sent for the two, and he asks them directly why she has done this.
Again, Guildenstern attempts to use diversionary tactics, rather than answer the question. Hamlet has described Elsinore as a "prison," to which Guildenstern replies "Prison!" and begins a course of discussion on this basis.
Hamlet does allow himself to be diverted for a space of several lines but eventually returns to his question: "what make you at Elsinore?" When Hamlet's questioning becomes more insistent ("Were you not sent for? Is it a free visitation?") the two eventually admit that they were indeed sent for by the queen.