Why is Charles Wallace annoyed with the three ladies?

Charles Wallace is annoyed because Mrs. Whatsit, Mrs. Which, and Mrs. Who whisk Meg, Calvin, and him off to the planet Uriel without any warning. He is especially annoyed for the sake of Meg, who doesn't have any idea what is going on. He also is annoyed because at first he fears they left her behind. Essentially, the element of surprise has confused and upset him. He says to them coldly:

You scared Meg half out of her wits, whisking her off this way without any warning.

Adding to his annoyance is the way the three women are laughing and giggling as if all of this is a great joke. Gales of laughter erupt from Mrs. Who and Mrs. Whatsit. Mrs. Who keeps quoting in Latin. Charles Wallace wants her to stop:

"Mrs. Who, I wish you'd stop quoting!" Charles Wallace sounded very annoyed.

Basically, now that they have startled and disoriented the children with the sudden, unannounced trip to Uriel, Charles wants the threesome to get serious, to focus, and to explain exactly what is going on. He is the bridge between the three women and Meg and Calvin, and he is performing the role of ensuring they all communicate clearly.