Where were Macduff's wife and son killed?

At the end of Act 4, scene 1, when Macbeth has spoken to the Weird Sisters for a second time, he learns from Lennox that Macduff has fled to England. In his anger at Macduff's escape from Scotland, Macbeth vows,

The castle of Macduff I will surprise,Seize upon Fife, give to th' edge o' th' swordHis wife, his babes, and all unfortunate soulsThat trace him in his line. (4.1.171-174)

This lets us know that Macbeth intends to have Lady Macduff and her children murdered in their own home, at Macduff's castle in Fife. The very next scene opens on Lady Macduff speaking with Ross about her husband having left Scotland without speaking to her or even explaining why he was leaving. We can assume, I think, given the proximity of Macbeth's vow and this, the murder scene he planned, that nothing has happened to alter the time and place of the intended murders. Further, later in the scene, after the messenger has told her to make her escape, Lady Macduff asks, "Whither should I fly?" (4.2.81). In other words, she seems to have nowhere else to go. Therefore, we can assume they are killed at home, as Macbeth planned.

Macduff's wife, Lady Macduff, and her unnamed son are killed in Macduff's own castle in Fife. Lady Macduff is in conversation with Ross, telling him she thinks it is unwise for her husband to have left his wife and children alone and unguarded. While Ross argues that Macduff is "noble" and knows best what he is doing, it transpires that Lady Macduff is right after all when, shortly after Ross leaves the castle, a Messenger arrives to suggest they leave the castle, as they are in danger.
Lady Macduff asks, "Whither should I fly? / I have done no harm." And, in fact, she has no time to leave the castle: a murderer immediately enters after she has asked herself this question, and her son, challenging the murderers, is killed onstage. Lady Macduff does not die on the stage but exits pursued by the murderers. Later, however, we are told that all Lady Macduff's children and Lady Macduff herself have been killed in their own home by the murderers.