Did Winston know that O’Brien was a part of the Party or not?

The answer to this is not a straightforward one. Initially, Winston does not know that O'Brien is really working on behalf of the Party. This is shown by the fact that he visits his apartment and receives from him a copy of Goldstein's book. In addition, in part 3, chapter 1, when Winston is imprisoned in the Ministry of Love, he wonders if O'Brien has also been caught and arrested. When O'Brien finally visits Winston in his cell, Winston says:

"They’ve got you too!" he cried.

This shows that Winston has no idea that O'Brien works for the Party. However, in the next paragraph, Winston admits that deep down, he always knew that O'Brien was working for the Party:

Yes, he saw now, he had always known it.

This suggests that instead of realizing that O'Brien really was a member of the Inner Party, working on behalf of Big Brother, Winston allowed himself to be lulled into a false sense of security. Why? Because Winston was desperate to rebel. He so badly wants to bring down Big Brother that he lets it cloud his judgment. He chooses to believe in O'Brien, the Brotherhood, and Goldstein's book instead of realizing that he could never overcome a force as strong as the Party.

No. Winston believes that O'Brien is an ally working against the Party from the first moment he spots O'Brien's seemingly unorthodox disposition at work. Winston is intrigued by O'Brien's behavior during the Two Minutes Hate period when he makes direct eye contact with him. Winston interprets O'Brien's stare as his way of saying,

I am with you...I know precisely what you are feeling. I know all about your contempt, your hatred, your disgust. But don’t worry, I am on your side! (Orwell, 22).

Winston believes that O'Brien shares the same ideas regarding the government as him and even begins to dream about colluding with O'Brien against the Party. Later on in the novel, Winston accepts O'Brien's invitation to pick up a Newspeak Dictionary at his home. Winston and Julia even travel to O'Brien's home, where O'Brien manipulates them into believing that he is a member of the Brotherhood. Winston pledges his allegiance to the Brotherhood and accepts Emmanuel Goldstein's book. Winston is utterly surprised and disheartened to learn that O'Brien is a government spy. O'Brien then becomes Winston's main torturer in the Ministry of Love. Overall, Winston trusted and believed that O'Brien was an enemy of the Party, when O'Brien actually worked for the government.