What was Machiavelli's theory about how rulers should act?

There is a great deal of popular misunderstanding about Machiavelli, largely due to misunderstandings people have about The Prince.
A lot of people believe a Machiavellian is someone who wants power so he can take what he wants for his own personal self-aggrandizement--"the ends justify the means." Nothing could be further from the truth. Machiavelli actually states that such behavior is liable to cost a prince his throne. From Ch. XIX:

It makes [the Prince] hated above all things, as I have said, to be rapacious, and to be a violator of the property and women of his subjects, from both of which he must abstain. And when neither their property nor honour is touched, the majority of men live content, and he has only to contend with the ambition of a few, whom he can curb with ease in many ways.

The truth of the matter is that being a Machiavellian prince is very demanding. The first requirement for a proper Machiavellian prince is to have virtù. Without this quality--which is rendered in English as everything from "manliness" to "ability" to "skill" to "power," depending on the translation you are reading--a man cannot hope to succeed as a ruler. It is virtù that will allow the prince to survive the slings and arrows of Fortune. (All reputable scholars agree that the conflict between virtù and fortuna is central to The Prince.)
According to Machiavelli, a man of virtù embodies the qualities of the lion (strength) and the fox (cleverness):

You must know there are two ways of contesting, the one by the law, the other by force; the first method is proper to men, the second to beasts; but because the first is frequently not sufficient, it is necessary to have recourse to the second. Therefore it is necessary for a prince to understand how to avail himself of the beast and the man. [...] A prince, therefore, being compelled knowingly to adopt the beast, ought to choose the fox and the lion; because the lion cannot defend himself against snares and the fox cannot defend himself against wolves. Therefore, it is necessary to be a fox to discover the snares and a lion to terrify the wolves. Those who rely simply on the lion do not understand what they are about (Ch. XVIII).

In addition to having virtù, a proper prince must constantly study the arts of war and make sure that he has a properly functioning army that does not depend on mercenary forces (Ch. VI, Chs. XII-XIV). He must be continually wary of the nobility and careful of how he cultivates the support of the broader populace (Ch. IX, Ch. XVII). The prince must realize that the most prudent course of action is likely to make him unpopular (see many chapters, but esp. Ch. XVI). Finally, the prince must accept that, no matter what, there is always a chance Fortune may undo everything he has accomplished (Ch. XXIV-XXV).
What is underscored in Chs. XVI-XVII (and echoed to some extent in Ch. IX) is that the prince must be concerned with what is prudent, not what will please the populace in the moment. As he says in Ch. XVII:

this is to be asserted in general of men, that they are ungrateful, fickle, false, cowardly, covetous, and as long as you succeed they are yours entirely; they will offer you their blood, property, life and children, as is said above, when the need is far distant; but when it approaches they turn against you.

For example, when it comes to practicing genrosity, contrary to the standard advice given to princes (which was influenced by the philosophy of Aristotle), Machiavelli suggests restraint. Machiavelli says a prince who throws lavish festivals or builds extravagantly is foolish, because this prince may not be able to provide for his people in tough times. However, a prince who initially appears stingy because he will not host festivals will be celebrated when he is able to provide for the people during a famine or other disaster.In summary, the Machiavellian ruler is a man of virtù, who uses a combination of force/skill with arms and clear, prudent long-term vision to strengthen his principality in order that it may withstand challenges from internal factions, outside enemies, or the acts of Fortune.