What does Romeo mean when he says that he loves Tybalt in Romeo and Juliet?

Tybalt has been spoiling for a fight with Romeo, especially since seeing him flirting with Juliet, Tybalt's cousin. Tybalt dislikes the Montagues intensely and wants to defend Juliet against a lovelorn specimen like Romeo. Romeo, having already stormed the Capulet citadel, and having, unbeknownst to Tybalt or any other Capulet, married Juliet, very much wants to avoid a quarrel.
Tybalt is now no longer Romeo's enemy, but his cousin. When Romeo says he loves Tybalt, stating "I . . . love thee better than thou canst devise [imagine]," he means he loves Tybalt as a relative because he has married Juliet. Romeo also says he loves Tybalt to try to convince him that he really doesn't want a fight: if you love a person you don't want to try to kill him.
Of course, sidestepping a fight with Tybalt only opens the ground for Mercutio to step into the breach and get killed, leading Romeo to kill Tybalt after all.

In act 3, scene 1, Tybalt enters the scene looking for Romeo and is met by Mercutio, who attempts to defend Romeo from his presumed enemy. When Tybalt sees Romeo, he refers to him as a villain. Romeo attempts to calm Tybalt by telling him,

Tybalt, the reason that I have to love thee Doth much excuse the appertaining rage To such a greeting. Villain am I none. Therefore, farewell. I see thou know’st me not (Shakespeare, 3.1.33-36).

When Tybalt challenges him to a duel, Romeo once again expresses his love for him by telling Tybalt,

I do protest I never injured thee, But love thee better than thou canst devise, Till thou shalt know the reason of my love (Shakespeare, 3.1.39-41).

Tybalt, like the other members of the Capulets and Montagues, is not aware that Romeo has married Juliet. The two lovers were forced to marry in secret because of the social pressure and ongoing feud between their families. Romeo tells Tybalt that he loves him because he is now part of Tybalt's family and related to him through marriage. Unfortunately, Romeo does not disclose this important information in time, and Mercutio ends up fighting Tybalt. In an attempt to stop the fight, Romeo obstructs Mercutio's view, allowing Tybalt to stab and kill him. In a fit of rage, Romeo ends up killing Tybalt and is banished from Verona.