After Jimmy Valentine is released from prison, how does he feel about committing crimes such as burglary?

Immediately after he is released from prison, Jimmy continues committing burglaries. He doesn't appear to have changed, and he certainly demonstrates no guilt regarding his actions.
The text tells us that the first place he visits after he gets out of prison is Mike Dolan's restaurant and inn. There, he retrieves a bag that was kept hidden for him while he served a short prison term; the bag contains all the tools of Jimmy's safe-breaking trade. In all, he spent more than nine hundred dollars for the special tools. With the tools, he steals eight hundred dollars from a safe in Richmond, Indiana. Next, he takes fifteen hundred dollars from a safe in Logansport. His largest heist is the safe in Jefferson City, where he steals five thousand dollars.
Because of Jimmy's actions, a local detective, Ben Price, is hot on his heels. It is only later when Jimmy meets the girl of his dreams (Annabel) that he contemplates leaving his life of crime behind. Prior to his wedding, he writes a letter to an old friend bequeathing him the tools of his trade. Jimmy tells this old friend that he is planning to go west after the wedding, where he'll never see anyone who knows how he used to make his living.
Ironically, even as he prepares to hand over his tools to his friend, he is called upon to use his safe-breaking skills to save Annabel's young niece. So, to reiterate, Jimmy feels no guilt committing burglaries after being released from prison. To Jimmy, safe-breaking is a valid way to make a living. He only contemplates leaving his old life after he meets Annabel, the young woman he wants to marry.