Throwing away the perfect guy?

My man is so sweet, like the sweetest guy ever. He looks tough and is strong, in a manual job, very steady and dependable. We've been together almost 4 years, living together and engaged for almost the whole time. I met him when I was pretty much over guys after being played bigtime by a guy I really liked so this man was a godsend. I pursued him, we fell in love, he looks after me. He has occasional anger problems but otherwise he is perfect by most girls standards. He puts me first in everything, he treats me all the time, he's respectful in front of his friends, he loves me so much...and I want to leave. I'm only 26 and not ready to get married and really just want to be single, not to meet other guys so much, just to not have to feel bad about doing things that I want to do that don't involve him. I want more excitement and spontinaity in my life. Am I being selfish? Will I regret leaving him? I wish I could be with someone who was more like me, adventurous.