I know that sort of thing does really annoy some people. I also know it can get annoying if it makes it difficult to read and to follow what they're saying. But if the character speaks in a rough, uncultured and uneducated kind of way, they would you really write them speaking perfect English? Of course not; that would be stupid, and completely misrepresents the character. So, personally, I say go for it. Besides, it's your character, if people don't like it then they can read something else.