With OSX Yosemite and IOS 8.x, I ran into a situation where the Dock icon for Messages was showing me that I had one unread message, but I could not access the conversation in order to clear the flag. I killed and restarted Messages on all my devices hoping that this would clear the flag, but without success. I couldn't get to the conversation because it was a notification from Facebook with some numeric ID that was not recoverable. I tried sending a message to that ID in the hopes that the stale conversation would re-open and I could kill the flag.
Here is what worked:
- Quit Messages
- Open Activity Monitor (Command-Spacebar "Activity Monitor")
- In the search field in the top right, type "dock"
- Select the row with "Dock" and click on the "X" in the top left and then click on "Quit"
This will kill the Dock application, which will restart right away and will clear the notification badge for the Messages application.