Rework server and socket closure
As an example, have a look at http://stackoverflow.com/questions/8051863/how-can-i-close-the-socket-in-a-proper-way
Closing streams and servers can be tricky. If after reading the thread you see no problems with your code, I'll explain further.