Previously, when select() failed, we assumed that there was an invalid
file descriptor in one of the client structs.  Thus we tried select()
one by one.  This is bogus, because we should never have invalid file
descriptors.  Remove it, and make select() errors fatal.