curl_multi_remove_handle() calls our socket function, and there's no
need to call curl_multi_socket_action().