Fix overflow of ConnectionOutput->size and ->count

commit 4b0d0df34f10a88c10cb23dd50087b59f5c4fece Author: Peter Harris <pharris@opentext.com> Date: Mon Nov 17 14:31:24 2014 -0500 Fix overflow of ConnectionOutput->size and ->count When (long) is larger than (int), and when realloc succeeds with sizes larger than INT_MAX, ConnectionOutput->size and ConnectionOutput->count overflow and become negative. When ConnectionOutput->count is negative, InsertIOV does not actually insert an IOV, and FlushClient goes into an infinite loop of writev(fd, iov, 0) [an empty list]. Avoid this situation by killing the client when it has more than INT_MAX unread bytes of data. Signed-off-by: Peter Harris <pharris@opentext.com> Reviewed-by: Keith Packard <keithp@keithp.com> Signed-off-by: Keith Packard <keithp@keithp.com> Backported-to-NX-by: Mike Gabriel <mike.gabriel@das-netzwerkteam.de>

Fix overflow of ConnectionOutput->size and ->count
cbc2d300 · Peter Harris · Mike Gabriel · 65b6a62b · cbc2d300
Commit cbc2d300 authored Feb 14, 2017 by Peter Harris Committed by Mike Gabriel Mar 20, 2017
Show whitespace changes
Inline Side-by-side

Showing with 4 additions and 3 deletions

io.c nx-X11/programs/Xserver/os/io.c +4 -3

No files found.
--- a/nx-X11/programs/Xserver/os/io.c
+++ b/nx-X11/programs/Xserver/os/io.c
@@ -1087,10 +1087,11 @@ FlushClient(ClientPtr who, OsCommPtr oc, const void *__extraBuf, int extraCount)

 	    if (notWritten > oco->size)
 	    {
-		unsigned char *obuf;
+		unsigned char *obuf = NULL;

-		obuf = (unsigned char *)realloc(oco->buf,
-						 notWritten + BUFSIZE);
+		if (notWritten + BUFSIZE <= INT_MAX) {
+		    obuf = realloc(oco->buf, notWritten + BUFSIZE);
+		}
 		if (!obuf)
 		{
 		    _XSERVTransDisconnect(oc->trans_conn);