glx: Add safe_{add,mul,pad} (v3) [CVE-2014-8093 4/6] (v4)

These are paranoid about integer overflow, and will return -1 if their operation would overflow a (signed) integer or if either argument is negative. Note that RenderLarge requests are sized with a uint32_t so in principle this could be sketchy there, but dix limits bigreqs to 128M so you shouldn't ever notice, and honestly if you're sending more than 2G of rendering commands you're already doing something very wrong. v2: Use INT_MAX for consistency with the rest of the server (jcristau) v3: Reject negative arguments (anholt) v4: RHEL5: add limits.h, use inline v5: backport to nx-libs 3.6.x (Mike DePaulo) Reviewed-by: Keith Packard <keithp@keithp.com> Reviewed-by: Julien Cristau <jcristau@debian.org> Reviewed-by: Michal Srb <msrb@suse.com> Reviewed-by: Andy Ritger <aritger@nvidia.com> Signed-off-by: Adam Jackson <ajax@redhat.com> Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com> Signed-off-by: Fedora X Ninjas <x@fedoraproject.org> Signed-off-by: Dave Airlie <airlied@redhat.com>

glx: Add safe_{add,mul,pad} (v3) [CVE-2014-8093 4/6] (v4)
1a9f2311 · Adam Jackson · Mike Gabriel · d0fcbc8a · 1a9f2311
Commit 1a9f2311 authored Nov 10, 2014 by Adam Jackson Committed by Mike Gabriel Feb 14, 2015
Hide whitespace changes
Inline Side-by-side

Showing with 41 additions and 0 deletions

glxserver.h nx-X11/programs/Xserver/GL/glx/glxserver.h +41 -0

No files found.
--- a/nx-X11/programs/Xserver/GL/glx/glxserver.h
+++ b/nx-X11/programs/Xserver/GL/glx/glxserver.h
@@ -54,6 +54,7 @@
 #include "GL/glx_ansic.h"


+#include <limits.h>
 /*
 ** The X header misc.h defines these math functions.
 */
@@ -223,6 +224,46 @@ extern void glxSwapQueryServerStringReply(ClientPtr client,
 /*
 * Routines for computing the size of variably-sized rendering commands.
 */
+static __inline__ int
+safe_add(int a, int b)
+{
+    if (a < 0 || b < 0)
+        return -1;
+
+    if (INT_MAX - a < b)
+        return -1;
+
+    return a + b;
+}
+
+static __inline__ int
+safe_mul(int a, int b)
+{
+    if (a < 0 || b < 0)
+        return -1;
+
+    if (a == 0 || b == 0)
+        return 0;
+
+    if (a > INT_MAX / b)
+        return -1;
+
+   return a * b;
+}
+
+static __inline__ int
+safe_pad(int a)
+{
+    int ret;
+
+    if (a < 0)
+        return -1;
+
+    if ((ret = safe_add(a, 3)) < 0)
+        return -1;
+
+    return ret & (GLuint)~3;
+}

 extern int __glXTypeSize(GLenum enm);
 extern int __glXImageSize(GLenum format, GLenum type,