swscale/ppc: fix ASAN stack-buffer-overflow in yuv2planeX

Fix two buffer overreads in the PowerPC yuv2planeX SIMD paths
that cause daily FATE checkasm-sw_scale ASAN failures on both
ppc64 (G5, altivec) and ppc64le (POWER9, VSX):

1. VSX LOAD_FILTER: vec_vsx_ld(joffset, filter) reads 16 bytes
   at the given byte offset.  When joffset >= filterSize*2 - 14
   (e.g. joffset=30 for filterSize=16), this reads up to 14 bytes
   past the 32-byte filter array.  Fix by replacing the vector
   load with vec_splats(f[j]) which only reads the single int16_t
   element needed (the result is splatted to all lanes anyway).

2. GET_LS look-ahead overread: yuv2planeX_8_16 calls
   yuv2planeX_8 twice per filter tap.  Each call's GET_LS macro
   speculatively loads the next 16-byte vector for pipelining.
   On the second call, this look-ahead reads 16 bytes past the
   last valid source element.  Fix by tightening the SIMD loop
   bound from (dstW - 15) to (dstW - 23), ensuring the farthest
   speculative load stays within src[j][0..dstW-1].  The scalar
   fallback handles the remaining 16-23 trailing pixels.

The ASAN reports from FATE:
  ppc64 (altivec): stack-buffer-overflow in yuv2planeX_8_16_altivec
                   at swscale_ppc_template.c:56
  ppc64le (VSX):   unknown-crash in yuv2planeX_8_16_vsx
                   at swscale_ppc_template.c:52

Signed-off-by: Scott Boudreaux <scott@elyanlabs.com>
(cherry picked from commit d4673a97ac)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This commit is contained in:
Scott Boudreaux
2026-03-18 18:28:09 +00:00
committed by Michael Niedermayer
parent 9825a8af1d
commit 733cce258b
+1 -1
View File
@@ -90,7 +90,7 @@ static void FUNC(yuv2planeX)(const int16_t *filter, int filterSize,
yuv2planeX_u(filter, filterSize, src, dest, dst_u, dither, offset, 0);
for (i = dst_u; i < dstW - 15; i += 16)
for (i = dst_u; i < dstW - 23; i += 16)
FUNC(yuv2planeX_8_16)(filter, filterSize, src, dest + i, dither,
offset, i);