Commit Graph

75080 Commits

Author SHA1 Message Date
Michael Niedermayer f87d2617d5 Changelog: Update
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-12 23:25:12 +01:00
zjh8890 cd83f899c9 avcodec/aarch64/neon.S: Update neon.s for transpose_4x4H
The transpose_4x4H is wrong which cost me much time to find this bug. The orders of r2 and r3 are wrong,
this bug waste me much time while I make aarch64 arm instruction which used the function.
(cherry picked from commit c18176bd55)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-12 15:05:06 +01:00
Rainer Hochecker 5e105aca01 avformat/utils: estimate_timings_from_pts - increase retry counter, fixes invalid duration for ts files with hevc codec
Fixes a mpegts file with hevc that fails estimating duration. Increasing number of
retries fixes the issue.

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 2d8c2f1a28)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-09 20:46:47 +01:00
Michael Niedermayer a9c721da12 avformat/matroskaenc: Check codecdelay before use
Fixes CID1238790

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e6971db12b)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-09 18:27:54 +01:00
Michael Niedermayer 4608cc176b avutil/mathematics: Fix division by 0
Fixes: CID1341571

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit bc8b1e694c)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-09 17:53:44 +01:00
Andreas Cadhalpun 2e54b8c379 mjpegdec: consider chroma subsampling in size check
If the chroma components are subsampled, smaller buffers are allocated
for them. In that case the maximal block_offset for the chroma
components is not as large as for the luma component.

This fixes out of bounds writes causing segmentation faults or memory
corruption.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 5adb5d9d89)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-07 01:34:13 +01:00
Michael Niedermayer 913c642c21 Update for 2.8.4
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 10:42:02 +01:00
James Zern cb4985ea9a libvpxenc: remove some unused ctrl id mappings
VP8E_UPD_ENTROPY, VP8E_UPD_REFERENCE, VP8E_USE_REFERENCE were removed
from libvpx and the remaining values were never used here

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Zern <jzern@google.com>
(cherry picked from commit 6540fe04a3)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:28 +01:00
Michael Niedermayer d27d59fef7 avcodec/vp3: ensure header is parsed successfully before tables
Fixes assertion failure
Fixes: 266ee543812e934f7b4a72923a2701d4/signal_sigabrt_7ffff6ae7cc9_7322_85218d61759d461bdf7387180e8000c9.ogg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 26379d4fdd)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:28 +01:00
Michael Niedermayer ea4d9cb3e5 avcodec/jpeg2000dec: Check bpno in decode_cblk()
Fixes: undefined shift
Fixes: c409ef86f892335a0a164b5871174d5a/asan_heap-oob_1dff564_2159_162b7234616deab02b544410455eb07b.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a85b02dcf7)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer e5a4044771 avcodec/pgssubdec: Fix left shift of 255 by 24 places cannot be represented in type int
Fixes: b293a6479bb4b5286cff24d356bfd955/asan_generic_225c3c9_7819_cc526b657450c6cdef1371b526499626.mkv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 4f2419888b)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer af44545613 swscale/utils: Fix for runtime error: left shift of negative value -1
Fixes: c106b36fa36db8ff8f3ed0c82be7bea2/asan_heap-oob_32699f0_6321_467b9a1d7e03d7cfd310b7e65dc53bcc.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 325b59368d)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 10fc3d690c avcodec/hevc: Fix integer overflow of entry_point_offset
Fixes out of array read
Fixes: d41d8cd98f00b204e9800998ecf8427e/signal_sigsegv_321165b_7641_077dfcd8cbc80b1c0b470c8554cd6ffb.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 2140858524)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 0fa92fee43 avcodec/dirac_parser: Check that there is a previous PU before accessing it
Fixes out of array read
Fixes: 99d142c47e6ba3510a74b872a1a2ae72/asan_heap-oob_11b36f4_3811_0f5c69e7609a88a580135678de1df844.dxa

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a08681f1e6)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer cf79fd0317 avcodec/dirac_parser: Add basic validity checks for next_pu_offset and prev_pu_offset
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit c7d6ec947c)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer b253035ab2 avcodec/dirac_parser: Fix potential overflows in pointer checks
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 79798f7c57)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer ded0a04153 avcodec/wmaprodec: Check bits per sample to be within the range not causing integer overflows
Fixes: 549d5aab1480d10f2a775ed90b0342f1/signal_sigabrt_7ffff6ae7cc9_5643_96bbb0cfe3e28be1dadfce1075016345.wma

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 66e05f6ff5)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 16c5da92d9 avcodec/wmaprodec: Fix overflow of cutoff
Fixes: 129ca3e28d73af7b1e24a9d4118e7a2d/signal_sigabrt_7ffff6ae7cc9_836_762b310fc3ef6087bd7771e5d8e90b9b.asf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0c56f8303e)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 0e3ec7db53 avformat/smacker: fix integer overflow with pts_inc
Fixes: ce19e41f0ef1e52a23edc488faecdb58/asan_heap-oob_2504e97_4202_ffa0df1baed14022b9bfd4f8ac23d0cb.smk

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 7ed47e9729)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 73966b0161 avcodec/vp3: Fix "runtime error: left shift of negative value"
Fixes: 5c6129154b356b80bcab86f9e3ee5d29/signal_sigabrt_7ffff6ae7cc9_7322_d26ac6d7cb6567db1b8be0159b387d0b.ogg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 18268f761b)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer b8621a2e98 avformat/riffdec: Initialize bitrate
Fixes CID1338334

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 32bf6550cb)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Timo Teräs aa9ac199b8 mpegencts: Fix overflow in cbr mode period calculations
ts->mux_rate is int (signed 32-bit) type. The period calculations
will start to overflow when mux_rate > 5mbps. This fixes overflows
by converting first to 64-bit type.

Fixes #5044.

Signed-off-by: Timo Teräs <timo.teras@iki.fi>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 64f7db554e)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer f2258e9899 avutil/timecode: Fix fps check
The fps variable is explicitly set to -1 in case of some errors, the check must
thus be signed or the code setting it needs to use 0 as error code
the type of the field could be changed as well but its in an installed header

Fixes: integer overflow
Fixes: 9982cc157b1ea90429435640a989122f/asan_generic_3ad004a_3799_22cf198d9cd09928e2d9ad250474fa58.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b46dcd5209)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer e04b039b15 avutil/mathematics: return INT64_MIN (=AV_NOPTS_VALUE) from av_rescale_rnd() for overflows
Fixes integer overflow
Fixes: mozilla bug 1229167

Found-by: Tyson Smith
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit f03c2ceec1)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer d295ddffe1 avcodec/apedec: Check length in long_filter_high_3800()
Fixes out of array read
Fixes: 0a7ff0c1d93da9cef28a315ec91b692a/asan_heap-oob_4a52e5_3604_9c56dbb20e308f4faeef7b35f688521a.ape

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit cd7524fdd1)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer aab65146af avcodec/vp3: always set pix_fmt in theora_decode_header()
Fixes assertion failure
Fixes: d0bb0662da342ec65f8f2a081222e6b9/signal_sigabrt_7ffff6ae7cc9_5471_82964f0a9ac2f4d3d59390c15473f6f7.ogg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a814f1d364)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 8cd3def81d avcodec/mpeg4videodec: Check available data before reading custom matrix
Fixes: out of array read
Fixes: 76c515fc3779d1b838667c61ea13ce92/asan_heap-oob_1fc0d07_8913_794a4629a264ebdb25b58d3a94ed1785.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 891dc8f875)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 460710500e avutil/mathematics: Do not treat INT64_MIN as positive in av_rescale_rnd
The code expects actual positive numbers and gives completely wrong
results if INT64_MIN is treated as positive
Instead clip it into the valid range that is add 1 and treat it as
negative

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 25e37f5ea9)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 4d9999705f avutil/integer: Fix av_mod_i() with negative dividend
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 3a9cb18855)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 3d69716bae avformat/dump: Fix integer overflow in av_dump_format()
Fixes part of mozilla bug 1229167

Found-by: Tyson Smith
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8e7f452022)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 2fbf723585 avcodec/h264_refs: Check that long references match before use
Fixes out of array read
Fixes: 59bb925e90201fa0f87f0a31945d43b5/asan_heap-oob_4a52e5_3388_66027f11e3d072f1e02401ecc6193361.jvt

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit aa427537b5)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 7372b42b18 avcodec/utils: Clear dimensions in ff_get_buffer() on failure
Fixes out of array access
Fixes: 482d8f2fd17c9f532b586458a33f267c/asan_heap-oob_4a52b6_7417_1d08d477736d66cdadd833d146bb8bae.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit abee0a1c60)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer fe89682d7c avcodec/utils: Use 64bit for aspect ratio calculation in avcodec_string()
Fixes integer overflow
Fixes: 3a45b2ae02f2cf12b7bd99543cdcdae5/asan_heap-oob_1dff502_8022_899f75e1e81046ebd7b6c2394a1419f4.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 4f03bebc79)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer ce6dd54711 avcodec/hevc: Check max ctb addresses for WPP
Fixes out of array read
Fixes: 2f95ddd996db8a6281d2e18c184595a7/asan_heap-oob_192fe91_3330_58e4441181e30a66c19f743dcb392347.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit dad354f38d)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer d32c9723a6 avcodec/vp3: Clear context on reinitialization failure
Fixes null pointer dereference
Fixes: 1536b9b096a8f95b742bae9d3d761cc6/signal_sigsegv_294aaed_2039_8d1797aeb823ea43858d0fa45c9eb899.ogv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 6105b7219a)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer 1cbc2cb508 avcodec/hevc: allocate entries unconditionally
Fixes out of array access
Fixes: 08664a2a7921ef48172f26495c7455be/asan_heap-oob_23036c6_3301_523388ef84285a0270caf67a43247b59.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit d85aa76115)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:27 +01:00
Michael Niedermayer e348769544 avcodec/hevc_cabac: Fix multiple integer overflows
Fixes: 04ec80eefa77aecd7a49a442cc02baea/asan_heap-oob_19544fa_3303_1905796cd9d8e15f86d664332caabc00.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit d5028f61e4)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:26 +01:00
Michael Niedermayer 9ce96a688b avcodec/jpeg2000dwt: Check ndeclevels before calling dwt_encode*()
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit feb3f39614)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:26 +01:00
Michael Niedermayer 9d5bdca3d8 avcodec/jpeg2000dwt: Check ndeclevels before calling dwt_decode*()
Fixes out of array access
Fixes: 01859c9a9ac6cd60a008274123275574/asan_heap-oob_1dff571_8250_50d3d1611e294c3519fd1fa82198b69b.avi

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 75422280fb)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:26 +01:00
Michael Niedermayer 31e54f41a0 avcodec/hevc: Check entry_point_offsets
Fixes out of array read
Fixes: 007c4a36608ebdf27ee260ad60a81184/asan_heap-oob_32076b4_2243_116b1cb29d91cc4974d6680e3d10bd91.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit ef9f7bbfa4)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:51:26 +01:00
Carl Eugen Hoyos aa3101a9e8 lavf/rtpenc_jpeg: Less strict check for standard Huffman tables.
There can be one or more Huffman table segments DHT.

Reported-by: Andrey Utkin
2015-12-02 14:56:53 +01:00
Michael Niedermayer ff3e717003 avcodec/ffv1dec: Clear quant_table_count if its invalid
Fixes deallocation of corrupted pointer
Fixes: 343dfbe142a38b521ed069dc4ea7c03b/signal_sigsegv_421427_4074_ffb11959610278cd40dbc153464aa254.avi
No releases affected

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e04126072e)

Fixes ticket #5052.
2015-12-02 12:03:17 +01:00
Michael Niedermayer 1c6243228c avcodec/ffv1dec: Print an error if the quant table count is invalid
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a8b254e436)
2015-12-02 11:57:34 +01:00
Andrey Utkin 482bece29b doc/filters/drawtext: fix centering example
Signed-off-by: Andrey Utkin <andrey.od.utkin@gmail.com>
Signed-off-by: Lou Logan <lou@lrcd.com>
(cherry picked from commit 648b26acc5)
Signed-off-by: Timothy Gu <timothygu99@gmail.com>
2015-12-01 17:32:05 -08:00
James Almer 644296e736 avutil/softfloat: use abort() instead of av_assert0(0)
Fixes compilation of host tool aacps_fixed_tablegen.

Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit 9f4a41bf99)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
n2.8.3
2015-11-27 15:01:22 +01:00
Michael Niedermayer a353cc44a6 Update for 2.8.3
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-11-27 14:29:04 +01:00
Michael Niedermayer 4c718691ea avcodec/cabac: Check initial cabac decoder state
Fixes integer overflows
Fixes: 1430e9c43fae47a24c179c7c54f94918/signal_sigsegv_421427_2340_591e9810c7b09efe501ad84638c9e9f8.264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Found-by: xiedingbao (Ticket4727)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8000d484b8)

Conflicts:

	libavcodec/cabac.h
2015-11-27 14:07:03 +01:00
Michael Niedermayer 24c504bd0a avcodec/cabac_functions: Fix "left shift of negative value -31767"
Fixes: 1430e9c43fae47a24c179c7c54f94918/signal_sigsegv_421427_2340_591e9810c7b09efe501ad84638c9e9f8.264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Found-by: xiedingbao (Ticket4727)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a1f6b05f52)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-11-27 14:03:01 +01:00
Michael Niedermayer 6b91701de3 avcodec/h264_slice: Limit max_contexts when slice_context_count is initialized
Fixes out of array access
Fixes: 1430e9c43fae47a24c179c7c54f94918/signal_sigsegv_421427_2049_f2192b6829ab6e0eefcb035329c03c60.264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 4ea4d2f438)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-11-26 16:06:39 +01:00
Martin Storsjö 1290c85c9d rtmpcrypt: Do the xtea decryption in little endian mode
The XTEA algorithm operates on 32 bit numbers, not on byte sequences.
The XTEA implementation in libavutil is written assuming big endian
numbers, while the rtmpe signature encryption assumes little endian.

This fixes rtmpe communication with rtmpe servers that use signature
type 8 (XTEA), e.g. crunchyroll.

CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit e7728319b92dbb4fb949155e33de7ff5358ddff3)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-11-26 16:06:39 +01:00