linux: m3u8: how to manually download and create an mp4 from segments listed in .m3u8

linux bash ffmpeg m3u8 aria2
Santhosh picture Santhosh · Jun 1, 2018 · Viewed 7.2k times · Source

I have an .m3u8 url i.e http://example.com/test.m3u8

I can download it by ffmpeg command:

ffmpeg -y -loglevel verbose -i "http://example.com/test.m3u8" -c copy -f mpegts test.mp4

But i am trying to do this manually:

I tried the following:

1) download the m3u8 file:

wget -O test.m3u8 "http://example.com/test.m3u8"

2) then download each segment into a folder:

aria2c -i test.m3u8

this downloads all the .ts files in the folder.

3) then i combine them

cat *.ts > out.ts

4) then i try to convert to mp4:

$ ffmpeg -i out.ts -c:v libx264 outputfilename.mp4
ffmpeg version 3.4.2 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 7.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-shared --enable-version3 --enable-omx
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
out.ts: Invalid data found when processing input

I am getting the following the above error finally. Invalid data found when processing input. Also i heard these files are encrypted. So i am not sure about how to do this.

Below is an example .m3u8 file:

#EXTM3U
#EXT-X-VERSION:5
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:5
#UPLYNK-SEGMENT: fadd0d418aea4c5da8b87feec9b0acbc,00000000,segment
#UPLYNK-KEY:https://content-aapm1.uplynk.com/check?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc
#EXT-X-PROGRAM-DATE-TIME:2018-06-01T05:25:50.630000+00:00
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000000
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000000.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000001
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000001.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000002
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000002.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000003
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000003.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000004
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000004.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000005
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000005.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000006
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000006.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000007
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000007.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000008
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000008.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000009
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000009.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000A
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000A.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000B
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000B.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000C
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000C.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000D
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000D.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000E
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000E.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000000F
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000000F.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000010
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000010.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000011
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000011.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000012
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000012.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000013
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000013.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000014
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000014.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000015
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000015.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000016
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000016.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000017
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000017.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000018
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000018.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x00000000000000000000000000000019
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G00000019.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000001A
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000001A.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000001B
#EXTINF:4.0960,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000001B.ts?x=0&si=0
#EXT-X-KEY:METHOD=AES-128,URI="https://content-aapm1.uplynk.com/check2?b=fadd0d418aea4c5da8b87feec9b0acbc&v=fadd0d418aea4c5da8b87feec9b0acbc&r=g",IV=0x0000000000000000000000000000001C
#EXTINF:3.6405,
https://stgec-ausw-tmp.uplynk.com/80C078/ausw/slices/fad/10b98e7c615f43a98b180d51797e74aa/fadd0d418aea4c5da8b87feec9b0acbc/G0000001C.ts?x=0&si=0
#EXT-X-ENDLIST

Answer

avolkmann picture avolkmann · Jun 6, 2018

Regarding the cat issue. Are you accounting for the BOM? All .ts files start with a BOM, so if you cat them, they will break.

According to specification you can find all required encryption details in the media playlist itself.

Example line:

#EXT-X-KEY:METHOD=AES-128,URI="http://example.com/path/to/resource.mp4/crypt.key?id=somekey"

This contains the encryption algorithm and the uri to the key. The encryption can be per list or per track. In your case it is per track.

Side note: I wrote a bit about my experiences with m3u8 / ts. Maybe it can help you or give you some new idea: https://andreasvolkmann.github.io/posts/2018-04-27-m3u8-and-ts-segments/

Related questions

How to extract duration time from ffmpeg output?

To get a lot of information about a media file one can do ffmpeg -i <filename> where it will output a lot of lines, one in particular Duration: 00:08:07.98, start: 0.000000, bitrate: 2080 kb/s I would like to output only 00:08:07.98, …

Read More
Looping through the content of a file in Bash

How do I iterate through each line of a text file with Bash? With this script: echo "Start!" for p in (peptides.txt) do echo "${p}" done I get this output on the screen: Start! ./runPep.sh: line 3: syntax error …

Read More
How to create a file in Linux from terminal window?

What's the easiest way to create a file in Linux terminal?

Read More