Add receiver client to OSS build + better readme(s)

Summary: - Added `flv_receiver_client` to OSS build (forgot to add before) - Better readme - Added readme for media clients - Added TODO to add de-jitter in the reciever Reviewed By: afrind Differential Revision: D68016786 fbshipit-source-id: 02ba6de00d68ebc3bb446e825cd8675e2eccee3c
facebookexperimental · Jan 10, 2025 · 59131b0 · 59131b0
1 parent 11e0f11
commit 59131b0
Show file tree

Hide file tree

Showing 9 changed files with 221 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -28,7 +28,7 @@ cd scripts
 
 - Execute date server (from project root dir)
 ```
-./_build/bin/moqdateserver -port 4433 -cert ./certs/certificate.pem -key ./certs/certificate.key --logging DBG
+./_build/bin/moqdateserver -port 4433 -cert ./certs/certificate.pem -key ./certs/certificate.key --logging DBG1
 ```
 
 - Execute text client
@@ -74,7 +74,20 @@ I0520 13:08:11.351163 7064889 MoQClient.cpp:137] onWebTransportUniStream
 11
 ```
 
-## Test with media client
+## Test with media streamer and media receiver
+
+To simplify testing MOQ at media level we added the following binaries to the repo `MoQFlvStreamerClient` and `MoQFlvReceiverClient`.
+
+![moq-relay-streamer-receiver](./pics/moq_streamer_receiver.png)
+Fig3: MoQFlvStreamerClient, MoQFlvReceiverClient, and moxygen
+
+- `MoQFlvStreamerClient`: Convert any FLV (h264 / AAC-LC) file or stream (fifo) into MOQ, publishing it to a relay using MoqMi packager (see RFC [draft-cenzano-moq-media-interop](https://datatracker.ietf.org/doc/draft-cenzano-moq-media-interop/))
+
+- `MoQFlvReceiverClient`: Subscribes to a relay for a video and audio track, demuxes them from MoqMi (expecting h264 / AAC-LC), transmuxes them to FLV and saves that to disc (or stream using fifo)
+
+They work with FLV packager. Since [ffmpeg](https://www.ffmpeg.org/ffmpeg.html) is able to mux and / or demux this packager in low latency and real time very nice you can build a huge variety of tests set ups, for more information and examples take a look to [READMOQMEDIA.md](./READMOQMEDIA.md)
+
+## Test with web media client
 - You can use [moq-encoder-player](https://github.com/facebookexperimental/moq-encoder-player) as encoder (publisher), and also as player (consumer)
 
 - You need to install that website ([moq-encoder-player](https://github.com/facebookexperimental/moq-encoder-player)) in a https server (apache2 recommended)
@@ -85,7 +98,9 @@ I0520 13:08:11.351163 7064889 MoQClient.cpp:137] onWebTransportUniStream
 ./_build/bin/moqrelayserver -port 4433 -cert ./certs/certificate.pem -key ./certs/certificate.key -endpoint "/moq" --logging DBG
 ```
 
-## Local test with media client (server running in localhost)
+## Local test with web media client
+
+Assuming all running in localhost
 
 - Execute (from project root dir)
 ```
@@ -95,7 +110,7 @@ I0520 13:08:11.351163 7064889 MoQClient.cpp:137] onWebTransportUniStream
 Note: [moq-encoder-player] indicate the root directory of that project. So you need to use those certs to enable connections from Chrome to localhost
 
 - Start client (MACOS)
-Run MOQ encoder / player locally
+Run MOQ encoder / player in the browser locally
    - Open a Chrome window and follow the instructions you will find in [moq-encoder-player](https://github.com/facebookexperimental/moq-encoder-player)
 
 

diff --git a/READMOQMEDIA.md b/READMOQMEDIA.md
@@ -0,0 +1,197 @@
+# moxygen test using `MoQFlvStreamerClient` and `MoQFlvReceiverClient`.
+
+## Introduction
+To simplify testing MOQ at media level we added the following binaries to the repo `MoQFlvStreamerClient` and `MoQFlvReceiverClient`.
+
+![moq-relay-streamer-receiver](./pics/moq_streamer_receiver.png)
+Fig1: MoQFlvStreamerClient, MoQFlvReceiverClient, and moxygen
+
+Remeber that ffmpeg allows you to install capture cards and playback cards (such DDD), this allows to send any broadcast feed (SDI) via MOQ or/and send any MOQ feed on to a broadcaster via SDI.
+
+![moq-to-sdi](./pics/sdi-to-moq.png)
+Fig2: Ingest SDI and send it via MOQ
+
+### `MoQFlvStreamerClient`
+Convert any FLV (h264 / AAC-LC) file or stream (fifo) into MOQ, publishing it to a relay using MoqMi packager (see RFC [draft-cenzano-moq-media-interop](https://datatracker.ietf.org/doc/draft-cenzano-moq-media-interop/))
+
+     - Arguments
+```
+-audio_track_name (Audio track Name) type: string default: "audio0"
+-connect_timeout (Connect timeout (ms)) type: int32 default: 1000
+-connect_url (URL for webtransport server) type: string
+    default: "https://localhost:4433/moq"
+-input_flv_file (FLV input fifo file) type: string default: ""
+-quic_transport (Use raw QUIC transport) type: bool default: false
+-track_namespace (Track Namespace) type: string default: "flvstreamer"
+-track_namespace_delimiter (Track Namespace Delimiter) type: string
+    default: "/"
+-transaction_timeout (Transaction timeout (s)) type: int32 default: 120
+-video_track_name (Video track Name) type: string default: "video0"
+```
+
+### `MoQFlvReceiverClient`
+Subscribes to a relay for a video and audio track, demuxes them from MoqMi (expecting h264 / AAC-LC), transmuxes them to FLV and saves that to disc (or stream using fifo)
+
+- Arguments
+```
+-audio_track_name (Track Name) type: string default: "audio0"
+-connect_timeout (Connect timeout (ms)) type: int32 default: 1000
+-connect_url (URL for webtransport server) type: string
+    default: "https://localhost:4433/moq"
+-fetch (Use fetch rather than subscribe) type: bool default: false
+-flv_outpath (File name to save the received FLV file to (ex:
+    /tmp/test.flv)) type: string default: ""
+-quic_transport (Use raw QUIC transport) type: bool default: false
+-track_namespace (Track Namespace) type: string default: "flvstreamer"
+-track_namespace_delimiter (Track Namespace Delimiter) type: string
+    default: "/"
+-transaction_timeout (Transaction timeout (s)) type: int32 default: 120
+-video_track_name (Video track Name) type: string default: "video0"
+```
+
+## Stream VOD file simulating live stream
+
+Important you follow the next steps in order (when fifo pipes are involved is important to send data before we connect the reader)
+
+- Start relay (terminal1)
+```
+./_build/bin/moqrelayserver -port 4433 -cert ./certs/certificate.pem -key ./certs/certificate.key -endpoint "/moq" --logging DBG1
+```
+
+Note: Look at [README.md](./README.md) on how to gernerate certificates
+
+- Stream FLV live stream in real time to a fifo (Speed = x1) (terminal2)
+```
+mkfifo ~/Movies/fifo.flv
+
+ffmpeg -y -f lavfi -re -i smptebars=duration=300:size=320x200:rate=30 -f lavfi -re -i sine=frequency=1000:duration=300:sample_rate=48000 -pix_fmt yuv420p -c:v libx264 -b:v 180k -g 60 -keyint_min 60 -profile:v baseline -preset veryfast -c:a aac -b:a 96k -vf "drawtext=fontfile=/usr/share/fonts/dejavu-sans-fonts/DejaVuSans.ttf: text=\'Local time %{localtime\: %Y\/%m\/%d %H.%M.%S} (%{n})\': x=10: y=10: fontsize=16: fontcolor=white: box=1: boxcolor=0x00000099" -f flv ~/Movies/fifo.flv
+```
+
+Note: fontfile location can change depending on OS
+
+- Connect streamer to that fifo (terminal3)
+```
+./_build/bin/moqflvstreamerclient -input_flv_file ~/Movies/fifo.flv --logging DBG1
+```
+This will read and demux FLV data (expecting 1 video in h264 and 1 audio in AAC-LC) and will announce the namespace `flvstreamer` (dafeult value) to the relay
+
+## Save stream to a file
+
+Assuming we already did all specified in [Stream VOD file simulating live stream](#stream-vod-file-simulating-live-stream)
+
+
+- Start receiver (terminal4)
+```
+./_build/bin/moqflvreceiverclient --flv_outpath ~/Movies/my-moq-out.flv --logging DBG1
+```
+This will subscribe to video `video0` and audio `audio0` tracks from `flvstreamer` namespace (default values), demux the data from MOQMi, and mux it into valid FLV data. It is expecting 1 video in h264 and 1 audio in AAC-LC.
+
+- Check / play the file
+
+You can check the correctness of the genated file with `ffprobe`
+```
+ffprobe ~/Movies/my-moq-out.flv
+...
+Input #0, flv, from '/Users/jcenzano/Movies/my-moq-out.flv':
+  Duration: 00:01:08.82, start: 55.573000, bitrate: 37 kb/s
+  Stream #0:0: Audio: aac (LC), 48000 Hz, mono, fltp
+  Stream #0:1: Video: h264 (Constrained Baseline), yuv420p(tv, bt470bg/unknown/unknown, progressive), 320x200 [SAR 1:1 DAR 8:5], 30.30 fps, 30 tbr, 1k tbn
+```
+
+Or if you system has UI, you can play it back with any media player, for instance:
+```
+ffplay ~/Movies/my-moq-out.flv
+```
+
+## Watch it live (OS needs UI)
+
+Assuming we already did all specified in [Stream VOD file simulating live stream](#stream-vod-file-simulating-live-stream)
+
+- Create out fifo (terminal4)
+```
+mkfifo ~/Movies/fifo-out.flv
+```
+
+- Start the player (terminal4)
+```
+ffplay ~/Movies/fifo-out.flv
+```
+
+- Start receiver (terminal5)
+```
+./_build/bin/moqflvreceiverclient --flv_outpath ~/Movies/fifo-out.flv --logging DBG1
+```
+
+![Live playback](./pics/moq-streamer-play.png)
+Fig3: Live playback via ffplay
+
+## Connect streamer app with browser (webcodecs) playback
+
+Assuming you have moxygen running in localhost with following params (terminal1):
+```
+./_build/bin/moqrelayserver -port 4433 -cert ../moq-encoder-player/certs/certificate.pem -key ../moq-encoder-player/certs/certificate.key  -endpoint "/moq" --logging DBG1
+```
+
+- Stream FLV live stream in real time to a fifo (Speed = x1) (terminal2)
+```
+mkfifo ~/Movies/fifo.flv
+
+ffmpeg -y -f lavfi -re -i smptebars=duration=300:size=320x200:rate=30 -f lavfi -re -i sine=frequency=1000:duration=300:sample_rate=48000 -pix_fmt yuv420p -c:v libx264 -b:v 180k -g 60 -keyint_min 60 -profile:v baseline -preset veryfast -c:a aac -b:a 96k -vf "drawtext=fontfile=/usr/share/fonts/dejavu-sans-fonts/DejaVuSans.ttf: text=\'Local time %{localtime\: %Y\/%m\/%d %H.%M.%S} (%{n})\': x=10: y=10: fontsize=16: fontcolor=white: box=1: boxcolor=0x00000099" -f flv ~/Movies/fifo.flv
+```
+
+Note: fontfile location can change depending on OS
+
+- Connect streamer to that fifo (terminal3)
+```
+./_build/bin/moqflvstreamerclient -input_flv_file ~/Movies/fifo.flv --logging DBG1
+```
+This will read and demux FLV data (expecting 1 video in h264 and 1 audio in AAC-LC) and will announce the namespace `flvstreamer` (dafeult value) to the relay
+
+- Install [moq-encoder-player](https://github.com/facebookexperimental/moq-encoder-player) following [this instructions](./README.md#local-test-with-web-media-client)
+
+- Open player page `http://localhost:8080/src-player/?local` (assuming port 8080 for webserver)
+    - Set "Namespace" to `flvstreamer`
+    - Remove "Track name"
+    - Click "Start"
+
+![Live playback in browser](./pics/moq-wc-play-from-app.png)
+Fig4: Live playback in the brower using Webcodecs
+
+## Connect browser streamer (webcodecs) and play it in application
+
+Assuming you have moxygen running in localhost with following params:
+```
+./_build/bin/moqrelayserver -port 4433 -cert ../moq-encoder-player/certs/certificate.pem -key ../moq-encoder-player/certs/certificate.key  -endpoint "/moq" --logging DBG1
+```
+
+- Install [moq-encoder-player](https://github.com/facebookexperimental/moq-encoder-player) following [this instructions](./README.md#local-test-with-web-media-client)
+
+- Open encoder page `http://localhost:8080/src-encoder/?local` (assuming port 8080 for webserver)
+    - Set "Namespace" to `flvstreamer`
+    - Remove "Track name prefix"
+    - Change audio codec to "AAC-LC"
+    - Click "Start"
+
+- Create out fifo (terminal4)
+```
+mkfifo ~/Movies/fifo-out.flv
+```
+
+- Start the player (terminal4)
+```
+ffplay ~/Movies/fifo-out.flv
+```
+
+- Start receiver (terminal5)
+```
+./_build/bin/moqflvreceiverclient --flv_outpath ~/Movies/fifo-out.flv --logging DBG1
+```
+
+**TODO: We need to define authinfo in the receiver and de-jitter!!!!**
+
+```
+src-encoder/?local&verbose=1:809 [MOQ-SENDER] Invalid subscribe authInfo undefined does not match with {"audio":{"namespace":"flvstreamer","name":"audio0","maxInFlightRequests":60,"isHipri":true,"authInfo":"secret","moqMapping":"SubGroupPerObj"},"video":{"namespace":"flvstreamer","name":"video0","maxInFlightRequests":39,"isHipri":false,"authInfo":"secret","moqMapping":"SubGroupPerObj"}}
+```
+
+![Live playback](./pics/moq-streamer-play-from-wc.png)
+Fig5: Live playback from WebCodecs via ffplay
diff --git a/moxygen/CMakeLists.txt b/moxygen/CMakeLists.txt
@@ -43,6 +43,7 @@ add_subdirectory(relay)
 add_subdirectory(samples/text-client)
 add_subdirectory(samples/date)
 add_subdirectory(samples/flv_streamer_client)
+add_subdirectory(samples/flv_receiver_client)
 add_subdirectory(moq_mi)
 add_subdirectory(flv_parser)
 add_subdirectory(test)
diff --git a/moxygen/samples/flv_receiver_client/CMakeLists.txt b/moxygen/samples/flv_receiver_client/CMakeLists.txt
@@ -25,6 +25,8 @@ target_compile_options(
 target_link_libraries(
   moqflvreceiverclient PUBLIC
   Folly::folly
+  flvparser
+  moqmi
   moxygen
 )
 

diff --git a/moxygen/samples/flv_receiver_client/MoQFlvReceiverClient.cpp b/moxygen/samples/flv_receiver_client/MoQFlvReceiverClient.cpp
@@ -154,6 +154,8 @@ class TrackReceiverHandler : public ObjectReceiverCallback {
       : trackMediaType_(TrackType(mediaType)) {}
   ~TrackReceiverHandler() override = default;
   FlowControlState onObject(const ObjectHeader&, Payload payload) override {
+    // TODO: Add jitter buffer to fix out of order packets, we will need latency
+    // parameter to determine how much to buffer we want
     if (payload) {
       auto payloadSize = payload->computeChainDataLength();
       XLOG(DBG1) << trackMediaType_.toStr()

diff --git a/pics/moq-streamer-play.png b/pics/moq-streamer-play.png
diff --git a/pics/moq-wc-play-from-app.png b/pics/moq-wc-play-from-app.png
diff --git a/pics/moq_streamer_receiver.png b/pics/moq_streamer_receiver.png
diff --git a/pics/sdi-to-moq.png b/pics/sdi-to-moq.png