From 3bd959b4ef53cffbddad6d605a036d085aa48969 Mon Sep 17 00:00:00 2001 From: sergystepanov Date: Mon, 9 Jan 2023 23:20:22 +0300 Subject: [PATCH] Refactored v3 (#350) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains refactored code. **Changelog** - Added new net code (the communication architecture was left intact). - All network client IDs now have custom type `network.Uid` backed by github.com/rs/xid lib. ``` The string representation of a UUID takes 32 bytes, and the new type will take just 16. Because of Golang JSON serialization problems with omitting zero-length empty slices (it can't) and the need to use UID values as map keys (maps don't support slices as keys), IDs are stored as strings (for now). ``` - A whole new WebSocket client/server implementation was added, as well as a new communication layer with synchronous and async call handlers. - WebSocket connections now support dedicated Ping/Pong frames as opposed to original ping text messages. - Used Gorilla WebSocket library doesn't allow concurrent (simultaneous) reads and writes, so this part was handled via send channel synchronization. - New API structures can be found in the `pkg/api` folder. - New communication protocol can be found in the `pkg/com/*` folder. - Updated communication protocol is based on JSON-encoded messaging through WebSocket and has the following structure: ``` Packet [id] string — a globally unique identification tag for the packet to track it trough a chain of requests. t uint8 — contains packet type information (i.e. INIT_PACKET, SDP_OFFER_PACKET, ...). [p] any — contains packet data (any type). Each packet is a text message in JSON-serialized form (WebSocket control frames obviously not). ``` ``` The main principle of this protocol and the duplex data exchange is: the one who initializes connection is called a client, and the one who is being connected to is called a server. With the current architecture, the coordinator is the server, the user browsers and workers are the clients. ____ ____ ↓ ↑ ↑ ↓ browser ⟶ coordinator ⟵ worker (c) (s) (c) One of the most crucial performance vise parts of these interactions is that all the server-initiated calls to clients should be asynchronous! ``` - In order to track synchronous calls (packets) with an asynchronous protocol, such as WebSocket, each packet may have an `id` that should be copied in all subsequent requests/responses. - The old `sessionID` param was replaced by `id` that should be stored inside the `p` (payload) part of the packet. - It is possible to skip the default ping check for all connected workers on every user connection and just pick the first available with the new roundRobin param in the coordinator config file `coordinator.roundRobin: true/false`. - Added a dedicated package for the system API (pkg/api/*). - Added structured logging system (zerolog) for better logging and cloud services integration. - Added a visual representation of the network message exchange in logs: ``` ... 01:00:01.1078 3f98 INF w → c Handshake ws://localhost:8000/wso 01:00:01.1138 994 INF c ← w Handshake localhost:8000 01:00:01.1148 994 INF c ← w Connect cid=cep.hrg 01:00:01.1158 994 DBG c connection id has been changed to cepl7obdrc3jv66kp2ug cid=cep.hrg 01:00:01.1158 3f98 INF w → c Connect cid=cep.2ug 01:00:01.1158 994 INF c New worker / addr: localhost, ... 01:00:01.1158 3f98 INF w Connected to the coordinator localhost:8000 cid=cep.2ug 01:00:02.5834 994 INF c ← u Handshake localhost:8000 01:00:02.6175 994 INF c ← u Connect cid=cep.hs0 01:00:02.6209 994 INF c Search available workers cid=cep.hs0 01:00:02.6214 994 INF c Found next free worker cid=cep.hs0 01:00:02.6220 994 INF c → u InitSession cid=cep.hs0 01:00:02.6527 994 INF c ← u WebrtcInit cid=cep.hs0 01:00:02.6527 994 INF c → w ᵇWebrtcInit cid=cep.hrg 01:00:02.6537 3f98 INF w ← c WebrtcInit cid=cep.2ug 01:00:02.6537 3f98 INF w WebRTC start cid=cep.2ug ... ``` - Replaced a monstrous Prometheus metrics lib. - Removed spflag dependency. - Added new `version` config file param/constant for compatibility reasons. - Bump the minimum required version for Go to 1.18 due to use of generics. - Opus encoder now is cached and the default config is 96Kbps, complexity 5 (was 196Kbps, 8). - Changed the default x264 quality parameters to `crf 23 / superfast / baseline` instead of `crf 17 / veryfast / main`. - Added a separate WebRTC logging config param `webrtc.logLevel`. - Worker now allocates much less memory. - Optimized and fixed RGB to YUV converter. - `--v=5` logging cmd flag was removed and replaced with the `debug` config parameter. **Breaking changes (migration to v3)** - Coordinator server API changes, see web/js/api/api.js. - Coordinator client event API changes: - c `GAME_PLAYER_IDX_CHANGE` (string) -> `GAME_PLAYER_IDX` (number) - c `GAME_PLAYER_IDX` -> `GAME_PLAYER_IDX_SET` - c `MEDIA_STREAM_INITIALIZED` -> `WEBRTC_NEW_CONNECTION` - c `MEDIA_STREAM_SDP_AVAILABLE` -> `WEBRTC_SDP_OFFER` - c `MEDIA_STREAM_CANDIDATE_ADD` -> `WEBRTC_ICE_CANDIDATE_RECEIVED` - c `MEDIA_STREAM_CANDIDATE_FLUSH` -> `WEBRTC_ICE_CANDIDATES_FLUSH` - x `MEDIA_STREAM_READY` -> **removed** - c `CONNECTION_READY` -> `WEBRTC_CONNECTION_READY` - c `CONNECTION_CLOSED` -> `WEBRTC_CONNECTION_CLOSED` - c `GET_SERVER_LIST` -> `WORKER_LIST_FETCHED` - x `KEY_STATE_UPDATED` -> **removed** - n `WEBRTC_ICE_CANDIDATE_FOUND` - n `WEBRTC_SDP_ANSWER` - n `MESSAGE` - `rtcp` module renamed to `webrtc`. - Controller state equals Libretro controller state (changed order of bits), see: web/js/input/input.js. - Added new `coordintaor.selector` config param that changes the selection algorithm for workers. By default it will select any free worker. Set this param to `ping` for the old behavior. - Changed the name of the `webrtc.iceServers.url` config param to `webrtc.iceServers.urls`. --- .github/workflows/cd/docker-compose.yml | 4 +- .../workflows/{release.yml => release.yml_} | 2 +- .gitignore | 4 + Dockerfile | 2 +- Makefile | 72 +- README.md | 5 - assets/cores/nestopia_libretro.cfg | 1 + cmd/coordinator/main.go | 33 +- cmd/worker/main.go | 39 +- configs/config.yaml | 43 +- docker-compose.yml | 2 +- docs/{designdoc/README.md => DESIGNv2.md} | 4 +- docs/STREAMING.md | 47 -- docs/designdoc/implementation/README.md | 43 - docs/img/crowdplay.gif | Bin 2029343 -> 0 bytes docs/img/landing-page-dark.png | Bin 374615 -> 0 bytes docs/img/landing-page-front.png | Bin 286790 -> 0 bytes docs/img/landing-page-gb.png | Bin 285742 -> 0 bytes docs/img/landing-page-ps-hm.png | Bin 373582 -> 0 bytes docs/img/landing-page-ps-x4.png | Bin 315101 -> 0 bytes docs/img/landing-page.gif | Bin 8818062 -> 0 bytes docs/img/landing-page2.png | Bin 27029 -> 0 bytes docs/userguide/instruction/README.md | 28 - go.mod | 63 +- go.sum | 588 +++----------- pkg/api/api.go | 151 ++++ pkg/api/coordinator.go | 47 ++ pkg/api/user.go | 30 + pkg/api/worker.go | 68 ++ pkg/codec/codec.go | 8 - pkg/com/com.go | 93 +++ pkg/com/map.go | 98 +++ pkg/com/map_test.go | 32 + pkg/com/net.go | 187 +++++ pkg/com/net_test.go | 171 ++++ pkg/config/coordinator/config.go | 40 +- pkg/config/emulator/config.go | 27 +- pkg/config/encoder/config.go | 17 +- pkg/config/loader.go | 4 +- pkg/config/shared/config.go | 4 +- pkg/config/webrtc/config.go | 24 +- pkg/config/worker/config.go | 34 +- pkg/coordinator/balancer.go | 105 +++ pkg/coordinator/browser.go | 34 - pkg/coordinator/coordinator.go | 74 +- pkg/coordinator/handlers.go | 389 ---------- pkg/coordinator/http.go | 52 -- pkg/coordinator/hub.go | 162 ++++ pkg/coordinator/internalhandlers.go | 75 -- pkg/coordinator/routes.go | 34 - pkg/coordinator/user.go | 89 +++ pkg/coordinator/useragenthandlers.go | 308 -------- pkg/coordinator/userapi.go | 37 + pkg/coordinator/userhandlers.go | 151 ++++ pkg/coordinator/worker.go | 121 +-- pkg/coordinator/workerapi.go | 62 ++ pkg/coordinator/workerhandlers.go | 31 + pkg/cws/api/api.go | 23 - pkg/cws/api/coordinator.go | 94 --- pkg/cws/api/worker.go | 21 - pkg/cws/cws.go | 219 ------ pkg/downloader/backend/backend.go | 10 - pkg/downloader/backend/grab.go | 58 -- pkg/downloader/downloader.go | 39 - pkg/downloader/pipe/pipe.go | 29 - pkg/emulator/emulator.go | 41 - pkg/emulator/graphics/opengl.go | 140 ---- pkg/emulator/graphics/sdl.go | 132 ---- pkg/emulator/image/color.go | 27 - pkg/emulator/image/draw.go | 82 -- pkg/emulator/libretro/nanoarch/cfuncs.go | 204 ----- .../libretro/nanoarch/configscanner.go | 46 -- pkg/emulator/libretro/nanoarch/input.go | 97 --- pkg/emulator/libretro/nanoarch/input_test.go | 27 - pkg/emulator/libretro/nanoarch/naemulator.go | 252 ------ pkg/emulator/libretro/nanoarch/nanoarch.go | 697 ----------------- pkg/emulator/libretro/nanoarch/persistence.go | 45 -- pkg/emulator/libretro/nanoarch/state.go | 100 --- pkg/emulator/libretro/nanoarch/storage.go | 29 - pkg/emulator/libretro/nanoarch/zipstorage.go | 42 - pkg/encoder/h264/options.go | 33 - pkg/encoder/h264/x264.go | 126 --- pkg/encoder/opus/encoder.go | 54 -- pkg/encoder/opus/opus.go | 200 ----- pkg/encoder/pipe.go | 65 -- pkg/encoder/type.go | 21 - pkg/encoder/vpx/options.go | 17 - pkg/encoder/yuv/options.go | 42 - pkg/encoder/yuv/yuv.c | 143 ---- pkg/encoder/yuv/yuv.go | 138 ---- pkg/games/launcher.go | 42 + pkg/games/{game_library.go => library.go} | 32 +- .../{game_library_test.go => library_test.go} | 5 +- pkg/{session => games}/session.go | 7 +- pkg/ice/ice.go | 57 -- pkg/ice/ice_test.go | 78 -- pkg/lock/lock.go | 48 -- pkg/lock/lock_test.go | 38 - pkg/logger/logger.go | 181 +++++ pkg/media/buffer.go | 40 - pkg/media/buffer_test.go | 69 -- pkg/media/resampler.go | 25 - pkg/monitoring/monitoring.go | 87 +-- pkg/network/address.go | 26 + pkg/network/address_test.go | 26 + pkg/network/httpx/listener_test.go | 2 +- pkg/network/httpx/options.go | 3 + pkg/network/httpx/server.go | 106 ++- pkg/network/socket/socket.go | 6 +- pkg/network/uid.go | 22 + pkg/network/webrtc/factory.go | 90 +++ pkg/network/webrtc/pionlogger.go | 33 + pkg/network/webrtc/webrtc.go | 216 ++++++ pkg/network/websocket/websocket.go | 198 ++++- pkg/os/os.go | 19 +- pkg/recorder/ffmpegstream.go | 121 --- pkg/service/service.go | 30 +- pkg/storage/noop.go | 19 - pkg/storage/storage.go | 6 - pkg/util/logging/init.go | 38 - pkg/webrtc/connection.go | 91 --- pkg/webrtc/webrtc.go | 370 --------- pkg/worker/cloudsave.go | 56 ++ pkg/{ => worker}/compression/compression.go | 7 +- .../compression/zip/compression.go | 25 +- .../compression/zip/compression_test.go | 4 + pkg/worker/coordinator.go | 140 +++- pkg/worker/coordinatorhandlers.go | 209 +++++ pkg/worker/emulator/emulator.go | 72 ++ pkg/{ => worker}/emulator/graphics/context.go | 0 .../emulator/graphics}/gl/KHR/khrplatform.h | 0 .../emulator/graphics}/gl/gl.go | 56 +- pkg/worker/emulator/graphics/opengl.go | 131 ++++ pkg/worker/emulator/graphics/sdl.go | 139 ++++ pkg/worker/emulator/image/draw.go | 112 +++ pkg/worker/emulator/image/draw_test.go | 71 ++ pkg/{ => worker}/emulator/image/rotation.go | 33 +- .../emulator/image/rotation_test.go | 0 pkg/{ => worker}/emulator/image/scale.go | 9 +- .../core => worker/emulator/libretro}/core.go | 2 +- pkg/worker/emulator/libretro/frontend.go | 240 ++++++ .../emulator/libretro/frontend_test.go} | 53 +- .../emulator/libretro}/libretro.h | 0 .../emulator/libretro}/loader.go | 22 +- .../emulator/libretro/manager/manager.go | 13 +- .../libretro/manager/remotehttp/downloader.go | 63 ++ .../libretro/manager/remotehttp/grab.go | 59 ++ .../libretro/manager/remotehttp/manager.go | 72 +- .../manager/remotehttp/manager_test.go | 0 pkg/worker/emulator/libretro/nanoarch.c | 206 +++++ pkg/worker/emulator/libretro/nanoarch.go | 734 ++++++++++++++++++ pkg/worker/emulator/libretro/nanoarch.h | 38 + .../emulator/libretro}/nanoarch_test.go | 121 ++- pkg/worker/emulator/libretro/properties.go | 71 ++ .../libretro/repo/buildbot/repository.go | 6 +- .../libretro/repo/buildbot/repository_test.go | 13 +- .../libretro/repo/github/repository.go | 6 +- .../libretro/repo/github/repository_test.go | 13 +- .../emulator/libretro/repo/raw/repository.go | 6 +- .../emulator/libretro/repo/repository.go | 10 +- pkg/worker/emulator/libretro/storage.go | 67 ++ .../emulator/libretro/storage_test.go} | 2 +- pkg/worker/encoder/encoder.go | 70 ++ pkg/{ => worker}/encoder/h264/libx264.go | 423 +--------- pkg/worker/encoder/h264/x264.go | 149 ++++ pkg/worker/encoder/opus/opus.go | 198 +++++ pkg/{ => worker}/encoder/vpx/libvpx.go | 42 +- pkg/worker/encoder/yuv/yuv.c | 130 ++++ pkg/worker/encoder/yuv/yuv.go | 125 +++ pkg/{ => worker}/encoder/yuv/yuv.h | 20 +- pkg/{ => worker}/encoder/yuv/yuv_test.go | 166 +++- pkg/worker/handlers.go | 233 ------ pkg/worker/http.go | 31 - pkg/worker/internalhandlers.go | 314 -------- pkg/worker/media.go | 170 ++++ pkg/worker/media_test.go | 214 +++++ pkg/{ => worker}/recorder/draw.go | 0 pkg/worker/recorder/ffmpegmux.go | 63 ++ pkg/{ => worker}/recorder/file.go | 0 pkg/{ => worker}/recorder/options.go | 3 +- pkg/worker/recorder/pngstream.go | 74 ++ pkg/{ => worker}/recorder/recorder.go | 106 +-- pkg/{ => worker}/recorder/recorder_test.go | 15 +- pkg/{ => worker}/recorder/wavstream.go | 44 +- pkg/{ => worker}/recorder/zipfile.go | 0 pkg/worker/recording.go | 71 ++ pkg/worker/room.go | 187 +++++ pkg/worker/room/media.go | 145 ---- pkg/worker/room/media_test.go | 87 --- pkg/worker/room/room.go | 480 ------------ pkg/worker/{room => }/room_test.go | 160 ++-- pkg/worker/router.go | 57 ++ pkg/worker/routes.go | 23 - pkg/worker/session.go | 21 - pkg/{storage/oracle.go => worker/storage.go} | 26 +- .../oracle_test.go => worker/storage_test.go} | 10 +- pkg/{ => worker}/thread/mainthread_darwin.go | 2 +- .../thread/mainthread_darwin_test.go | 0 pkg/{ => worker}/thread/thread.go | 1 - pkg/{ => worker}/thread/thread_darwin.go | 0 pkg/worker/worker.go | 88 ++- tests/e2e/main_test.go | 516 ------------ web/index.html | 31 +- web/js/api/api.js | 64 ++ web/js/controller.js | 167 ++-- web/js/event/event.js | 22 +- web/js/input/input.js | 92 ++- web/js/input/keyboard.js | 57 +- web/js/input/touch.js | 3 +- web/js/network/socket.js | 145 +--- web/js/network/{rtcp.js => webrtc.js} | 124 ++- web/js/stats/stats.js | 25 +- web/js/workerManager.js | 19 +- 213 files changed, 7977 insertions(+), 9280 deletions(-) rename .github/workflows/{release.yml => release.yml_} (99%) create mode 100644 assets/cores/nestopia_libretro.cfg rename docs/{designdoc/README.md => DESIGNv2.md} (96%) delete mode 100644 docs/STREAMING.md delete mode 100644 docs/designdoc/implementation/README.md delete mode 100644 docs/img/crowdplay.gif delete mode 100644 docs/img/landing-page-dark.png delete mode 100644 docs/img/landing-page-front.png delete mode 100644 docs/img/landing-page-gb.png delete mode 100644 docs/img/landing-page-ps-hm.png delete mode 100644 docs/img/landing-page-ps-x4.png delete mode 100644 docs/img/landing-page.gif delete mode 100644 docs/img/landing-page2.png delete mode 100644 docs/userguide/instruction/README.md create mode 100644 pkg/api/api.go create mode 100644 pkg/api/coordinator.go create mode 100644 pkg/api/user.go create mode 100644 pkg/api/worker.go delete mode 100644 pkg/codec/codec.go create mode 100644 pkg/com/com.go create mode 100644 pkg/com/map.go create mode 100644 pkg/com/map_test.go create mode 100644 pkg/com/net.go create mode 100644 pkg/com/net_test.go create mode 100644 pkg/coordinator/balancer.go delete mode 100644 pkg/coordinator/browser.go delete mode 100644 pkg/coordinator/handlers.go delete mode 100644 pkg/coordinator/http.go create mode 100644 pkg/coordinator/hub.go delete mode 100644 pkg/coordinator/internalhandlers.go delete mode 100644 pkg/coordinator/routes.go create mode 100644 pkg/coordinator/user.go delete mode 100644 pkg/coordinator/useragenthandlers.go create mode 100644 pkg/coordinator/userapi.go create mode 100644 pkg/coordinator/userhandlers.go create mode 100644 pkg/coordinator/workerapi.go create mode 100644 pkg/coordinator/workerhandlers.go delete mode 100644 pkg/cws/api/api.go delete mode 100644 pkg/cws/api/coordinator.go delete mode 100644 pkg/cws/api/worker.go delete mode 100644 pkg/cws/cws.go delete mode 100644 pkg/downloader/backend/backend.go delete mode 100644 pkg/downloader/backend/grab.go delete mode 100644 pkg/downloader/downloader.go delete mode 100644 pkg/downloader/pipe/pipe.go delete mode 100644 pkg/emulator/emulator.go delete mode 100644 pkg/emulator/graphics/opengl.go delete mode 100644 pkg/emulator/graphics/sdl.go delete mode 100644 pkg/emulator/image/color.go delete mode 100644 pkg/emulator/image/draw.go delete mode 100644 pkg/emulator/libretro/nanoarch/cfuncs.go delete mode 100644 pkg/emulator/libretro/nanoarch/configscanner.go delete mode 100644 pkg/emulator/libretro/nanoarch/input.go delete mode 100644 pkg/emulator/libretro/nanoarch/input_test.go delete mode 100644 pkg/emulator/libretro/nanoarch/naemulator.go delete mode 100644 pkg/emulator/libretro/nanoarch/nanoarch.go delete mode 100644 pkg/emulator/libretro/nanoarch/persistence.go delete mode 100644 pkg/emulator/libretro/nanoarch/state.go delete mode 100644 pkg/emulator/libretro/nanoarch/storage.go delete mode 100644 pkg/emulator/libretro/nanoarch/zipstorage.go delete mode 100644 pkg/encoder/h264/options.go delete mode 100644 pkg/encoder/h264/x264.go delete mode 100644 pkg/encoder/opus/encoder.go delete mode 100644 pkg/encoder/opus/opus.go delete mode 100644 pkg/encoder/pipe.go delete mode 100644 pkg/encoder/type.go delete mode 100644 pkg/encoder/vpx/options.go delete mode 100644 pkg/encoder/yuv/options.go delete mode 100644 pkg/encoder/yuv/yuv.c delete mode 100644 pkg/encoder/yuv/yuv.go create mode 100644 pkg/games/launcher.go rename pkg/games/{game_library.go => library.go} (86%) rename pkg/games/{game_library_test.go => library_test.go} (91%) rename pkg/{session => games}/session.go (76%) delete mode 100644 pkg/ice/ice.go delete mode 100644 pkg/ice/ice_test.go delete mode 100644 pkg/lock/lock.go delete mode 100644 pkg/lock/lock_test.go create mode 100644 pkg/logger/logger.go delete mode 100644 pkg/media/buffer.go delete mode 100644 pkg/media/buffer_test.go delete mode 100644 pkg/media/resampler.go create mode 100644 pkg/network/address.go create mode 100644 pkg/network/address_test.go create mode 100644 pkg/network/uid.go create mode 100644 pkg/network/webrtc/factory.go create mode 100644 pkg/network/webrtc/pionlogger.go create mode 100644 pkg/network/webrtc/webrtc.go delete mode 100644 pkg/recorder/ffmpegstream.go delete mode 100644 pkg/storage/noop.go delete mode 100644 pkg/storage/storage.go delete mode 100644 pkg/util/logging/init.go delete mode 100644 pkg/webrtc/connection.go delete mode 100644 pkg/webrtc/webrtc.go create mode 100644 pkg/worker/cloudsave.go rename pkg/{ => worker}/compression/compression.go (50%) rename pkg/{ => worker}/compression/zip/compression.go (87%) rename pkg/{ => worker}/compression/zip/compression_test.go (92%) create mode 100644 pkg/worker/coordinatorhandlers.go create mode 100644 pkg/worker/emulator/emulator.go rename pkg/{ => worker}/emulator/graphics/context.go (100%) rename pkg/{emulator/backend => worker/emulator/graphics}/gl/KHR/khrplatform.h (100%) rename pkg/{emulator/backend => worker/emulator/graphics}/gl/gl.go (91%) create mode 100644 pkg/worker/emulator/graphics/opengl.go create mode 100644 pkg/worker/emulator/graphics/sdl.go create mode 100644 pkg/worker/emulator/image/draw.go create mode 100644 pkg/worker/emulator/image/draw_test.go rename pkg/{ => worker}/emulator/image/rotation.go (70%) rename pkg/{ => worker}/emulator/image/rotation_test.go (100%) rename pkg/{ => worker}/emulator/image/scale.go (73%) rename pkg/{emulator/libretro/core => worker/emulator/libretro}/core.go (98%) create mode 100644 pkg/worker/emulator/libretro/frontend.go rename pkg/{emulator/libretro/nanoarch/persistence_test.go => worker/emulator/libretro/frontend_test.go} (83%) rename pkg/{emulator/libretro/nanoarch => worker/emulator/libretro}/libretro.h (100%) rename pkg/{emulator/libretro/nanoarch => worker/emulator/libretro}/loader.go (92%) rename pkg/{ => worker}/emulator/libretro/manager/manager.go (69%) create mode 100644 pkg/worker/emulator/libretro/manager/remotehttp/downloader.go create mode 100644 pkg/worker/emulator/libretro/manager/remotehttp/grab.go rename pkg/{ => worker}/emulator/libretro/manager/remotehttp/manager.go (53%) rename pkg/{ => worker}/emulator/libretro/manager/remotehttp/manager_test.go (100%) create mode 100644 pkg/worker/emulator/libretro/nanoarch.c create mode 100644 pkg/worker/emulator/libretro/nanoarch.go create mode 100644 pkg/worker/emulator/libretro/nanoarch.h rename pkg/{emulator/libretro/nanoarch => worker/emulator/libretro}/nanoarch_test.go (71%) create mode 100644 pkg/worker/emulator/libretro/properties.go rename pkg/{ => worker}/emulator/libretro/repo/buildbot/repository.go (71%) rename pkg/{ => worker}/emulator/libretro/repo/buildbot/repository_test.go (82%) rename pkg/{ => worker}/emulator/libretro/repo/github/repository.go (56%) rename pkg/{ => worker}/emulator/libretro/repo/github/repository_test.go (82%) rename pkg/{ => worker}/emulator/libretro/repo/raw/repository.go (66%) rename pkg/{ => worker}/emulator/libretro/repo/repository.go (60%) create mode 100644 pkg/worker/emulator/libretro/storage.go rename pkg/{emulator/libretro/nanoarch/zipstorage_test.go => worker/emulator/libretro/storage_test.go} (97%) create mode 100644 pkg/worker/encoder/encoder.go rename pkg/{ => worker}/encoder/h264/libx264.go (56%) create mode 100644 pkg/worker/encoder/h264/x264.go create mode 100644 pkg/worker/encoder/opus/opus.go rename pkg/{ => worker}/encoder/vpx/libvpx.go (85%) create mode 100644 pkg/worker/encoder/yuv/yuv.c create mode 100644 pkg/worker/encoder/yuv/yuv.go rename pkg/{ => worker}/encoder/yuv/yuv.h (61%) rename pkg/{ => worker}/encoder/yuv/yuv_test.go (87%) delete mode 100644 pkg/worker/handlers.go delete mode 100644 pkg/worker/http.go delete mode 100644 pkg/worker/internalhandlers.go create mode 100644 pkg/worker/media.go create mode 100644 pkg/worker/media_test.go rename pkg/{ => worker}/recorder/draw.go (100%) create mode 100644 pkg/worker/recorder/ffmpegmux.go rename pkg/{ => worker}/recorder/file.go (100%) rename pkg/{ => worker}/recorder/options.go (91%) create mode 100644 pkg/worker/recorder/pngstream.go rename pkg/{ => worker}/recorder/recorder.go (57%) rename pkg/{ => worker}/recorder/recorder_test.go (90%) rename pkg/{ => worker}/recorder/wavstream.go (68%) rename pkg/{ => worker}/recorder/zipfile.go (100%) create mode 100644 pkg/worker/recording.go create mode 100644 pkg/worker/room.go delete mode 100644 pkg/worker/room/media.go delete mode 100644 pkg/worker/room/media_test.go delete mode 100644 pkg/worker/room/room.go rename pkg/worker/{room => }/room_test.go (71%) create mode 100644 pkg/worker/router.go delete mode 100644 pkg/worker/routes.go delete mode 100644 pkg/worker/session.go rename pkg/{storage/oracle.go => worker/storage.go} (82%) rename pkg/{storage/oracle_test.go => worker/storage_test.go} (81%) rename pkg/{ => worker}/thread/mainthread_darwin.go (89%) rename pkg/{ => worker}/thread/mainthread_darwin_test.go (100%) rename pkg/{ => worker}/thread/thread.go (83%) rename pkg/{ => worker}/thread/thread_darwin.go (100%) delete mode 100644 tests/e2e/main_test.go create mode 100644 web/js/api/api.js rename web/js/network/{rtcp.js => webrtc.js} (58%) diff --git a/.github/workflows/cd/docker-compose.yml b/.github/workflows/cd/docker-compose.yml index ec68b8ac..2a4e31b1 100644 --- a/.github/workflows/cd/docker-compose.yml +++ b/.github/workflows/cd/docker-compose.yml @@ -18,7 +18,7 @@ services: coordinator: <<: *default-params - command: coordinator --v=5 + command: coordinator volumes: - ${APP_DIR:-/cloud-game}/cache:/usr/local/share/cloud-game/assets/cache - ${APP_DIR:-/cloud-game}/games:/usr/local/share/cloud-game/assets/games @@ -28,7 +28,7 @@ services: environment: - MESA_GL_VERSION_OVERRIDE=3.3 entrypoint: [ "/bin/sh", "-c", "xvfb-run -a $$@", "" ] - command: worker --v=5 --zone=${ZONE:-} + command: worker --zone=${ZONE:-} volumes: - ${APP_DIR:-/cloud-game}/cache:/usr/local/share/cloud-game/assets/cache - ${APP_DIR:-/cloud-game}/cores:/usr/local/share/cloud-game/assets/cores diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml_ similarity index 99% rename from .github/workflows/release.yml rename to .github/workflows/release.yml_ index e76f6050..9df1f646 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml_ @@ -41,7 +41,7 @@ jobs: - uses: actions/setup-go@v2 with: - go-version: ^1.18 + go-version: ^1.19 - name: Get Linux dev libraries and tools if: matrix.os == 'ubuntu-latest' diff --git a/.gitignore b/.gitignore index c5d83e3e..c2ecdfe2 100644 --- a/.gitignore +++ b/.gitignore @@ -67,6 +67,9 @@ _output/ ./build release/ vendor/ +tests/ +!tests/e2e/ +*.exe .dockerignore @@ -75,3 +78,4 @@ fbneo/ hi/ nvram/ *.mcd + diff --git a/Dockerfile b/Dockerfile index 349de259..23d50dc7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -20,7 +20,7 @@ RUN apt-get -qq update && apt-get -qq install --no-install-recommends -y \ && rm -rf /var/lib/apt/lists/* # go setup layer -ARG GO=go1.18.linux-amd64.tar.gz +ARG GO=go1.19.linux-amd64.tar.gz RUN wget -q https://golang.org/dl/$GO \ && rm -rf /usr/local/go \ && tar -C /usr/local -xzf $GO \ diff --git a/Makefile b/Makefile index 3960ba0b..af45bf55 100644 --- a/Makefile +++ b/Makefile @@ -1,11 +1,10 @@ -# Makefile includes some useful commands to build or format incentives -# More commands could be added - -# Variables PROJECT = cloud-game REPO_ROOT = github.com/giongto35 ROOT = ${REPO_ROOT}/${PROJECT} +CGO_CFLAGS='-g -O3 -funroll-loops' +CGO_LDFLAGS='-g -O3' + fmt: @goimports -w cmd pkg tests @gofmt -s -w cmd pkg tests @@ -13,47 +12,6 @@ fmt: compile: fmt @go install ./cmd/... -check: fmt - @golangci-lint run cmd/... pkg/... -# @staticcheck -checks="all,-S1*" ./cmd/... ./pkg/... ./tests/... - -dep: - go mod download -# go mod tidy - -# NOTE: there is problem with go mod vendor when it delete github.com/gen2brain/x264-go/x264c causing unable to build. https://github.com/golang/go/issues/26366 -#build.cross: build -# CGO_ENABLED=1 GOOS=darwin GOARC=amd64 go build --ldflags '-linkmode external -extldflags "-static"' -o bin/coordinator-darwin ./cmd/coordinator -# CGO_ENABLED=1 GOOS=darwin GOARC=amd64 go build --ldflags '-linkmode external -extldflags "-static"' -o bin/worker-darwin ./cmd/worker -# CC=arm-linux-musleabihf-gcc GOOS=linux GOARC=amd64 CGO_ENABLED=1 go build --ldflags '-linkmode external -extldflags "-static"' -o bin/coordinator-linu ./cmd/coordinator -# CC=arm-linux-musleabihf-gcc GOOS=linux GOARC=amd64 CGO_ENABLED=1 go build --ldflags '-linkmode external -extldflags "-static"' -o bin/worker-linux ./cmd/worker - -# A user can invoke tests in different ways: -# - make test runs all tests; -# - make test TEST_TIMEOUT=10 runs all tests with a timeout of 10 seconds; -# - make test TEST_PKG=./model/... only runs tests for the model package; -# - make test TEST_ARGS="-v -short" runs tests with the specified arguments; -# - make test-race runs tests with race detector enabled. -TEST_TIMEOUT = 60 -TEST_PKGS ?= ./cmd/... ./pkg/... -TEST_TARGETS := test-short test-verbose test-race test-cover -.PHONY: $(TEST_TARGETS) test tests -test-short: TEST_ARGS=-short -test-verbose: TEST_ARGS=-v -test-race: TEST_ARGS=-race -test-cover: TEST_ARGS=-cover -$(TEST_TARGETS): test - -test: compile - @go test -timeout $(TEST_TIMEOUT)s $(TEST_ARGS) $(TEST_PKGS) - -test-e2e: compile - @go test ./tests/e2e/... - -cover: - @go test -v -covermode=count -coverprofile=coverage.out $(TEST_PKGS) -# @$(GOPATH)/bin/goveralls -coverprofile=coverage.out -service=travis-ci -repotoken $(COVERALLS_TOKEN) - clean: @rm -rf bin @rm -rf build @@ -62,25 +20,33 @@ clean: build: mkdir -p bin/ go build -ldflags "-w -s -X 'main.Version=$(GIT_VERSION)'" -o bin/ ./cmd/coordinator - go build -buildmode=exe -tags static -ldflags "-w -s -X 'main.Version=$(GIT_VERSION)'" $(EXT_WFLAGS) -o bin/ ./cmd/worker + CGO_CFLAGS=${CGO_CFLAGS} CGO_LDFLAGS=${CGO_LDFLAGS} \ + go build -buildmode=exe -tags static \ + -ldflags "-w -s -X 'main.Version=$(GIT_VERSION)'" $(EXT_WFLAGS) \ + -o bin/ ./cmd/worker verify-cores: - go test -run TestAllEmulatorRooms ./pkg/worker/room -v -renderFrames $(GL_CTX) -outputPath "../../../_rendered" + go test -run TestAllEmulatorRooms ./pkg/worker -v -renderFrames $(GL_CTX) -outputPath "../../_rendered" dev.build: compile build dev.build-local: mkdir -p bin/ go build -o bin/ ./cmd/coordinator - go build -o bin/ ./cmd/worker + CGO_CFLAGS=${CGO_CFLAGS} CGO_LDFLAGS=${CGO_LDFLAGS} go build -o bin/ ./cmd/worker dev.run: dev.build-local - ./bin/coordinator --v=5 & - ./bin/worker --v=5 + ./bin/coordinator & ./bin/worker + +dev.run.debug: + go build -race -o bin/ ./cmd/coordinator + CGO_CFLAGS=${CGO_CFLAGS} CGO_LDFLAGS=${CGO_LDFLAGS} \ + go build -race -gcflags=all=-d=checkptr -o bin/ ./cmd/worker + ./bin/coordinator & ./bin/worker dev.run-docker: docker rm cloud-game-local -f || true - CLOUD_GAME_GAMES_PATH=$(PWD)/assets/games docker-compose up --build + docker-compose up --build # RELEASE # Builds the app for new release. @@ -97,8 +63,8 @@ dev.run-docker: # Config params: # - RELEASE_DIR: the name of the output folder (default: release). # - CONFIG_DIR: search dir for core config files. -# - DLIB_TOOL: the name of a dynamic lib copy tool (with params) (e.g., ldd -x -y; defalut: ldd). -# - DLIB_SEARCH_PATTERN: a grep filter of the output of the DLIB_TOOL (e.g., mylib.so; default: .*so). +# - DLIB_TOOL: the name of a dynamic lib copy tool (with params) (e.g., ldd -x -y; default: ldd). +# - DLIB_SEARCH_PATTERN: a grep filter of the output of the DLIB_TOOL (e.g., my_lib.so; default: .*so). # Be aware that this search pattern will return only matched regular expression part and not the whole line. # de. -> abc def ghj -> def # Makefile special symbols should be escaped with \. diff --git a/README.md b/README.md index ec092fd6..58734c9b 100644 --- a/README.md +++ b/README.md @@ -31,11 +31,6 @@ Because I only hosted the platform on limited servers in US East, US West, Eu, S latency issues + connection problem. You can try hosting the service following the instruction the next section to have a better sense of performance. -| Screenshot | Screenshot | -| :--------------------------------------------: | :--------------------------------------------: | -| ![screenshot](docs/img/landing-page-ps-hm.png) | ![screenshot](docs/img/landing-page-ps-x4.png) | -| ![screenshot](docs/img/landing-page-gb.png) | ![screenshot](docs/img/landing-page-front.png) | - ## Feature 1. **Cloud gaming**: Game logic and storage is hosted on cloud service. It reduces the cumbersome of game diff --git a/assets/cores/nestopia_libretro.cfg b/assets/cores/nestopia_libretro.cfg new file mode 100644 index 00000000..fd06bde7 --- /dev/null +++ b/assets/cores/nestopia_libretro.cfg @@ -0,0 +1 @@ +nestopia_audio_type=stereo diff --git a/cmd/coordinator/main.go b/cmd/coordinator/main.go index 40dc25d5..816390e6 100644 --- a/cmd/coordinator/main.go +++ b/cmd/coordinator/main.go @@ -1,40 +1,33 @@ package main import ( - "context" - goflag "flag" "math/rand" "time" config "github.com/giongto35/cloud-game/v2/pkg/config/coordinator" "github.com/giongto35/cloud-game/v2/pkg/coordinator" + "github.com/giongto35/cloud-game/v2/pkg/logger" "github.com/giongto35/cloud-game/v2/pkg/os" - "github.com/giongto35/cloud-game/v2/pkg/util/logging" - "github.com/golang/glog" - flag "github.com/spf13/pflag" ) -var Version = "" - -func init() { - rand.Seed(time.Now().UTC().UnixNano()) -} +var Version = "?" func main() { + rand.Seed(time.Now().UTC().UnixNano()) conf := config.NewConfig() - flag.CommandLine.AddGoFlagSet(goflag.CommandLine) conf.ParseFlags() - logging.Init() - defer logging.Flush() + log := logger.NewConsole(conf.Coordinator.Debug, "c", true) - glog.Infof("[coordinator] version: %v", Version) - glog.V(4).Infof("[coordinator] Local configuration %+v", conf) - c := coordinator.New(conf) + log.Info().Msgf("version %s", Version) + log.Info().Msgf("conf version: %v", conf.Version) + if log.GetLevel() < logger.InfoLevel { + log.Debug().Msgf("config: %+v", conf) + } + c := coordinator.New(conf, log) c.Start() - - ctx, cancelCtx := context.WithCancel(context.Background()) - defer c.Shutdown(ctx) <-os.ExpectTermination() - cancelCtx() + if err := c.Stop(); err != nil { + log.Error().Err(err).Msg("service shutdown errors") + } } diff --git a/cmd/worker/main.go b/cmd/worker/main.go index e625fd4f..b4fb1352 100644 --- a/cmd/worker/main.go +++ b/cmd/worker/main.go @@ -1,43 +1,38 @@ package main import ( - "context" - goflag "flag" "math/rand" "time" config "github.com/giongto35/cloud-game/v2/pkg/config/worker" + "github.com/giongto35/cloud-game/v2/pkg/logger" "github.com/giongto35/cloud-game/v2/pkg/os" - "github.com/giongto35/cloud-game/v2/pkg/thread" - "github.com/giongto35/cloud-game/v2/pkg/util/logging" "github.com/giongto35/cloud-game/v2/pkg/worker" - "github.com/golang/glog" - flag "github.com/spf13/pflag" + "github.com/giongto35/cloud-game/v2/pkg/worker/thread" ) -var Version = "" - -func init() { - rand.Seed(time.Now().UTC().UnixNano()) -} +var Version = "?" func run() { + rand.Seed(time.Now().UTC().UnixNano()) conf := config.NewConfig() - flag.CommandLine.AddGoFlagSet(goflag.CommandLine) conf.ParseFlags() - logging.Init() - defer logging.Flush() + log := logger.NewConsole(conf.Worker.Debug, "w", true) + log.Info().Msgf("version %s", Version) + log.Info().Msgf("conf version: %v", conf.Version) + if log.GetLevel() < logger.InfoLevel { + log.Debug().Msgf("config: %+v", conf) + } - glog.Infof("[worker] version: %v", Version) - glog.V(4).Infof("[worker] Local configuration %+v", conf) - wrk := worker.New(conf) + done := os.ExpectTermination() + wrk := worker.New(conf, log, done) wrk.Start() - - ctx, cancelCtx := context.WithCancel(context.Background()) - defer wrk.Shutdown(ctx) - <-os.ExpectTermination() - cancelCtx() + <-done + time.Sleep(100 * time.Millisecond) + if err := wrk.Stop(); err != nil { + log.Error().Err(err).Msg("service shutdown errors") + } } func main() { diff --git a/configs/config.yaml b/configs/config.yaml index 85217353..8ef789bf 100644 --- a/configs/config.yaml +++ b/configs/config.yaml @@ -2,14 +2,21 @@ # Application configuration file # +# for the compatibility purposes +version: 3 + coordinator: # debugging switch + # - shows debug logs # - allows selecting worker instances debug: false + # selects free workers: + # - any (default, any free) + # - ping (with the lowest ping) + selector: any # games library library: - # some directory which is gonna be the root folder for the library - # where games are stored + # root folder for the library (where games are stored) basePath: assets/games # an explicit list of supported file extensions # which overrides Libretro emulator ROMs configs @@ -54,6 +61,8 @@ coordinator: gtag: worker: + # show more logs + debug: false network: # a coordinator address to connect to coordinatorAddress: localhost:8000 @@ -111,9 +120,14 @@ emulator: # special tag {user} will be replaced with current user's home dir storage: "{user}/.cr/save" + # path for storing emulator generated files + localPath: "./libretro" + libretro: # use zip compression for emulator save states - saveCompression: false + saveCompression: true + # Libretro cores logging level: DEBUG = 0, INFO, WARN, ERROR, DUMMY = INT_MAX + logLevel: 1 cores: paths: libs: assets/cores @@ -176,6 +190,7 @@ emulator: roms: [ "zip" ] nes: lib: nestopia_libretro + config: nestopia_libretro.cfg roms: [ "nes" ] snes: lib: snes9x_libretro @@ -191,21 +206,23 @@ emulator: encoder: audio: - channels: 2 # audio frame duration needed for WebRTC (Opus) - frame: 20 - frequency: 48000 + # most of the emulators have ~1400 samples per a video frame, + # so we keep the frame buffer roughly half of that size or 2 RTC packets per frame + frame: 10 video: # h264, vpx (VP8) codec: h264 + # concurrent execution units (0 - disabled) + concurrency: 0 # see: https://trac.ffmpeg.org/wiki/Encode/H.264 h264: # Constant Rate Factor (CRF) 0-51 (default: 23) - crf: 17 + crf: 23 # ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow, placebo - preset: veryfast + preset: superfast # baseline, main, high, high10, high422, high444 - profile: main + profile: baseline # film, animation, grain, stillimage, psnr, ssim, fastdecode, zerolatency tune: zerolatency # 0-3 @@ -216,9 +233,6 @@ encoder: bitrate: 1200 # force keyframe interval keyframeInterval: 5 - # run without a game - # (experimental) - withoutGame: false # game recording # (experimental) @@ -269,7 +283,7 @@ webrtc: dtlsRole: # a list of STUN/TURN servers to use iceServers: - - url: stun:stun.l.google.com:19302 + - urls: stun:stun.l.google.com:19302 # configures whether the ice agent should be a lite agent (true/false) # (performance) # don't use iceServers when enabled @@ -287,3 +301,6 @@ webrtc: # override ICE candidate IP, see: https://github.com/pion/webrtc/issues/835, # can be used for Docker bridged network internal IP override iceIpMap: + # set additional log level for WebRTC separately + # -1 - trace, 6 - nothing, ..., debug - 0 + logLevel: 6 diff --git a/docker-compose.yml b/docker-compose.yml index 7dcef1a1..5fb3725c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -16,7 +16,7 @@ services: - 9000:9000 - 8443:8443/udp command: > - bash -c "Xvfb :99 & coordinator --v=5 & worker --v=5" + bash -c "Xvfb :99 & coordinator & worker" volumes: # keep cores persistent in the cloud-game_cores volume - cores:/usr/local/share/cloud-game/assets/cores diff --git a/docs/designdoc/README.md b/docs/DESIGNv2.md similarity index 96% rename from docs/designdoc/README.md rename to docs/DESIGNv2.md index b15cd503..f6ade821 100644 --- a/docs/designdoc/README.md +++ b/docs/DESIGNv2.md @@ -5,7 +5,7 @@ Web-based Cloud Gaming Service contains multiple workers for gaming stream and a ## Worker Worker is responsible for streaming game to frontend -![worker](../img/worker.png) +![worker](img/worker-internal.png) - After Coordinator matches the most appropriate server to the user, webRTC peer-to-peer handshake will be conducted. The coordinator will exchange the signature (WebRTC Session Remote Description) between two peers over Web Socket connection. - On worker, each user session will spawn a new room running a gaming emulator. Image stream and audio stream from emulator is captured and encoded to WebRTC streaming format. We applied Vp8 for Video compression and Opus for audio compression to ensure the smoothest experience. After finish encoded, these stream is then piped out to user and observers joining that room. @@ -16,7 +16,7 @@ Worker is responsible for streaming game to frontend Coordinator is loadbalancer and coordinator, which is in charge of picking the most suitable workers for a user. Every time a user connects to Coordinator, it will collect all the metric from all workers, i.e free CPU resources and latency from worker to user. Coordinator will decide the best candidate based on the metric and setup peer-to-peer connection between worker and user based on WebRTC protocol -![Architecture](../img/coordinator.png) +![Architecture](img/coordinator.png) 1. A user connected to Coordinator . 2. Coordinator will find the most suitable worker to serve the user. diff --git a/docs/STREAMING.md b/docs/STREAMING.md deleted file mode 100644 index 5eeb54ad..00000000 --- a/docs/STREAMING.md +++ /dev/null @@ -1,47 +0,0 @@ -## Streaming process description - -This document describes the step-by-step process of media streaming in all parts of the application. - -``` -┌──────────────┐ ┌───────────────┐ ┌──────────────┐ -│ USER AGENT │ │ COORDINATOR │ │ WORKER...n │ -├──────────────┤ ├───────────────┤ ├──────────────┤ -│ TCP/WS ├──1──►│ WS ──────► WS │◄───┤ TCP/WS │ -│ │ │ ▲ 2 │ │ │ │ -│ │ │ └───────────┘ │ │ │ -│ │ └───────────────┘ │ │ -│ UDP/RTP │◄─────────────3────────────┤ UDP/RTP │ -│ AUDIO < │ OPUS │ AUDIO │ -│ VIDEO < │ VP8/H264 │ VIDEO │ -│ DATA > │ 010101 │ DATA │ -└──────────────┘ └──────────────┘ -``` - -The app is based on WebRTC technology which allows the server to stream media and exchange data with ultra-low latencies. An essential part of these types of P2P connections is the signaling process. It's implemented as a custom text-based messaging protocol on top of WebSocket (quite similarly to [WAMP](https://wamp-proto.org)). The app supports both STUN and TURN protocols for NAT traversal or ICE. In terms of supported codecs, it can stream h264, VP8, and OPUS media. - -The streaming process begins when a user opens the main application page (index.html) served by the coordinator. -- The user's browser tries to open a new WebSocket connection to the coordinator — socket.init(roomId, zone) [web/js/network/socket.js:32](https://github.com/giongto35/cloud-game/blob/ae5260fb4726fd34cc0b0b05100dcc8457f52883/web/js/network/socket.js#L32) -> In the initial WebSocket Upgrade request query it may send two params: roomId — an identification number for existing game rooms stored in the URL query of the application page (i.e. app.com/?id=xxxxxx), zone — or, more precisely, region — serves the purpose of CDN and geographical segmentation of the streaming. -- On the coordinator side this request goes into a dedicated handler (/ws) — func (o *Server) WS(w http.ResponseWriter, r *http.Request) [pkg/coordinator/handlers.go:150](https://github.com/giongto35/cloud-game/blob/ae5260fb4726fd34cc0b0b05100dcc8457f52883/pkg/coordinator/handlers.go#L150) -- There, it unconditionally accepts the WebSocket connection and tags it with some ID, so it will be listening to messages from the user's side. Here a new client connection should be considered as established. -- Next, given provided query params, the coordinator tries to find a suitable worker whose job — directly stream games to a user. -> This process of choosing the right worker is following: if there is no roomId param, then the coordinator gathers the full list of available workers, filters them by a zone value (if provided), returns the user a list of public URLs, which he can ping and send results back to the coordinator. After that, the coordinator links the fastest one with the user. Alternatively, if the user did provide some roomId, then the coordinator directly assigns a worker with that room (workers have 1:1 mapping to rooms or games). -> All the information exchange initiated from the worker side is handled in a separate endpoint (/wso) [pkg/coordinator/handlers.go#L81](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/pkg/coordinator/handlers.go#L81). -- Coordinator sends to the user ICE servers and the list of games available for playing. That's handled in [web/js/network/socket.js:57](https://github.com/giongto35/cloud-game/blob/ae5260fb4726fd34cc0b0b05100dcc8457f52883/web/js/network/socket.js#L57). -- From this point, the user's browser begins to initialize WebRTC connection to the worker — web/js/controller.js:413 → [web/js/network/rtcp.js:16](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/web/js/network/rtcp.js#L16). -- First, it sends init request through the WebSocket connection to the coordinator handler in [pkg/coordinator/useragenthandlers.go:17](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/pkg/coordinator/useragenthandlers.go#L17). -> Following a standard WebRTC call [negotiation procedure](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Signaling_and_video_calling), the coordinator acts as a mediator between users and workers. The signaling protocol here is a text messaging through WebSocket transport. -- Coordinator notifies the user's worker that it wants to establish a new PeerConnection (call). That part is being handled in [pkg/worker/internalhandlers.go:42](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/pkg/worker/internalhandlers.go#L42). It is worth noting that it is a worker who makes SDP offer and waits for an SDP answer. -- Worker initializes new WebRTC connection handler in func (w *WebRTC) StartClient(isMobile bool, iceCB OnIceCallback) (string, error) [pkg/webrtc/webrtc.go:103](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/pkg/webrtc/webrtc.go#L103). -- Then through the coordinator it makes simultaneously an SDP offer as well as sends ICE candidates that are handled on the coordinator side (from the user) in [pkg/coordinator/useragenthandlers.go](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/pkg/coordinator/useragenthandlers.go), -(from the worker) in [pkg/coordinator/internalhandlers.go](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/pkg/coordinator/internalhandlers.go), and on the user side both in [web/js/network/socket.js:56](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/web/js/network/socket.js#L56) and inside [web/js/network/rtcp.js](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/web/js/network/rtcp.js). - - Browser on the user's side after SDP offer links remote streams to the HTML Video element in [web/js/controller.js:417](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/web/js/controller.js#L417), makes SDP answer and gathers remote ICE candidates until it's done (if receive an empty ICE candidate). - - For the user's side a successful WebRTC connection should be considered established when WebRTC datachannel is opened here [web/js/network/rtcp.js:31](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/web/js/network/rtcp.js#L31). - *And that should be it for the streaming part.* - > At this point all the connections should be successfully established and the user's ready for a game to start. The coordinator should notify the worker about that fact and the worker starts pushing media frames, listen to the input through the direct to the user WebRTC data channel. - - Then the user may send the game start request to the coordinator in [web/js/controller.js:153](https://github.com/giongto35/cloud-game/blob/a7d8e53dac2bbcf8306e0dafe3878644c760d368/web/js/controller.js#L153). - -#### Streaming requirements -- Workers should not have any closed UDP ports to be able to provide suitable ICE candidates. -- Coordinator should have at least one non-blocked TCP port (default: 8000) for HTTP/WebSocket signaling connections from users and workers. -- Browser should not block WebRTC and support it (check [here](https://test.webrtc.org/)). diff --git a/docs/designdoc/implementation/README.md b/docs/designdoc/implementation/README.md deleted file mode 100644 index 17626a35..00000000 --- a/docs/designdoc/implementation/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# Web-based Cloud Gaming Service Implementation Document - -## Code structure -``` -. -├── cmd: service entrypoint -│ ├── main.go: Spawn coordinator or worker based on flag -│ └── main_test.go -├── static: static file for front end -│ ├── js -│ │ └── ws.js: client logic -│ ├── game.html: frontend with gameboy ui -│ └── index_ws.html: raw frontend without ui -├── coordinator: coordinator -│ ├── handlers.go: coordinator entrypoint -│ ├── browser.go: router listening to browser -│ └── worker.go: router listening to worker -├── games: roms list, no code logic -├── worker: integration between emulator + webrtc (communication) -│ ├── room: -│ │ ├── room.go: room logic -│ │ └── media.go: video + audio encoding -│ ├── handlers.go: worker entrypoint -│ └── coordinator.go: router listening to coordinator -├── emulator: emulator internal -│ ├── nes: NES device internal -│ ├── director.go: coordinator of views -│ └── gameview.go: in game logic -├── cws -│ └── cws.go: socket multiplexer library, used for signaling -└── webrtc: webrtc streaming logic -``` - -## Room -Room is a fundamental part of the system. Each user session will spawn a room with a game running inside. There is a pipeline to encode images and audio and stream them out from emulator to user. The pipeline also listens to all input and streams to the emulator. - -## Worker -Worker is an instance that can be provisioned to scale up the traffic. There are multiple rooms inside a worker. Worker will listen to coordinator events in `coordinator.go`. - -## Coordinator -Coordinator is the coordinator, which handles all communication with workers and frontend. -Coordinator will pair up a worker and a user for peer streaming. In WebRTC handshaking, two peers need to exchange their signature (Session Description Protocol) to initiate a peerconnection. -Events come from frontend will be handled in `coordinator/browser.go`. Events come from worker will be handled in `coordinator/worker.go`. Coordinator stays in the middle and relays handshake packages between workers and user. diff --git a/docs/img/crowdplay.gif b/docs/img/crowdplay.gif deleted file mode 100644 index 66514a5e133710459f55bbbe5543ace6af8a3682..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 2029343 zcmcF}hgTEN_x7d$q4(YbhTc0!F;qiGBy1VSz0 zvi93v6VnX_!AKb0RO+p=u1)|g>9`Rph$H9L5hq(0|0IiqkR+9CvZ6$S)FkF&o z5(x-2{(tQxIsgG*iy{d^(hvpUFNv-G0D=_ff3wu0`~iOgshxO9LW?I}l4L{xC{kKT zNLq;iNfwg$t!Pry<^KQx2of+7Qv?y^Pf8Q18I3^H0wfJdKM_DH9zb295tk$wB)&x8 z5{)8JK-6C10R%}a((DLQvPeE89fxdLxMjkX8;U#l-(DiT|IM0Fl~E z-|DumwuQdDv@8TfnoHm}6iNjU0m=WHmvox|Fd}dV#*?#Sh=q_bFd#!+Ty^587!4vs z8B0eJImCSP$4<8lp7SciUWzi6kEe+m*7A&>?HFZC-F_P(##}l5Le=L>{`mJjvm(tP z@+&cx>UX861aaPpAC-7Ph%67!+zGH4V^eQEJ@M$JP2HU~9Ir&XzfH5(bgb!QNWDYr zgT=uDiP!fXx*i|Dot}LB>Q2vpKf!#G9RYU+<7g?&rb3(ChM%+Q7D{%W0hAC0KeIH9 z>*v`*dW;sVOTp(|wfS>{-mc*NrMf#+DfeEqAFVW?XJ_s?sC*^#J)ovH>v{cfZ4AqA zQ^?c)eRZnr22nHe;_S;vp6=+=$nM8q*6aO<6z0x7KTeisv(0BCdw!pPmP2V`ihZwQ zejR*Ve_r(F-#_4=h|5#x9~&181)cW>4klmi62+h;>WX8zZtEJ&Zsxri%Mn=Z8V7$W zdN&pkZhJRgB*l9xQM9@G?lajwQMW|JuPPW%)jlLSCG}6+tmjB;)*ZYKo8eBXfpEf3 znvwGOPP&-^n${fe)a9k)*q$X7jRFGTet5Y-s{*2eAdvpRqOEe z^+Dm^uU#k2+z-0Xx?Vo$yWBeb)(d=;2_f0?+9n zgSvmyTTYu#a&K9pG`=Pm)0IOhtxQKOY{a7$n>|vM405MrXOFTbDAdZ)GZG5rK@;dr zo*y&1kgJwi&F_*w-Wf4Z2fstf6eup3a%=vax4l!}w&1AV+xEf9fj4BqC0nX}*}bSf zWX0cRRe3SMOSAoB@aWagl}D4ukHBmu#0{S+3{fg{Bbxqx=w>{BL+DnL%-7Iu{Ee@n zdn>uZRxobtwB35IjSGbb%KGJ6oc8v z`pK`KU>TB%q4b1SrJ4gA+Z3E!rFwua@Fa?lMgVp*z7qU{)+}7SA8Io;$kufdBkS+7 z<0-1fwbB@?+EYRoG&aP$H2$B=22Bl-0f`iRdlHW_DP^)j4R5JE47G?TWhq0Ah}kzK z+V*TU#COzrhIwKP97{PyVMDTPb4mJoa4z;xj542$^8-a2o^GYl>w~8$fy38$e|nE< z{CW8NzoTn#3b8Srvr{})4M!5CzMYVkPOWjRRS z4r$3Wmr+PlA+9|>ajW5dX1(ODga_y3?G^bW8(hf*Yi$atzTPw;B75w*R9qQ z_+t@cpHag8$XiEl;#RVm)nkf-Ro{H{T-u8t<&-^lu9jZEKX0?D`0M6l;&lCX&KdoG zs*yhb?$^%eF9lc0?;e;(iZ7IsqjiIzUlJ6@%LGe))L2M~&FZ1hO!jTHw!O91Efb6N zDY8-)wJvEL2|8*e^0nTkFR9UQXWP%`>jNTRI&K@;XkvHc{xRS65$-T(J6n9ka3sMI zl|C})`>ruwv+geQBw^&AF9g2`xU2uJ1+i=@SO}oOT`-w`EsI7Inq!c*#GG=tq&suB zt*=__N^*g9TLUYnpX=YIMPgq_dq;2meUEO3k7wtv`)C__S5%{6CjA|6M}YQ!N*L~Qb^hva8`mLucMwZn@}h$(u1kaicg%S{K^tle*k_z=@i=Vz56)kRm8L( z(u>?8_rrql)0WOVSC;c{W{+NBi0T^v^F=?o*@+^n}(Z)N%p9UnV>;QM|t zs+A%UD+Dal7$YGt(F_Jjmt{)T59UNYGq8#*h}xoaqMR2_ZbdTyu|O(P{b+#5$pU2e zkq{0x+F)u9RK>RjGPn~b@*e+s2n zj{EvIM+b9wb-bHfumZ?)D}kS@?MR2{wR^=6nBhpsYaZ>jWlro}zP5b&cqk1Id@#tm zfsw)mq(F~}XY4xy%%_pgEdko2-}|9f5mqm_76~KmKFyi6{_g7C%X6OXtqEQJzBit% zXilG7Jn6#1@;$*5^#|IbvRI+lP5=FaY|PE91W_qIB+If9SA5&g0MBW* zNDG4)z=P7v#VB@P$dCNcif2GP0mxk78!x3dUdkjjtHcEkL!y<)%2=vksOu6=w4FBk zW@o8+Jv*XA5!dL~@^_^?I$*b@(kR!hL{y@>6+3*S*um&9u%FbfRwZ*bhVK~+T_iIA+z(QJk49s;nAA30GO*WkxX_2 zJ3Lu61x2)IK#BmttbUk5{!w>P@sD2ifJv8-(9g>e*cqhFP(hP_uHlfuZv@|InJJoC zr1A6HKWj~I=(SGbWG=Tx$=ekTLH-_PKRVmV*`uh(Y(e*TGoZ5rH1$0C`F zy&&b`-3G{?dw#0yAS=)l{cK>NDwqbPc>bPK3F&`v*Dy8OFk=VHbAjbm#R=Nt1VVA} z92_DZCsBHt_ghs4dCdLC^vgqBFVrCx zm?I1QyX47@1Q{yxb6^09P2bkNXuTg$ZkL!jrI=D>Bv*i)aR-Q8&P$d;#8Nv{FzfCP zg%iVLfcitUb0z5So}qvB-Dnl4p>>%3>D?3e#~n9~J#+zmyc5$#(7k<^%NQ4G;(2@$ z%mshplERTdrks#G=8{pJkQ<+{vIEiuSc_pn6N*4)2vCB05WdZD`!NI8o-&WM$c}z+ z!@e*h^673xh=(vg*~Wc-Yd^2Km?H8}9lX0I{_ay|OKUk$YvooM#sW{jR6ZC@JxkbeJCmM1++x5m{>mA`3 zkv)+IULc)0FnaI4hYH@y1`iFxXInxOZNpteL^;?%6*obQ824yFe-q(2_FR!$A>1$o zG=U>JD;9KSjr1}=(!_;2D~D$5hyGaNw7m$LMq#DwLLon(nK!{1#OyGulIH{=t~2sb zh!yzbVz@8IlOUapa^bKXiU2)mm;;;ZGZD|aROGT(_|Ht9R$+|4iht}|?}R1qM7As! zwX8dKS@iHCkYnL6&d1Lp6##PYlp8j=?iLTMl>v7+S0LsQlS?u)K6(O84$<># z%C>`SkZIoZzl9A84EWE^3)#Dk$*x2$tE5o`-_Yd%^{IiRHhgIV{oi@s8>zWBoN(`B zd;Yjj{={+!>cb8N)m2XhXL6?`+br|mdCh7w2u?&1CiS&b3+NH?I zYapf#3}iT@OfQs^B8UU-dkYlmlvDKI&6u-teASN>$BwX>>ad@`lP<(q0~?*h5x%fr zt~5Q+Tr8e|PAO^tX-E;C(ee&zV_e`-(Ca$6)*rx&J2NSWg z7Nw}osWRxiD`kM}$#6Hxaxd^s$m~cc3{41~Oh_ZVev~GmJ@lFTv%6R_iLATZ~~1aXRG(No)+U6v!?YYA>zN ztsQ#UhPd|HFt>K<*J{e5hvL;e>ean}yZfHupLKD&E){aUG#LA9SK1cdNnz)j;UNj_ zMU|^(=x2G0c@sWn`gdk6A9}C8%=$ol*}Lx7`;oivutf63xs#Q18wgGRoL&E{SU*KiKh>vx=u|)bCI6Ss`@$pl z$sPqmJO*wCGbs5Y#JI}fm!*O{g9wd5VaGxF$2!-)*s;|O&_S`k|LXXY^(gZ6C`Ak@ zAGr;w=XtOUU2ykmebQUHKK#LG*uZfZ6)`*%1xzqSPUyY?KYAk_U3cSXh~6D5L%`YR zk2p@@oQ`nLJa1hz-a1IW^)PvRC*rMd{)nyv>sO&W-=L$)=gt}*%mOdpKH?b-(-;k* zd479H-uF~Ke}@(KNkN);NOmZS7Vj}s|cty`dwZ?c|{zUnwiM*rnn?t4{e%9;Jqn#0>Jw2mcCZpX) zlf67suM~&Wl7_}2rY7>I(iI-RavhoDnck!B+jW@Up6uPqn_hc4y$~@yZ_>BsKfThO zy(2iim;c5PT|mt1XZ$one$@Z%Xy%A#_J_vodEE>wX_jG1fcc2w^X2T*#Q_nGIZ=~2 za;G_1Cgy7X{c*`HjtLPC#67KeJ>k1KXZGYqGY}@b6!D_1)byt zy~qWVf(6sw1vB173(ZA!%>{J*g3Z-}oz$X(>7r)+;+^S5*W*PErzLC6B_~Z*t@v(* zShstT-4CXh?rVMsF#Qnd^kG}bnTMyHvgrele@r+p!AgShsVMSOou$APQ6ItpVWSEFN*pC;1sezwZzrh3Ge!u=K2q( zb?#~PKDYHhz3Z2&>;JCSfi+HOzm+u42KmzsSm6d$--grMIrqs82EI)utxXobb*s&_ zAEqOe9h)uVC4ql8dA@E6&~5RWZ3#MWT`SxY?b{Mr+Yl4Z?AJyI z{PkLY?YdYKhI!lF==P#$h-rw`%BY5!b1+47_KHUrE`yBf8^Mk_AcT)&{ z_ha1e$6)^LMmc|eaR0MEv5?`kVsy~-Zc^Xw^O;?Tulvb`yJ=c(sb&ZLs~^*y5BS$q zXdfP=YJG{wKL||OOVIip8M*&t=F9zmUt;MFBO1Q6rtDWWe2y$Ud|mjZeeE##>E2W6 z!;Xe8&dy)n{+ld)demk5Wo+hXh~{8Idg}fCgIu}~Gf%DA3*Y^5dXHE={$$22q_nGP zqPNo*zO|FPcEID5k!kK#^GTT4Q;ZSZ{#sh&l3-5z&E;`bY|HA(6_T>(|x8l zXSB0tCO6K^WX>##PT}9qZu9@-y7N=w!4J0QKTRL}6n?djCjN9J{*a>oDUtEr2Yu%H z=Pi}{_v`f2LILE=C(tWpgJmY7_?g+a|JHwHD`;ENG{A!2(27t{1=N8n z03jDLp^Xdtw+n_HkjqI^%zqcFvqGcC>78a?*i_sgubJx$Ib08Ps+eR(m z0@Z#f{G@!{BgD?Plj@3zm@R+%-!F1~@>fLgovcWAcUFM^1Nrq&_nbeRcw*S%?$ZAE zf@>qlfML(M@6dStOEet=B@>G(9A7*DAtPsVj1SNq!opa@KUwU+^>ED0_c`E<*TB&N za%>r6veyz)1dN%Esvnm2V`Zr^m;3Ui+POlMJ65ouU2KNRjl}$fAf!&ZqRe>`rGgh2 zdh4Oty%$LBR1kut*hF+*VzA5=3$CBAD#=vI-K%!WjJqdL>&YBFu)Q5e&yrd=a&;ZA zmwkiE5Fa3Jvq4VHB58u^QGICB2d3~RV6fm=kjG!kR*P#o(ZhKv{2eAGsGVlyGYdWp zy45}1nc6?p4N$D4{JiB;Pe+MwuM^5}5q$gDrl=x-z?GRSIfP$eb4p(^<(4XWR<=vN~QGf{V_FWjB1Z`-?IP?9~zkj2;- zXba?B$(vw6%ykT+?PIboCUS3a!|)+UQX>uITBzg9C`=;{sm;yUEvdlmzT`uxqqW9r<&uu(^}v< zCEPAxD%C+EGX7u-@k3OXI?~GWrbNXLOJm*wsd*i#(0#ZGwIG&;R@D><5!Yo!$(d2~ z*TJAme!-gb_5}?}C~U3Lf}_YJT6aMYhum&5TIapfOK72bz(!n;nLzJC z$pgpj1ZbhG8>;Zi$|OWzE|zC2?!ToVxn~l6;oeblIboA!;$?7LqCcD2*3ZH@2(|po zTsZ8LBuW!>Jx+SY^y?jXRCL;}e3#HdL$lua39_@|t-#&j=l45%BrWMstZ)a36U;%93s*-mF*UJj-a8TjjMAm`(qluyJQ55eym1(Qf z8<6ib4za}sA@T!MX3@}FgEU(xTe1Y&J1%P2+Ad1LjpvZV?iO;w@F3C zQRsmeEHzxK0ffAD`Bp4o5{?6U;$63`B7i6)IV+jg*xMF{#yHK(2dRefp#8fxFQ^H0 z{tw&;~eDcMp}%LI-Npk<d0T8pjBBZ<~bouTyxSb7285V^_lX5=gK>>Mwv*W+_DbNMueMp&}R%B9kl3b zh~d~&OP8npz5r5ev@HIK8mhv zw%xY(5H1D{H}3g8{(_-}Hc~)ourS9@x%sW~;>XeE;ensQUa}1d7g)5(kxEChurovUUiv2J#;u8$-OfiMtEfaqvx7_T70(Oa9xo(%c z;&nTD!Eo)lTFxMF#BpT%iq}{U1_zP(dgM~m7_0jn-hNoxr6g8A+Cwj=rjypTa39ixIJsY@w9Z36v0yZe>giP^vV^{m_2x zbWglAldIm%^`@SCeWhImTVaV&OkIp+_CUn;LVyY!%Q5u0FXYcd;Dmvd7gkKxWOU6a ztv^~N2Q!$Q$9eaUadjoy4sI_7H9H=|sO7jwJx{;=^grO47eRulN_JJO^*EgY24^xa zMl!86(pb?PkLac$&_&<31;$SVq77$=(3!rfg5)v2jRbJ9-FMO9U2i1U_`I^f(2 z&E!tPv5LBkh=vECYG83F9O%)a(q}`T2GvYweH^b2uX!l zc@%RfJ{2hC3YBLF#l}1VNqMIUYrlEdSiA6%=>u0$ZzC#X_tOfx$~^W-CE-v zoqxq}5;bbMvIz~c$wMc!eo=5@S#jx54jYRA^r8sBNzMmcZ|+uNKY~RLVk|+K6ch0x z3^h|okw$fZQdEc<-hFx?P8N?5IALt>PJPaf9}xYvFGfS7_3* zMUv;`NDUIWcKU`QDIqzx_~MIzg`mMcj84Z6VZoG5NiyhR{DS5^UKs7>RWTVUub`5P7=Y2IdCtp?OW+oo)rh zpkNS=Vh~3^JH0}!W@s#*OU$tX28Aq0Z@O89Ov>#9mF_Umb^^3HhMMD?qQO+pw7E)# z3kH|dIsYVGlWUf1Nk_cokauC0vQ=dOKrmDR$hKl(fU_VsH_DG@Y)9U4{N@4=3vqy} z(*w(*w>+_c)26AGboa?Q@uMnG)!`pWO=DLo)lb93yhQbbTmYWbjzM7@tp@##-j*YI^h64*D&F?oPA6AwG z8*%$7)kMGuoEUBwncHDz`=Scf9Ne#dV5k59WD!lnL9Mx*2r34yh2oenV0f{z)qBn# zsfc|-#pjK7^+w!-JjhNSq$(|46^6ZqiiyC2Y;9FIkk}+pg6P zOz<7W>;0}Qg^T=BouOO7h)mH4*v<$g@zCeo?tM1c_N<;O6;4R)u1|}fg9}p0s`uw% z)bywr#o1nN!|2CEl*)^5*YLa7G+}eH;zi~8qiOD34ct^(+8fT9(qR*`6iiMOz9tYq zR9p1y&wHq76x)7O<3{fC5J2G)ZA}0^&1>GcVwr#P?lu#h1PV+J*WywFwOeSZQ$m!g zwVt>}>n1`mWL!)EkV;FOhy}#j0wSWxxz-BP4vpVT==)I~=qK8l%Ls?g_vY&UES;-N%nJfw%?gSa?G&N|0bkmdQH00!*~zZ7oZu(&ndZ!ZJEAn=84r@fP|uN^w4IU($@e<6hO& zmBe9QDPC(C*Di2nrqc!`vg`v4LN>(W01Ofd=ELBYqw*;URhZ)| zK))9lw_tc6cQ!cJpCFnjuk+b(E)IpO;NHO*LNOE;V0k3?pb63`KXiC-lX61-6*MpH z^3DfVdqlTd4tztsF9Y&49t5;2mRbCH=yDjm)tb8@aFUVK>Qana9$R&JaPHD8CT-Zg zygdLguZi=Yf+=6Xchta-Q(UP>0EPILXFRaP@mz?g!(k^7x`uJoO|bke`AWR)-_$7A zgw%KOW(-e5Bk7JSogsRjwUL!2iP*{-h2t!v8fZ7Xj|8$3`;6cTwZ}ZB>0&T$Z!E+` zSyO;!ZUBo^#mU)XS%IR>lF94i6&z@c4hEbb9;FO>{qRr>%JCjWfbgPX=zu6~@)vsW zD31V$kOi1G9>PnA61ie~`Zvo<4MP*32y(GJ+Qr0b=04LFj{azuDwUV7U$~o?GA1Bl zBsotp7v<<|rm(NtJbA19_b#Y$UcH_!AK&!BHs$9=+K^VE9*DhB(;uQ1&!(B!vr;l9 zoHtL+neVQ