Libretro cores are selected based on the file extensions of the ROMs. However, ROM file extensions are not unique across cores (e.g., .zip is used in both DosBox and MAME). To load a specific core correctly, it's necessary to place the corresponding ROMs in designated folders specified in the configuration. By default, you can use the keys from cores.list as the folder names, or you can specify your own custom folder names using the 'folder' parameter for each core.
This PR contains refactored code.
**Changelog**
- Added new net code (the communication architecture was left intact).
- All network client IDs now have custom type `network.Uid` backed by github.com/rs/xid lib.
```
The string representation of a UUID takes 32 bytes, and the new type will take just 16.
Because of Golang JSON serialization problems with omitting zero-length empty slices (it can't)
and the need to use UID values as map keys (maps don't support slices as keys),
IDs are stored as strings (for now).
```
- A whole new WebSocket client/server implementation was added, as well as a new communication layer with synchronous and async call handlers.
- WebSocket connections now support dedicated Ping/Pong frames as opposed to original ping text messages.
- Used Gorilla WebSocket library doesn't allow concurrent (simultaneous) reads and writes, so this part was handled via send channel synchronization.
- New API structures can be found in the `pkg/api` folder.
- New communication protocol can be found in the `pkg/com/*` folder.
- Updated communication protocol is based on JSON-encoded messaging through WebSocket and has the following structure:
```
Packet
[id] string — a globally unique identification tag for the packet to track it trough a chain of requests.
t uint8 — contains packet type information (i.e. INIT_PACKET, SDP_OFFER_PACKET, ...).
[p] any — contains packet data (any type).
Each packet is a text message in JSON-serialized form (WebSocket control frames obviously not).
```
```
The main principle of this protocol and the duplex data exchange is:
the one who initializes connection is called a client, and
the one who is being connected to is called a server.
With the current architecture, the coordinator is the server, the user browsers and workers are the clients.
____ ____
↓ ↑ ↑ ↓
browser ⟶ coordinator ⟵ worker
(c) (s) (c)
One of the most crucial performance vise parts of these interactions is that
all the server-initiated calls to clients should be asynchronous!
```
- In order to track synchronous calls (packets) with an asynchronous protocol, such as WebSocket, each packet may have an `id` that should be copied in all subsequent requests/responses.
- The old `sessionID` param was replaced by `id` that should be stored inside the `p` (payload) part of the packet.
- It is possible to skip the default ping check for all connected workers on every user connection and just pick the first available with the new roundRobin param in the coordinator config file `coordinator.roundRobin: true/false`.
- Added a dedicated package for the system API (pkg/api/*).
- Added structured logging system (zerolog) for better logging and cloud services integration.
- Added a visual representation of the network message exchange in logs:
```
...
01:00:01.1078 3f98 INF w → c Handshake ws://localhost:8000/wso
01:00:01.1138 994 INF c ← w Handshake localhost:8000
01:00:01.1148 994 INF c ← w Connect cid=cep.hrg
01:00:01.1158 994 DBG c connection id has been changed to cepl7obdrc3jv66kp2ug cid=cep.hrg
01:00:01.1158 3f98 INF w → c Connect cid=cep.2ug
01:00:01.1158 994 INF c New worker / addr: localhost, ...
01:00:01.1158 3f98 INF w Connected to the coordinator localhost:8000 cid=cep.2ug
01:00:02.5834 994 INF c ← u Handshake localhost:8000
01:00:02.6175 994 INF c ← u Connect cid=cep.hs0
01:00:02.6209 994 INF c Search available workers cid=cep.hs0
01:00:02.6214 994 INF c Found next free worker cid=cep.hs0
01:00:02.6220 994 INF c → u InitSession cid=cep.hs0
01:00:02.6527 994 INF c ← u WebrtcInit cid=cep.hs0
01:00:02.6527 994 INF c → w ᵇWebrtcInit cid=cep.hrg
01:00:02.6537 3f98 INF w ← c WebrtcInit cid=cep.2ug
01:00:02.6537 3f98 INF w WebRTC start cid=cep.2ug
...
```
- Replaced a monstrous Prometheus metrics lib.
- Removed spflag dependency.
- Added new `version` config file param/constant for compatibility reasons.
- Bump the minimum required version for Go to 1.18 due to use of generics.
- Opus encoder now is cached and the default config is 96Kbps, complexity 5 (was 196Kbps, 8).
- Changed the default x264 quality parameters to `crf 23 / superfast / baseline` instead of `crf 17 / veryfast / main`.
- Added a separate WebRTC logging config param `webrtc.logLevel`.
- Worker now allocates much less memory.
- Optimized and fixed RGB to YUV converter.
- `--v=5` logging cmd flag was removed and replaced with the `debug` config parameter.
**Breaking changes (migration to v3)**
- Coordinator server API changes, see web/js/api/api.js.
- Coordinator client event API changes:
- c `GAME_PLAYER_IDX_CHANGE` (string) -> `GAME_PLAYER_IDX` (number)
- c `GAME_PLAYER_IDX` -> `GAME_PLAYER_IDX_SET`
- c `MEDIA_STREAM_INITIALIZED` -> `WEBRTC_NEW_CONNECTION`
- c `MEDIA_STREAM_SDP_AVAILABLE` -> `WEBRTC_SDP_OFFER`
- c `MEDIA_STREAM_CANDIDATE_ADD` -> `WEBRTC_ICE_CANDIDATE_RECEIVED`
- c `MEDIA_STREAM_CANDIDATE_FLUSH` -> `WEBRTC_ICE_CANDIDATES_FLUSH`
- x `MEDIA_STREAM_READY` -> **removed**
- c `CONNECTION_READY` -> `WEBRTC_CONNECTION_READY`
- c `CONNECTION_CLOSED` -> `WEBRTC_CONNECTION_CLOSED`
- c `GET_SERVER_LIST` -> `WORKER_LIST_FETCHED`
- x `KEY_STATE_UPDATED` -> **removed**
- n `WEBRTC_ICE_CANDIDATE_FOUND`
- n `WEBRTC_SDP_ANSWER`
- n `MESSAGE`
- `rtcp` module renamed to `webrtc`.
- Controller state equals Libretro controller state (changed order of bits), see: web/js/input/input.js.
- Added new `coordintaor.selector` config param that changes the selection algorithm for workers. By default it will select any free worker. Set this param to `ping` for the old behavior.
- Changed the name of the `webrtc.iceServers.url` config param to `webrtc.iceServers.urls`.
* Add initial external configuration files support.
These external configuration files allow changing app params at the runtime without recompilation.
* Find config files with specified directory in the tests
* Add aspect ratio recalculation config
* Clean code
* Add new configuration files into the Docker container image
* Add shared core and config paths into the Libretro cores config
* Split ROM <-> Emulator mapping between workers and coordinators
* Extract coordinator config
* Add shared worker/coordinator server config
* Add explicit embedded shared worker/coordinator struct for auto-config reflection fill
* Remove default stun/turn servers from the config
* Extract and add new ice servers config structures
* Update coordinator config params
* Add auto emulation lib loader based on the runtime OS/arch
* Update configuration structures
* Remove shared config embedding
* Add missing network config params
* Add game library external config
* Remove unused config parameters
* Add WebRTC encoder external options
* Add user dir for config search
* Update config loader
* Update config
* Add generic downloader with Grab lib implementation
* Add a simple file downloader backed by the grab lib
* Add initial Libretro core repos abstractions
* Expose compression info for Libretro cores repository records
* Add pipe-based abstract file downloader
* Refactor downloader
* Refactor Libretro repos
* Add worker coresync stubs
* Add multiprocess-safe HTTP-based core manager implementation
* Remove Libretro cores from the repo
* Keep custom N64 cores in te repo for now
* Add Libretro cores repo select in the config
* Fix http manager repo switch
* Cleanup code
* Add greedy Libretro lib loader
* Don't crash when arch map is not set
* Disable dynamic recompiler for pcsx core by default since it's could cause a crash
* Use global Libretro dynalib handler
* Shorten the default Libretro cores store path
* Update zip extractor implementation
* Remove explicit fig lib field markings
* Add config note to the README file
* Add GitHub repo backend for the core downloader
* Fix GitHub repo param list in the manager factory
* Add env variables reader with CLOUD_GAME prefix
* Re-optimize ice server info struct custom marshaler
* Follow Go standard for naming constants
* Use reformatted pixFormats for Libretro cores
* Use OpenGL 2.1 Core profile bindings for render instead 4.1
* Cleanup the code
* SDL attributes should be set before the sdl.Init call
* Use simple vertical frame flip function instead imaging lib with OpenGL renderer
* Use the separate control flow for the macOS OpenGL context handling
* Add OpenGL pixel type/format switch based on cores callback
* Use unified log instead of fmt
* Clean code
* Remove unnecessary SDL init flag
* Printout errors with SDL / OpenGL functions
* Add CGO Libretro logging output
* Use main thread lock for windows and OpenGL context
* Remove Darwin OS switch
* Add extended OpenGL version info print
* Update Libretro cores info print
* Add game library module (#232)
* Add game library
* Add missing local game lib files
* Add missing return statement
* Use v2 suffix
* Bump the dependencies
* Update Libretro modules to support headless test runners
* Port old savestates tests as example for Libretro cores runner testing
* Add n64 core example game and a test
* Update room tests for various games
* Add frame dump support for CI builds
* Add frame rendering to image output for core testing
* Update ROM frame exporter in tests
* Disable Docker image publishing
* Add frame rendering output for non-gl cores for CI
* Add auto GL context override for headless, gpu-less machines (e.g. Github CI Xeon)
* Add Windows CI headless cores frame render config
* Add missing Mesa OpenGL drivers to Ubuntu CI
* Add mupen n64 core download into CI tests
* Add Linux, macOS, Windows core frame render tests into CI
* Remove unnecessary var
* Add some comments
* Revert Y flip
* Move OpenGL into a separate package
* Add SDL package
* Update modules
* Allow HTTP access to Raspberry Pi over local network
Lower audio buffer maximum theoretical size to get the worker code to compile on Raspberry Pi
* Add https port flag to run https worker and coordinator on the same machine
Add https chain and key flags to allow to use an existing certificate and bypass letsencrypt
Note the ping address resolution is still broken with this configuration
* Add option to define a ping server in the coordinator
This is useful when it is not predicatable what address and port the worker will be runnning at
This only works when there is a single worker
* Free temporarily allocated CStrings
Store constant CString
* Only load core once and unload it when done
* Add Nintendo 64 support!
Disclaimer: only tested with Mupen64plus and Mupen64plusNext on Raspberry Pi.
It probably needs more work to run on every system and with other OpenGL libretro libraries.
Input controls are hacked together, it really needs analog stick and remapping support to play in a nicer way.
I am worried there might be a memory leak when unloading Mupen64plus but this needs further investigation.
* Add analog sticks + R2,L2,R3,L3 support
* Add client logic to control left analog stick via keyboard and touch
Add client logic to toggle between dpad mode and analog mode (even for joystick)
Might need to revisit if and when remapping is implemented
Tocuh sensitivity of analog stick is pretty high, might need tweaking
* Add cores for Raspberry Pi
Add N64 core for linux x86_64
* Reset use OpenGL flag on nanoarch shutdown (line lost in refactoring)
* Introduce Makefile and go vendor directory for faster build
* WIP: Refactor cloud-game codebase with spliting overlord and worker binary
* Fix all issues and have a running build
* Complete first version of refactor
