This article explains the server-side implementation of the LoadBalancing application.
- [Differences to the Photon Cloud](#diffcloudlb)
- [Basic Workflow](#bwlb)
- [Master Server](#mslb)
- [Master: Handling Client Peers](#mhcplb)
- [Master: Handling Game Server Peers](#mhscplb)
- [Game Server](#gslb)
- [Game Server: Handling Client Peers](#gshcplb)
- [Game Server: Reporting Game States to the Master](#gsrgsttmlb)
- [LoadBalancing implementation](#lbilb)
- [Game Servers: Determine Workload](#gsdwlb)
- [Implementation Details](#idlb)
- [Master Server: LoadBalancing Algorithm](#mslbalb)
- [Configuration and Deployment](#cadlb)
- [Deploying a Game Server](#dagslb)
- [Deploying a Master Server](#damslb)
- [Take a Game Server out of rotation](#oorlb)
In Photon 3, the LoadBalancing Application was (literally) extending the [Lite](/zh-tw/onpremise/current/applications/lite-application) Application.
It provided all the well-known Lite functions - like Rooms, Events, Properties and so on, and adds a layer of scalability, that enables you to run the application on multiple servers.
LoadBalancing also adds lobby support and matchmaking capabilities.
In Photon 4, Lite was deprecated in favor of Hive. So LoadBalancing is now extending Hive which is more complex yet more extensible.
The basic setup did not change and is always simple: There is always 1 Master Server and 1..N Game Servers.
The Master Server handles these tasks:
- keep track of the games that are currently open on the Game Servers.
- keep track of the workload of the connected Game Servers and assign peers to the appropriate Game Servers.
- keep and update a list of available rooms for clients in the "Lobby".
- find rooms (random or by name) for clients and forward the game server address to them.
The Game Servers has these tasks:
- host the game rooms. They run a slightly modified version of the [Lite](/zh-tw/onpremise/current/applications/lite-application) Application for this purpose.
- regularly report their current work load and the list of their games to the Master Server.
#### Differences to the Photon Cloud
The LoadBalancing Application provides almost the same logic as the Photon Cloud service. Some requirements to run as Cloud service just don't apply to a special server with custom logic, so they are removed. This greatly simplifies the code, too.
- No virtual Applications. Only one game logic is running per LB instance. The AppId in operation Authorize is ignored.
- No player separation by "game version" parameters. This is part if the virtual apps.
[Back to Content](#contentlb)
### Basic Workflow
The workflow from a client side perspective is quite simple as well:
Clients connect to the Master Server, where they can join the lobby and retrieve a list of open games.
When they call a **CreateGame** operation on the Master, the game is not actually created - the Master Server only determines the Game Server with the least workload and returns its IP to the client.
When clients call a **JoinGame** or **JoinRandomGame** operation on the Master, the Master looks up the Game Server on which the game is running, and returns its IP to the client.
The client disconnects from the Master Server, connects to the Game Server with the IP it just received, and calls the **CreateGame** or **JoinGame** operation again.
From then on, everything works the same way as with a [Lite](/zh-tw/onpremise/current/applications/lite-application) application.
[Back to Content](#contentlb)
## Master Server
This section explains the Master Server implementation - see the LoadBalancing.MasterServer namespace in the \src-server\Loadbalancing\Loadbalancing.sln solution.
The MasterApplication decides if incoming connections are originated by game clients (on the "client port") or by game servers (on the "game server port").
### Master: Handling Client Peers
The MasterClientPeer represents a client connection to the Master Server. The following operations are available for a MasterClientPeer:
The **Authenticate** operation has only a dummy implementation. The developer should use it as a starting point to implement his own authentication mechanism:
The **JoinLobby** operation is used to add the MasterClientPeer to the AppLobby, which contains a GameList - the list of all open games on any Game Server.
The peer receives an initial **GameListEvent**, which contains the current list of games in the GameList (filtered by the optional Properties of the **JoinLobby** operation):
Afterwards, a GameListUpdateEvent is send to the client at regular intervals,
which contains the list of changed games (also filtered by the optional Properties of the **JoinLobby** operation).
The client will receive the update events as long as it is connected.
- **JoinGame** / **JoinRandomGame**
The **JoinGame** operation is called when a client wants to join an existing game that is listed in the AppLobby's GameList, specified by a unique GameId.
If the game exists and the peer is allowed to join, the Master Server returns the IP of the Game Server, on which the Game is actually running, to the client.
The Master Server also updates the GameState and adds the peer to its list of "joining peers". It will be removed once it has joined the game on the Game Server (or after a certain timeout).
This way, the Master Server can keep track of peers that are in transition between the Master and the Game Server.
**JoinRandomGame** works in a similar way, except that the game is chosen by random by the Master Server and the GameId is returned to the client.
The **CreateGame** operation is called when a client wants to create a new game.
The Master Server determines a Game Server on which the new game will be created and returns the Game Server's IP to the client. See the "LoadBalancing Algorithm" section for more details.
In addition, a GameState object is created and added to the GameList, and the peer is stored as a "joining peer".
Note that this GameState is only used to keep track of the games - the game itself only exists on a Game Server.
[Back to Content](#contentlb)
### Master: Handling Game Server Peers
The Master Server always knows which Game Servers are available, how many games they host and how the current workload is.
To achieve this, each Game Server connects to the Master Server on startup. The MasterApplication maintains a GameServerCollection, in which IncomingGameServerPeers are stored.
The Game Server can only call one operation:
The Game Servers call the **RegisterGameServer** operation once after they are connected to the Master Server.
The Game Server is added to the Master's GameServerCollection and to its LoadBalancer (see the "LoadBalancing Algorithm" seection below).
It will be removed from the GameServerCollection on disconnect.
Check the "Game Server" section to see how the Game Server sends further updates about its games and its workload to the Master.
This section describes the Game Server implementation. See the LoadBalancing.GameServer namespace in the \src-server\Loadbalancing\Loadbalancing.sln solution.
Game Server: Handling Client Peers
The Game Server is derived from the Lite application.
As soon as a client has received a Game Server address from the Master, the client can call any operation on the Game Server that is available in Lite.
The only difference is that we have separate operation codes for JoinGame and CreateGame on the Game Server, while Lite handles both with JoinGame.
The connection to the Master server is represented as an OutgoingMasterServerPeer in the Game Server.
Once the connection is established, the Game Server calls a Register operation on the Master Server.
Afterwards, the Game Server publishes all existing game states to the master server:
This is done by sending a message to each Game that tells it to send it's game state to the master:
The Game handles this in the ProcessMessage method and calls the UpdateGameStateOnMaster method to send an UpdateGameEvent to the master:
The game state is also updated on the master whenever a game is created, joined or left by a client or its properties are changed.
The next section describes how the game servers report their current work load to the Master Server,
and how the Master Server determines the Game Server that is best suited to handle new CreateGame requests - the actual LoadBalancing algorithm.
See the LoadBalancing.LoadShedding namespace in the \src-server\Loadbalancing\Loadbalancing.sln solution for implementation details.
The Game Servers regularly report their current work load to the master server. The work load includes, for example:
- CPU usage
- some Photon-specific values, like ENet + Business Queue Length, the average time the server spends on each request, etc.
- Latency (when sending requests to itself)
The most important (and easiest to understand) factor is the CPU load, so we will focus on the CPU load in this documentation.
All these factors are summarized in a single value - the "Load Level" of a Game Server, which is reported to the Master.
The lower the load level, the better is the Game Server suited to host new games.
The Game Server collects "Feedback" about the above-mentioned factors. There is one FeedbackController object for each factor - it consists of a FeedbackName and a FeedbackLevel:
The DefaultConfiguration class defines the thresholds for each value -
e.g., a server has the "lowest" FeedbackLevel up to 20% CPU usage, it reaches the "highest" FeedbackLevel at 90% CPU and so on.
These values can also be configured in a workload.config file.
The LoadBalancing.LoadShedding.Configuration namespaces takes care of reading values from a config file, or applies the DefaultConfiguration if no config exists.
At regular intervals, the Game Server checks some Windows Performance Counters, sets the current values for all its FeedbackControllers and calculates a new "overall feedback".
This is done in the WorkloadController class:
If the overall feedback level changes, the OutgoingMasterServerPeer will report the new server state to the Master:
See the LoadBalancing.LoadBalancer class in the \src-server\Loadbalancing\Loadbalancing.sln solution for implementation details.
The Master Server stores the LoadLevel for each Game Server in the LoadBalancer class.
It also holds an additional list of all the servers that currently have the lowest load level.
Whenever a client calls the CreateGame operation, the Master Server fetches the address of a server with the lowest load level from the LoadBalancer and returns it to the client,
which then connects to that server.
For demonstration purposes, the SDK contains a setup of 1 Master and 2 Game Servers in its deploy directory:
This setup is only intended for local development.
Deploying a Game Server
When you deploy your LoadBalancing project to a production server, you should not host 2 Game Server applications on one server.
Remove all settings for "GameServer2" from the PhotonServer.config, and delete the /deploy/LoadBalancing/GameServer2 directory.
You need to make sure that the Game Servers can register at the Master Server. Set the MasterIPAddress in the Photon.LoadBalancing.dll.config to the Master's public IP.
You also need to make sure that the game clients can reach the Game Servers. On each Game Server, you need to set the Public IP address of that Game Server.
If you leave the value empty, the Public IP Address will be auto detected.
You can also use the Photon Control to set a Public IP.
Deploying a Master Server
You need to make sure that you only have 1 Master server: either remove all settings for the "Master" application from the PhotonServer.config on your Game Servers,
or at least make sure that your game servers and clients all use the same IP to connect to the same, single Master server.
Otherwise, no special configuration is required on the master server.