This article explains the server-side implementation of the LoadBalancing application.
- Take a Game Server out of rotation
In Photon 3, the LoadBalancing Application was (literally) extending the "Lite application". It provided all the well-known Lite functions - like Rooms, Events, Properties and so on and adds a layer of scalability, that enables you to run the application on multiple servers. LoadBalancing also adds lobby support and matchmaking capabilities.
In Photon 4, Lite was deprecated in favor of Hive. So LoadBalancing is now extending Hive which is more complex yet more extensible.
The basic setup did not change and is always simple: There is always 1 Master Server and 1..N Game Servers.
The Master Server handles these tasks:
- keep track of the games that are currently open on the Game Servers.
- keep track of the workload of the connected Game Servers and assign peers to the appropriate Game Servers.
- keep and update a list of available rooms for clients in the "Lobby".
- find rooms (random or by name) for clients and forward the game server address to them.
The Game Servers have these tasks:
- host the game rooms. They run a slightly modified version of the "Lite application" for this purpose.
- regularly report their current work load and the list of their games to the Master Server.
Differences with Photon Cloud
The LoadBalancing Application provides almost the same logic as the Photon Realtime Cloud service. Some requirements to run as Cloud service just don't apply to a special server with custom logic, so they are removed. This greatly simplifies the code, too.
- No virtual Applications. Only one game logic is running per LB instance. The AppId parameter in operation Authenticate is ignored.
- No player separation by AppVersion parameter. This is part of the virtual apps also.
The workflow from a client side perspective is quite simple as well:
Clients connect to the Master Server, where they can join the lobby and retrieve a list of open games.
When they call a CreateGame operation on the Master, the game is not actually created - the Master Server only determines the Game Server with the least workload and returns its IP to the client.
When clients call a JoinGame or JoinRandomGame operation on the Master, the Master looks up the Game Server on which the game is running and returns its IP to the client.
The client disconnects from the Master Server, connects to the Game Server with the IP it just received and calls the CreateGame or JoinGame operation again.
From then on, everything works the same way as with a "Lite application".
This section explains the Master Server implementation - see the LoadBalancing.MasterServer namespace in the "\src-server\Loadbalancing\Loadbalancing.sln" solution.
MasterApplication decides if incoming connections are originated by game clients (on the "client port") or by game servers (on the "game server port").
Master Server: Handling Client Peers
MasterClientPeer represents a client connection to the Master Server. T
he following operations are available for a MasterClientPeer:
The Authenticate operation has only a dummy implementation. The developer should use it as a starting point to implement his own authentication mechanism:
The JoinLobby operation is used to add the MasterClientPeer to the AppLobby, which contains a GameList - the list of all open games on any Game Server. The peer receives an initial GameListEvent, which contains the current list of games in the GameList (filtered by the optional Properties of the JoinLobby operation):
GameListUpdateEventis send to the client at regular intervals, which contains the list of changed games (also filtered by the optional Properties of the JoinLobby operation). The client will receive the update events as long as it is connected.
- JoinGame / JoinRandomGame
The JoinGame operation is called when a client wants to join an existing game that is listed in the AppLobby's GameList, specified by a unique GameId. If the game exists and the peer is allowed to join, the Master Server returns the IP of the Game Server, on which the Game is actually running, to the client.
The Master Server also updates the GameState and adds the peer to its list of "joining peers". It will be removed once it has joined the game on the Game Server (or after a certain timeout). This way, the Master Server can keep track of peers that are in transition between the Master and the Game Server.
JoinRandomGame works in a similar way, except that the game is chosen by random by the Master Server and the GameId is returned to the client.
The CreateGame operation is called when a client wants to create a new game. The Master Server determines a Game Server on which the new game will be created and returns the Game Server's IP to the client. See the "LoadBalancing Algorithm" section for more details.
In addition, a GameState object is created and added to the GameList and the peer is stored as a "joining peer".
Note that this GameState is only used to keep track of the games - the game itself only exists on a Game Server.
Handling Game Server Peers
The Master Server always knows which Game Servers are available, how many games they host and how the current workload is.
To achieve this, each Game Server connects to the Master Server on startup.
MasterApplication maintains a
GameServerCollection, in which
IncomingGameServerPeers are stored.
The Game Server can only call one operation:
The Game Servers call the RegisterGameServer operation once after they are connected to the Master Server. The Game Server is added to the Master's
GameServerCollectionand to its LoadBalancer (see the "LoadBalancing Algorithm"). It will be removed from the
Check the "Game Server" section to see how the Game Server sends further updates about its games and its workload to the Master.
This section describes the Game Server implementation.
LoadBalancing.GameServer namespace in the "\src-server\Loadbalancing\Loadbalancing.sln" solution.
Game Server: Handling Client Peers
The Game Server is derived from the "Lite application". As soon as a client has received a Game Server address from the Master, the client can call any operation on the Game Server that is available in Lite. The only difference is that we have separate operation codes for JoinGame and CreateGame on the Game Server, while Lite handles both with JoinGame.
Reporting Game States to the Master
The connection to the Master server is represented as an
OutgoingMasterServerPeer in the Game Server.
Once the connection is established, the Game Server calls a Register operation on the Master Server.
Afterwards, the Game Server publishes all existing game states to the master server:
This is done by sending a message to each Game that tells it to send it's game state to the Master:
The Game handles this in the
ProcessMessage method and calls the
UpdateGameStateOnMaster method to send an UpdateGameEvent to the Master:
The game state is also updated on the master whenever a game is created, joined or left by a client or its properties are changed.
The next section describes how the game servers report their current work load to the Master Server and how the Master Server determines the Game Server that is best suited to handle new CreateGame requests - the actual LoadBalancing algorithm.
LoadBalancing.LoadShedding namespace in the "\src-server\Loadbalancing\Loadbalancing.sln" solution for implementation details.
The Game Servers regularly report their current work load to the master server. The work load includes, for example:
- CPU usage
- some Photon-specific values, like ENet + Business Queue Length, the average time the server spends on each request, etc.
- Latency (when sending requests to itself)
The most important (and easiest to understand) factor is the CPU load, so we will focus on the CPU load in this documentation.
All these factors are summarized in a single value - the "Load Level" of a Game Server, which is reported to the Master.
The lower the load level, the better is the Game Server suited to host new games.
The Game Server collects "Feedback" about the above-mentioned factors. There is one FeedbackController object for each factor - it consists of a FeedbackName and a FeedbackLevel:
DefaultConfiguration class defines the thresholds for each value -
e.g., a server has the "lowest" FeedbackLevel up to 20% CPU usage, it reaches the "highest" FeedbackLevel at 90% CPU and so on.
These values can also be configured in a "workload.config" file.
See "workload.config Example".
LoadBalancing.LoadShedding.Configuration namespaces takes care of reading values from a config file, or applies the DefaultConfiguration if no config exists.
At regular intervals, the Game Server checks some Windows Performance Counters, sets the current values for all its FeedbackControllers and calculates a new "overall feedback".
This is done in the WorkloadController class:
If the overall feedback level changes, the
OutgoingMasterServerPeer will report the new server state to the Master:
If you want to know how we configure the workload for traffic then just take 90% of the maximum traffic for "Highest", 70% for "High", 50% for "Normal", 35% for "Low" and 20% for "Lowest". But you are free to declare lowest/low/normal/high/highest as you want if you don't want to use those percentage values.
If your server's bandwidth is 20 Mbps (20000000 b/s) then you can have the following values:
- Lowest: 4000000
- Low: 7000000
- Normal: 10000000
- High: 14000000
- Highest: 18000000
Only "Highest" has consequences, where if its value is exceeded then no more games are created on this Game Server. If you want to use higher limits you should test if the application can handle the number of messages.
LoadBalancing.LoadBalancer class in the "\src-server\Loadbalancing\Loadbalancing.sln" solution for implementation details.
The Master Server stores the LoadLevel for each Game Server in the
It also holds an additional list of all the servers that currently have the lowest load level.
Whenever a client calls the CreateGame operation, the Master Server fetches the address of a server with the lowest load level from the
LoadBalancer and returns it to the client,
which then connects to that server.
Configuration and Deployment
For demonstration purposes, the SDK contains a setup of 1 Master and 2 Game Servers in its deploy directory:
This setup is only intended for local development.
Deploying a Game Server
When you deploy your LoadBalancing project to a production server, you should not host 2 Game Server applications on one server.
You need to make sure that the Game Servers can register at the Master Server. Set the MasterIPAddress in the "Photon.LoadBalancing.dll.config" to the Master's public IP.
You also need to make sure that the game clients can reach the Game Servers. On each Game Server, you need to set the Public IP address of that Game Server. If you leave the value empty, the Public IP Address will be auto detected.
You can also use the Photon Control to set a Public IP.
Deploying a Master Server
You need to make sure that you only have 1 Master server: either remove all settings for the "Master" application from the PhotonServer.config on your Game Servers, or at least make sure that your game servers and clients all use the same IP to connect to the same, single Master server.
Otherwise, no special configuration is required on the master server.
Take a Game Server out of Rotation
As discussed in the "Loadbalancing Implementation" section, the Master server knows the state of the Game Servers, as noted in the "ServerState" enumeration:
- "online" (that's the default)
- "out of rotation" (= open games are still listed in the Lobby and players can join existing games on that server, but no new games are created)
- "offline" (existing games on that server can not be joined and no new games are created on that GS)
The GS send their server state to the Master regularly - see the
If you want to set a server state programmatically, what you need to do is:
- modify the WorkloadController class so that it determines the current server state
- for example, you can add a "file watcher" and read the server state from a text file (0 / 1 / 2).
You can also build an operation that is called from a client, read it from a database, or whatever comes to your mind.