Beaconizer

Beaconizer is a different build of mapserver, which loads maps and generates .bcn files.

Input files

Beaconizer will not read from pigged files. It needs an unpigged copy of the the bin and scenes directories, and either geobin or the .txt files from object_library that create it. It also needs the underlying map data produced by the editor. (For now, a copy of i24 maps is available in [1])

Operating modes

Beaconizer has to be started in one of the following modes:

• BeaconClient: the worker processes that run the calculations to construct beacon files
• Master-BeaconServer: the coordinator for all the other processes
• Auto-BeaconServer: loads every map and rebuilds the beacons if the map crc does not match the bcn file
• Manual-BeaconServer
• Request-BeaconServer: processes requests for beaconization that mapservers send to master

Selecting one of these modes causes the mapserver to start as a beaconizer, and alters normal mapserver startup in the following ways:

• d_ prefixes in data files would normally cause the file to be ignored by mapserver; beaconizer does not do this
• Popup error dialogs are suppressed
• FolderCache is set to filesystem-only mode, so piggs will not be read
• Shared memory is disabled, so it loads files directly from disk instead of sharing preloaded copies with other things running on this machine
• Server command processing is not done
• Animations are not loaded
• Container management and the usual dbserver client do not start
• The beaconizer client or server is started, according to the mode

Beaconizer can run in development or production mode. This is unrelated to all other forms of development or production mode in mapserver, and controlled exclusively by the -beaconproductionmode flag.

BeaconServer listens on the first free port in the range port 48812 (0xbeac) to 48913, except for Master-BeaconServer which listens on 48813, and no other BeaconServer will listen on that port. All beaconservers accept the same network commands.

Launcher has special handling for launching Master-BeaconServer, BeaconClients, and Request-BeaconServer instances.

Beaconizer will install itself as c:/beaconizer/BeaconServer.exe and c:/beaconizer/BeaconClient.exe. It will copy these executables to c:/beaconizer/beaconcopy.XX.XX.(YYYY-mm-dd).(HH-MM-SS)/BeaconServer.exe as it runs, and relaunch processes from those paths.

Running Beaconizer

In order to run beaconizer, you will need a running dbserver (which itself requires authserver). A launcher is not strictly required, although it may make things simpler.

You must run exactly one instance of Master-BeaconServer. Launch it with:

beaconizer -beaconmasterserver

You will also need to start multiple copies of BeaconClient. These are the workers which do most of the computations involved in creating beacons. The more of them you have, the faster it will go. The first instance of BeaconClient that you start on any machine will become Sentry-BeaconClient, which does not do any computation itself. You will need to start more than one instance of BeaconClient for anything useful to happen. There are two options for this. If you run the Sentry-BeaconClient with -beaconproductionmode, then other BeaconClient instances will be started manually; this is intended for use from the launcher. If you run the Sentry-BeaconClient without the -beaconproductionmode flag, then it will launch its own BeaconClient instances and let them work only when this machine is not "active", based on keyboard and mouse activity. This is intended to run on a desktop that is used part of the time.

BeaconClient is launched with:

beaconizer -beaconclient 127.0.0.1

filling in the IP address where you are running the Master-BeaconServer.

With a Master-BeaconServer and some BeaconClients running, you now need to start a BeaconServer to do the work. There are three options for this.

1. If you want to process beaconization requests from MapServers as part of a live server, you want to run a Request-BeaconServer.
2. If you want to process all the maps that you have locally, you want to run an Auto-BeaconServer.
3. If you want to control the process directly, you want to run a Manual-BeaconServer.

You can do all of these at the same time, as long as you have enough BeaconClients running to process their work. In the case of Auto and Manual modes, it will load the maps and bins itself, so you must pass -beacondatatoolsrootpath. Here we will assume that you have installed the server into c:\Ouroboros and have your data files in c:\Ouroboros\data.

If you are using Auto-BeaconServer to process maps locally, this command will cause it to process every map once (skipping any that do not need doing) and then exit:

beaconizer -beaconautoserver 127.0.0.1 -beacondatatoolsrootpath c:\Ouroboros -beacononepassonly

Be aware that on a single machine, a City_Zone map will take several hours to process.

Master-BeaconServer

The master server is responsible for handling requests from mapservers, and for coordinating other BeaconServer and BeaconClient instances.

MapServer requests

Every MapServer connects to the Master-BeaconServer when it needs a map beaconizing. It gets the address of the Master-BeaconServer from the -beaconusemasterserver command-line argument, which launcher sets based on MasterBeaconServer in servers.cfg. It connects as a "Requester", using BMSG_C2S_REQUESTER_CONNECT. It then uploads its own map data as a series of RequesterMapData messages.

The Master-BeaconServer creates a process node and file load request for the map data it receives, for its own worker threads. When all map data has been received, the file loader thread will write the map data to a .beaconrequest file, which it uses to manage a queue of pending requests. When a Request-BeaconServer is available, the Master-BeaconServer will assign one of these requests to it and transmit the map data to the Request-BeaconServer as MapData messages. When the Request-BeaconServer is finished, it sends a beacon file back to Master-BeaconServer as BeaconFile messages. The Request-BeaconServer then becomes available to pick up more work, and the Master-BeaconServer sends the beacon file back to the MapServer as BeaconFile messages.

Coordinating workers

Every other BeaconClient and BeaconServer connects to the Master-BeaconServer. When a BeaconServer is processing a map, the Master-BeaconServer assigns some BeaconClients to it, sending each BeaconClient a TransferToServer message, which causes the BeaconClient to disconnect from the Master-BeaconServer and instead connect to the BeaconServer it was directed to.

Every Request-BeaconServer waits for the Master-BeaconServer to send it maps to process.

An Auto-BeaconServer loads the list of maps that it finds locally, and treats this as a work queue: it loads each map in turn and waits for the Master-BeaconServer to send it clients to process them.

A Manual-BeaconServer does no work until instructed via the console menu.

Master internals

The Master-BeaconServer starts two worker threads.

The file loader waits for load requests to arrive, reads the corresponding .beaconrequest file from the cache dir (checking that the file version and crc match), then marks the load request as complete.

The request processor queues all requests in the ${cache_dir}/pending on startup, then scans the process queue for work it can do. On each pass it will:

• Take up to three nodes in state WAITING_FOR_WRITE_TO_DISK and write their map data packet to ${cache_dir}/pending/${name}.pending
• For the first 50Mb of compressed map data in the queue, load the map data from the pending file if it is in state WAITING_FOR_MAP_DATA_FROM_DISK and move it to state WAITING_FOR_REQUEST_SERVER. For the rest of the map data in the queue, free it and park it in state WAITING_FOR_MAP_DATA_FROM_DISK. This caps the amount of data held in memory at 50Mb.

In production mode, the master server will connect to dbserver and send BEACON2DB_BEACONSERVER_STATUS messages roughly once a second, with the longest outstanding client request wait time.

In its main processing loop, the master server:

• processes all clients to assign them work and check pings
  • any connected client of type BCT_SERVER which is a Request server and does not currently have a process node assigned, will select the first process node in the queue in state WAITING_FOR_REQUEST_SERVER and assign it to this server. This sends BMSG_S2CT_MAP_DATA to that server.
• scans the process queue for completed work
  • any nodes without a requester are cancelled
  • any process node in state WAITING_FOR_LOAD_REQUEST which has completed loading (by the file loader thread) is checked for load success. If the file was loaded, then it is sent to the client and the state moves to WAITING_FOR_CLIENT_TO_RECEIVE_BEACON_FILE. Otherwise, it moves to state WAITING_TO_REQUEST_MAP_DATA_FROM_CLIENT
  • any process node in state WAITING_TO_REQUEST_MAP_DATA_FROM_CLIENT is checked to see if all map data has been received, and moved to state WAITING_FOR_WRITE_TO_DISK. If not all map data has been received, then the first five process nodes in this state will send BMSG_S2CT_REQUEST_CHUNK_RECEIVED to the client and move to state WAITING_FOR_MAP_DATA_FROM_CLIENT
  • process nodes in state WAITING_FOR_MAP_DATA_FROM_CLIENT count towards the five node limit if the request was sent less than three seconds ago
• assigns available clients to servers which need them

This gives us the following state machine for process nodes:

• Created in state WAITING_FOR_LOAD_REQUEST in response to BMSG_C2ST_REQUESTER_MAP_DATA
• File loader thread attempts to load the data
• If valid data is found, moves to state WAITING_FOR_CLIENT_TO_RECEIVE_BEACON_FILE. This is a terminal state; it stays here until the client is done.
• If no valid data is found, the main processing loop advances it to WAITING_TO_REQUEST_MAP_DATA_FROM_CLIENT
• When the master server has capacity to continue (5 at a time), it sends BMSG_S2CT_REQUEST_CHUNK_RECEIVED to the client and moves to state WAITING_FOR_MAP_DATA_FROM_CLIENT
• When a chunk of map data is received, moves to WAITING_TO_REQUEST_MAP_DATA_FROM_CLIENT (loop back up)
• When all map data is received, moves to WAITING_FOR_WRITE_TO_DISK
• When the processing thread has capacity (3 at a time), writes the map data to a pending file
• If too much data is pending, unloads some so that only 50Mb of map data is held in memory waiting for request servers, and queues it for reloading when there is capacity (state WAITING_FOR_MAP_DATA_FROM_DISK)
• When capacity is available, moves to WAITING_FOR_REQUEST_SERVER
• When a Request server is available, assigns the work to that server and sends BMSG_S2CT_MAP_DATA
• When the request server sends BMSG_C2ST_BEACON_FILE, moves the state to WAITING_FOR_CLIENT_TO_RECEIVE_BEACON_FILE and begins sending the file

Command-line flags

-beaconclient <master name>	Start as BeaconClient. Set the master server name.
-beaconserver <master name>	Start as Manual-BeaconServer. Set the master server name.
-beaconautoserver <master name>	Start as Auto-BeaconServer. Set the master server name.
-beaconmasterserver	Start as Master-BeaconServer.
-beaconrequestserver <master name>	Start as Request-BeaconServer. Set the master server name.
-beaconclientsubserver <subserver name>	Set the subserver name for this client.
-beaconusemasterserver <master address>	Set the network address to connect to the master server.
-beaconrequestcachedir <cache dir>	Set the request cache directory. Default is c:/beaconizer/requestcache
-beaconproductionmode	Set the beaconizer into production mode. If this flag is not used then it will be in development mode.
-beaconallownovodex
-beaconsymstore	Only used on a master server. Runs symstoreVS8.exe to add BeaconServer.exe, BeaconClient.exe, BeaconServer.pdb and BeaconClient.pdb to a symstore.
-beaconworkduringuseractivity	Allows a BeaconClient to work while the user of the machine is "active" (keyboard or mouse moved recently).
-beaconnonetstart
-beaconnoperforce
-beacondatatoolsrootpath <path>	Set the data/tools root path, where maps and data files will be loaded from. Default is c:/game. This is used by the Manual and Auto BeaconServers to load maps. Request-BeaconServer gets the maps sent by MapServer instead.