MAK Legion: Scalability & Interoperability Framework
Distributed simulation at unprecedented scale
Legion is a scalability and interoperability framework for exchanging information, exchanging state and interactions between different simulation applications - between virtual simulators, constructive simulations, and other tools that go into a simulation system.
Legion is composed of
- Legion Data Store Library – to enable optimized multi-threaded simulation applications
- Legion Network Library – to efficiently pass updates and events between applications
- Legion Entity Server – to maintain and serve authoritative state based on client interest
Proven Scalability and Flexibility
- 3.8-million cloud-hosted entities demonstrated for the US Army Synthetic Training Environment
- Initial integrations with VR-Forces, VR-Vantage, VBS4, Unreal, Unity, OneSAF proven before launch
- Designed for both cloud and local deployment on multiple platforms
Designed for Distributed Simulation: Use Cases
- BRIGADE-LEVEL EXERCISES — MEGA CITIES — MODELING GLOBAL AIR TRAFFIC
- THEATER LEVEL WARGAMES — PATTERN OF LIFE TO FILL YOUR SIMULATIONS WITH CIVILIAN ACTIVITY
Key Features
1. Thread-safe Data Store
- Enables applications to simulate, read, and write many entities in parallel – taking full advantage of available hardware
2. Data-Oriented Design
- Highly-optimized run-time data storage - in contiguous memory for cache coherency
- Object-oriented API layer provides ease-of-use
3. Data Remains in Single Consistent Format Throughout the System
- Network Library, Data Store, and Entity Server designed as one coherent system
- Eliminates expensive data copies, marshalling, and conversions
4. Optimized Network Utilization
- Thousands of entities bundled into a single large packet each frame
- Entity Server sends only the data needed by each client, and only what’s changed
5. Relevance Filtering on the Server
- Server’s Data Store spatially organized
- Server filters against interest criteria
6. Spatial Organization & Ownership Transfer
- Keeps entities spatially organized as they move through the virtual world
7. Optimized for Late Joiners
- All clients get their updates from the server, so late joiners don’t burden every simulator to provide initial state
Legion Overview
The Legion framework has three main components:
- The Data Store Library – which supports the implementation of efficient, multi-threaded simulation applications
- The Network Library – which efficiently communications state updates and events between participating applications
- The Entity Server – which maintains and serves authoritative state for entities based on client interest
The two libraries (Data Store and Network) are what you would use to link your participating federate into a simulation with potentially millions of entities.
Legion Data Store Library
The Legion Data Store Library addresses an important component of scalability: the ability of simulations to massively increase the number of entities each simulation engine can simulate by parallelizing access to the entity state and interaction Data Store.
Enabling Massively Parallel Simulation
Many traditional simulation engines are not able to take full advantage of today’s multi-core hardware. Such systems use a single simulation execution thread that executes one task at a time.
A more efficient approach is to process entity tasks in parallel, taking better advantage of multiple cores and allowing the application to represent many more entities for each frame.
The Problem: Multi-Threading Issues
A challenge with multi-threading is that when one thread may be writing the state of an entity, another thread may also be reading it. Using locking to prevent this problem creates bottlenecks in execution and reduces the overall performance of the execution – basically, reducing the performance gains that multi-threading was meant to achieve.
The Solution: Double-Buffering
The solution is to implement a double-buffering scheme where simulations read from the current frame’s copy while writing to the next frame’s copy. Once the entities have been updated in the current frame, the next frame becomes the current frame. This ensures the framework is threadsafe.
Making the Data Store part of the framework, versus part of a federate’s implementation, allows any application using the framework to benefit from this multi-threading capability.
Data-Oriented Design
Legion’s Data-Oriented Design is a significant contributor to the overall efficiency of the Legion framework. Arranging the entity data into large, contiguous blocks of memory allows efficient use of the computer’s memory cache. Allowing the application to iterate over many entities without needing to move data in and out of the cache (“cache misses”) reduces the processing overhead, thus reducing the time needed to process each entity.
Legion Network Library
The Legion Network Library handles the communication of data from the Data Store to the network and from the network to the Data Store.
The Legion Network Library is responsible for communicating information between simulation applications. Although multiple different implementations are possible, our primary implementation uses a TCP-based client-server approach involving a central Legion Server. The Legion Network Library’s job is to synchronize the application’s local Data Store with a mirrored copy in the Legion Server: It pulls the state of locally-simulated entities from the Data Store, and sends any required updates to the Server; and writes data to the Data Store based on updates from the Server about remote entities that match the application’s interest criteria.
The networking details are abstracted behind a standard API which allows different configurations based on exercise unique needs. For example, if minimizing latency is more important than scalability, a peer-to-peer implementation is possible, without the use of an Entity Server.
Legion Entity Server
The Legion Entity Server is a centralized server which acts as a “shared blackboard”, maintaining a mirror of each Sim Engine’s Data Store.
This figure shows an example of Sim Engine instance A and Sim Engine instance B as clients working with the Legion Server. The server maintains a copy of the Data Stores from clients A and B as well as for C and D (which represent any arbitrary number of other, not shown, simulation clients). Any Sim Engine instance may be simulating a single entity (as in a first-person simulator) or may be simulating thousands of entities (as in a computer generated forces (CGF) simulation).
The data stays in the same, contiguous layout from the sim engine’s Data Store through the network and into the entity’s Data Store. Keeping the data aligned like this eliminates the need for expensive marshalling and copying of data.
Legion’s design also supports the use of multiple Entity Servers if a single server cannot handle the entity load from the simulation engines.This configuration may also be useful as a risk mitigation strategy (redundant servers) or to support wide area network (WAN) implementations with Entity Servers at each site.
Legion Interest Management
A significant contributor to Legion efficiency lies in the interest management scheme.
From the client simulation application’s perspective, the client expresses interest in entity and event data through the Network API. Then the Legion Network Library on the client receives large messages from the server that represent all remotely-simulated entities matching the client’s interest criteria. These “reflected entities” are placed directly into the client's Data Store, ready for use by the client simulation in the next frame.
The client’s simulation code uses the same abstract Data Store API to access data about the reflected objects that it uses to store and access the entities that it is simulating locally.
The Network API:
The server’s Data Store is coarsely organized spatially by simulation instance. The Network Library in the Entity Server takes care of the fine filtering against the interest criteria, then sends only the data required by the client using the same data representation as defined in the Data Store, thus no data marshalling is required.
Comparing Legion with other Standard Frameworks
So how does the Legion Framework compare with other frameworks such as Distributed Interactive Simulation (DIS) or High Level Architecture (HLA)?
DIS - Interface is at the Network
With DIS, each application implements its own unique Data Store as well as its own unique networking code (though some use 3rd party applications for this like VR-Link). Really, the only “Framework” is the network itself, where the interface between the application and the framework is at the network protocol level.
HLA - Interface is at the Network API (RTI)
In HLA, the Federate application implements its own Data Store, but the Framework provides a common networking implementation (in this case the Run-time Infrastructure). The interface between the Application and the HLA Framework is at the Network API level. This requires a lot of marshalling/copying to encode and decode each attribute from the internal Data Store for the federate into the FOM specific representation to be passed to the HLA API.
Legion - Interface is at the Data Store API and Network API
With Legion, there is a common, optimized Data Store Library with a common, optimized network implementation. The interface between applications and the framework is at the Data Store API and the Network API level. This allows the Network Library to be more efficient since it can reach directly into the Data Store implementation.
Status, Future and Standardization
Initial Integrations
Initial integrations of Legion with several simulation clients and game engines have already been developed, demonstrating Legion's ability to bring any application into million-entity simulation scenarios.
- MAK has demonstrated scenarios of up to 3.8 million moving entities for the US Army Synthetic Training Environment using MAK’s reference implementation which included:
- 380 instances of VR-Forces CGF, simulating 10,000 entities each
- State of the entire scenario managed by a single Legion Server
- Sim Engines and Legion Server hosted on AWS cloud, with Legion Monitor, VR-Forces GUI and VBS4 clients connected from the STE facility in Orlando
- Initial integrations achieved with Legion include: VR-Forces, VR-Engage, VBS4, Unreal, Unity, OneSAF, and Exonaut.
- Legion has a well defined API with source code examples in several languages including: C++, C, Java, and C# APIs.
- The Built-In API covers the full breadth of DIS/RPR FOM semantics, supporting DIS type implementations out of the box.
- Development of a Legion/HLA/DIS Gateway and Legion Logger are nearly complete.
