MAK Legion Capabilities
MAK Legion Capabilities
Click through the tabs below to learn about MAK Legion.
Introduction
Today’s simulations require millions of high-fidelity entities to support representation of megacities, crowds, and large brigade and above training scenarios. Scaling distributed simulation exercises has been envisioned for many years. Achieving such scales requires a design that looks at the entire system, to include the simulation engines and a framework for interoperability that will support the needs of high performing client systems.
The current distributed simulation frameworks were not really designed for this level of scale, nor did they consider operation in a cloud-based, client-sever paradigm (similar to Massive Multiplayer Online Games (MMOGs)).
The US Army Synthetic Training Environment program asked MAK to develop a next-generation framework as part of the Common Synthetic Environment (CSE) OTA. The result of this effort was a prototype high performance scalability framework called Legion.
MAK is continuing development and productization and is supporting the Simulation Interoperability Standards Organization (SISO) to make Legion a standard for the entire distributed simulation community.
Legion Overview
The Legion framework has three main components:
- The Data Store Library – which supports the implementation of efficient, multi-threaded simulation applications
- The Network Library – which efficiently communications state updates and events between participating applications
- The Entity Server – which maintains and serves authoritative state for entities based on client interest
The two libraries (Data Store and Network) are what you would use to link your participating federate into a simulation with potentially millions of entities.
Legion Data Store Library
The Legion Data Store Library addresses an important component of scalability: the ability of simulations to massively increase the number of entities each simulation engine can simulate by parallelizing access to the entity state and interaction Data Store.
Enabling Massively Parallel Simulation
Many traditional simulation engines are not able to take full advantage of today’s multi-core hardware. Such systems use a single simulation execution thread that executes one task at a time.
A more efficient approach is to process entity tasks in parallel, taking better advantage of multiple cores and allowing the application to represent many more entities for each frame.
The Problem: Multi-Threading Issues
A challenge with multi-threading is that when one thread may be writing the state of an entity, another thread may also be reading it. Using locking to prevent this problem creates bottlenecks in execution and reduces the overall performance of the execution – basically, reducing the performance gains that multi-threading was meant to achieve.
The Solution: Double-Buffering
The solution is to implement a double-buffering scheme where simulations read from the current frame’s copy while writing to the next frame’s copy. Once the entities have been updated in the current frame, the next frame becomes the current frame. This ensures the framework is threadsafe.
Making the Data Store part of the framework, versus part of a federate’s implementation, allows any application using the framework to benefit from this multi-threading capability.
Legion Data Store Application Programmer Interface (API)
Legion provides a simple Object-Oriented API to the Data Store, allowing applications to interface with the framework while shielding the application from the details of double buffering. Large numbers of entities can be simulated without concern for locking or synchronization issues. Implementing the Legion API does not, however, impose a requirement that the application use multi-threading. Even applications that are not multi-threaded can use the framework. Single-threaded applications will just not get the full performance benefit of this part of the Legion architecture.
Data-Oriented Design
Legion’s Data-Oriented Design is a significant contributor to the overall efficiency of the Legion framework. Arranging the entity data into large, contiguous blocks of memory allows efficient use of the computer’s memory cache. Allowing the application to iterate over many entities without needing to move data in and out of the cache (“cache misses”) reduces the processing overhead, thus reducing the time needed to process each entity.
Summary of Legion Data Store - Key Points
To summarize the Legion Data Store Library:
- The Data Store Library is a standard, optimized, multithread-safe way for applications to store and access state data, that also doesn’t require the applications to be multithreaded.
- The Data Store Library is not dependent on the Network Library. Because of this, the Legion Data Store Library can be used to improve the performance of high volume simulation engines that are non-networked or that use different distributed simulation standards, such as DIS and HLA.
- The built-in standard object model and the type-safe API are based on DIS-style entity types and the Real-time Platform Reference (RPR) Federation Object Model (FOM), and they support SISO enums, geocentric coordinates, dead-reckoning, etc.
- Code generation is used to extend the Data Store to support custom object models.
The focus here is not to reinvent the interoperability wheel, but to be more efficient in how we send the data we have always sent during distributed simulation exercises.
Legion Network Library
The Legion Network Library handles the communication of data from the Data Store to the network and from the network to the Data Store.
The Legion Network Library is responsible for communicating information between simulation applications. Although multiple different implementations are possible, our primary implementation uses a TCP-based client-server approach involving a central Legion Server. The Legion Network Library’s job is to synchronize the application’s local Data Store with a mirrored copy in the Legion Server: It pulls the state of locally-simulated entities from the Data Store, and sends any required updates to the Server; and writes data to the Data Store based on updates from the Server about remote entities that match the application’s interest criteria.
The networking details are abstracted behind a standard API which allows different configurations based on exercise unique needs. For example, if minimizing latency is more important than scalability, a peer-to-peer implementation is possible, without the use of an Entity Server.
Legion Entity Server
The Legion Entity Server is a centralized server which acts as a “shared blackboard”, maintaining a mirror of each Sim Engine’s Data Store.
This figure shows an example of Sim Engine instance A and Sim Engine instance B as clients working with the Legion Server. The server maintains a copy of the Data Stores from clients A and B as well as for C and D (which represent any arbitrary number of other, not shown, simulation clients). Any Sim Engine instance may be simulating a single entity (as in a first-person simulator) or may be simulating thousands of entities (as in a computer generated forces (CGF) simulation).
The data stays in the same, contiguous layout from the sim engine’s Data Store through the network and into the entity’s Data Store. Keeping the data aligned like this eliminates the need for expensive marshalling and copying of data.
Legion’s design also supports the use of multiple Entity Servers if a single server cannot handle the entity load from the simulation engines.This configuration may also be useful as a risk mitigation strategy (redundant servers) or to support wide area network (WAN) implementations with Entity Servers at each site.
Legion Interest Management
A significant contributor to Legion efficiency lies in the interest management scheme.
From the client simulation application’s perspective, the client expresses interest in entity and event data through the Network API. Then the Legion Network Library on the client receives large messages from the server that represent all remotely-simulated entities matching the client’s interest criteria. These “reflected entities” are placed directly into the client's Data Store, ready for use by the client simulation in the next frame.
The client’s simulation code uses the same abstract Data Store API to access data about the reflected objects that it uses to store and access the entities that it is simulating locally.
The Network API:
The server’s Data Store is coarsely organized spatially by simulation instance. The Network Library in the Entity Server takes care of the fine filtering against the interest criteria, then sends only the data required by the client using the same data representation as defined in the Data Store, thus no data marshalling is required.
Summary of Legion Networking Library - Key Points
To summarize the Legion Networking Library:
- Using TCP for the transport mechanism ensures simpler and more efficient code
- Bundling the state for thousands of entities into a single large packet makes more efficient use of the network
- Entity Server sends only the data needed by each client
- Networking Library offloads responsibility of sending to many clients to the Entity Server (vs the clients)
- This Networking Library and Entity Server framework is well suited for WAN and cloud implementations
Comparing Legion with other Standard Frameworks
So how does the Legion Framework compare with other frameworks such as Distributed Interactive Simulation (DIS) or High Level Architecture (HLA)?
DIS - Interface is at the Network
With DIS, each application implements its own unique Data Store as well as its own unique networking code (though some use 3rd party applications for this like VR-Link). Really, the only “Framework” is the network itself, where the interface between the application and the framework is at the network protocol level.
HLA - Interface is at the Network API (RTI)
In HLA, the Federate application implements its own Data Store, but the Framework provides a common networking implementation (in this case the Run-time Infrastructure). The interface between the Application and the HLA Framework is at the Network API level. This requires a lot of marshalling/copying to encode and decode each attribute from the internal Data Store for the federate into the FOM specific representation to be passed to the HLA API.
Legion - Interface is at the Data Store API and Network API
With Legion, there is a common, optimized Data Store Library with a common, optimized network implementation. The interface between applications and the framework is at the Data Store API and the Network API level. This allows the Network Library to be more efficient since it can reach directly into the Data Store implementation.
Key Differences between Legion and HLA
There are a number of key differences between HLA and the Legion Framework design which are summarized in the table below:
HLA Feature | Legion Approach |
RTI | |
Stateless RTI – Attribute requests passed through to federates | Entity server mirrors entity state – can respond to requests automatically. |
Late Joiners | |
All clients responsible for informing late joiners of their state. | Server takes the responsibility off the clients to inform the late joiners. |
API | |
Generic attribute API – Typically required higher-level SDKs above | Built-in semantics for commonly used data, type-safe APIs, etc. |
Interest Management | |
Generic interest management – No built-in geographic filters | Built-in knowledge of locations, spatial organization, data-based filters |
Data Store | |
All federates responsible for maintaining state efficiently | No Data Store, all federates are standard, optimized, thread-safe implementation of Data Store |
Filtering | |
| Filtering typically performed by RTI library linked into federate | Filtering largely outsourced to Entity Server |
Frame | |
No concept of "frame" – Separate message usually sent for each entity | Bulk-sending state for many entities each frame is vastly more efficient |
Why Not Just Evolve HLA?
HLA has been designed to be stateless and is unaware of Object Model semantics. The HLA API requires federates to pass data, and not “pull” data from the federate’s Data Store. The HLA API was also built around passing attributes one at a time. The concept of a simulation frame, used by Legion for double buffering, does not exist in HLA.
Legion’s fundamental basis is on Data-Oriented Design with an approach that is geared more toward efficient data access and distribution to take advantage of multi-core hardware environments and efficient cache access. It has been optimized for scalability and ease of use by trading off some of the flexibility that is required by HLA. For example, there is no Time Management in Legion, which has greatly simplified its implementation. It also implements a simplified ownership transfer mechanism - Legion cannot transfer individual attributes but must transfer the entire entity. In Legion, a simplified interest management approach is based on knowledge of semantics (geography, etc.).
Legion is not intended to be a general-purpose replacement to HLA. It is presented as an alternative when very large entity counts are required or when the benefits of a client-server architecture are desirable.
Status, Future and Standardization
Initial Integrations
Initial integrations of Legion with several simulation clients and game engines have already been developed, demonstrating Legion's ability to bring any application into million-entity simulation scenarios.
- MAK has demonstrated scenarios of up to 3.8 million moving entities for the US Army Synthetic Training Environment using MAK’s reference implementation which included:
- 380 instances of VR-Forces CGF, simulating 10,000 entities each
- State of the entire scenario managed by a single Legion Server
- Sim Engines and Legion Server hosted on AWS cloud, with Legion Monitor, VR-Forces GUI and VBS4 clients connected from the STE facility in Orlando
- Initial integrations achieved with Legion include: VR-Forces, VR-Engage, VBS4, Unreal, Unity, OneSAF, and Exonaut.
- Legion has a well defined API with source code examples in several languages including: C++, C, Java, and C# APIs.
- The Built-In API covers the full breadth of DIS/RPR FOM semantics, supporting DIS type implementations out of the box.
- Development of a Legion/HLA/DIS Gateway and Legion Logger are nearly complete.
SISO Standardization
MAK has been a member of the simulation interoperability standards community since its founding nearly 30 years ago. We believe that open standards are best for the overall community. We are currently working with the Simulation Interoperability Standards Organization (SISO) to provide the Legion Interface Specification as the basis for a new Scalability/Interoperability Standard. As part of our participation in the 2021 Virtual Simulation Innovation Workshop, MAK presented the Legion Framework and hosted a kick-off of a Legion Study group to begin the process of Legion standardization.
