OpenSim Performance and Scalability Project

• Developers corner for local OpenSim efforts
• Testbed setups and hardware configurations
• OpenSim configuration guides
• Building OpenSim and Mono from Sources

Targets

1. Setup a carrier-grade testing infrastructure for network simulations and various loads of OpenSim servers.
   This will be used to conduct networking performance and scalability testing.
2. Prepare an automated build and verification toolchain for building the latest OpenSim together with all the dependencies.
   This will be used to verify correct dependencies, and tune upstream libraries for better performance through small optimisations and compiler flag tuning. E.g. for mono it may be worth trying out various memory allocators and garbage collectors etc.
3. Prepare a text-based, scriptable client for automated stress-testing and funcitonality verification of the OpenSim servers.
   This will be used to actually conduct functionality verification tests, and stress-test the server to obtain performance measures for tuning the OpenSim server itself.

Introduction

OpenSimulator is a community-driven effort aiming at implementing SecondLife compliant server for virtual worlds. The entire system is written in C# on .Net platform. World45 and Praeteritio are looking into way to make the OpenSimulator scalable and perform well on carrier-grade servers, e.g. Sun Microsystesms UltraSPARC multicore systems.

As an example, consider a typical client-server application, such as an airline ticket booking system, there is a set of travel agents, who may be logging into the booking system throughout the day; there is a server managing requests for information, and there is an underlying database processing requests and assembling replies. Such a system is designed against well-defined criteria: number of simultaneous users, maximum acceptable response time, average size of request, number of requests per user and average size of response. On wider level, the system is also designed on number of accesses per day, size of database, available bandwidth, backup and downtime strategies and so on.

OpenSim is similar to these systems, with the added complexity of providing a chunk of calculation with each client request. SecondLife as a Client provides little more than a fast rendering engine, so the position and interaction has to be delivered within good response times for the user experience to be satisfactory.

Many of these basic performance and scalability criteria are not clearly defined yet in OpenSim. As many other community-driven efforts the project grows organically, tested initially with low usage, with higher usage added mostly as an afterthought. This is something this project is trying to address. We will provide more rigid specifications that will help in shaping OpenSimulator? as a truly carrier-grade, high-performance scalable alternative to other commercially viable simulators. OpenSim needs target figures against which its performance can be judged.
We are investigating the possibility of offloading majority of requests to be handled by a database system - the transaction processing on modern databases is now extremely sophisticated, it would be tempting to suggest that ALL incoming client requests are handled by the database engine. However, how much can be handled this way, and how much various OpenSim servers need to handle themselves is not yet known. We are trying to provide appropriate architecture, and identify key response times and adjust the underlying system to meet the key performance and scalability requirements.

The feasibility stage of the project is somewhat constrained - initially, we are only to identify bottlemecks in the processing, limitations laid down by the number of simultaneous users, and bandwidth. We will suggest possible ways of improving the overall performance. But from the above discussion, all such work needs to be seen in the context of the overall system performance.

Picking on the Physics engine, it ought to be possible to built such an engine working autonomously, so test requests can be thrown at it up to the target rates defined, and optimise its performance. if this means more hardware, more parallelism, or revised software, then that can be included in the conclusions. But adding more and more functionality with no regard to either current performance or ultimate performance is guaranteed to lead to failure.

The role of parallelism is key to much of the improvement potential. But often the algorithms need to be designed with that parallelism in mind, rather than dividing the regions up in the hope this will resolve the difficulties. Database engines are typically re-designed from the bottom up to use available parallelism, and can now use this technology most effectively.

Performance Harness Notes

Logging

Logging will be best done directly to log files rather than to the console windows. Each log entry needs a timestamp.

Startup Scripts

An automatic installation script needs to be established.

Automatic testing

A set of automatic tests to explore repeatability and performance. The number of initial concurrent clients and the automated activities needs to be defined eg Logging on and off, dressing, movement, interaction.

Learning the Jargon

In terms of learning the terminology and the like, you could use the second life wikis (see below) or just go into second life and play for a while (second life proper as opposed to an opensim instance, just because things are already set up). Going to a sand-box and building stuff gives you some idea of how everything is constructed and what the system has to accommodate. This makes understanding opensim a lot easier.

Experiences with the OpenSim on Win32 Servers

XP Experiences

Key URLs

• http://www.libsecondlife.org/wiki/Main_Page - This is a library which should make it easy to provide sample load for the server. What we need is covered in the examples.
• https://wiki.secondlife.com/wiki/Main_Page - The second life wiki. Generically useful. Don't go into it in detail, but a decent reference.
• OpenSim Developers Docs
• CIL ECMA standard - the main web-site for CIL documentation
• Mono Performance Tips - how to use mono profiler and related information