Recommended threading strategies and OS platform

Recommended threading strategies and OS platform

The size of the thread pool and the host operating system impact performance and processor utilization. For example, on the Solaris operating system, which provides an efficient hierarchical threads system, the MDEX Engine exhibits little decrease in performance at higher thread pool sizes. On other systems, such as Windows and Linux, performance degrades at larger thread pool sizes.

In general, Endeca recommends using one to four threads per processor for good performance in most cases. The actual optimal number of threads for a given application depends on many factors, and is best determined through experimental performance measurements using expected query load on production data.

If high performance is not required, enable two threads by specifying --threads 2. This mode gains all side benefits of multithreaded mode, and helps to prevent long request queues by allowing for simultaneous processing of queries. It also conserves available machine resources by only allowing two threads.
If high performance is required, enable more than two threads. Determine the optimal number of threads through load testing of different configurations. As a starting point, enable the following number of threads:
On a dual-core processor, enable 3-4 threads per CPU
On a quad-core processor, enable 5-7 threads per CPU
On a hyperthreaded processor, enable 2-3 threads per CPU
On a standard processor, enable 2 threads per CPU

Generally, configure one MDEX Engine per physical CPU. You may decide to deploy fewer Engines. Deploy more than one Engine per CPU only after extensive performance testing.

For example, consider a server with two hyperthreaded CPUs and sufficient disk resources and RAM, on which a high-performance application will be deployed. The appropriate starting point for such an architecture would be two MDEX Engines, each running multithreaded with 2-3 threads.