Recommended threading strategies and OS platform

좀비프로세스가 발생하는 이유를 찾던 중에 dgraph 큐에 큰 job이 들어왔을 때 이 job이 처리되는 시간의 delay로 인해 좀비프로세스가 발생할 수 있다는 얘기는 저 job을 빠른 시간안에 처리를 해줄 수 있다면 좀비프로세스가 발생하지 않는 다는 얘기와 같다.
Endeca 에서는 일반적으로 쿼드코어 프로세스의 경우 스레드를 4개정도 사용할 수 있도록 하는 것을 가이드 하고 있다.
하지만 이렇게 스레드를 할당 했음에도 불구하고 해당 dgraph 이 정상종료 되지 않고 좀비프로세스가 되어 버린다면 스레드를 늘려 볼 필요가 있어 보인다. 물론 이 경우는 프로젝트 초기에 H/W estimation을 넘어선 경우일 거라 여겨지기 때문에 H/W를 늘려주는 게 더 맞는 방법일 수도 있다.

참조: http://mdex61.endeca.com/help/index.jsp?topic=/com.endeca.MDEX.PerfTuning.doc/src/tperf_configuring_the_number_mdex_engine_query_threads.html

The size of the thread pool and the host operating system impact performance and processor utilization.

In general, Endeca recommends using one thread per processor or core for good performance in most cases. The actual optimal number of threads for a given application depends on many factors, and is best determined through experimental performance measurements using expected query load on production data.

If high performance is required, enable more than one thread. Determine the optimal number of threads through load testing of different configurations.

As a starting point, enable the following number of threads:

• On a quad-core processor, enable 4 threads per processor
• On a hyperthreaded processor, enable 2 threads per processor
• On a standard processor, enable 1 thread per processor

For example, consider a server with two hyperthreaded processors and sufficient disk resources and RAM, on which a high-performance application will be deployed. The appropriate starting point for such an architecture would be one MDEX Engine running multithreaded with 4 threads.

Multithreaded MDEX Engine on Linux and Solaris

On Linux and Solaris, the MDEX Engine uses the POSIX Thread Library, Pthreads. You can examine the thread count using standard tools, such as top.

Multithreaded MDEX Engine on Windows

On Windows, the MDEX Engine uses native Windows threads. The thread count for an MDEX Engine can be examined in the Windows Task Manager in the Threads column.

Note: The number of threads listed may be greater than the value specified for the --threads flag; the additional threads that could be listed are those that are used infrequently by processes that are not CPU-intensive and represent internal maintenance tasks. All the CPU-intensive, query processing-related threads are controlled by the --threads flag.

Multithreaded MDEX Engine on VMware

On VMware, use the following configuration:

• Be aware of the limitation of four virtual CPUs per virtual machine.
• Specify four threads for each Dgraph. Overall, the number of threads should not exceed the number of VCPUs.