1.The first step in diagnosing a GPU crash is to identify, if possible, the error code associated with the crash. This can be done by enabling the Logfile option in an MPC Trace Advanced call. If the program crashes, the last entry in the logfile should have an error code that may be used to help identify the path forward to resolving the issue. 2.A common reason that the program may crash has to do with the over-allocation of memory resources on the GPU board. On the GPUs, memory for raytracing must be allocated prior to the start of the raytrace (unlike the CPU raytrace, where memory is dynamically allocated and can overflow to temporary buffer files on the hard disk). When ray splitting, in particular, is involved, additional memory is pre-allocated to support up to N total ancestry levels. However, depending on the size of the model (when loaded into GPU memory) and the available GPU memory, the raytrace may end up using more memory than is physically available on the GPU device. In such a case, the GPU device itself will crash and take the program with it. An easy way to check for this scenario is to use the "Force Monte-Carlo raytrace" option on the MPC Trace Advanced feature and see if the program still crashes or not.
Failed to set pipeline stack size
The following messages may be reported at the start of a GPU raytrace when ray splitting is allowed by the raytrace properties:
failed to set pipeline stack size failed: MPCTasks:1198 OPTIX_ERROR_INVALID_VALUE
These errors indicate that a raytrace with the maximum requested split-ray ancestry cannot be executed. When ray splitting is allowed during the GPU raytrace, the maximum possible ancestry level that can be supported will be limited by one of two items:
•The NVIDIA libraries have a maximum ancestry limit of 31 •GPU hardware boards have different intrinsic capacities to support ancestry levels. For example, an RTX 3070 board is limited to 13 total ancestry levels.
The user must either (a) decrease the requested maximum ancestry level until the raytrace can be initialized, or (b) switch to the Monte-Carlo raytrace mode so that no ray-splitting occurs. Decreasing the maximum ancestry level can be done by using the "Requested maximum ray ancestry level during a split-mode raytrace" in the MPC Advanced Raytrace dialog or by adjusting the allowed specular and scatter ancestry levels on the document's raytrace properties.
There are no sources defined on the device
Not all source definitions are supported directly on the device, meaning that the GPU cannot generate the rays itself during a Trace GPU Rays call. This error message indicates that rays are unable to be generated directly on the GPU device for the active source definitions. The active sources will either, (a) need to be generated by the CPU using a Trace CPU Rays call, or (b) the source definitions will need to be modified to conform to the options supported by the GPU. Refer to the Sources section of the Supported Features help topic for more information.
What is the NVIDIA compute capability? The compute capability can be thought of as a version number that refers to the architectural and supported feature specifications of a given GPU board. In general, the higher the compute capability number, the better the card. FREDmpc requires a graphics card with compute capability greater than 6.0, though this number is subject to change in future releases.
When comparing the results from GPU and CPU calculations, there are some key issues to consider:
1.Comparisons are most easily made in Monte-Carlo mode due to the fact that in ray splitting mode, only the parent rays are available in the CPU ray buffer following an MPC Trace CPU Rays call. The "missing" child rays in ray splitting mode make it difficult (if not impossible) to compare results between GPU and CPU. 2.The GPU raytrace does not track "absorbed rays" at each surface, which the CPU raytrace does. Consequently, this absorbed ray data is missing from the GPU raytrace. 3.Compare the analysis results from your detectors between the GPU and CPUs, considering both the total power as well as the features in the distributions. If these appear consistent, then crank up the ray counts on the GPU raytrace to start getting better statistical sampling and faster convergence! 4.You can use the MPC > Toggle MPC Render Mode option to view the system model in the 3D viewer using the GPUs. The model, as it appears in the 3D view in this mode, appears how the GPUs "see the hardware". Visually inspect the hardware model in the 3D view to check for open seams, gaps or missing hardware that may be the result of meshed approximations of geometry in the GPU representation. Hardware components with gaps may be the result of poor tessellation or may need to be replaced with a different geometry type that is better supported by the GPUs. 5.Use the MPC Trace Advanced option with the "Log MPC raytrace activity to a file" and the "Reload the FRED document to the GPU before performing the Raytrace Action" options enabled. Inspect the output log file to see if there are any unsupported or approximated elements in the system that may be leading to inconsistent results between the GPU and CPU raytrace results.
The MPC Trace Advanced option allows for the specification of a GPU ray buffer size that determines how many rays will be simultaneously loaded into the GPU memory and traced in one single "launch". The size of the GPU ray buffer should typically be set between 1 and 10 million rays. The overall performance of the GPU will have very little dependence on the size of the ray buffer itself, so it isn't recommended that any significant effort be put into "tuning" the size of the GPU ray buffer. If the GPUs are working at peak capacity for a given ray buffer size, then no additional gains are expected by increasing the buffer size beyond the current amount.
Surface Incident/Absorbed Power When a raytrace is performed on the GPUs, surface incident/absorbed power data is collected. Please refer to Surface Incident/Absorbed Power.
Can I use the GPUs to render my model? Yes, the GPUs can be used to provide a real-time raytrace rendering of your model by going to MPC > Toggle MPC Render Mode. Please refer to Rendering (3D View).
A GPU benchmarking file is provided with your installation of FRED. Please refer to the benchmark file information provided here. Benchmark data is compiled to produce the relative GPU performance graph found here.
When running the GPU raytrace in 32-bit mode, it may be observed that certain analyses such as the Surface Incident/Absorbed Power calculation will report "Infinite" ray powers. This behavior can be mitigated in the following ways: •Run the the MPC raytrace in 64-bit mode •Set the model's raytrace properties to use the Monte-Carlo option for the parent ray specifier (or, alternatively, using the "Force Monte-Carlo" option on the MPC Advanced Raytrace settings)
When running the $cudainfo command from FRED's command line, you may observe an error stating that the CUDA driver and runtime could not be initialized (see error message highlighted in red below). This indicates that the NVIDIA driver is too old to be used with FREDMPC. The resolution is to download and install the latest driver for your GPU from the NVIDIA website: https://www.nvidia.com/Download/index.aspx?lang=en-us
--- NVIDIA GPU COMPUTE DEVICE INFORMATION
NVML initialization: success NVML version: 10.430.86 NVML driver version: 430.86 NVML CUDA driver version: 10020 NVML device count: 2 NVML cuda compute capability: 7.5 (Quadro RTX 4000) NVML cuda compute capability: 7.5 (Quadro RTX 4000) NVML shutdown: success
CUDA runtime latest supported driver: 10020 CUDA runtime version: [3]error CUDA driver and runtime could not be initialized. CUDA runtime device count: [3]error No compute ready devices found
MPC Toolbar and Menu Items Greyed Out Upon starting FRED and opening a document, you observe that your MPC toolbar and menu options are greyed out. Reasons for this behavior are itemized below: •You are not currently accessing a FREDMPC license, which is required in order to use the GPU capability. •The computer does not have a GPU installed, or the GPU lacks sufficient compute capability (i.e. is too old) to be used with FREDMPC. Please refer to the GPU Hardware Configuration help page for more information. •The computer has a GPU which is supported, but the driver is too old to work with FREDMPC. In this case, the driver for your GPU should be updated to its latest version. NVIDIA drivers can be found at the following link: https://www.nvidia.com/Download/index.aspx?lang=en-us
|