When performing raytracing with FREDmpc, the available analyses options depend on which mode of MPC raytracing was used.
When using the MPC Trace CPU Rays mode of operation, the ray data is available in the CPU ray buffer following the GPU raytrace. Consequently, any of FRED's analysis functionality may be called on the ray data as long as the selected analysis is appropriate given the ray attributes supported on the GPUs. If ray splitting is allowed by the raytrace properties, child rays will not be available for analyses post-trace. It is recommended that the Monte-Carlo option be used on the raytrace property settings when possible.
When using the MPC Trace GPU Rays mode of operation, only the analyses supported by Detector Entities and Directional Analysis Entities are allowed.
Surface Incident/Absorbed Power Surface incident/absorbed power data is collected during a GPU raytrace and results can be accessed from Analyses > Surface Incident/Absorbed Power after the raytrace. Using the Surface Incident/Absorbed Power option on the Analyses menu will report the appropriate values from the most recent raytrace, whether it was performed on the CPU or GPU. Be aware that when using MPC in 32-bit mode, single precision power accumulation will result in differences between the values reported by the CPU and GPU raytraces. Surface incident/absorbed power collection can be suppressed on the GPUs using the MPC Advanced Raytrace controls.
The Data Collector Surface functionality is supported when performing raytracing on the GPUs and results can be accessed from Analyses > Data Collector Surface Visualization after the raytrace. Be aware that when using MPC in 32-bit mode, single precision power accumulation will result in differences between the values reported by the CPU and GPU raytraces.
When performing coherent ray calculations using FRED's implementation of Gaussian Beam Decomposition, the coherent wave field is calculated by determining the complex field contribution that each beamlet makes to each pixel in an analysis grid and then coherently summing together the results from all beamlets to recover the total optical field. This procedure is generally referred to as a coherent field summation and is required during various calculations such as coherent source power scaling, irradiance spread function analysis, coherent scalar wave field analysis, etc.
If coherent rays, scalar or polarized, exist in the CPU ray buffer and one of the MPC > Toggle MPC Coherent Ray Sum options is active, any calculations that require a coherent field summation will be automatically performed on the GPU. This applies during analyses such as Irradiance, Color Image, Coherent Scalar Wave Field, etc. when coherent rays are present in the CPU ray buffer, but also applies to operations such as coherent source power scaling and coherent field resampling during ray creation. Other than toggling the desired MPC Coherent Ray Sum toggle from the Analyses menu (or toolbar buttons), no additional actions are required on behalf of the user. FRED will simply perform the coherent field summation on the GPUs transparently whenever the calculation is needed. The state of the MPC Coherent Ray Sum toggle is saved with the FRED document itself.
Single vs. Double Precision Two forms of the MPC Coherent Ray Sum are provided; single and double precision. In cases where the rays have been propagated over long distances, or in cases where the ray's current locations have large displacements from the analysis grid being used in the calculation, double precision may be required to ensure accuracy in the field summation. When using the double precision calculation mode, GPU performance will be reduced by a factor that is dependent on the GPU board's architecture. The following procedure can be used to determine whether the double precision coherent field summation mode is required for a given analysis: 1.Turn off the MPC Coherent Ray Sum by making sure that neither toggle, single or double, is active. When called, the analysis will use the double precision CPU algorithm (default). Store the resulting calculation in an Analysis Result Node (ARN). 2.Activate the single precision MPC Coherent Ray Sum toggle. Perform the analysis again and store the resulting calculation in an Analysis Result Node (ARN). 3.Select the two ARNs on the tree and then use the ARN Linear Combination tool to difference the results. If there is a non-negligible residual that cannot be tolerated for your analysis, then the double precision MPC Coherent Ray Sum should be used.
CPU vs. GPU Performance The MPC Coherent Ray Sum will not perform the calculation faster than the CPU in all cases. When determining a coherent ray's complex field contribution to an analysis grid, the CPU field summation algorithm performs a series of conditional checks on the ray to determine whether various approximations can be made . Depending on the result of those checks, the required complex field calculations can be reduced and allow the CPU calculations to be performed orders of magnitude faster than they otherwise would be. The result of this CPU algorithm is that for a group of N rays, each ray's complex field contribution will be calculated using an algorithm tuned for its specific properties. However, this type of conditional code branching inside of an algorithm is incompatible with maximizing GPU performance (best scaling is achieved when all rays on the GPUs do the exact same thing at the exact same time). Consequently, the GPU Coherent Ray Sum algorithm does not implement the conditional checks on the ray data that allow for the reduced set of calculations to be made.
This difference between GPU and CPU calculation time will be most noticeable in cases where individual beamlet wavefronts are flat (or approximately flat), which is often the case in the source construction plane or if the field being represented is a plane wave. The GPU Coherent Ray Sum will outperform the CPU calculation in cases where the CPU algorithm cannot use its reduced calculation modes. Test runs using CPU and GPU modes will be required to determine which mode yields the highest performance for any given system layout.
Analysis Surfaces are unsupported and ignored by the GPUs when using the MPC Trace GPU Rays mode of operation. •In the MPC Trace GPU Rays mode of operation, Plane type Detector Entities should be used in place of Analysis Surfaces when raytracing on the GPUs. •In the MPC Trace CPU Rays mode of operation, Analysis Surfaces can be used to post-process the rays (ex. perform an Irradiance Spread Function analysis) that return to the CPU ray buffer after the GPU raytrace is completed. However, note that only parent rays are returned to the CPU ray buffer, and so for analyses of systems that include ray splitting (e.g. scattered or ghost child rays) those child rays will not be included in the Analysis Surface results. To address this issue either switch to using a detector entity (which includes child rays generated on the GPU) or consider performing a Monte-Carlo trace which does not generate child rays.
If a properly configured and supported Detector Entity (DE) type is present in the model, it can be used by the GPU raytrace to generate Analysis Results Nodes (ARNs). The table below itemizes the support for specific Detector Entity types on the GPUs. Any DE with an unsupported type or unsupported parameter is ignored by the GPUs.
Each detector entity uses a set of parameters that define its size, pixel resolutions, the type of calculation, etc. The table below itemizes the GPU support for each parameter.
1.Detector entities with the “Illuminance” analysis configured will not perform the requested analysis, but will halt rays on the GPU if the “Absorb rays” flag is set to True. 2.Ray filters a.In the context of Detector Entities, ray filters are criteria that are applied in addition to the base requirement that rays have intersected the DE. If, for example, the ray filter of a DE in the model is “Scattered rays”, the expectation is that only scattered rays intersecting the DE contribute to the resulting analysis. This behavior is identical during a CPU raytrace. b.Ray filters operate on the attributes associated with each ray, though not all ray attributes are applicable to a GPU raytrace. The following rules apply when using ray selection filters on the GPUs: •The "Polarized rays" filter will always evaluate to False during a GPU raytrace. Support for polarized rays on the GPUs is not yet implemented. •The "Coherent rays" filter will always evaluate to False during a GPU raytrace. Support for raytracing coherent rays on the GPUs is not yet implemented. •The "Rays on the specified ray path" filter will always evaluate to True during a GPU raytrace. Specification of a path number filter only makes sense in a post-trace context when the path ID of interest is known. •Ray filters involving ray ID numbers (ex. "Ray number N", "Ray numbers <= N", "Every N'th Ray") will be evaluated, but caution is advised for the following scenarios: •When applying these filters during a split mode raytrace, split child rays on the GPUs do NOT have unique ray ID numbers. Split child rays simply inherit the ray ID of their parent ray. This means that in a split mode raytrace there can be multiple different rays with the same ray ID. There is no such ambiguity when the rays are traced in Monte-Carlo mode. •When the GPUs generate the source rays (Trace GPU Rays mode) and multiple GPU boards are being used, the ray ID numbers will be duplicated on each board. For example, if a GPU raytrace is executed in Trace GPU Rays mode using a system with 3 GPU boards, there will be three rays (one on each board) with a ray ID of 0.
Directional Analysis Entities (DAEs) will auto-generate an Intensity on Polar Grid type Analysis Results Node (ARN) at the conclusion of a GPU raytrace subject to the following behaviors: •The ray selection criteria of a DAE is applied at the conclusion of the raytrace, such that the timing of the calculation is equivalent to the “At Trace End” mode of a detector entity construct. Child rays in a split-mode raytrace will be included in the DAE analysis as long as they satisfy the ray selection criteria at the end of the raytrace. •Ray selection filters will be applied in either the Trace CPU Rays or Trace GPU Rays modes of operation.
Please refer to the Detector Entities section of this document for a more complete description of ray selection criteria.
Analysis Results may be generated by the GPUs if a properly configured detector entity (DE) or directional analysis entity (DAE) is active in the model. If a properly configured DE or DAE is present in the GPU raytrace, the Analysis Results Nodes will be automatically added to the FRED document object tree in the Analysis Results folder at the conclusion of the GPU raytrace.
The table below itemizes how specific coating types are handled by the GPUs. Coatings on the GPUs will have no spatial dependence.
1.Simple table lookup rounding down to lowest angle value defined with no interpolation (e.g. if angles 0, 20, and 45 degrees are defined in the coating then a ray at 39 degrees will see the coating definition defined at 20 degrees). 2.Approximated as a General Sampled coating type with incident material Air and substrate material Simple Glass. The coating will be sampled at 8 angles between 0 and 0.9999 in direction cosine space for each active wavelength of all active sources.
GPU raytracing is allowed on remote nodes when using FRED’s Distributed Computing capability.
When a surface is represented on the GPUs, the representation can either be exact or it can be an approximation. For a given surface to be represented exactly, the following requirements must be met: •Surface Type has a GPU implementation •Surface is Traceable •No Surface or Curve Trimming Specifications are applied •Surface is not part of an Element Composite (boolean solid) element
When the above conditions are not met for a given surface, the representation of that surface on the GPUs will be an approximation using triangle meshes (think of the *.OBJ or *.STL formats from CAD, here). For example, a plane surface whose aperture is defined by trimming with a racetrack curve would be converted to a meshed approximation on the GPUs. Meshing a surface will generally result in reduced accuracy in its representation (except, for example, in the case of a meshed planar surface) and can be a source of error in the GPU raytrace when comparing results with the CPU raytrace.
When a surface is approximated by triangle meshing, the triangle patches that approximate the surface on the GPUs are generated using two different algorithms which depend on the surface type. We refer to the two different algorithms here as the "Parametric Tessellation Algorithm" and the "Volume Based Algorithm".
In the Volume Based Algorithm, the surface's volume bounding box is recursively subdivided when forming the tessellation representation of the surface. In this algorithm, the user has NO ability to access controls that would improve the quality of the resulting mesh. Surfaces which use the Volume Based Algorithm can be multi-valued for a given (x,y) position in its aperture (ex. a sphere). The following surfaces use the Volume Based Algorithm when being represented as meshes on the GPU: •Conicoid, General Asphere, Standard Asphere, Conic Focii, Polynomial Surface, Scripted Surface (Implicit) and Cylinder
All other surface types use the Parametric Tessellation Algorithm. The user can improve the quality of the mesh representation for these objects by increasing the tessellation using FRED’s Visualization Attributes dialog or by reducing the tessellation scale factor values on the Visualization tab of the surface dialog. Note that it is possible to represent certain Conicoid, General Asphere and Standard Asphere surfaces using a Zernike surface type as an alternative, in which case the meshing of the Zernike surface can then be user-controlled.
The table below indicates which surface types have exact GPU implementations and which use the triangle mesh approximation.
1.The Cylinder surface type will only be fully supported when the semi-apertures of the Front End and Back End are identical (i.e. the cylinder wall is constant along the Z axis). If the Cylinder wall is sloped, then the surface is approximated on the GPUs as a triangle mesh representation. 2.Surface types not specifically identified in the above table will be approximated on the GPUs as a triangle mesh.
Each surface in a FRED model has a set of attributes that describe the properties of the geometry representation. These attributes can be physical properties, FRED specific properties needed for raytracing, or visualization properties needed for rendering.
The table below indicates whether a given surface property is supported, partially supported, or ignored by the GPUs.
A supported property does not mean the implementation on the GPU is identical to the native implementation in FRED for CPU raytracing. Please refer to the associated section of this document for each supported or partially supported attribute for additional information.
1.If a surface has Surface Trimming Specifications applied, the surface is converted to an approximation using triangle meshing.
During the raytrace, rays intersect surfaces. However, non-surface nodes in the Geometry hierarchy may act to augment the properties of the surface representations. Curves, for example, can be used in FRED to construct surfaces or to define custom apertures on surfaces. Or, the location of a surface may depend upon the location of a Custom Element above it on the tree.
The table below itemizes non-surface geometry node support for the GPUs.
1.Surfaces which are part of an Element Composite structure will be automatically converted to an approximate triangle mesh representation on the GPUs.
Keywords are unsupported and ignored. The utility of keywords is in model construction and model management. As such, keywords have no value to the GPUs for the purposes of raytracing.
The table below itemizes how specific material types are handled by the GPUs.
1.The real and imaginary refractive index values are computed for all active wavelengths of all sources. 2.All unsupported material types are automatically converted to sampled materials. The real and imaginary refractive index values are computed for all active wavelengths of all sources at normal incidence.
The table below itemizes how specific material attributes are handled by the GPUs.
GPU raytrace and analyses can be performed with either 32-bit or 64-bit precision. •The fastest raytrace speed will be achieved in 32-bit mode, at the expense of accuracy. For example, long propagation distances or systems with many intersections may be more susceptible to error stack-up due to single precision raytracing. The 32-bit GPU raytrace may not be able to resolve surfaces separated by distances smaller than 1e-04 (in document units). A comparison of the 32-bit and 64-bit GPU raytrace results can be made in order to determine whether 32-bit mode is sufficient for a given model. Not all system models require the use of 64-bit precision raytracing. •The best accuracy will be achieved in 64-bit mode, at the expense of raytrace speed. The performance penalty incurred with 64-bit mode is dependent on the GPU hardware itself. •A 64-bit GPU raytrace is not guaranteed to recover the same result as the 64-bit CPU raytrace in all cases. This is, in part, due to the fact that certain portions of the GPU raytrace are handled directly by the NVIDIA API and remain limited by 32-bit precision. In spite of this 32-bit bottleneck in the GPU raytrace, the 64-bit GPU raytrace can be expected to have orders of magnitude better resolution in comparison to the 32-bit raytrace mode when extremely large, or extremely small, distance scales are involved.
A precision state toggle is stored and saved with the FRED document and determines what level of precision the MPC Trace CPU Rays and MPC Trace GPU Rays commands are executed with. Double float precision (64-bit) can be selected by going to MPC > MPC Set Raytrace to Double Precision, or by choosing the associated toolbar button. This state toggle does not affect the MPC Advanced Raytrace, which has its own flag to determine the precision with which the advanced raytrace will be executed.
GPU support for ray types is itemized in the table below and provides a good starting point for evaluating whether FREDMPC can be utilized for a given analysis.
1.Coherent Rays a.Coherent rays will not be generated on the GPUs using the Trace GPU Rays raytrace mode. b.When using the Trace CPU Rays raytrace mode to push rays from the FRED ray buffer to the GPUs, the coherence attributes will be removed from the rays prior to GPU raytracing and then replaced on the CPUs when the rays are returned to FRED. These rays will trace on the GPUs but should not be used for radiometric calculations. c.If coherent rays exist in the CPU ray buffer and the MPC Coherent Ray Sum Mode is toggled active (Analyses menu or toolbar button), the GPUs will be used to perform coherent field summation calculations. The GPU coherent field summation will happen transparently for any operation that requires it (ex. coherent source power scaling, irradiance spread function, coherent vector wave field).
Each ray has a set of attributes (ex. position, direction, power, wavelength, etc.) that are used by various analyses functions. The table below itemizes which ray attributes are supported on the GPUs.
1.Light “strength” is an attribute that is only applicable to the Trace Render raytrace mode in order to generate rendered graphics. 2.Only a limited set of the status attribute codes are supported at this time. Supported status codes are: ray is halted, reason ray is halted, ray intersected, ray interacted, ray reflected, ray transmitted, scatter ray, specular ray, unresolvable material error, TIR error. 3.It is important to note that although raytrace path data can be requested using the MPC Trace Advanced options, the ray attribute for path # is not updated during a GPU raytrace. Consequently, there is no support for ray selection filtering based on path # after a GPU raytrace has been performed or for using the image artifact diagnostic "lasso" tool in a chart view. Path based analyses are limited to the information presented in the Raytrace Paths table and the Stray Light Report table.
Raytrace Paths can be tracked on the GPUs when the model uses the Monte-Carlo parent ray specifier on the applicable raytrace property definitions. Raytrace Paths will not be tracked when ray splitting occurs on the GPUs.
A user-supplied maximum total event count for path tracking must be supplied to the GPU prior to raytracing. This option is found on the MPC Trace Advanced dialog in the GUI. Only paths with total event counts less than or equal to this user-supplied value (default = 10 events) will be available for review in the Raytrace Paths table or Stray Light Report at the conclusion of a GPU raytrace.
The number of raytrace paths exceeding the maximum total event count limit, as well as the total power in these paths, is reported in the output window at the conclusion of a GPU raytrace.
There is currently no support for redrawing raytrace paths on the CPU if they have been generated by the GPU.
There is currently no support for performing path-based analyses with rays (ex. analyzing the irradiance distribution for path #X or using the image artifact diagnostic "lasso" tool in a chart window).
The table below itemizes how specific attributes of a raytrace property are handled by the GPUs.
1.Parent ray specification and ray splitting a.When using a parent ray specifier other than Monte-Carlo, care must be taken to properly configure the GPU buffer to ensure that an appropriate amount of resource is reserved for ray generation. In particular, GPU memory allocation for ray splitting is specified on the MPC Trace Advanced dialog by setting the maximum total ancestry level (specular + scatter) to be supported by the GPU device. b.When using a parent ray specifier other than Monte-Carlo in combination with the Trace CPU Rays mode, only the parent rays will be returned to the CPU ray buffer. This limits the available post-trace analyses of ray data using the CPU when ray splitting is active during the GPU raytrace. c.When using the Monte-Carlo (one ray only) option on the GPU, selection of the final ray leaving the interaction proceeds in a 2-step manner that is different than the CPU algorithm for ray selection. The GPU algorithm is the following: Step 1: There are four mechanisms for power leaving the surface interaction; specular reflected power, specular transmitted power, total reflected scatter power and total transmitted scatter power. The amount of power in each of these four quantities is used in a Monte-Carlo selection process to choose what type of ray will ultimately leave the surface (specularly reflected ray, specularly transmitted ray, scatter reflected ray or scatter transmitted ray). Note that the GPU uses the total scatter as the quantity of consideration for each of the scattered mechanisms rather than the fractional scatter indicated by the importance sample applied to the surface. This last point is subtle, but allows for better convergence in the statistical sampling of the scattered contributions. Step 2: The candidate rays of the mechanism selected in Step 1 are formed and a second Monte-Carlo selection process using the candidate ray fluxes is performed to select the final output ray for propagation. For example, if the mechanism selected in Step 1 is transmitted scatter and the importance sample on the surface allows 10 scattered rays, each of those 10 scattered rays become the candidates in Step 2 and only one of them is selected as the final output ray in the Monte-Carlo selection process.
A real-time raytrace rendering of your model can be displayed in the 3D viewer by going to MPC > Toggle MPC Render Mode. This is particularly valuable when debugging MPC raytracing issues since it allows the user to see the model in the 3D viewer as seen by the GPU raytrace. Defects in the geometry representation can be identified in the 3D rendering and are usually the result of the associated geometry element being represented on the GPU as a triangle mesh approximation. In other cases, the geometry may appear to have "pinholes" through the surface and are an indication of a numeric precision limitation associated with the scale of the model.
In this 3D GPU rendering mode, only a limited subset of the 3D visualization view tools and viewing modes are available.
The table below itemizes support for scatter models on the GPUs. The following behaviors apply regarding scatter models on the GPUs:
If a scatter model is unsupported, it will be ignored by the GPUs. A table of total integrated scatter (TIS) values is pre-calculated at 7 angles of incidence as part of the scatter model representation and then interpolated during the GPU raytrace.
1.Unsupported scatter model types are ignored by the GPUs.
Each scatter model definition contains additional attributes that further describe the scattering behavior. These attributes appear in the FRED GUI as “Additional Data” at the bottom of a scatter model’s dialog or as part of the definition for a user-scripted scatter model. The table below itemizes the GPU support for these additional attributes.
A surface with an active scatter model must have at least one active importance sample (or “Scatter Directions Regions of Interest”, as it appears in FRED’s GUI) in order for scattered rays to be generated by the surface. Although importance samples do not affect the radiometry of the scattered rays themselves, it affects the statistical sampling of a given direction by the scattered rays and, ultimately, the ray statistics at the analysis plane(s).
The table below itemizes importance sample support for the GPUs.
1.The GPU implementation calculates an equivalent "Into a given direction" specification for each scattering interaction such that the designated ellipsoidal volume, closed curve, or entity, is fully subtended by the scattered rays. Consequently, the radiometry of the resulting scattered distribution will be correct, but the encompassing sampled directions may be different. The GPU will, generally, overfill the requested importance sample target with rays as compared to the corresponding CPU importance sample definition.
Each importance sample specification has an additional set of attributes (seen in the FRED GUI as “Other Data”) that are used to further optimize the efficiency of an importance sample. The table below itemizes the GPU support for these attributes.
1.The direction type flag will be forced to use the Monte-Carlo option on the GPUs when the scattering surface’s raytrace property has Monte-Carlo set as the parent ray specifier. In all other cases, the requested direction type will be used. This option generates statistically more scattered rays where the BSDF values are highest. 2.When the scattering surface’s raytrace property has Monte-Carlo set as the parent ray specifier, the number of scatter rays will be forced to 1 during the GPU raytrace. In all other cases, the requested number of scatter rays will be used (up to a maximum of 10 rays).
Scripted elements (ex. materials, surfaces, scatter, etc.) are approximated on the GPUs. Refer to the corresponding section of this document for more information regarding how scripted elements become approximated on the GPUs.
Scripts that modify a FRED document (including pre/post update scripts) should be executed prior to sending the document to the GPUs.
The following script commands are used to support MPC raytracing:
There are three raytrace modes of operation for performing a raytrace using the GPUs, described in the Raytrace Modes section of this document. The information in this section is only relevant when the Trace GPU Rays mode is used. When using Trace CPU Rays mode, source features are only restricted by the supported attributes of the ray data on the GPUs (see the Rays section for more information).
When the Trace GPU Rays mode is used, the source definition is loaded into the GPUs and the GPUs are then used to generate and trace the rays. For this to execute properly, the attributes of the source definition need to be supported by the GPU implementation. If an attribute of the source is unsupported, no rays will be generated or traced by the GPUs.
The table below itemizes source attribute support for the GPUs.
1.Only the active wavelengths in the list will be used by the GPUs. This also affects the representation of sampled materials on the GPUs, whose refractive index values are evaluated at the active wavelengths for each active source node that uses the “As specified by list” wavelengths option. 2.Rays generated from sources using the "Random according to spectrum" option should not be traced on the GPUs. See the Spectra section for more information. 3.Refer to the Materials section of this document for details on how each material type is represented by the GPUs. 4.Power units will always be interpreted as Watts, regardless of the actual setting in the source. If, for example, a source designated 50 Lumens, the GPUs would generate rays having 50 Watts worth of total power. The rays will be properly traced, but the radiometry will be incorrect for a 50 Lumen source.
The following Source Primitives have constructions that are supported on the GPU device during a Trace GPU Rays call. •Plane Wave (incoherent) •Lambertian Plane (incoherent) •Random volume into a sphere
All Spectra types are unsupported and ignored. A source whose wavelength specification is “Randomly according to spectrum” will not be able to generate rays on the GPUs using the Trace GPU Rays raytrace mode. Rays created from sources that use Spectra should not be traced on the GPUs using Trace CPU Rays mode. Due to how the information for wavelength and sampled properties (coatings, materials, etc.) are stored on the GPUs, rays generated from sources using Spectra may be returned to the CPU ray buffer with incorrect wavelength values.
As an alternative to spectra, multiple wavelength values should be assigned to a source node's wavelength list and can then be weighted according to a desired spectrum (right mouse click menu option in the source's wavelength list control).
Surface Roughness is not supported on the GPUs. If a Surface Roughness attribute is assigned to a surface in the FRED model, it will be ignored when translated to the GPUs.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||