SPOT Performance Visualization Tool

In order to protect against performance regression of SUNDIALS at all scales, we leverage the GitLab CI infrastructure setup for CI testing to perform continuous performance testing.

To track and visualize performance over time, we collect data with Caliper and Adiak. Steps to setting up continuous performance testing(CPT) can be found in Continuous Performance Testing.

Documentation to using the SPOT performance visualization tool can be found here.

Setting Up Your Own SPOT Visualizations

To display data from non-GitLab jobs or create a local collection of runs, input into the SPOT search bar the absolute path to the directory containing the .cali files and refresh the page. By default .cali files will be generated in the build directory under Benchmarking/caliper and Testing/caliper. To specify where .cali output files are placed, define the CMake options SUNDIALS_BENCHMARK_CALIPER_OUTPUT_DIR and SUNDIALS_TEST_CALIPER_OUTPUT_DIR with the desired directory path.

To retain the same filters as a given SPOT visualization link in Bookmarks to Notable SPOT Visualizations, swap out the sf value in the URL with the directory path containing the .cali files.

Bookmarks to Notable SPOT Visualizations

Notes:

SPOT will be put into maintenance mode in the coming months (as of July 2023). The team behind SPOT plans to bring the functionality to Thicket as SPOT’s successor.

Aggregate results for example runs on SPOT may not be accurate. The aggregation method involves combining .cali files in ways that do not necessarily equal the true sum of the results.

Examples

Aggregated Runs

Benchmarks

Advection Reaction 3D

Diffusion 2D

Note: CUDA Diffusion 2D visualizations are not available as the benchmark errors out before completion.