Remote Trace Viewer Tool

The Remote Trace Viewer (RTV) tool allows you to load, view, and analyze multiple remote trace profiles simultaneously in your local browser. It operates on your local machine and establishes a secure connection to the Intel Gaudi server. This user-friendly tool requires only the Intel Gaudi server address and the remote location of the trace profile.

Note

RTV currently supports the https://perfetto.habana.ai/ profile viewer only. For further details, refer to Getting Started with Intel Gaudi Profiler.

The following features are included in the tool:

  • Efficient Resource Management - Avoids downloading large (multi-GB) trace files onto your local machine to save time and storage space.

  • Avoids Memory Constraints - Bypasses the WASM Trace Processor Out-of-Memory error encountered when loading large trace files locally.

  • Facilitates Comparative Analysis - Allows for active comparative analysis of multiple trace profiles simultaneously.

Prerequisites

  • Read access rights to the location of the trace profile on Intel Gaudi server.

  • For Windows users accessing remote trace files, ensure the setup of WSL (Windows Subsystem for Linux) is available in the Start Menu >> Microsoft Store.

Using RTV

  1. To download the RTV tool, open https://perfetto.habana.ai/, select from left bar “Trace Processor Shell” and download the tool for windows or Linux.

  2. Open a terminal window (or a console on Windows) and navigate to the directory where the trace_processor_shell was downloaded.

  3. Execute the script with the following arguments:

    Windows: trace_processor_shell.exe -httpd <TRACE_FILE_PATH> --httpd_port <PORT_NUMBER>

Linux: ./trace_processor_shell -httpd <TRACE_FILE_PATH> --httpd_port <PORT_NUMBER>

    • <TRACE_FILE_PATH> is the full path to the trace profile on the Intel Gaudi server.

    • <PORT_NUMBER> is the port number used to create a socket for viewing the trace at https://perfetto.habana.ai/?rpc_port=<PORT_NUMBER>. The default port is 9001.

  4. Wait for the trace file to load. Upon running the script, if needed, enter the password for the SSH connection to Intel Gaudi server. The script will then start loading the remote trace file. Allow the file to fully load, and upon completion, a confirmation message indicating the successful loading of the trace will appear. For example:

    processor_shell.cc:1636 Trace loaded: 3671.31 MB in 197.14s (18.6 MB/s)

  5. Launch the local browser manually. Launch a browser tab with the following URL ending with PORT_NUMBER. Make sure to replace 9002 in this URL with the PORT_NUMBER used in Step 1.

    https://perfetto.habana.ai/#!/?rpc_port=9002

  6. Select the native accelerator. To load the trace file, in your browser, select “YES, use loaded trace” button as shown in the screenshot below.

    ../../_images/Use_loaded_trace.png

  7. (Optional) Run Steps 2-5 to load more trace files on a different port number. You can now view multiple active traces together in different browser tabs with various ports. This is useful for making direct comparisons.

    ../../_images/analyse_multiple_traces.png

Troubleshooting

When starting a new trace viewing session on a previously used port number, the following error message may appear:

http_server.cc:63 Failed to listen on IPv4 socket: "127.0.0.1:9001" (errno: 98, Address already in use)

This occurs because the trace_processor_shell process from a previous session might still be running on the server, even after the session has been disconnected.

To resolve this, you have the following options:

  • Use a different port number for your new trace viewing session.

  • Stop the trace_processor_shell process manually, and restart the session. Make sure to use the lsof -i Linux command to find the process ID (PID) of trace_processor_shell, and kill -9 <PID> command to terminate the identified process.