Remote Trace Viewer Tool

The Remote Trace Viewer (RTV) tool allows you to load, view, and analyze multiple remote trace profiles simultaneously in your local browser. It operates on your local machine and establishes a secure connection to the Intel Gaudi server. This user-friendly tool requires only the Intel Gaudi server address and the remote location of the trace profile.

Note

RTV currently supports the https://ui.perfetto.dev/ profile viewer only. For further details, refer to Getting Started with Intel Gaudi Profiler.

The following features are included in the tool:

• Efficient Resource Management - Avoids downloading large (multi-GB) trace files onto your local machine to save time and storage space.

• Avoids Memory Constraints - Bypasses the WASM Trace Processor Out-of-Memory error encountered when loading large trace files locally.

• Facilitates Comparative Analysis - Allows for active comparative analysis of multiple trace profiles simultaneously.

Prerequisites

• Read access rights to the location of the trace profile on Intel Gaudi server.

• For Windows users accessing remote trace files, ensure the setup of WSL (Windows Subsystem for Linux) is available in the Start Menu >> Microsoft Store.

Note

For Linux/Mac users, the script should function natively from your local terminal.

Using RTV

1. To download the RTV tool, save the code below as a bash script named remote_trace_viewer.sh on your local machine. On Windows, make sure to replace CRLF with LF:

   #!/bin/bash
   
   # Check for the necessary number of arguments
   if [[ $# -ne 2 ]]; then
     echo "Usage: $0 <username@Intel-Gaudi-Server-address>:<remote_trace_path> <port_number>"
     exit 1
   fi
   
   # Parse the arguments into username_and_server and trace_path
   IFS=":" read -r username_and_server trace_path <<< "$1"
   echo
   echo "::RTV:: SSHing to $username_and_server "
   echo
   port=$2
   
   # Establish the SSH connection with the port forwarding using parsed values
   ssh -L ${port}:localhost:${port} ${username_and_server} << EOF
     # Download and prepare trace_processor on the Intel Gaudi server
     curl -LO https://get.perfetto.dev/trace_processor &&
     chmod +x ./trace_processor &&
     echo  "****************************************************************************************************"
     echo  "::RTV:: Launch https://ui.perfetto.dev/#!/?rpc_port=${port} in your local browser to view the trace."
     echo  "****************************************************************************************************"
   
     # Launch trace_processor with specified path and port
     ./trace_processor --httpd ${trace_path} --http-port ${port} &
   EOF
   
   echo "::RTV:: Exited Remote Trace Viewer."
   

2. Run the script:

   1. Open a terminal window (a WSL terminal if on Windows) and navigate to the directory where the remote_trace_viewer.sh script is saved.

   2. Execute the script with the following arguments:

      ./remote_trace_viewer.sh <username@Intel-Gaudi-Server-address>:<TRACE_FILE_PATH> <PORT_NUMBER>
      

      • <TRACE_FILE_PATH> is the full path to the trace profile on the Intel Gaudi server.

      • <PORT_NUMBER>: Choose an unused port number (e.g. 4489, 4490, 4491 etc.) for the browser to load the trace file.

      Use different port numbers for each file when loading multiple traces.

      The following are example commands for running the scipt in different terminals to load multiple trace files in separate browser tabs:

      bash remote_trace_viewer.sh [email protected]:"/home/myuser/hpu_profile/1.pt.trace.json" 4489
      bash remote_trace_viewer.sh [email protected]:"/home/myuser/hpu_profile/2.pt.trace.json" 4490
      

3. Wait for the trace file to load. Upon running the script, if needed, enter the password for the SSH connection to Intel Gaudi server. The script will then start loading the remote trace file. Allow the file to fully load, and upon completion, a confirmation message indicating the successful loading of the trace will appear. For example:

   processor_shell.cc:1636 Trace loaded: 3671.31 MB in 197.14s (18.6 MB/s)
   

4. Launch the local browser manually. Launch a browser tab with the following URL ending with PORT_NUMBER. Make sure to replace 4489 in this URL with the PORT_NUMBER used in Step 1.

   https://ui.perfetto.dev/#!/?rpc_port=4489
   

5. Select the native accelerator. To load the trace file, in your browser, select “YES, use loaded trace” button as shown in the screenshot below.

   

6. (Optional) Run Steps 2-5 to load more trace files on a different port number. You can now view multiple active traces together in different browser tabs with various ports. This is useful for making direct comparisons.

   

Troubleshooting

When starting a new trace viewing session on a previously used port number, the following error message may appear:

http_server.cc:63 Failed to listen on IPv4 socket: "127.0.0.1:4489" (errno: 98, Address already in use)

This occurs because the trace_processor_shell process from a previous session might still be running on the server, even after the session has been disconnected.

To resolve this, you have the following options:

• Use a different port number for your new trace viewing session.

• Stop the trace_processor_shell process manually, and restart the session. Make sure to use the lsof -i Linux command to find the process ID (PID) of trace_processor_shell, and kill -9 <PID> command to terminate the identified process.