Debugging Slow Convergence

• Use TensorBoard to visualize the model and its training progression

• View the Model Graph

• Use the Profiler to identify bottlenecks

TensorBoard Usage

Graph Visualization

Visualizing topology graphs in TensorBoard does not require performing any Gaudi-specific steps. To save and display a model graph, see the example in TensorBoard.

Model Graph

Set the following environment variables to generate a dump of the training graph:

$ export GRAPH_VISUALIZATION=1
$ # Train your model as usual

The graphs will be written to .graph_dumps directory as *.pbtx files.