Debugging Slow Convergence

• Use TensorBoard to visualize the model and its training progression

• View the Model Graph

• Use the Profiler to identify bottlenecks

Graph Visualization with TensorBoard

Visualizing topology graphs in TensorBoard does not require any Gaudi-specific steps. To save and display a model graph, see the example in TensorBoard.

Model Graph

Set the following environment variables to generate a dump of the training graph:

$ export GRAPH_VISUALIZATION=1
$ # Train your model as usual

The graphs are then written to .graph_dumps directory as *.pbtx files.