# Torchview/hiddenlayer produces blank nodes in visualisation

**URL:** <https://discuss.huggingface.co/t/torchview-hiddenlayer-produces-blank-nodes-in-visualisation/113368>\
**Category:** Models\
**Created:** [October 22, 2024, 10:29am UTC](https://discuss.huggingface.co/t/torchview-hiddenlayer-produces-blank-nodes-in-visualisation/113368 "2024-10-22T10:29:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maklirt](https://avatars.discourse-cdn.com/v4/letter/m/eb9ed0/32.png) [@Maklirt](https://discuss.huggingface.co/u/Maklirt)\
**Post date:** [October 22, 2024, 10:29am UTC](https://discuss.huggingface.co/t/torchview-hiddenlayer-produces-blank-nodes-in-visualisation/113368/1 "2024-10-22T10:29:56Z")

</div>

I have tested _torchview_ and _hiddenlayer_ attempting to visualize operations within _LlamaDecoderLayer_ and both ways produced similarly flawed results. Since both methods lead to the same error I suspect there is something wrong with transformers library (or my understanding of it) or my code.

**TL;DR:**  
torchview and hiddenlayer produce incorrect visualization of LlamaDecoderLayer. The problem is with content of nodes in the visualization that represent tensor operations. They do not include any information. The flow (edges) correctly connect nodes.

**Model/Module to visualize:**  
My code (below) wraps the “LlamaDecoderLayer”, passes “LlamaConfig” and “layer\_idx” such that I can simply pass it to _torchview_ or _hiddenlayer_.

```python
    class SingleLayerLlama(nn.Module):
        def __init__ (self, attn_mask, config, layer_idx):
            super(). __init__ ()
            self.attn_mask = attn_mask
            self.layer = LlamaDecoderLayer(config=config, layer_idx=layer_idx)
        
        def forward(self, hidden_states):
            position_embedding = torch.rand(1, int(hidden_states.shape[1]), int(hidden_states.shape[2]/32))
            position_embeddings = (position_embedding, position_embedding)
            hidden_states = self.layer(
                hidden_states, 
                attention_mask=attn_mask, 
                # position_ids=position_ids, 
                position_embeddings = position_embeddings,
                layer_head_mask=None,
                past_key_value=None, 
                use_cache=False 
            )[0]
            return hidden_states

```

**Visualization results:**  
The results have the following flaw, the nodes include no information.

_torchview_

 ![torchview](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/f/3f9f81ece76585219127f941c502d11bb43aa3e3.png)

_hiddenlayer_  
(my first post, only 1 image is allowed)

**Questions:**

1. Is this expected behavior for _transformers_ models/modules?
2. Is there something wrong with my approach?

**Other info:**  
Code used to produce the visualizations:

_torchview_

```python
    from torchview import draw_graph
    model_graph = draw_graph(
        _layer, 
        device="cpu",
        input_size=hidden_states.shape, 
        expand_nested=True,
        save_graph = True,
        hide_inner_tensors = False,
        hide_module_functions = False,
        show_shapes = True,
        graph_name='LlamaDecoderLayer',
        directory=''<path>/', # Save in the current directory
        )
    with open(''<path>/LlamaDecoderLayer.gv', 'r') as f:
        dot_graph = f.read()
    import graphviz
    graph = graphviz.Source(dot_graph)
    graph.render(
        '<path>/LlamaDecoderLayer', 
        format='svg', 
        cleanup=True,
        engine='neato',
        )

```

_hiddenlayer_

```python
    # From 
https://github.com/pytorch/pytorch/blob/2efe4d809fdc94501fc38bf429e9a8d4205b51b6/torch/utils/tensorboard/_pytorch_graph.py#L384
    def _node_get(node: torch._C.Node, key: str):
        """Gets attributes of a node which is polymorphic over return type."""
        sel = node.kindOf(key)
        return getattr(node, sel)(key)
    torch._C.Node. __getitem__ = _node_get

    import hiddenlayer as hl
    im = hl.build_graph(_layer, torch.zeros(hidden_states.shape))
    print(f" >>> im = {type(im)}")
    im.save(path="<path>/LlamaDecoderLayer_hl" , format="svg")

```

P.S.  
I have tested loading the weights from Llama3.1 model, but the result is the same.

---

<div class="post-metadata">

**Author:** ![Maklirt](https://avatars.discourse-cdn.com/v4/letter/m/eb9ed0/32.png) [@Maklirt](https://discuss.huggingface.co/u/Maklirt)\
**Post date:** [October 23, 2024, 4:02am UTC](https://discuss.huggingface.co/t/torchview-hiddenlayer-produces-blank-nodes-in-visualisation/113368/2 "2024-10-23T04:02:15Z")

</div>

I have found the answer.

The problem was in installation of graphviz. I have installed graphviz from source code ([Source Code | Graphviz](https://graphviz.org/download/source/)) rather than using package manager. For graphviz to correctly display HTML tables, it requires **expat-devel**.

The lesson is: install things using package manager.

Interestingly, graphviz provided very little information on what was wrong when rendering graphs. There was a very sparse warning about rendering HTML tables, which slipped through my fingers several times.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [October 23, 2024, 4:03pm UTC](https://discuss.huggingface.co/t/torchview-hiddenlayer-produces-blank-nodes-in-visualisation/113368/3 "2024-10-23T16:03:14Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.