--- title: Granularity keywords: fastai sidebar: home_sidebar summary: "What block of parameter to remove in a neural network ?" description: "What block of parameter to remove in a neural network ?" nb_path: "nbs/0a_granularity.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
 
{% endraw %} {% raw %}
{% endraw %}

Conv2d Pruning

{% raw %}
{% endraw %}

A Conv2d layer possess a 4d-tensor as weights. This means that there exist many ways of removing blocks from it.

0-D Blocks

In the case of convolution filters, removing 0-D elements is equivalent to removing individual weights.

  • weight granularity
{% raw %}
get_pruned_conv(0)
{% endraw %}

1-D Blocks

1-D blocks of elements is equivalent to removing vectors from the convolution filters. There are several ways to chose the vectors, that will be represented below.

  • shared_weight: this granularity is very particular as it removes individual weights from a filter, but with a pattern that is shared across all filters.
{% raw %}
get_pruned_conv(1)
{% endraw %}
  • channel: remove vector of weights along the channel axis.
{% raw %}
get_pruned_conv(2)
{% endraw %}
  • column: remove vector of weights along the height axis.
{% raw %}
get_pruned_conv(3)
{% endraw %}
  • row: remove vector of weights along the width axis.
{% raw %}
get_pruned_conv(4)
{% endraw %}

2-D Blocks

  • shared_channel: remove vector of weight along the channel axis, but with a pattern that is shared across all filters.
{% raw %}
get_pruned_conv((1,2))
{% endraw %}
  • shared_column: remove vector of weight along the height axis, but with a pattern that is shared across all filters.
{% raw %}
get_pruned_conv((1,3))
{% endraw %}
  • shared_row: remove vector of weight along the width axis, but with a pattern that is shared across all filters.
{% raw %}
get_pruned_conv((1,4))
{% endraw %}
  • vertical_slice: remove vertical slices of weight along the height axis.
{% raw %}
get_pruned_conv((2,3))
{% endraw %}
  • horizontal_slice: remove vertical slices of weight along the width axis.
{% raw %}
get_pruned_conv((2,4))
{% endraw %}
  • kernel: remove kernels of from the convolution filters.
{% raw %}
get_pruned_conv((3,4))
{% endraw %}

3-D Blocks

  • shared_vertical_slice: remove vertical slices of weight along the height axis, with a pattern that is shared across all filters.
{% raw %}
get_pruned_conv((1,2,3))
{% endraw %}
  • shared_horizontal_slice: remove horizontal slices of weight along the width axis, with a pattern that is shared across all filters.
{% raw %}
get_pruned_conv((1,2,4))
{% endraw %}
  • shared_kernel: remove kernels of weight from the convolution filters, with a pattern that is shared across all filters.
{% raw %}
get_pruned_conv((1,3,4))
{% endraw %}
  • filter: remove entire filters.
{% raw %}
get_pruned_conv((2,3,4))
{% endraw %}

Linear Pruning

{% raw %}
{% endraw %}

0-D Blocks

As for the convolution filters, weights from a Linear layer can be removed independently.

  • weight: remove individual weights.
{% raw %}
get_pruned_linear(0)
{% endraw %}

1-D Blocks

  • column: remove column of weight, which corresponds to removing input neurons.
{% raw %}
get_pruned_linear(1)
{% endraw %}
  • row: remove rows of weight, which corresponds to removing output neurons.
{% raw %}
get_pruned_linear(2)
{% endraw %}

Transformer Pruning

{% include note.html content='This is an experimental part of the library' %}

{% raw %}
{% endraw %}