--- title: Criteria keywords: fastai sidebar: home_sidebar summary: "Which parameter is important in a neural network ?" description: "Which parameter is important in a neural network ?" nb_path: "nbs/0b_criteria.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
 
{% endraw %} {% raw %}
{% endraw %}

The criteria implemented come from this paper.

{% raw %}

class Criteria[source]

Criteria(f, needs_init=False, needs_update=False, output_f=None, return_init=False)

{% endraw %} {% raw %}
{% endraw %}

Weight Based Criteria

Random

{% raw %}
{% endraw %} {% raw %}
demo_model(random)
{% endraw %}

Large Final Value

{% raw %}
{% endraw %} {% raw %}
demo_model(large_final)
{% endraw %}

Squared Final Value

{% raw %}
{% endraw %} {% raw %}
demo_model(squared_final)
{% endraw %}

Small Final Value

{% raw %}
{% endraw %} {% raw %}
demo_model(small_final)
{% endraw %}

Large Init Value

{% raw %}
{% endraw %} {% raw %}
demo_model(large_init)
{% endraw %}

Small Init Value

{% raw %}
{% endraw %} {% raw %}
demo_model(small_init)
{% endraw %}

Large Init Large Final Value

{% raw %}
{% endraw %} {% raw %}
demo_model(large_init_large_final, 80)
{% endraw %}

Small Init Small Final Value

{% raw %}
{% endraw %} {% raw %}
demo_model(small_init_small_final)
{% endraw %}

Increasing Magnitude

{% raw %}
{% endraw %} {% raw %}
demo_model(magnitude_increase)
{% endraw %}

Movement Pruning

{% raw %}
{% endraw %} {% raw %}
demo_model(movement)
{% endraw %}

Updating Versions

The following criteria use an updating value of the weights, i.e. the value from the previous iteration of training, instead of the initialization value to better capture the training dynamics.

Updating Magnitude Increase

{% raw %}
{% endraw %} {% raw %}
demo_model(updating_magnitude_increase)
{% endraw %}

Updating Movement

{% raw %}
{% endraw %} {% raw %}
demo_model(updating_movement)
{% endraw %}

Updating mov-magnitude

{% raw %}
{% endraw %} {% raw %}
demo_model(updating_movmag)
{% endraw %}

Gradient Based Criteria

First order Taylor expansion on the weight (as per Nvidia Taylor Pruning)

{% raw %}

grad_crit[source]

grad_crit(m, g)

{% endraw %} {% raw %}
{% endraw %}