gromo.modules.growing_module.GrowingModule#
- class gromo.modules.growing_module.GrowingModule(layer: Module, tensor_s_shape: tuple[int, int] | None = None, tensor_m_shape: tuple[int, int] | None = None, post_layer_function: Module = Identity(), extended_post_layer_function: Module | None = None, allow_growing: bool = True, previous_module: Module | None = None, next_module: Module | None = None, device: device | None = None, name: str | None = None, target_in_neurons: int | None = None, initial_in_neurons: int | None = None)[source]#
- property activation_gradient: Tensor#
Return the derivative of the activation function before this layer at 0+.
/!/ A caching mechanism is used to avoid recomputing the value multiple times. Therefore, if the previous module changes its post layer function, the cache must be cleared manually by setting _activation_gradient_previous_module to None.
- Returns:
derivative of the activation function before this layer at 0+
- Return type:
torch.Tensor
- add_parameters(**kwargs) None[source]#
Grow the module by adding new parameters to the layer.
- Parameters:
kwargs (dict) – typically include the values of the new parameters to add to the layer
- apply_change(scaling_factor: float | Tensor | None = None, apply_previous: bool = True, apply_delta: bool = True, apply_extension: bool = True, extension_size: int | None = None) None[source]#
Apply the optimal delta and extend the layer with current optimal delta and layer extension with the current scaling factor. This means that the layer input is extended with the current layer output extension and the previous layer output is extended with the previous layer output extension both scaled by the current scaling factor. This also means that the layer output is not extended.
- Parameters:
scaling_factor (float | torch.Tensor | None) –
- scaling factor to apply to the optimal delta,
if None use the current scaling factor
apply_previous (bool) – if True apply the change to the previous layer, by default True
apply_delta (bool) – if True apply the optimal delta to the layer, by default True
apply_extension (bool) – if True apply the extension to the layer, by default True
extension_size (int | None) – size of the extension to apply, by default None and get automatically determined using self.eigenvalues_extension.shape[0]
- complete_growth(extension_kwargs: dict) None[source]#
Complete the growth to the target size.
- Parameters:
extension_kwargs (dict) – Additional arguments for creating layer extensions.
- compute_cross_covariance_update() tuple[Tensor, int][source]#
Compute the update of the tensor C := B[-1] B[-2]^T.
- Returns:
torch.Tensor – update of the tensor C
int – number of samples used to compute the update
- compute_m_prev_update(desired_activation: Tensor | None = None) tuple[Tensor, int][source]#
Compute the update of the tensor M_{-2} := dA B[-2]^T.
- Parameters:
desired_activation (torch.Tensor | None) – desired variation direction of the output of the layer
- Returns:
torch.Tensor – update of the tensor M_{-2}
int – number of samples used to compute the update
- compute_m_update(desired_activation: Tensor | None = None) tuple[Tensor, int][source]#
Compute the update of the tensor M. Should be added to the type of layer.
- Parameters:
desired_activation (torch.Tensor | None) – desired variation direction of the output of the layer
- Returns:
torch.Tensor – update of the tensor M
int – number of samples used to compute the update
- compute_n_update()[source]#
Compute the update of the tensor N. Should be added to the type of layer.
- Returns:
update of the tensor N
- Return type:
torch.Tensor
- compute_optimal_added_parameters(numerical_threshold: float = 1e-15, statistical_threshold: float = 0.001, maximum_added_neurons: int | None = None, update_previous: bool = True, dtype: dtype = torch.float32, use_projected_gradient: bool = True) tuple[Tensor, Tensor | None, Tensor, Tensor][source]#
Compute the optimal added parameters to extend the input layer. Update the extended_input_layer and the eigenvalues_extension.
- Parameters:
numerical_threshold (float) – threshold to consider an eigenvalue as zero in the square root of the inverse of S
statistical_threshold (float) – threshold to consider an eigenvalue as zero in the SVD of S{-1/2} N
maximum_added_neurons (int | None) – maximum number of added neurons, if None all significant neurons are kept
update_previous (bool) – whether to change the previous layer extended_output_layer
dtype (torch.dtype) – dtype for S and N during the computation
use_projected_gradient (bool) – whereas to use the projected gradient ie tensor_n or the raw tensor_m
- Returns:
optimal added weights alpha weights, alpha bias, omega and eigenvalues lambda
- Return type:
tuple[torch.Tensor, torch.Tensor | None, torch.Tensor, torch.Tensor]
- compute_optimal_delta(update: bool = True, dtype: dtype = torch.float32, force_pseudo_inverse: bool = False) tuple[Tensor, Tensor | None, Tensor | float][source]#
Compute the optimal delta for the layer using current S and M tensors.
dW* = M S[-1]^-1 (if needed we use the pseudo-inverse)
Compute dW* (and dBias* if needed) and update the optimal_delta_layer attribute. L(A + gamma * B * dW) = L(A) - gamma * d + o(gamma) where d is the first order decrease and gamma the scaling factor.
- compute_optimal_updates(numerical_threshold: float = 1e-10, statistical_threshold: float = 1e-05, maximum_added_neurons: int | None = None, update_previous: bool = True, dtype: dtype = torch.float32, use_projected_gradient: bool = True) tuple[Tensor, Tensor | None][source]#
Compute the optimal update and additional neurons.
- Parameters:
numerical_threshold (float) – threshold to consider an eigenvalue as zero in the square root of the inverse of S
statistical_threshold (float) – threshold to consider an eigenvalue as zero in the SVD of S{-1/2} N
maximum_added_neurons (int | None) – maximum number of added neurons, if None all significant neurons are kept
update_previous (bool) – whether to change the previous layer extended_output_layer
dtype (torch.dtype) – dtype for the computation of the optimal delta and added parameters
use_projected_gradient (bool) – whereas to use the projected gradient ie tensor_n or the raw tensor_m
- Returns:
optimal extension for the previous layer (weights and biases)
- Return type:
tuple[torch.Tensor, torch.Tensor | None]
- compute_s_update() tuple[Tensor, int][source]#
Compute the update of the tensor S. Should be added to the type of layer.
- Returns:
torch.Tensor – update of the tensor S
int – number of samples used to compute the update
- copy_uniform_initialization(tensor: Tensor, reference_tensor: Tensor, fan_in: int) None[source]#
Initialize tensor with uniform law aligned on reference
Initialize the tensor with a uniform law with bounds -sqrt(std(W)), sqrt(std(W)) where std(W) is the empirical standard deviation of the reference_tensor if the reference_tensor has a non-zero variance. Otherwise, use bounds -1 / sqrt(fan_in), 1 / sqrt(fan_in) where fan_in is the number of input features of the extension.
- Parameters:
tensor (torch.Tensor) – tensor to initialize
reference_tensor (torch.Tensor) – tensor to get the standard deviation from
fan_in (int) – number of input features of the extension
- create_layer_extensions(extension_size: int, output_extension_size: int | None = None, input_extension_size: int | None = None, output_extension_init: str = 'copy_uniform', input_extension_init: str = 'copy_uniform') None[source]#
Create extension for layer input and output.
Create the layer input and output extensions of given sizes. Allow to have different sizes for input and output extensions, this is useful for example if you connect a convolutional layer to a linear layer.
- Parameters:
extension_size (int) – size of the extension to create
output_extension_size (int | None) – size of the output extension to create, if None use extension_size
input_extension_size (int | None) – size of the input extension to create, if None use extension_size
output_extension_init (str) – Initialization method for the output extension. Must be one of the keys in known_inits (e.g., “copy_uniform”, “zeros”). Default is “copy_uniform”.
input_extension_init (str) – Initialization method for the input extension. Must be one of the keys in known_inits (e.g., “copy_uniform”, “zeros”). Default is “copy_uniform”.
- create_layer_in_extension(extension_size: int) None[source]#
Create the layer input extension of given size.
- Parameters:
extension_size (int) – size of the extension to create
- create_layer_out_extension(extension_size: int) None[source]#
Create the layer output extension of given size.
- Parameters:
extension_size (int) – size of the extension to create
- delete_update(include_previous: bool = True, delete_delta: bool = True, delete_input: bool = True, delete_output: bool = False) None[source]#
Delete the updates of the layer: - optimal_delta_layer - extended_input_layer and associated extensions
By default, we do not delete the extended_output_layer of this layer because it could be required by the next layer.
- Parameters:
include_previous (bool, optional) – delete the extended_output_layer of the previous layer, by default True
delete_delta (bool, optional) – delete the optimal_delta_layer of the module, by default True
delete_input (bool, optional) – delete the extended_input_layer of this module, by default True
delete_output (bool, optional) – delete the extended_output_layer of this layer, by default False warning: this does not delete the extended_input_layer of the next layer
- Raises:
NotImplementedError – raised when include_previous is True and the previous module is of type MergeGrowingModule
TypeError – raised when the previous module is not of type GrowingModule or MergeGrowingModule
- extended_forward(x: Tensor, x_ext: Tensor | None = None, use_optimal_delta: bool = True, use_extended_input: bool = True, use_extended_output: bool = True) tuple[Tensor, Tensor | None][source]#
Forward pass of the module with layer extension and layer update scaled according to the scaling factor. WARNING: does not store the input and pre-activity tensors. WARNING: the scaling factor is squared for the optimal delta and linear for the extension. (Instead of linear for the optimal delta and squared for the extension as in the theory).
- Parameters:
x (torch.Tensor) – input tensor
x_ext (torch.Tensor | None) – extension tensor
use_optimal_delta (bool, optional) – if True, use the optimal delta layer, default True
use_extended_input (bool, optional) – if True, use the extended input layer, default True
use_extended_output (bool, optional) – if True, use the extended output layer, default True
- Returns:
output tensor and extension tensor
- Return type:
tuple[torch.Tensor, torch.Tensor]
- property first_order_improvement: Tensor#
Get the first order improvement of the block.
- Returns:
first order improvement
- Return type:
torch.Tensor
- forward(x)[source]#
Forward pass of the module. If needed, store the activity and pre-activity tensors.
- Parameters:
x (torch.Tensor) – input tensor
- Returns:
output tensor
- Return type:
torch.Tensor
- static get_fan_in_from_layer(layer: Module) int[source]#
Get the fan_in (number of input features) from a given layer.
- Parameters:
layer (torch.nn.Module) – layer to get the fan_in from
- Returns:
fan_in of the layer
- Return type:
- property input_extended: Tensor#
Return the input extended ones if the bias is used.
- Returns:
input extended
- Return type:
torch.Tensor
- layer_in_extension(weight: Tensor) None[source]#
Extend the layer with the parameters of layer assuming that the input of the layer is extended but not the output.
- Parameters:
weight (torch.Tensor) – weight of the extension
- layer_of_tensor(weight: Tensor, bias: Tensor | None = None, force_bias: bool = True) Module[source]#
- Create a layer with the same characteristics (excepted the shape)
with weight as parameter and bias as bias.
- Parameters:
weight (torch.Tensor) – weight of the layer
bias (torch.Tensor | None) – bias of the layer
force_bias (bool) – if True, the created layer require a bias if self.use_bias is True
- Returns:
layer with the same characteristics
- Return type:
torch.nn.Module
- layer_out_extension(weight: Tensor, bias: Tensor | None = None) None[source]#
Extend the layer with the parameters of layer assuming that the output of the layer is extended but not the input.
- Parameters:
weight (torch.Tensor) – weight of the extension
bias (torch.Tensor | None) – bias of the extension if needed
- missing_neurons() int[source]#
Get the number of missing neurons to reach the target hidden features.
- Returns:
number of missing neurons
- Return type:
- normalize_optimal_updates(std_target: float | None = None, normalization_type: str = 'legacy_normalization') None[source]#
Normalize optimal update to target standard deviation
Normalize the optimal updates so that the standard deviation of the weights of the updates is equal to std_target. If std_target is None, we automatically determine it. We use the standard deviation of the weights of the layer if it has weights. If the layer has no weights, we aim to have a std of 1 / sqrt(in_features).
If normalization_type is “equalize_second_layer”: Let s be the target standard deviation then: - optimal_delta_layer is scaled to have a std of s (so by s / std(optimal_delta_layer)) - extended_input_layer is scaled to have a std of s (so by s / std(extended_input_layer)) - extended_output_layer is scaled to match the scaling of the extended_input_layer and the optimal_delta_layer (so by std(extended_input_layer) / std(optimal_delta_layer))
If normalization_type is “equalize_extensions”: Let s be the target standard deviation then: - extended_input_layer is scaled to have a std of s (so by s / std(extended_input_layer)) - extended_output_layer is scaled to have a std of s (so by s / std(extended_output_layer)) - optimal_delta_layer is scaled to match the scaling of the extended_input_layer and the extended_output_layer (so by s ** 2 / (std(extended_input_layer) * std(extended_output_layer)))
- number_of_neurons_to_add(method: str = 'fixed_proportional', number_of_growth_steps: int = 1) int[source]#
Get the number of neurons to add in the next growth step.
- - fixed_proportional: add a fixed proportion of the total number of neurons
to add at each growth step. The amount to add is computed as an integer division as a consequence a few neurons may remain to be added after all growth steps have been performed.
- number_of_parameters() int[source]#
Return the number of parameters of the layer.
- Returns:
number of parameters
- Return type:
- parameter_step(delta_weights: Tensor, delta_biases: Tensor | None = None) None[source]#
Update the parameters of the layer with the given deltas.
- Parameters:
delta_weights (torch.Tensor) – delta values for the weights
delta_biases (torch.Tensor | None) – delta values for the biases, if None, the biases are not updated
- parameters(recurse: bool = True) Iterator[Parameter][source]#
Return the parameters of the layer.
- Parameters:
recurse (bool) – if True, return the parameters of the submodules
- Returns:
iterator over the parameters of the layer
- Return type:
Iterator[Parameter]
- projected_v_goal(input_vector: Tensor) Tensor[source]#
Compute the projected gradient of the goal with respect to the activity of the layer.
dLoss/dA_proj := dLoss/dA - dW B[-1] where A is the pre-activation vector of the layer, and dW the optimal delta for the layer
- Parameters:
input_vector (torch.Tensor of shape (n_samples, in_features)) – input vector B[-1]
- Returns:
projected gradient of the goal with respect to the activity of the next layer dLoss/dA - dW B[-1]
- Return type:
torch.Tensor
- static scale_layer(layer: Module, scale: float) Module[source]#
Scale the weights and biases of a given layer by a specified factor.
- Parameters:
layer (torch.nn.Module) – The layer whose parameters are to be scaled.
scale (float) – The factor by which to scale the layer’s parameters.
- Returns:
The layer with scaled parameters.
- Return type:
torch.nn.Module
- scale_layer_extension(scale: float | None, scale_output: float | None, scale_input: float | None) None[source]#
Scale the layer extension by a given factor. This means scaling the extended_input_layer, the extended_output_layer and the eigenvalues_extension. However as the eigenvalues_extension will be squared they will be scaled by sqrt(scale_input * scale_output).
- Parameters:
scale (float | None) – The factor by which to scale the layer extension. If not None, replace both scale_input and scale_output if they are not None.
scale_output (float | None) – The factor by which to scale the layer output extension.
scale_input (float | None) – The factor by which to scale the layer input extension. If not None, scale must be None.
- scale_parameter_update(scale: float) None[source]#
Scale the parameter update by a given factor. This means scaling the optimal delta and the parameter_update_decrease.
- Parameters:
scale (float) – The factor by which to scale the parameter update.
- set_scaling_factor(factor: float) None[source]#
Assign scaling factor to all growing layers
- Parameters:
factor (float) – scaling factor
- sub_select_optimal_added_parameters(keep_neurons: int | None = None, threshold: float | None = None, sub_select_previous: bool = True, zeros_if_not_enough: bool = False, zeros_fan_in: bool = True, zeros_fan_out: bool = False) None[source]#
Select the first keep_neurons neurons of the optimal added parameters linked to this layer.
- Parameters:
keep_neurons (int | None) – number of neurons to keep, if None, the number of neurons is determined by the threshold
threshold (float | None) – threshold to determine the number of neurons to keep, if None, keep_neurons must be provided
sub_select_previous (bool) – if True, sub-select the previous layer added parameters as well
zeros_if_not_enough (bool) – if True, will keep the all neurons and set the non selected ones to zero (either first or last depending on zeros_fan_in and zeros_fan_out)
zeros_fan_in (bool) – if True and zeros_if_not_enough is True, will set the non selected fan-in parameters to zero
zeros_fan_out (bool) – if True and zeros_if_not_enough is True, will set the non selected fan-out parameters to zero
- property tensor_n: Tensor#
Compute the tensor N for the layer with the current M_{-2}, C and optimal delta.
- Returns:
N
- Return type:
torch.Tensor
- property tensor_s: TensorStatistic#
Return the tensor S of the layer. Either the tensor S computed locally or the tensor S of the previous merge layer.
- Returns:
tensor S
- Return type:
TensorStatistic
- property tensor_s_growth#
Redirect to the tensor S of the previous module.
- update_input_size(input_size: tuple[int, ...] | None = None, compute_from_previous: bool = False, force_update: bool = True) tuple[int, ...] | None[source]#
Update the input size of the layer. Either according to the parameter or the input currently stored.
- Parameters:
- Returns:
updated input size if it could be computed, None otherwise
- Return type:
