attribench.distributed.metrics.SensitivityN

class attribench.distributed.metrics.SensitivityN(model_factory, attributions_dataset, batch_size, maskers, activation_fns, min_subset_size, max_subset_size, num_steps, num_subsets, segmented=False, address='localhost', port='12355', devices=None)

Bases: Metric

Compute the Sensitivity-n metric for a given AttributionsDataset and model using multiple processes.

Sensitivity-n is computed by iteratively masking a random subset of n features of the input samples and computing the output of the model on the masked samples.

For each random subset of masked features, the sum of the attributions is also computed. This results in two series of values: the model output and the sum of the attributions. The Sensitivity-n metric is the correlation between these two series.

This is repeated for different values of n between min_subset_size and max_subset_size in num_steps steps. min_subset_size and max_subset_size are percentages of the total number of features. For each value of n, num_subsets random subsets are generated.

If segmented is True, then the Seg-Sensitivity-n metric is computed. This metric is analogous to Sensitivity-n, but instead of using random subsets of features, the images are first segmented into superpixels and then random subsets of superpixels are masked. This improves the signal-to-noise ratio of the metric for high-resolution images.

The Sensitivity-n metric is computed for each masker in maskers and for each activation function in activation_fns. The number of processes is determined by the number of devices. If devices is None, then all available devices are used. Samples are distributed evenly across the processes.

Parameters:

model_factoryModelFactory

ModelFactory instance or callable that returns a model. Used to create a model for each subprocess.

attributions_datasetAttributionsDataset

Dataset containing the attributions to compute Sensitivity-n on.

batch_sizeint

Batch size to use when computing the model output.

maskersDict[str, Masker]

Dictionary of maskers to use. Keys are the names of the maskers.

activation_fnsUnion[Tuple[str], str]

Activation functions to use. If a single string is passed, then the it is converted to a single-element list.

min_subset_sizefloat

Minimum percentage of features to mask.

max_subset_sizefloat

Maximum percentage of features to mask.

num_stepsint

Number of steps between min_subset_size and max_subset_size.

num_subsetsint

Number of random subsets to generate for each value of n.

segmentedbool

If True, then the Seg-Sensitivity-n metric is computed.

addressstr, optional

Address to use for the distributed computation. Defaults to “localhost”.

portstr | int, optional

Port to use for the distributed computation. Defaults to “12355”.

devicesTuple | None

Devices to use for the distributed computation. If None, then all available devices are used.

Methods

run	Runs the metric computation and optionally saves the result.
save_result	Save the result to disk.

Attributes

result

run(result_path=None, progress=True)

Runs the metric computation and optionally saves the result. If no result path is given, the result will not be saved to disk. It can still be accessed via the result property.

Parameters:

result_pathstr, optional

Path to save the result to. If None, the result is not saved to disk.

progressbool, optional

Whether to show a progress bar. Defaults to True.

save_result(path, format='hdf5')

Save the result to disk.

Parameters:

pathstr

Path to save the result to.

formatstr, optional

Format to save the result in. If "hdf5", the result is saved as an HDF5 file. If "csv", the result is saved as a directory structure containing CSV files. Default: "hdf5".

Raises:

ValueError

If the result is None.