mlm_insights.core.metrics.classification_metrics package¶

Submodules¶

mlm_insights.core.metrics.classification_metrics.accuracy_score module¶

class mlm_insights.core.metrics.classification_metrics.accuracy_score.AccuracyScore(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, target_column: str = 'y_true', prediction_column: str = 'y_predict', actual_score: float = 0.0, total_count: int = 0)¶

Bases: DatasetMetricBase

This metric calculates the Accuracy score of a classification model. Accuracy is the ratio of the number of
correct predictions to the total number of input samples.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will
throw InvalidTargetPredictionException

Configuration¶

None

Returns¶

float: number of correctly classified samples
float: fraction of correctly classified samples

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.accuracy_score import AccuracyScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=AccuracyScore)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["AccuracyScore"])


if __name__ == "__main__":
    main()

Returns the standard metric result as:
{
    'metric_name': 'AccuracyScore',
    'metric_description': 'Computes Accuracy classification score',
    'variable_count': 2,
    'variable_names': ['actual_score', 'normalized_accuracy_score'],
    'variable_types': [CONTINUOUS, CONTINUOUS],
    'variable_dtypes': [FLOAT, FLOAT],
    'variable_dimensions': [0, 0],
    'metric_data': [4.0, 0.5714],
    'metadata': {},
    'error': None
}

actual_score: float = 0.0¶

compute(dataset: DataFrame, **kwargs: Any) → None¶: Computes accuracy score for the passed in dataset

Parameters¶

dataset : pd.DataFrame DataFrame object for either the entire dataset for a partition on which a Metric is being computed

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → AccuracyScore¶

Create an accuracy score metric using the configuration and kwargs

Parameters¶

config : Metric configuration kwargs: Key value pair for dynamic arguments. The current kwargs contains:

features_metadata: Contains input schema for each feature

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns Accuracy score Metric in Standard format.

Returns¶

StandardMetricResult: Accuracy score Metric in Standard format.

merge(other_metric: AccuracyScore, **kwargs: Any) → AccuracyScore¶

Merge two Accuracy Score into one, without mutating the others.

Parameters¶

other_metricAccuracy Score metric: Other Accuracy Score that need be merged.

Returns¶

TypeMetric: A new instance of Accuracy Score

prediction_column: str = 'y_predict'¶

target_column: str = 'y_true'¶

total_count: int = 0¶

mlm_insights.core.metrics.classification_metrics.common module¶

mlm_insights.core.metrics.classification_metrics.common.compute_average_micro(multi_class_cm: MultiClassConfusionMatrix) → float¶

mlm_insights.core.metrics.classification_metrics.common.compute_precision_score_macro(multi_class_cm: MultiClassConfusionMatrix) → float¶

mlm_insights.core.metrics.classification_metrics.common.compute_precision_score_weighted(multi_class_cm: MultiClassConfusionMatrix) → float¶

This method returns metric result for Precision, Recall, Fbeta metrics. These metrics have the same shape , hence the method has been refactored to a common location

Parameters¶

ovr_list : Score for each label versus the rest of the labels

labels: List of labels present in the model dataset

macro: Macro averaging output where metrics is calculated for each label and then return their unweighted mean

micro: Micro averaging output where metrics is computed globally by counting the total true positives, false negatives and false positives

weighted: Weighted averaging output where metrics is computed for each label, and then return the average weights

mlm_insights.core.metrics.classification_metrics.confusion_matrix module¶

class mlm_insights.core.metrics.classification_metrics.confusion_matrix.ConfusionMatrix(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>)¶

Bases: DatasetMetricBase

This metric calculates the Confusion Matrix for a classification model. A confusion matrix is a table that is
used to define the performance of a classification model.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

The result returned by this metric is a Confusion matrix whose i-th row and j-th column entry indicates the number of samples with true label being i-th class and predicted label being j-th class

Configuration¶

None

Returns¶

matrix: List[List[Integer]]

confusion matrix for the classification model

labels: List[String]

Labels or classes within the classification model

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.confusion_matrix import ConfusionMatrix
from mlm_insights.core.metrics.classification_metrics.recall_score import RecallScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=RecallScore),
                                                   MetricMetadata(klass=ConfusionMatrix)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["ConfusionMatrix"])


if __name__ == "__main__":
    main()

Returns the standard metric result as:
{
    'metric_name': 'ConfusionMatrix',
    'metric_description': 'Metric to compute confusion matrix to evaluate the accuracy of a classification.'
    'variable_count': 2,
    'variable_names': ['matrix', 'labels'],
    'variable_types': [DISCRETE, NOMINAL],
    'variable_dtypes': [INTEGER, STRING],
    'variable_dimensions': [2, 1],
    'metric_data': [[[2,2],[2,1]], ['ham', 'spam']],
    'metadata': {},
    'error': None
}

config: Dict[str, ConfigParameter]¶

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → ConfusionMatrix¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns Confusion Matrix Metric in Standard format.

Returns¶

StandardMetricResult: Confusion Matrix Metric in Standard format.

mlm_insights.core.metrics.classification_metrics.false_negative_rate module¶

class mlm_insights.core.metrics.classification_metrics.false_negative_rate.FalseNegativeRate(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)¶

Bases: DatasetMetricBase

This metric calculates the False Negative Rate of a classification model.
FalseNegativeRate is the ratio fn / (fn + tp) where fn is the number of false negatives and tp the
number of true positives. The false negative rate is the proportion of positive transactions that were incorrectly
scored as negative by the classifier.
The best value is 0 and the worst value is 1.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration¶

labels: List[Union[str, int]], default=None

When no labels are provided to the metric, the metric returns empty result.
When labels are provided to the metric, the metric returns one-versus-rest FNR for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.false_negative_rate import FalseNegativeRate
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=FalseNegativeRate)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["FalseNegativeRate"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'FalseNegativeRate',
    'metric_description': 'Metric computes the False Negative Rate.'                          ' FalseNegativeRate is the ratio ``fn / (fn + tp)``'                          ' where ``fn`` is the number of false negatives and ``tp`` the number of true positives.'                          ' The false negative rate is the proportion of positive transactions that'                          ' were incorrectly scored as negative by the classifier.'
    'variable_count': 2,
    'variable_names': ['ovr', 'labels'],
    'variable_types': [CONTINUOUS, NOMINAL],
    'variable_dtypes': [FLOAT, STRING],
    'variable_dimensions': [1, 1],
    'metric_data': [[], []],
    'metadata': {},
    'error': None
}

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → FalseNegativeRate¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns FalseNegativeRate Metric in Standard format.

Returns¶

StandardMetricResult: FalseNegativeRate Metric in Standard format.

labels: List[str | int] = None¶

merge(other_metric: FalseNegativeRate, **kwargs: Any) → FalseNegativeRate¶

Merge two FalseNegativeRate metric into one, without mutating the others.

Parameters¶

other_metricFalseNegativeRate: Other FalseNegativeRate metric that need be merged.

Returns¶

TypeMetric: A new instance of FalseNegativeRate

mlm_insights.core.metrics.classification_metrics.false_positive_rate module¶

class mlm_insights.core.metrics.classification_metrics.false_positive_rate.FalsePositiveRate(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)¶

Bases: DatasetMetricBase

This metric calculates the False Positive Rate of a classification model (Type 1 Error).
False Positive Rate is the ratio fp / (fp + tn) where fp is the number of false positives and tn the
number of true negatives. It indicates number of items wrongly classified as positive out of the total actual negatives
The best value is 0 and the worst value is 1.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration¶

labels: List[Union[str, int]], default=None

When no labels are provided to the metric, the metric returns empty result
When labels are provided to the metric, the metric returns one-versus-rest FPR for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.false_positive_rate import FalsePositiveRate
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=FalsePositiveRate)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["FalsePositiveRate"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'FalsePositiveRate',
    'metric_description': 'Metric computes the False Positive Rate (Type 1 Error).'                      ' False Positive Rate is the ratio ``fp / (fp + tn)`` where'                      ' ``fp`` is the number of false positives and ``tn`` the number of true negatives.'                      ' It indicates number of items wrongly classified as positive out of the total actual negatives'
    'variable_count': 2,
    'variable_names': ['ovr', 'labels'],
    'variable_types': [CONTINUOUS, NOMINAL],
    'variable_dtypes': [FLOAT, STRING],
    'variable_dimensions': [1, 1],
    'metric_data': [[], []],
    'metadata': {},
    'error': None
}

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → FalsePositiveRate¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns FalsePositiveRate Metric in Standard format.

Returns¶

StandardMetricResult: FalsePositiveRate Metric in Standard format.

labels: List[str | int] = None¶

merge(other_metric: FalsePositiveRate, **kwargs: Any) → FalsePositiveRate¶

Merge two FalsePositiveRate metric into one, without mutating the others.

Parameters¶

other_metricFalsePositiveRate: Other FalsePositiveRate metric that need be merged.

Returns¶

TypeMetric: A new instance of FalsePositiveRate

mlm_insights.core.metrics.classification_metrics.fbeta_score module¶

class mlm_insights.core.metrics.classification_metrics.fbeta_score.FBetaScore(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None, beta: float = 1)¶

Bases: DatasetMetricBase

This metric calculates the F-beta score of a classification model.
F-beta score is the weighted harmonic mean of precision and recall, reaching its optimal value at 1
and its worst value at 0.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

The beta parameter determines the weight of recall in the combined score.

beta = 1, recall and the precision are equally important
beta < 1 lends more weight to precision
beta > 1 favors recall

Configuration¶

labels: List[Union[str, int]], default=None

When no labels are provided to the metric, the metric returns micro, macro and weighted average scores
When labels are provided to the metric, the metric returns one-versus-rest fbeta score for each of

the provided labels

beta: float, default=1

The default value of beta is set to 1, thus by default this metric computes F1 score

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.fbeta_score import FBetaScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=FBetaScore)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["FBetaScore"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'FBetaScore',
    'metric_description': 'Metric computes the F-beta score.'                          ' F-beta score is the weighted harmonic mean of precision and recall,'                          ' reaching its optimal value at 1 and its worst value at 0.'                          ' The beta parameter determines the weight of recall in the combined score.'
    'variable_count': 6,
    'variable_names': ['macro', 'micro', 'weighted', 'ovr', 'labels', 'beta_value'],
    'variable_types': [1, 1, 1, 1, 6, 1],
    'variable_dtypes': [2, 2, 2, 2, 6, 2],
    'variable_dtypes': [FLOAT, FLOAT, FLOAT, FLOAT, STRING, FLOAT],
    'variable_types': [CONTINUOUS, CONTINUOUS, CONTINUOUS, CONTINUOUS, TEXT, CONTINUOUS],
    'variable_dimensions': [0, 0, 0, 1, 1, 0],
    'metric_data': [4.0, 0.5714, 3.25, None, None, 1],
    'metadata': {},
    'error': None
}

beta: float = 1¶

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → FBetaScore¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns FBetaScore Metric in Standard format.

Returns¶

StandardMetricResult: FBetaScore Metric in Standard format.

labels: List[str | int] = None¶

merge(other_metric: FBetaScore, **kwargs: Any) → FBetaScore¶

Merge two FBetaScore metric into one, without mutating the others.

Parameters¶

other_metricFBetaScore: Other FBetaScore metric that need be merged.

Returns¶

TypeMetric: A new instance of FBetaScore

mlm_insights.core.metrics.classification_metrics.log_loss module¶

class mlm_insights.core.metrics.classification_metrics.log_loss.LogLoss(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, target_column: str = 'y_true', prediction_score_column: str = 'y_score', actual_score: float = 0.0, total_count: int = 0, labels: ~typing.List[~typing.Any] | None = None)¶

Bases: DatasetMetricBase

This metric calculates the Log loss of a classification model.
This is the loss function used in (multinomial) logistic regression, defined as the negative log-likelihood
of a logistic model that returns y_pred probabilities for its training data y_true.
The log loss is only defined for two or more labels. The log loss is non-negative and value closer to Zero is
considered better.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.
Along with target and prediction columns, Log loss needs a prediction score column as well. Without the prediction
score column defined, the metric will throw an Exception.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will
throw InvalidTargetPredictionException

Configuration¶

labels: List[Union[str, int]], default=None

When no labels are provided to the metric, the metric assumes that data is of binary classification and 1 is considered as the positive label. If the target data is of multi-classification then loss and normalized loss is not calculated and None is returned.
When only 1 label is provided then the label is considered as positive label and data is assumed to be of binary classification.
When labels are provided to the metric, the labels are used to hot encode the prediction scores and the result is used to calculate the log loss. labels need to be provided in the same order as the values in prediction score column.
In case of any mismatch in labels and target data, loss and normalized loss is not calculated and None is returned.

Examples

from mlm_insights.core.metrics.classification_metrics.log_loss import LogLoss
classification_metrics = [
    MetricMetadata(klass=LogLoss)
]
metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

runner = InsightsBuilder().             with_input_schema(input_schema).             with_data_frame(data_frame=multi_class_data_frame).             with_metrics(metrics=metric_details).             with_engine(engine=EngineDetail(engine_name="native")).             build()

run_result = runner.run().profile.to_json()
dataset_metrics = profile_json['dataset_metrics']
dataset_metrics["LogLoss"]["value"]["loss"]

Returns the metric result as:
{
    'value': {
        'loss': log_loss,
        'normalized_loss' : normalized_log_loss,
    }
}

actual_score: float = 0.0¶

compute(dataset: DataFrame, **kwargs: Any) → None¶: Computes log loss for the passed in dataset

Parameters¶

dataset : pd.DataFrame DataFrame object for either the entire dataset for a partition on which a Metric is being computed

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → LogLoss¶

Create a log loss metric using the configuration and kwargs

Parameters¶

config : Metric configuration kwargs: Key value pair for dynamic arguments. The current kwargs contains:

features_metadata: Contains input schema for each feature

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns log loss Score Metric in Standard format.

Returns¶

StandardMetricResult: log loss Score Metric in Standard format.

labels: List[Any] = None¶

merge(other_metric: LogLoss, **kwargs: Any) → LogLoss¶

Merge two log loss Score into one, without mutating the others.

Parameters¶

other_metricLog Loss metric: Other Log Loss metric that need be merged.

Returns¶

TypeMetric: A new instance of Log Loss Metric

prediction_score_column: str = 'y_score'¶

target_column: str = 'y_true'¶

total_count: int = 0¶

mlm_insights.core.metrics.classification_metrics.precision_recall_auc module¶

class mlm_insights.core.metrics.classification_metrics.precision_recall_auc.PrecisionRecallAreaUnderCurve(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, positive_label: str | int = 1)¶

Bases: DatasetMetricBase

This metric calculates the Area Under Curve for the Precision-Recall Curve.
A precision-recall curve is a graphical plot which illustrates the performance of a binary classifier system as
its discrimination threshold is varied.
It is created by plotting precision (precision = tp / (tp + fp)) vs. recall (recall = tp / (tp + fn)),
at various threshold settings. Once the precision-recall pairs are computed for different threshold values, the
Area Under Curve is computed.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an approximate metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will
throw InvalidTargetPredictionException

Part of the implementation is taken from sklearn: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py

Configuration¶

positive_label: int or str , default=1

The label of the positive class.

Examples

import pandas
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.precision_recall_auc import PrecisionRecallAreaUnderCurve
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        "y_true": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "y_score": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)
    model = LogisticRegression(solver='lbfgs')
    model.fit(x_train, y_train)
    y_score = model.predict_proba(x_test)[:, 1]
    df = pandas.DataFrame({'y_true': y_test, 'y_score': y_score})

    classification_metrics = [
        MetricMetadata(klass=PrecisionRecallAreaUnderCurve, config={"positive_label": 1})
    ]
    metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

    runner = InsightsBuilder().                 with_input_schema(input_schema).                 with_data_frame(data_frame=df).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    run_result = runner.run().profile.to_json()
    dataset_metrics = run_result['dataset_metrics']
    print(dataset_metrics["PrecisionRecallAreaUnderCurve"])


if __name__ == "__main__":
    main()


Returns the metric result as:
  return {
        "value": auc
    }

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → PrecisionRecallAreaUnderCurve¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns PrecisionRecallAreaUnderCurve Metric in Standard format.

Returns¶

StandardMetricResult: PrecisionRecallAreaUnderCurve Metric in Standard format.

merge(other_metric: PrecisionRecallAreaUnderCurve, **kwargs: Any) → PrecisionRecallAreaUnderCurve¶

Merge two PrecisionRecallAreaUnderCurve metric into one, without mutating the others.

Parameters¶

other_metricPrecisionRecallAreaUnderCurve: Other PrecisionScore metric that need be merged.

Returns¶

PrecisionRecallAreaUnderCurve: A new instance of PrecisionRecallAreaUnderCurve

positive_label: str | int = 1¶

mlm_insights.core.metrics.classification_metrics.precision_recall_curve module¶

class mlm_insights.core.metrics.classification_metrics.precision_recall_curve.PrecisionRecallCurve(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, positive_label: str | int = 1)¶

Bases: DatasetMetricBase

This metric computes the Precision-Recall Curve, by computing precision-recall pairs for different probability
thresholds.
A precision-recall curve is a graphical plot which illustrates the performance of a binary classifier system as
its discrimination threshold is varied.
It is created by plotting precision (precision = tp / (tp + fp)) vs. recall (recall = tp / (tp + fn)),
at various threshold settings.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an approximate metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will
throw InvalidTargetPredictionException

Configuration¶

positive_label: int or str , default=1

The label of the positive class.

Examples

import pandas
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.precision_recall_curve import PrecisionRecallCurve
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        "y_true": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "y_score": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)
    model = LogisticRegression(solver='lbfgs')
    model.fit(x_train, y_train)
    y_score = model.predict_proba(x_test)[:, 1]
    df = pandas.DataFrame({'y_true': y_test, 'y_score': y_score})

    classification_metrics = [
        MetricMetadata(klass=PrecisionRecallCurve, config={"positive_label": 1})
    ]
    metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

    runner = InsightsBuilder().                 with_input_schema(input_schema).                 with_data_frame(data_frame=df).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    run_result = runner.run().profile.to_json()
    dataset_metrics = run_result['dataset_metrics']
    print(dataset_metrics["PrecisionRecallCurve"])

if __name__ == "__main__":
    main()

Returns the metric result as:
{
    "value": {
        "precision": precision,
        "recall": recall,
        "thresholds": thresholds
    }
}

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → PrecisionRecallCurve¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns PrecisionRecallCurve Metric in Standard format.

Returns¶

StandardMetricResult: PrecisionRecallCurve Metric in Standard format.

merge(other_metric: PrecisionRecallCurve, **kwargs: Any) → PrecisionRecallCurve¶

Merge two PrecisionRecallCurve metric into one, without mutating the others.

Parameters¶

other_metricPrecisionRecallCurve: Other PrecisionScore metric that need be merged.

Returns¶

PrecisionRecallCurve: A new instance of PrecisionRecallCurve

positive_label: str | int = 1¶

mlm_insights.core.metrics.classification_metrics.precision_score module¶

class mlm_insights.core.metrics.classification_metrics.precision_score.PrecisionScore(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)¶

Bases: DatasetMetricBase

This metric calculates the Precision score of a classification model.
The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number
of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample
that is negative.
The best value is 1 and the worst value is 0.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration¶

labels: List[Union[str, int]], default=None

When no labels are provided to the metric, the metric returns micro, macro and weighted average scores
When labels are provided to the metric, the metric returns one-versus-rest precision score for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.precision_score import PrecisionScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=PrecisionScore)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["PrecisionScore"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'PrecisionScore',
    'metric_description': 'This metric computes the Precision.The precision is the ratio ``tp / (tp + fp)``'                          ' where ``tp`` is the number of true positives and ``fp`` the number of false positives.'                          ' The precision is intuitively the ability of the classifier not to label as positive'                          ' a sample that is negative.'                          'The best value is 1 and the worst value is 0.',
    'variable_count': 5,
    'variable_names': ['macro', 'micro', 'weighted', 'ovr', 'labels'],
    'variable_dtypes': [FLOAT, FLOAT, FLOAT, FLOAT, STRING],
    'variable_types': [CONTINUOUS, CONTINUOUS, CONTINUOUS, CONTINUOUS, NOMINAL],
    'variable_dimensions': [0, 0, 0, 1, 1],
    'metric_data': [4.0, 0.5714],
    'metadata': {},
    'error': None
}

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → PrecisionScore¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns PrecisionScore Metric in Standard format.

Returns¶

StandardMetricResult: PrecisionScore Metric in Standard format.

labels: List[str | int] = None¶

merge(other_metric: PrecisionScore, **kwargs: Any) → PrecisionScore¶

Merge two PrecisionScore metric into one, without mutating the others.

Parameters¶

other_metricPrecisionScore: Other PrecisionScore metric that need be merged.

Returns¶

TypeMetric: A new instance of PrecisionScore

mlm_insights.core.metrics.classification_metrics.recall_score module¶

class mlm_insights.core.metrics.classification_metrics.recall_score.RecallScore(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)¶

Bases: DatasetMetricBase

This metric computes the recall score of a classification model.
Recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives.
The recall is intuitively the ability of the classifier to find all the positive samples.
The best value is 1 and the worst value is 0.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration¶

labels: List[Union[str, int]], default=None

When no labels are provided to the metric, the metric returns micro, macro and weighted average scores
When labels are provided to the metric, the metric returns one-versus-rest recall score for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.recall_score import RecallScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        'spam_or_ham_prediction': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        'spam_or_ham_target': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=RecallScore)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["RecallScore"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'RecallScore',
    'metric_description': 'Metric computes the recall.Recall is the ratio tp / (tp + fn)'                          ' where tp is the number of true positives and fn the number of false negatives.'                          'The recall is intuitively the ability of the classifier to find all the positive samples',
    'variable_count': 5,
    'variable_names': ['macro', 'micro', 'weighted', 'ovr', 'labels'],
    'variable_dtypes': [FLOAT, FLOAT, FLOAT, FLOAT, STRING],
    'variable_types': [CONTINUOUS, CONTINUOUS, CONTINUOUS,
                        CONTINUOUS, NOMINAL],
    'variable_dimensions': [0, 0, 0, 1, 1],
    'metric_data': [4.0, 0.5714],
    'metadata': {},
    'error': None
}

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → RecallScore¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns RecallScore Metric in Standard format.

Returns¶

StandardMetricResult: RecallScore Metric in Standard format.

labels: List[str | int] = None¶

merge(other_metric: RecallScore, **kwargs: Any) → RecallScore¶

Merge two RecallScore metric into one, without mutating the others.

Parameters¶

other_metricRecallScore: Other RecallScore metric that need be merged.

Returns¶

TypeMetric: A new instance of RecallScore

mlm_insights.core.metrics.classification_metrics.roc module¶

class mlm_insights.core.metrics.classification_metrics.roc.ROCCurve(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, positive_label: str | int = 1)¶

Bases: DatasetMetricBase

This metric computes the ROC Curve (receiver operating characteristic curve).
A receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the
performance of a binary classifier system as its discrimination threshold is varied.
It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate)
vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold
settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an approximate metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will
throw InvalidTargetPredictionException

Configuration¶

positive_label: int or str , default=1

The label of the positive class.

Examples

import pandas
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.roc import ROCCurve
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        "y_true": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "y_score": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)
    model = LogisticRegression(solver='lbfgs')
    model.fit(x_train, y_train)
    y_score = model.predict_proba(x_test)[:, 1]
    df = pandas.DataFrame({'y_true': y_test, 'y_score': y_score})

    classification_metrics = [
        MetricMetadata(klass=ROCCurve)
    ]
    metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

    runner = InsightsBuilder().                 with_input_schema(input_schema).                 with_data_frame(data_frame=df).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    run_result = runner.run().profile.to_json()
    dataset_metrics = run_result['dataset_metrics']
    print(dataset_metrics["ROCCurve"])

if __name__ == "__main__":
    main()

Returns the metric result as:
{
    "value": {
        "fpr": fpr,
        "tpr": tpr,
        "thresholds": thresholds
    }
}

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → ROCCurve¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns ROC Curve Metric in Standard format.

Returns¶

StandardMetricResult: ROC Curve Metric in Standard format.

merge(other_metric: ROCCurve, **kwargs: Any) → ROCCurve¶

Merge two ROCCurve metric into one, without mutating the others.

Parameters¶

other_metricROCCurve: Other ROCCurve metric that need be merged.

Returns¶

TypeMetric: A new instance of ROCCurve

positive_label: str | int = 1¶

mlm_insights.core.metrics.classification_metrics.roc_auc module¶

class mlm_insights.core.metrics.classification_metrics.roc_auc.ROCAreaUnderCurve(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, positive_label: str | int = 1)¶

Bases: DatasetMetricBase

This metric computes the ROC Area under Curve
A receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the
performance of a binary classifier system as its discrimination threshold is varied.
It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate)
vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold
settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an approximate metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will
throw InvalidTargetPredictionException

Configuration¶

positive_label: int or str, default=None

The label of the positive class. When pos_label=1,

Examples

import pandas
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.roc_auc import ROCAreaUnderCurve
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        "y_true": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "y_score": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)
    model = LogisticRegression(solver='lbfgs')
    model.fit(x_train, y_train)
    y_score = model.predict_proba(x_test)[:, 1]
    df = pandas.DataFrame({'y_true': y_test, 'y_score': y_score})

    classification_metrics = [
        MetricMetadata(klass=ROCAreaUnderCurve, config={"positive_label": 1})
    ]
    metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

    runner = InsightsBuilder().                 with_input_schema(input_schema).                 with_data_frame(data_frame=df).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    run_result = runner.run().profile.to_json()
    dataset_metrics = run_result['dataset_metrics']
    print(dataset_metrics["ROCAreaUnderCurve"])


if __name__ == "__main__":
    main()


Returns the metric result as:
{
    "value": auc
}

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → ROCAreaUnderCurve¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns ROC Area Under Curve Metric in Standard format.

Returns¶

StandardMetricResult: ROC Area Under Curve Metric in Standard format.

merge(other_metric: ROCAreaUnderCurve, **kwargs: Any) → ROCAreaUnderCurve¶

Merge two ROCAreaUnderCurve metric into one, without mutating the others.

Parameters¶

other_metricROCAreaUnderCurve: Other ROCAreaUnderCurve metric that need be merged.

Returns¶

TypeMetric: A new instance of ROCAreaUnderCurve

positive_label: str | int = 1¶

mlm_insights.core.metrics.classification_metrics.specificity module¶

class mlm_insights.core.metrics.classification_metrics.specificity.Specificity(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)¶

Bases: DatasetMetricBase

This metric computes the Specificity or True Negative Rate of a classification model.
Specificity is the ratio tn / (fp + tn) where tn is the number of true negatives and fp the number
of false positives.
The specificity is intuitively the ability of the classifier to find all the negative samples.
The best value is 1 and the worst value is 0.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration¶

labels: List[Union[str, int]], default=None

When no labels are provided to the metric, the metric returns empty result
When labels are provided to the metric, the metric returns one-versus-rest specificity for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.false_positive_rate import FalsePositiveRate
from mlm_insights.core.metrics.metric_metadata import MetricMetadata

def main():
    input_schema = {
            'email_subject': FeatureType(
                data_type=DataType.STRING,
                variable_type=VariableType.TEXT,
                column_type=ColumnType.INPUT),
            "spam_or_ham_prediction": FeatureType(
                data_type=DataType.STRING,
                variable_type=VariableType.NOMINAL,
                column_type=ColumnType.PREDICTION),
            "spam_or_ham_target": FeatureType(
                data_type=DataType.STRING,
                variable_type=VariableType.NOMINAL,
                column_type=ColumnType.TARGET)
        }
    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                                'spam_or_ham_prediction': y_pred_binary,
                                                'spam_or_ham_target': y_true_binary
                                                })
    metric_details = MetricDetail(univariate_metric={},
                                      dataset_metrics=[MetricMetadata(klass=FalsePositiveRate)])

    runner = InsightsBuilder().                     with_input_schema(input_schema=input_schema).                     with_data_frame(data_frame=binary_class_data_frame).                     with_metrics(metrics=metric_details).                     with_engine(engine=EngineDetail(engine_name="native")).                     build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["FalsePositiveRate"])

if __name__ == "__main__":
        main()


Returns the standard metric result as:
{
    'metric_name': 'Specificity',
    'metric_description': 'Metric computes the Specificity or True Negative Rate.'                      ' Specificity is the ratio ``tn / (fp + tn)`` where'                      ' ``tn`` is the number of true negatives and ``fp`` the number of false positives.'                      ' The specificity is intuitively the ability of the classifier to find all the negative samples.'
    'variable_count': 2,
    'variable_names': ['ovr', 'labels'],
    'variable_dtypes': [FLOAT, STRING],
    'variable_types': [CONTINUOUS, NOMINAL],
    'variable_dimensions': [1, 1],
    'metric_data': [[], []],
    'metadata': {},
    'error': None
}

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) → Specificity¶

Factory Method to create an object. The configuration will be available in config.

Returns¶

MetricBase: An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) → List[SDCMetaData]¶: Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns¶

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) → Dict[str, Any]¶: Returns the computed value of the metric

Returns¶

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) → StandardMetricResult¶: Returns Specificity Metric in Standard format.

Returns¶

StandardMetricResult: Specificity Metric in Standard format.

labels: List[str | int] = None¶

merge(other_metric: Specificity, **kwargs: Any) → Specificity¶

Merge two Specificity metric into one, without mutating the others.

Parameters¶

other_metricSpecificity: Other Specificity metric that need be merged.

Returns¶

TypeMetric: A new instance of Specificity