mlm_insights.core.transformers.interfaces package

Submodules

mlm_insights.core.transformers.interfaces.transformer module

class mlm_insights.core.transformers.interfaces.transformer.Transformer

Bases: ABC

Abstract Base Class for defining an Insights Transformer.

Write a custom transformer to modify the input data frame. For eg: Adding a new column to the input data frame, deriving new columns from the existing data frame, encoding the columns, flattening the columns, or removing unused columns.

Multiple transformers can be chained to operate on the input data frame and produce a final data frame.

Derived class must implement the abstract methods: create, transform and get_output_schema

abstract classmethod create(config: Any) → Transformer

Factory Method to create a transformer. The configuration will be available in config.

Returns

Transformer

An Instance of Transformer.

get_output_schema(input_schema: Schema, **kwargs: Any) → Schema

Override this method if the transform method changes the schema of the input data frame for eg: add/remove columns. Users need familiarity with PyArrow Schema API. Refer here: https://arrow.apache.org/docs/python/generated/pyarrow.Schema.html

Suppose one is adding a column of type float to an input Dataframe. The function can look as below:

def get_output_schema(input_schema: pyarrow.Schema, **kwargs) -> pyarrow.Schema:
  return input_schema.append(pa.field('column_name', pyarrow.float()))

Parameters

input_schema: pa.Schema

Schema of the input data frame

Returns

output_schema: pa.Schema

Modified schema

abstract transform(data_frame: DataFrame, **kwargs: Any) → DataFrame

Transform the input data frame to produce a new data frame with the modifications. If this method changes the schema of the data_frame, you must override get_output_schema as well to return the modified schema

Parameters

data_frame: pd.DataFrame

Data Frame to operate on

Returns

output_data_frame: pd.DataFrame

Transformed Data Frame

Module contents