Metrics#
Here we describe the available metrics in Argilla:
Common metrics: Metrics available for all datasets
Text classification: Metrics for text classification
Token classification: Metrics for token classification
Common metrics#
- argilla.metrics.commons.keywords(name: str, query: Optional[str] = None, size: int = 20) MetricSummary #
Computes the keywords occurrence distribution in dataset
- Parameters:
name โ The dataset name.
query โ An ElasticSearch query with the [query string syntax]( https://argilla.readthedocs.io/en/stable/guides/queries.html)
size โ The number of kewords to retrieve. Default to 20
- Returns:
The dataset keywords occurrence distribution
Examples
>>> from argilla.metrics.commons import keywords >>> summary = keywords(name="example-dataset") >>> summary.visualize() # will plot an histogram with results >>> summary.data # returns the raw result data
- argilla.metrics.commons.records_status(name: str, query: Optional[str] = None) MetricSummary #
Computes the records status distribution for a dataset
- Parameters:
name โ The dataset name.
query โ An ElasticSearch query with the [query string syntax](https://argilla.readthedocs.io/en/stable/guides/queries.html)
- Returns:
The status distribution metric summary
Examples
>>> from argilla.metrics.commons import records_status >>> summary = records_status(name="example-dataset") >>> summary.visualize() # will plot an histogram with results >>> summary.data # returns the raw result data
- argilla.metrics.commons.text_length(name: str, query: Optional[str] = None) MetricSummary #
Computes the input text length metrics for a dataset
- Parameters:
name โ The dataset name.
query โ An ElasticSearch query with the [query string syntax](https://argilla.readthedocs.io/en/stable/guides/queries.html)
- Returns:
The text length metric summary
Examples
>>> from argilla.metrics.commons import text_length >>> summary = text_length(name="example-dataset") >>> summary.visualize() # will plot an histogram with results >>> summary.data # returns the raw result data
Text classification#
- argilla.metrics.text_classification.metrics.f1(name: str, query: Optional[str] = None) MetricSummary #
Computes the single label f1 metric for a dataset
- Parameters:
name โ The dataset name.
query โ An ElasticSearch query with the [query string syntax](https://argilla.readthedocs.io/en/stable/guides/queries.html)
- Returns:
The f1 metric summary
Examples
>>> from argilla.metrics.text_classification import f1 >>> summary = f1(name="example-dataset") >>> summary.visualize() # will plot a bar chart with results >>> summary.data # returns the raw result data
- argilla.metrics.text_classification.metrics.f1_multilabel(name: str, query: Optional[str] = None) MetricSummary #
Computes the multi-label label f1 metric for a dataset
- Parameters:
name โ The dataset name.
query โ An ElasticSearch query with the [query string syntax](https://argilla.readthedocs.io/en/stable/guides/queries.html)
- Returns:
The f1 metric summary
Examples
>>> from argilla.metrics.text_classification import f1_multilabel >>> summary = f1_multilabel(name="example-dataset") >>> summary.visualize() # will plot a bar chart with results >>> summary.data # returns the raw result data
Token classification#
- class argilla.metrics.token_classification.metrics.ComputeFor(value)#
An enumeration.
- argilla.metrics.token_classification.metrics.entity_capitalness(name: str, query: Optional[str] = None, compute_for: Union[str, ComputeFor] = ComputeFor.PREDICTIONS) MetricSummary #
Computes the entity capitalness. The entity capitalness splits the entity mention shape in 4 groups:
UPPER
: All characters in entity mention are upper case.LOWER
: All characters in entity mention are lower case.FIRST
: The first character in the mention is upper case.MIDDLE
: First character in the mention is lower case and at least one other character is upper case.- Parameters:
name โ The dataset name.
query โ An ElasticSearch query with the query string syntax
compute_for โ Metric can be computed for annotations or predictions. Accepted values are
Annotations
andPredictions
. Default toPredictions
.
- Returns:
The summary entity capitalness distribution
Examples
>>> from argilla.metrics.token_classification import entity_capitalness >>> summary = entity_capitalness(name="example-dataset") >>> summary.visualize()
- argilla.metrics.token_classification.metrics.entity_density(name: str, query: Optional[str] = None, compute_for: Union[str, ComputeFor] = ComputeFor.PREDICTIONS, interval: float = 0.005) MetricSummary #
Computes the entity density distribution. Then entity density is calculated at record level for each mention as
mention_length/tokens_length
- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
compute_for โ Metric can be computed for annotations or predictions. Accepted values are
Annotations
andPredictions
. Default toPredictions
.interval โ The interval for histogram. The entity density is defined in the range 0-1.
- Returns:
The summary entity density distribution
Examples
>>> from argilla.metrics.token_classification import entity_density >>> summary = entity_density(name="example-dataset") >>> summary.visualize()
- argilla.metrics.token_classification.metrics.entity_labels(name: str, query: Optional[str] = None, compute_for: Union[str, ComputeFor] = ComputeFor.PREDICTIONS, labels: int = 50) MetricSummary #
Computes the entity labels distribution
- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
compute_for โ Metric can be computed for annotations or predictions. Accepted values are
Annotations
andPredictions
. Default toPredictions
labels โ The number of top entities to retrieve. Lower numbers will be better performants
- Returns:
The summary for entity tags distribution
Examples
>>> from argilla.metrics.token_classification import entity_labels >>> summary = entity_labels(name="example-dataset", labels=20) >>> summary.visualize() # will plot a bar chart with results >>> summary.data # The top-20 entity tags
- argilla.metrics.token_classification.metrics.f1(name: str, query: Optional[str] = None) MetricSummary #
Computes F1 metrics for a dataset based on entity-level.
- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
- Returns:
The F1 metric summary containing precision, recall and the F1 score (averaged and per label).
Examples
>>> from argilla.metrics.token_classification import f1 >>> summary = f1(name="example-dataset") >>> summary.visualize() # will plot three bar charts with the results >>> summary.data # returns the raw result data
To display the results as a table:
>>> import pandas as pd >>> pd.DataFrame(summary.data.values(), index=summary.data.keys())
- argilla.metrics.token_classification.metrics.mention_length(name: str, query: Optional[str] = None, level: str = 'token', compute_for: Union[str, ComputeFor] = ComputeFor.PREDICTIONS, interval: int = 1) MetricSummary #
Computes mentions length distribution (in number of tokens).
- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
level โ The mention length level. Accepted values are โtokenโ and โcharโ
compute_for โ Metric can be computed for annotations or predictions. Accepted values are
Annotations
andPredictions
. Defaults toPredictions
.interval โ The bins or bucket for result histogram
- Returns:
The summary for mention token distribution
Examples
>>> from argilla.metrics.token_classification import mention_length >>> summary = mention_length(name="example-dataset", interval=2) >>> summary.visualize() # will plot a histogram chart with results >>> summary.data # the raw histogram data with bins of size 2
- argilla.metrics.token_classification.metrics.token_capitalness(name: str, query: Optional[str] = None) MetricSummary #
Computes the token capitalness distribution
UPPER
: All characters in the token are upper case.LOWER
: All characters in the token are lower case.FIRST
: The first character in the token is upper case.MIDDLE
: First character in the token is lower case and at least one other character is upper case.- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
- Returns:
The summary for token length distribution
Examples
>>> from argilla.metrics.token_classification import token_capitalness >>> summary = token_capitalness(name="example-dataset") >>> summary.visualize() # will plot a histogram with results >>> summary.data # The token capitalness distribution
- argilla.metrics.token_classification.metrics.token_frequency(name: str, query: Optional[str] = None, tokens: int = 1000) MetricSummary #
Computes the token frequency distribution for a numbe of tokens.
- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
tokens โ The top-k number of tokens to retrieve
- Returns:
The summary for token frequency distribution
Examples
>>> from argilla.metrics.token_classification import token_frequency >>> summary = token_frequency(name="example-dataset", token=50) >>> summary.visualize() # will plot a histogram with results >>> summary.data # the top-50 tokens frequency
- argilla.metrics.token_classification.metrics.token_length(name: str, query: Optional[str] = None) MetricSummary #
Computes the token size distribution in terms of number of characters
- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
- Returns:
The summary for token length distribution
Examples
>>> from argilla.metrics.token_classification import token_length >>> summary = token_length(name="example-dataset") >>> summary.visualize() # will plot a histogram with results >>> summary.data # The token length distribution
- argilla.metrics.token_classification.metrics.tokens_length(name: str, query: Optional[str] = None, interval: int = 1) MetricSummary #
Computes the text length distribution measured in number of tokens.
- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
interval โ The bins or bucket for result histogram
- Returns:
The summary for token distribution
Examples
>>> from argilla.metrics.token_classification import tokens_length >>> summary = tokens_length(name="example-dataset", interval=5) >>> summary.visualize() # will plot a histogram with results >>> summary.data # the raw histogram data with bins of size 5
- argilla.metrics.token_classification.metrics.top_k_mentions(name: str, query: Optional[str] = None, compute_for: Union[str, ComputeFor] = ComputeFor.PREDICTIONS, k: int = 100, threshold: int = 2, post_label_filter: Optional[Set[str]] = None)#
Computes the consistency for top k mentions in the dataset.
Entity consistency defines the label variability for a given mention. For example, a mention first identified in the whole dataset as Cardinal, Person and Time is less consistent than a mention Peter identified as Person in the dataset.
- Parameters:
name โ The dataset name.
query โ
An ElasticSearch query with the query string syntax
compute_for โ Metric can be computed for annotations or predictions. Accepted values are
Annotations
andPredictions
. Default toPredictions
k โ The number of mentions to retrieve.
threshold โ The entity variability threshold (must be greater or equal to 1).
post_label_filter โ A set of labels used for filtering the results. This filter may affect to the expected
mentions (number of) โ
- Returns:
The summary top k mentions distribution
Examples
>>> from argilla.metrics.token_classification import top_k_mentions >>> summary = top_k_mentions(name="example-dataset") >>> summary.visualize()