Client#

Here we describe the Python client of Argilla that we divide into four basic modules:

  • Methods: These methods make up the interface to interact with Argilla’s REST API.

  • Records: You need to wrap your data in these Records for Argilla to understand it.

  • Datasets: Datasets: You can wrap your records around these Datasets for extra functionality.

  • FeedbackDataset: FeedbackDataset: the dataset format for FeedbackTask and LLM support.

Methods#

argilla.active_client() → Argilla#

Returns the active argilla client.

If Active client is None, initialize a default one.

argilla.copy(dataset: str, name_of_copy: str, workspace: Optional[str] = None) → None#

Creates a copy of a dataset including its tags and metadata

Parameters:

  • dataset – Name of the source dataset

  • name_of_copy – Name of the copied dataset

  • workspace – If provided, dataset will be copied to that workspace

Examples

>>> import argilla as rg
>>> rg.copy("my_dataset", name_of_copy="new_dataset")
>>> rg.load("new_dataset")
argilla.delete(name: str, workspace: Optional[str] = None) → None#

Deletes a dataset.

Parameters:

  • name – The dataset name.

  • workspace – The workspace to which records will be logged/loaded. If None (default) and the env variable ARGILLA_WORKSPACE is not set, it will default to the private user workspace.

Examples

>>> import argilla as rg
>>> rg.delete(name="example-dataset")
argilla.delete_records(name: str, workspace: Optional[str] = None, query: Optional[str] = None, ids: Optional[List[Union[str, int]]] = None, discard_only: bool = False, discard_when_forbidden: bool = True) → Tuple[int, int]#

Delete records from a argilla dataset.

Parameters:

  • name – The dataset name.

  • workspace – The workspace to which records will be logged/loaded. If None (default) and the env variable ARGILLA_WORKSPACE is not set, it will default to the private user workspace.

  • query – An ElasticSearch query with the query string syntax

  • ids – If provided, deletes dataset records with given ids.

  • discard_only – If True, matched records won’t be deleted. Instead, they will be marked as Discarded

  • discard_when_forbidden – Only super-user or dataset creator can delete records from a dataset. So, running “hard” deletion for other users will raise an ForbiddenApiError error. If this parameter is True, the client API will automatically try to mark as Discarded records instead. Default, True

Returns:

The total of matched records and real number of processed errors. These numbers could not be the same if some data conflicts are found during operations (some matched records change during deletion).

Examples

>>> ## Delete by id
>>> import argilla as rg
>>> rg.delete_records(name="example-dataset", ids=[1,3,5])
>>> ## Discard records by query
>>> import argilla as rg
>>> rg.delete_records(name="example-dataset", query="metadata.code=33", discard_only=True)
argilla.get_workspace() → str#

Returns the name of the active workspace.

Returns:

The name of the active workspace as a string.

argilla.init(api_url: Optional[str] = None, api_key: Optional[str] = None, workspace: Optional[str] = None, timeout: int = 60, extra_headers: Optional[Dict[str, str]] = None) → None#

Init the Python client.

We will automatically init a default client for you when calling other client methods. The arguments provided here will overwrite your corresponding environment variables.

Parameters:

  • api_url – Address of the REST API. If None (default) and the env variable ARGILLA_API_URL is not set, it will default to http://localhost:6900.

  • api_key – Authentification key for the REST API. If None (default) and the env variable ARGILLA_API_KEY is not set, it will default to argilla.apikey.

  • workspace – The workspace to which records will be logged/loaded. If None (default) and the env variable ARGILLA_WORKSPACE is not set, it will default to the private user workspace.

  • timeout – Wait timeout seconds for the connection to timeout. Default: 60.

  • extra_headers – Extra HTTP headers sent to the server. You can use this to customize the headers of argilla client requests, like additional security restrictions. Default: None.

Examples

>>> import argilla as rg
>>>
>>> rg.init(api_url="http://localhost:9090", api_key="4AkeAPIk3Y")
>>> # Customizing request headers
>>> headers = {"X-Client-id":"id","X-Secret":"secret"}
>>> rg.init(api_url="http://localhost:9090", api_key="4AkeAPIk3Y", extra_headers=headers)
argilla.load(name: str, workspace: Optional[str] = None, query: Optional[str] = None, vector: Optional[Tuple[str, List[float]]] = None, ids: Optional[List[Union[str, int]]] = None, limit: Optional[int] = None, sort: Optional[List[Tuple[str, str]]] = None, id_from: Optional[str] = None, batch_size: int = 250, include_vectors: bool = True, include_metrics: bool = True, as_pandas: Optional[bool] = None) → Union[DatasetForTextClassification, DatasetForTokenClassification, DatasetForText2Text]#

Loads a argilla dataset.

Parameters:

  • name – The dataset name.

  • workspace – The workspace to which records will be logged/loaded. If None (default) and the env variable ARGILLA_WORKSPACE is not set, it will default to the private user workspace.

  • query –

    An ElasticSearch query with the query string syntax

  • vector – Vector configuration for a semantic search

  • ids – If provided, load dataset records with given ids.

  • limit – The number of records to retrieve.

  • sort – The fields on which to sort [(<field_name>, ‘asc|decs’)].

  • id_from – If provided, starts gathering the records starting from that Record. As the Records returned with the load method are sorted by ID, id_from can be used to load using batches.

  • batch_size – If provided, load batch_size samples per request. A lower batch size may help avoid timeouts.

  • include_vectors – When set to False, indicates that records will be retrieved excluding their vectors, if any. By default, this parameter is set to True, meaning that vectors will be included.

  • include_metrics – When set to False, indicates that records will be retrieved excluding their metrics. By default, this parameter is set to True, meaning that metrics will be included.

  • as_pandas – DEPRECATED! To get a pandas DataFrame do rg.load('my_dataset').to_pandas().

Returns:

A argilla dataset.

Examples

Basic Loading: load the samples sorted by their ID

>>> import argilla as rg
>>> dataset = rg.load(name="example-dataset")
Iterate over a large dataset:

When dealing with a large dataset you might want to load it in batches to optimize memory consumption and avoid network timeouts. To that end, a simple batch-iteration over the whole database can be done employing the from_id parameter. This parameter will act as a delimiter, retrieving the N items after the given id, where N is determined by the limit parameter. NOTE If no limit is given the whole dataset after that ID will be retrieved.

>>> import argilla as rg
>>> dataset_batch_1 = rg.load(name="example-dataset", limit=1000)
>>> dataset_batch_2 = rg.load(name="example-dataset", limit=1000, id_from=dataset_batch_1[-1].id)
argilla.log(records: Union[TextClassificationRecord, TokenClassificationRecord, Text2TextRecord, TextGenerationRecord, Iterable[Union[TextClassificationRecord, TokenClassificationRecord, Text2TextRecord, TextGenerationRecord]], DatasetForTextClassification, DatasetForTokenClassification, DatasetForText2Text], name: str, workspace: Optional[str] = None, tags: Optional[Dict[str, str]] = None, metadata: Optional[Dict[str, Any]] = None, batch_size: int = 100, verbose: bool = True, background: bool = False, chunk_size: Optional[int] = None, num_threads: int = 0, max_retries: int = 3) → Union[BulkResponse, Future]#

Logs Records to argilla.

The logging happens asynchronously in a background thread.

Parameters:

  • records – The record, an iterable of records, or a dataset to log.

  • name – The dataset name.

  • workspace – The workspace to which records will be logged/loaded. If None (default) and the env variable ARGILLA_WORKSPACE is not set, it will default to the private user workspace.

  • tags – A dictionary of tags related to the dataset.

  • metadata – A dictionary of extra info for the dataset.

  • batch_size – The batch size for a data bulk.

  • verbose – If True, shows a progress bar and prints out a quick summary at the end.

  • background – If True, we will NOT wait for the logging process to finish and return an asyncio.Future object. You probably want to set verbose to False in that case.

  • chunk_size – DEPRECATED! Use batch_size instead.

  • num_threads – If > 0, will use num_thread separate number threads to batches, sending data concurrently. Default to 0, which means no threading at all.

  • max_retries – Number of retries when logging a batch of records if a httpx.TransportError occurs. Default 3.

Returns:

Summary of the response from the REST API. If the background argument is set to True, an asyncio.Future will be returned instead.

Examples

>>> import argilla as rg
>>> record = rg.TextClassificationRecord(
...     text="my first argilla example",
...     prediction=[('spam', 0.8), ('ham', 0.2)]
... )
>>> rg.log(record, name="example-dataset")
1 records logged to http://localhost:6900/datasets/argilla/example-dataset
BulkResponse(dataset='example-dataset', processed=1, failed=0)
>>>
>>> # Logging records in the background
>>> rg.log(record, name="example-dataset", background=True, verbose=False)
<Future at 0x7f675a1fffa0 state=pending>
argilla.set_workspace(workspace: str) → None#

Sets the active workspace.

Parameters:

workspace – The new workspace

Records#

This module contains the data models for the interface

class argilla.client.models.Framework(value)#

Frameworks supported by Argilla

Options:

transformers: Transformers peft: PEFT Transformers library setfit: SetFit Transformers library spacy: Spacy Explosion spacy-transformers: Spacy Transformers Explosion library span_marker: SpanMarker Tom Aarsen library spark-nlp: Spark NLP John Snow Labs library openai: OpenAI LLMs trl: Transformer Reinforcement Learning trlx: Transformer Reinforcement Learning X

class argilla.client.models.Text2TextRecord(*, text: str, prediction: Optional[List[Union[str, Tuple[str, float]]]] = None, prediction_agent: Optional[str] = None, annotation: Optional[str] = None, annotation_agent: Optional[str] = None, vectors: Optional[Dict[str, List[float]]] = None, id: Optional[Union[int, str]] = None, metadata: Optional[Dict[str, Any]] = None, status: Optional[str] = None, event_timestamp: Optional[datetime] = None, metrics: Optional[Dict[str, Any]] = None, search_keywords: Optional[List[str]] = None)#

Record for a text to text task

Parameters:

  • text – The input of the record

  • prediction – A list of strings or tuples containing predictions for the input text. If tuples, the first entry is the predicted text, the second entry is its corresponding score.

  • prediction_agent – Name of the prediction agent. By default, this is set to the hostname of your machine.

  • annotation – A string representing the expected output text for the given input text.

  • annotation_agent – Name of the prediction agent. By default, this is set to the hostname of your machine.

  • vectors – Embedding data mappings of the natural language text containing class attributes’

  • id – The id of the record. By default (None), we will generate a unique ID for you.

  • metadata – Metadata for the record. Defaults to {}.

  • status – The status of the record. Options: ‘Default’, ‘Edited’, ‘Discarded’, ‘Validated’. If an annotation is provided, this defaults to ‘Validated’, otherwise ‘Default’.

  • event_timestamp – The timestamp for the creation of the record. Defaults to datetime.datetime.now().

  • metrics – READ ONLY! Metrics at record level provided by the server when using rg.load. This attribute will be ignored when using rg.log.

  • search_keywords – READ ONLY! Relevant record keywords/terms for provided query when using rg.load. This attribute will be ignored when using rg.log.

Examples

>>> import argilla as rg
>>> record = rg.Text2TextRecord(
...     text="My name is Sarah and I love my dog.",
...     prediction=["Je m'appelle Sarah et j'aime mon chien."],
...     vectors = {
...         "bert_base_uncased": [1.2, 2.3, 3.4, 5.2, 6.5],
...         "xlm_multilingual_uncased": [2.2, 5.3, 5.4, 3.2, 2.5]
...     }
... )
classmethod prediction_as_tuples(prediction: Optional[List[Union[str, Tuple[str, float]]]])#

Preprocess the predictions and wraps them in a tuple if needed

class argilla.client.models.TextClassificationRecord(*, text: Optional[str] = None, inputs: Optional[Union[str, List[str], Dict[str, Union[str, List[str]]]]] = None, prediction: Optional[List[Tuple[str, float]]] = None, prediction_agent: Optional[str] = None, annotation: Optional[Union[str, List[str]]] = None, annotation_agent: Optional[str] = None, vectors: Optional[Dict[str, List[float]]] = None, multi_label: bool = False, explanation: Optional[Dict[str, List[TokenAttributions]]] = None, id: Optional[Union[int, str]] = None, metadata: Optional[Dict[str, Any]] = None, status: Optional[str] = None, event_timestamp: Optional[datetime] = None, metrics: Optional[Dict[str, Any]] = None, search_keywords: Optional[List[str]] = None)#

Record for text classification

Parameters:

  • text – The input of the record. Provide either ‘text’ or ‘inputs’.

  • inputs – Various inputs of the record (see examples below). Provide either ‘text’ or ‘inputs’.

  • prediction – A list of tuples containing the predictions for the record. The first entry of the tuple is the predicted label, the second entry is its corresponding score.

  • prediction_agent – Name of the prediction agent. By default, this is set to the hostname of your machine.

  • annotation – A string or a list of strings (multilabel) corresponding to the annotation (gold label) for the record.

  • annotation_agent – Name of the prediction agent. By default, this is set to the hostname of your machine.

  • vectors – Vectors data mappings of the natural language text containing class attributes

  • multi_label – Is the prediction/annotation for a multi label classification task? Defaults to False.

  • explanation – A dictionary containing the attributions of each token to the prediction. The keys map the input of the record (see inputs) to the TokenAttributions.

  • id – The id of the record. By default (None), we will generate a unique ID for you.

  • metadata – Metadata for the record. Defaults to {}.

  • status – The status of the record. Options: ‘Default’, ‘Edited’, ‘Discarded’, ‘Validated’. If an annotation is provided, this defaults to ‘Validated’, otherwise ‘Default’.

  • event_timestamp – The timestamp for the creation of the record. Defaults to datetime.datetime.now().

  • metrics – READ ONLY! Metrics at record level provided by the server when using rg.load. This attribute will be ignored when using rg.log.

  • search_keywords – READ ONLY! Relevant record keywords/terms for provided query when using rg.load. This attribute will be ignored when using rg.log.

Examples

>>> # Single text input
>>> import argilla as rg
>>> record = rg.TextClassificationRecord(
...     text="My first argilla example",
...     prediction=[('eng', 0.9), ('esp', 0.1)],
...     vectors = {
...         "english_bert_vector": [1.2, 2.3, 3.1, 3.3]
...     }
... )
>>>
>>> # Various inputs
>>> record = rg.TextClassificationRecord(
...     inputs={
...         "subject": "Has ganado 1 million!",
...         "body": "Por usar argilla te ha tocado este premio: <link>"
...     },
...     prediction=[('spam', 0.99), ('ham', 0.01)],
...     annotation="spam",
...     vectors = {
...                     "distilbert_uncased":  [1.13, 4.1, 6.3, 4.2, 9.1],
...                     "xlm_roberta_cased": [1.1, 2.1, 3.3, 4.2, 2.1],
...             }
...     )
class argilla.client.models.TextGenerationRecord(*, text: str, prediction: Optional[List[Union[str, Tuple[str, float]]]] = None, prediction_agent: Optional[str] = None, annotation: Optional[str] = None, annotation_agent: Optional[str] = None, vectors: Optional[Dict[str, List[float]]] = None, id: Optional[Union[int, str]] = None, metadata: Optional[Dict[str, Any]] = None, status: Optional[str] = None, event_timestamp: Optional[datetime] = None, metrics: Optional[Dict[str, Any]] = None, search_keywords: Optional[List[str]] = None)#
class argilla.client.models.TokenAttributions(*, token: str, attributions: Dict[str, float] = None)#

Attribution of the token to the predicted label.

In the argilla app this is only supported for TextClassificationRecord and the multi_label=False case.

Parameters:

  • token – The input token.

  • attributions – A dictionary containing label-attribution pairs.

class argilla.client.models.TokenClassificationRecord(text: str = None, tokens: List[str] = None, tags: Optional[List[str]] = None, *, prediction: Optional[List[Union[Tuple[str, int, int], Tuple[str, int, int, Optional[float]]]]] = None, prediction_agent: Optional[str] = None, annotation: Optional[List[Tuple[str, int, int]]] = None, annotation_agent: Optional[str] = None, vectors: Optional[Dict[str, List[float]]] = None, id: Optional[Union[int, str]] = None, metadata: Optional[Dict[str, Any]] = None, status: Optional[str] = None, event_timestamp: Optional[datetime] = None, metrics: Optional[Dict[str, Any]] = None, search_keywords: Optional[List[str]] = None)#

Record for a token classification task

Parameters:

  • text – The input of the record

  • tokens – The tokenized input of the record. We use this to guide the annotation process and to cross-check the spans of your prediction/annotation.

  • prediction – A list of tuples containing the predictions for the record. The first entry of the tuple is the name of predicted entity, the second and third entry correspond to the start and stop character index of the entity. The fourth entry is optional and corresponds to the score of the entity (a float number between 0 and 1).

  • prediction_agent – Name of the prediction agent. By default, this is set to the hostname of your machine.

  • annotation – A list of tuples containing annotations (gold labels) for the record. The first entry of the tuple is the name of the entity, the second and third entry correspond to the start and stop char index of the entity.

  • annotation_agent – Name of the prediction agent. By default, this is set to the hostname of your machine.

  • vectors – Vector data mappings of the natural language text containing class attributes’

  • id – The id of the record. By default (None), we will generate a unique ID for you.

  • metadata – Metadata for the record. Defaults to {}.

  • status – The status of the record. Options: ‘Default’, ‘Edited’, ‘Discarded’, ‘Validated’. If an annotation is provided, this defaults to ‘Validated’, otherwise ‘Default’.

  • event_timestamp – The timestamp for the creation of the record. Defaults to datetime.datetime.now().

  • metrics – READ ONLY! Metrics at record level provided by the server when using rg.load. This attribute will be ignored when using rg.log.

  • search_keywords – READ ONLY! Relevant record keywords/terms for provided query when using rg.load. This attribute will be ignored when using rg.log.

Examples

>>> import argilla as rg
>>> record = rg.TokenClassificationRecord(
...     text = "Michael is a professor at Harvard",
...     tokens = ["Michael", "is", "a", "professor", "at", "Harvard"],
...     prediction = [('NAME', 0, 7), ('LOC', 26, 33)],
...     vectors = {
...            "bert_base_uncased": [3.2, 4.5, 5.6, 8.9]
...          }
... )
char_id2token_id(char_idx: int) → Optional[int]#

DEPRECATED, please use the argilla.utisl.span_utils.SpanUtils.char_to_token_idx dict instead.

spans2iob(spans: Optional[List[Tuple[str, int, int]]] = None) → Optional[List[str]]#

DEPRECATED, please use the argilla.utils.SpanUtils.to_tags() method.

token_span(token_idx: int) → Tuple[int, int]#

DEPRECATED, please use the argilla.utisl.span_utils.SpanUtils.token_to_char_idx dict instead.

Datasets#

class argilla.client.datasets.DatasetForText2Text(records: Optional[List[Text2TextRecord]] = None)#

This Dataset contains Text2TextRecord records.

It allows you to export/import records into/from different formats, loop over the records, and access them by index.

Parameters:

records – A list of `Text2TextRecord`s.

Raises:

WrongRecordTypeError – When the record type in the provided list does not correspond to the dataset type.

Examples

>>> # Import/export records:
>>> import argilla as rg
>>> dataset = rg.DatasetForText2Text.from_pandas(my_dataframe)
>>> dataset.to_datasets()
>>>
>>> # Passing in a list of records:
>>> records = [
...     rg.Text2TextRecord(text="example"),
...     rg.Text2TextRecord(text="another example"),
... ]
>>> dataset = rg.DatasetForText2Text(records)
>>> assert len(dataset) == 2
>>>
>>> # Looping over the dataset:
>>> for record in dataset:
...     print(record)
>>>
>>> # Indexing into the dataset:
>>> dataset[0]
... rg.Text2TextRecord(text="example"})
>>> dataset[0] = rg.Text2TextRecord(text="replaced example")
classmethod from_datasets(dataset: datasets.Dataset, text: Optional[str] = None, annotation: Optional[str] = None, metadata: Optional[Union[str, List[str]]] = None, id: Optional[str] = None) → DatasetForText2Text#

Imports records from a datasets.Dataset.

Columns that are not supported are ignored.

Parameters:

  • dataset – A datasets Dataset from which to import the records.

  • text – The field name used as record text. Default: None

  • annotation – The field name used as record annotation. Default: None

  • metadata – The field name used as record metadata. Default: None

Returns:

The imported records in a argilla Dataset.

Examples

>>> import datasets
>>> ds = datasets.Dataset.from_dict({
...     "text": ["my example"],
...     "prediction": [["mi ejemplo", "ejemplo mio"]]
... })
>>> # or
>>> ds = datasets.Dataset.from_dict({
...     "text": ["my example"],
...     "prediction": [[{"text": "mi ejemplo", "score": 0.9}]]
... })
>>> DatasetForText2Text.from_datasets(ds)
classmethod from_pandas(dataframe: DataFrame) → DatasetForText2Text#

Imports records from a pandas.DataFrame.

Columns that are not supported are ignored.

Parameters:

dataframe – A pandas DataFrame from which to import the records.

Returns:

The imported records in a argilla Dataset.

class argilla.client.datasets.DatasetForTextClassification(records: Optional[List[TextClassificationRecord]] = None)#

This Dataset contains TextClassificationRecord records.

It allows you to export/import records into/from different formats, loop over the records, and access them by index.

Parameters:

records – A list of `TextClassificationRecord`s.

Raises:

WrongRecordTypeError – When the record type in the provided list does not correspond to the dataset type.

Examples

>>> # Import/export records:
>>> import argilla as rg
>>> dataset = rg.DatasetForTextClassification.from_pandas(my_dataframe)
>>> dataset.to_datasets()
>>>
>>> # Looping over the dataset:
>>> for record in dataset:
...     print(record)
>>>
>>> # Passing in a list of records:
>>> records = [
...     rg.TextClassificationRecord(text="example"),
...     rg.TextClassificationRecord(text="another example"),
... ]
>>> dataset = rg.DatasetForTextClassification(records)
>>> assert len(dataset) == 2
>>>
>>> # Indexing into the dataset:
>>> dataset[0]
... rg.TextClassificationRecord(text="example")
>>> dataset[0] = rg.TextClassificationRecord(text="replaced example")
classmethod from_datasets(dataset: datasets.Dataset, text: Optional[str] = None, id: Optional[str] = None, inputs: Optional[Union[str, List[str]]] = None, annotation: Optional[str] = None, metadata: Optional[Union[str, List[str]]] = None) → DatasetForTextClassification#

Imports records from a datasets.Dataset.

Columns that are not supported are ignored.

Parameters:

  • dataset – A datasets Dataset from which to import the records.

  • text – The field name used as record text. Default: None

  • id – The field name used as record id. Default: None

  • inputs – A list of field names used for record inputs. Default: None

  • annotation – The field name used as record annotation. Default: None

  • metadata – The field name used as record metadata. Default: None

Returns:

The imported records in a argilla Dataset.

Examples

>>> import datasets
>>> ds = datasets.Dataset.from_dict({
...     "inputs": ["example"],
...     "prediction": [
...         [{"label": "LABEL1", "score": 0.9}, {"label": "LABEL2", "score": 0.1}]
...     ]
... })
>>> DatasetForTextClassification.from_datasets(ds)
classmethod from_pandas(dataframe: DataFrame) → DatasetForTextClassification#

Imports records from a pandas.DataFrame.

Columns that are not supported are ignored.

Parameters:

dataframe – A pandas DataFrame from which to import the records.

Returns:

The imported records in a argilla Dataset.

class argilla.client.datasets.DatasetForTokenClassification(records: Optional[List[TokenClassificationRecord]] = None)#

This Dataset contains TokenClassificationRecord records.

It allows you to export/import records into/from different formats, loop over the records, and access them by index.

Parameters:

records – A list of `TokenClassificationRecord`s.

Raises:

WrongRecordTypeError – When the record type in the provided list does not correspond to the dataset type.

Examples

>>> # Import/export records:
>>> import argilla as rg
>>> dataset = rg.DatasetForTokenClassification.from_pandas(my_dataframe)
>>> dataset.to_datasets()
>>>
>>> # Looping over the dataset:
>>> assert len(dataset) == 2
>>> for record in dataset:
...     print(record)
>>>
>>> # Passing in a list of records:
>>> import argilla as rg
>>> records = [
...     rg.TokenClassificationRecord(text="example", tokens=["example"]),
...     rg.TokenClassificationRecord(text="another example", tokens=["another", "example"]),
... ]
>>> dataset = rg.DatasetForTokenClassification(records)
>>>
>>> # Indexing into the dataset:
>>> dataset[0]
... rg.TokenClassificationRecord(text="example", tokens=["example"])
>>> dataset[0] = rg.TokenClassificationRecord(text="replace example", tokens=["replace", "example"])
classmethod from_datasets(dataset: datasets.Dataset, text: Optional[str] = None, id: Optional[str] = None, tokens: Optional[str] = None, tags: Optional[str] = None, metadata: Optional[Union[str, List[str]]] = None) → DatasetForTokenClassification#

Imports records from a datasets.Dataset.

Columns that are not supported are ignored.

Parameters:

  • dataset – A datasets Dataset from which to import the records.

  • text – The field name used as record text. Default: None

  • id – The field name used as record id. Default: None

  • tokens – The field name used as record tokens. Default: None

  • tags – The field name used as record tags. Default: None

  • metadata – The field name used as record metadata. Default: None

Returns:

The imported records in a argilla Dataset.

Examples

>>> import datasets
>>> ds = datasets.Dataset.from_dict({
...     "text": ["my example"],
...     "tokens": [["my", "example"]],
...     "prediction": [
...         [{"label": "LABEL1", "start": 3, "end": 10, "score": 1.0}]
...     ]
... })
>>> DatasetForTokenClassification.from_datasets(ds)
classmethod from_pandas(dataframe: DataFrame) → DatasetForTokenClassification#

Imports records from a pandas.DataFrame.

Columns that are not supported are ignored.

Parameters:

dataframe – A pandas DataFrame from which to import the records.

Returns:

The imported records in a argilla Dataset.

argilla.client.datasets.read_datasets(dataset: datasets.Dataset, task: Union[str, TaskType], **kwargs) → Union[DatasetForTextClassification, DatasetForTokenClassification, DatasetForText2Text]#

Reads a datasets Dataset and returns a argilla Dataset

Columns not supported by the Record instance corresponding with the task are ignored.

Parameters:

  • dataset – Dataset to be read in.

  • task – Task for the dataset, one of: [“TextClassification”, “TokenClassification”, “Text2Text”].

  • **kwargs – Passed on to the task-specific DatasetFor*.from_datasets() method.

Returns:

A argilla dataset for the given task.

Examples

>>> # Read text classification records from a datasets Dataset
>>> import datasets
>>> ds = datasets.Dataset.from_dict({
...     "inputs": ["example"],
...     "prediction": [
...         [{"label": "LABEL1", "score": 0.9}, {"label": "LABEL2", "score": 0.1}]
...     ]
... })
>>> read_datasets(ds, task="TextClassification")
>>>
>>> # Read token classification records from a datasets Dataset
>>> ds = datasets.Dataset.from_dict({
...     "text": ["my example"],
...     "tokens": [["my", "example"]],
...     "prediction": [
...         [{"label": "LABEL1", "start": 3, "end": 10}]
...     ]
... })
>>> read_datasets(ds, task="TokenClassification")
>>>
>>> # Read text2text records from a datasets Dataset
>>> ds = datasets.Dataset.from_dict({
...     "text": ["my example"],
...     "prediction": [["mi ejemplo", "ejemplo mio"]]
... })
>>> # or
>>> ds = datasets.Dataset.from_dict({
...     "text": ["my example"],
...     "prediction": [[{"text": "mi ejemplo", "score": 0.9}]]
... })
>>> read_datasets(ds, task="Text2Text")
argilla.client.datasets.read_pandas(dataframe: DataFrame, task: Union[str, TaskType]) → Union[DatasetForTextClassification, DatasetForTokenClassification, DatasetForText2Text]#

Reads a pandas DataFrame and returns a argilla Dataset

Columns not supported by the Record instance corresponding with the task are ignored.

Parameters:

  • dataframe – Dataframe to be read in.

  • task – Task for the dataset, one of: [“TextClassification”, “TokenClassification”, “Text2Text”]

Returns:

A argilla dataset for the given task.

Examples

>>> # Read text classification records from a pandas DataFrame
>>> import pandas as pd
>>> df = pd.DataFrame({
...     "inputs": ["example"],
...     "prediction": [
...         [("LABEL1", 0.9), ("LABEL2", 0.1)]
...     ]
... })
>>> read_pandas(df, task="TextClassification")
>>>
>>> # Read token classification records from a datasets Dataset
>>> df = pd.DataFrame({
...     "text": ["my example"],
...     "tokens": [["my", "example"]],
...     "prediction": [
...         [("LABEL1", 3, 10)]
...     ]
... })
>>> read_pandas(df, task="TokenClassification")
>>>
>>> # Read text2text records from a datasets Dataset
>>> df = pd.DataFrame({
...     "text": ["my example"],
...     "prediction": [["mi ejemplo", "ejemplo mio"]]
... })
>>> # or
>>> ds = pd.DataFrame({
...     "text": ["my example"],
...     "prediction": [[("mi ejemplo", 0.9)]]
... })
>>> read_pandas(df, task="Text2Text")

FeedbackDataset#

class argilla.client.feedback.dataset.local.FeedbackDataset(*, fields: List[TextField], questions: List[Union[TextQuestion, RatingQuestion, LabelQuestion, MultiLabelQuestion, RankingQuestion]], guidelines: Optional[str] = None)#
add_records(records: Union[FeedbackRecord, Dict[str, Any], List[Union[FeedbackRecord, Dict[str, Any]]]]) → None#

Adds the given records to the dataset, and stores them locally. If you are planning to push those to Argilla, you will need to call push_to_argilla afterwards, to both create the dataset in Argilla and push the records to it. Then, from a FeedbackDataset pushed to Argilla, you’ll just need to call add_records and those will be automatically uploaded to Argilla.

Parameters:

records – can be a single FeedbackRecord, a list of FeedbackRecord, a single dictionary, or a list of dictionaries. If a dictionary is provided, it will be converted to a FeedbackRecord internally.

Raises:

  • ValueError – if the given records are an empty list.

  • ValueError – if the given records are neither: FeedbackRecord, list of FeedbackRecord, list of dictionaries as a record or dictionary as a record.

  • ValueError – if the given records do not match the expected schema.

iter(batch_size: Optional[int] = 250) → Iterator[List[FeedbackRecord]]#

Returns an iterator over the records in the dataset.

Parameters:

batch_size – the size of the batches to return. Defaults to 100.

property records: List[FeedbackRecord]#

Returns the records in the dataset.

class argilla.client.feedback.dataset.remote.dataset.RemoteFeedbackDataset(*, client: httpx.Client, id: UUID, name: str, workspace: Workspace, fields: List[AllowedFieldTypes], questions: List[AllowedQuestionTypes], guidelines: Optional[str] = None)#
delete() → None#

Deletes the current FeedbackDataset from Argilla. This method is just working if the user has either owner or admin role.

Raises:

  • PermissionError – if the user does not have either owner or admin role.

  • RuntimeError – if the FeedbackDataset cannot be deleted from Argilla.

filter_by(response_status: Union[FeedbackResponseStatusFilter, List[FeedbackResponseStatusFilter]]) → FilteredRemoteFeedbackDataset#

Filters the current RemoteFeedbackDataset based on the response_status of the responses of the records in Argilla. This method creates a new class instance of FilteredRemoteFeedbackDataset with the given filters.

Parameters:

response_status – the response status/es to filter the dataset by. Can be one of: draft, pending, submitted, and discarded.

Returns:

A new instance of FilteredRemoteFeedbackDataset with the given filters.

records_cls#

alias of RemoteFeedbackRecords

class argilla.client.feedback.dataset.remote.dataset.RemoteFeedbackRecords(dataset: RemoteFeedbackDataset)#
add(records: Union[FeedbackRecord, Dict[str, Any], List[Union[FeedbackRecord, Dict[str, Any]]]], show_progress: bool = True) → None#

Pushes a list of `FeedbackRecord`s to Argilla.

Parameters:

  • records – can be a single FeedbackRecord, a list of FeedbackRecord, a single dictionary, or a list of dictionaries. If a dictionary is provided, it will be converted to a FeedbackRecord internally.

  • show_progress – Whether to show a tqdm progress bar while pushing the records.

Raises:

  • PermissionError – if the user does not have either owner or admin role.

  • Exception – If the pushing of the records to Argilla fails.

delete(records: List[RemoteFeedbackRecord]) → None#

Deletes a list of `RemoteFeedbackRecord`s from Argilla.

Parameters:

records – A list of `RemoteFeedbackRecord`s to delete from Argilla.

Raises:

  • PermissionError – if the user does not have either owner or admin role.

  • RuntimeError – If the deletion of the records from Argilla fails.

class argilla.client.feedback.schemas.questions.LabelQuestion(*, id: Optional[UUID] = None, name: ConstrainedStrValue, title: Optional[str] = None, description: Optional[str] = None, required: bool = True, type: Literal['label_selection'] = 'label_selection', settings: Dict[str, Any] = None, labels: Union[ConstrainedListValue[str], Dict[str, str]], visible_labels: Optional[ConstrainedIntValue] = 20)#

Schema for the FeedbackDataset label questions, which are the ones that will require a label response from the user. This class should be used when the user can only select one label.

Parameters:

  • type – The type of the question. Defaults to ‘label_selection’ and cannot/shouldn’t be modified.

  • labels – The list of labels of the label question. The labels must be unique, and the list must contain at least two unique labels. Additionally, labels can also be a dictionary of labels, where the keys are the labels, and the values are the labels that will be shown in the UI.

  • visible_labels – The number of visible labels in the UI. Defaults to 20, and must be 3 or greater.

Examples

>>> from argilla.client.feedback.schemas.questions import LabelQuestion
>>> LabelQuestion(name="label_question", title="Label Question", labels=["label_1", "label_2"])
class argilla.client.feedback.schemas.questions.MultiLabelQuestion(*, id: Optional[UUID] = None, name: ConstrainedStrValue, title: Optional[str] = None, description: Optional[str] = None, required: bool = True, type: Literal['multi_label_selection'] = 'multi_label_selection', settings: Dict[str, Any] = None, labels: Union[ConstrainedListValue[str], Dict[str, str]], visible_labels: Optional[ConstrainedIntValue] = 20)#

Schema for the FeedbackDataset label questions, which are the ones that will require a label response from the user. This class should be used when the user can select multiple labels.

Parameters:

  • type – The type of the question. Defaults to ‘multi_label_selection’ and cannot/shouldn’t be modified.

  • labels – The list of labels of the label question. The labels must be unique, and the list must contain at least two unique labels. Additionally, labels can also be a dictionary of labels, where the keys are the labels, and the values are the labels that will be shown in the UI.

  • visible_labels – The number of visible labels in the UI. Defaults to 20, and must be 3 or greater.

Examples

>>> from argilla.client.feedback.schemas.questions import MultiLabelQuestion
>>> MultiLabelQuestion(name="multi_label_question", title="Multi Label Question", labels=["label_1", "label_2"])
class argilla.client.feedback.schemas.questions.QuestionSchema(*, id: Optional[UUID] = None, name: ConstrainedStrValue, title: Optional[str] = None, description: Optional[str] = None, required: bool = True, type: Optional[Literal['text', 'rating', 'label_selection', 'multi_label_selection', 'ranking']] = None, settings: Dict[str, Any] = None)#

Base schema for the FeedbackDataset questions. Which means that all the questions in the dataset will have at least these fields.

Parameters:

  • id – The ID of the question in Argilla. Defaults to None, and is automatically fulfilled internally once the question is pushed to Argilla.

  • name – The name of the question. This is the only required field.

  • title – The title of the question. If not provided, it will be capitalized from the name field. And its what will be shown in the UI.

  • description – The description of the question. Defaults to None, and is not shown in the UI, otherwise, it will be shown in the tooltip close to each question.

  • required – Whether the question is required or not. Defaults to True. Note that at least one question must be required.

  • type – The type of the question. Defaults to None, and ideally it should be defined in the class inheriting from this one to be able to use a discriminated union based on the type field.

  • settings – The settings of the question. Defaults to an empty dict, and it is automatically fulfilled internally before the question is pushed to Argilla, as the settings is part of the payload that will be sent to Argilla.

Disclaimer:

You should not use this class directly, but instead use the classes that inherit from this one, as they will have the type field already defined, and ensured to be supported by Argilla.

class argilla.client.feedback.schemas.questions.RankingQuestion(*, id: Optional[UUID] = None, name: ConstrainedStrValue, title: Optional[str] = None, description: Optional[str] = None, required: bool = True, type: Literal['ranking'] = 'ranking', settings: Dict[str, Any] = None, values: Union[ConstrainedListValue[str], Dict[str, str]])#

Schema for the FeedbackDataset ranking questions, which are the ones that will require a ranking response from the user. More specifically, the user will be asked to rank the labels, all the labels need to be assigned (if either the question is required or if at least one label has been ranked), and there can be ties/draws.

Parameters:

  • type – The type of the question. Defaults to ‘ranking’ and cannot/shouldn’t be modified.

  • values – The list of labels of the ranking question. The labels must be unique, and the list must contain at least two unique labels. Additionally, labels can also be a dictionary of labels, where the keys are the labels, and the values are the labels that will be shown in the UI.

Examples

>>> from argilla.client.feedback.schemas.questions import RankingQuestion
>>> RankingQuestion(name="ranking_question", title="Ranking Question", labels=["label_1", "label_2"])
class argilla.client.feedback.schemas.questions.RatingQuestion(*, id: Optional[UUID] = None, name: ConstrainedStrValue, title: Optional[str] = None, description: Optional[str] = None, required: bool = True, type: Literal['rating'] = 'rating', settings: Dict[str, Any] = None, values: ConstrainedListValue[ConstrainedIntValue])#

Schema for the FeedbackDataset rating questions, which are the ones that will require a rating response from the user.

Parameters:

  • type – The type of the question. Defaults to ‘rating’ and cannot/shouldn’t be modified.

  • values – The list of interger values of the rating question. There is not need for the values to be sequential, but they must be unique, contain at least two unique integers in the range [1, 10].

Examples

>>> from argilla.client.feedback.schemas.questions import RatingQuestion
>>> RatingQuestion(name="rating_question", title="Rating Question", values=[1, 2, 3, 4, 5])
class argilla.client.feedback.schemas.questions.TextQuestion(*, id: Optional[UUID] = None, name: ConstrainedStrValue, title: Optional[str] = None, description: Optional[str] = None, required: bool = True, type: Literal['text'] = 'text', settings: Dict[str, Any] = None, use_markdown: bool = False)#

Schema for the FeedbackDataset text questions, which are the ones that will require a text response from the user.

Parameters:

  • type – The type of the question. Defaults to ‘text’ and cannot/shouldn’t be modified.

  • use_markdown – Whether the question should be rendered using markdown or not. Defaults to False.

Examples

>>> from argilla.client.feedback.schemas.questions import TextQuestion
>>> TextQuestion(name="text_question", title="Text Question")
class argilla.client.feedback.schemas.fields.FieldSchema(*, id: Optional[UUID] = None, name: ConstrainedStrValue, title: Optional[str] = None, required: bool = True, type: Optional[Literal['text']] = None, settings: Dict[str, Any] = None)#

Base schema for the FeedbackDataset fields.

Parameters:

  • id – The ID of the field in Argilla. Defaults to None, and is automatically fulfilled internally once the field is pushed to Argilla.

  • name – The name of the field. This is the only required field.

  • title – The title of the field. If not provided, it will be capitalized from the name field. And its what will be shown in the UI.

  • required – Whether the field is required or not. Defaults to True. Note that at least one field must be required.

  • type – The type of the field. Defaults to None, and ideally it should be defined in the class inheriting from this one to be able to use a discriminated union based on the type field.

  • settings – The settings of the field. Defaults to an empty dict, and it is automatically fulfilled internally before the field is pushed to Argilla, as the settings is part of the payload that will be sent to Argilla.

Disclaimer:

You should not use this class directly, but instead use the classes that inherit from this one, as they will have the type field already defined, and ensured to be supported by Argilla.

class argilla.client.feedback.schemas.fields.TextField(*, id: Optional[UUID] = None, name: ConstrainedStrValue, title: Optional[str] = None, required: bool = True, type: Literal['text'] = 'text', settings: Dict[str, Any] = None, use_markdown: bool = False)#

Schema for the FeedbackDataset text fields, which are the ones that will require a text to be defined as part of the record.

Parameters:

  • type – The type of the field. Defaults to ‘text’ and cannot/shouldn’t be modified.

  • use_markdown – Whether the field should be rendered using markdown or not. Defaults to False.

Examples

>>> from argilla.client.feedback.schemas.fields import TextField
>>> TextField(name="text_field", title="Text Field")
class argilla.client.feedback.schemas.records.FeedbackRecord(*, fields: Dict[str, str], metadata: Dict[str, Any] = None, responses: List[ResponseSchema] = None, suggestions: Union[Tuple[SuggestionSchema], List[SuggestionSchema]] = None, external_id: Optional[str] = None)#

Schema for the records of a FeedbackDataset.

Parameters:

  • fields – Fields that match the FeedbackDataset defined fields. So this attribute contains the actual information shown in the UI for each record, being the record itself.

  • metadata – Metadata to be included to enrich the information for a given record. Note that the metadata is not shown in the UI so you’ll just be able to see that programatically after pulling the records. Defaults to None.

  • responses – Responses given by either the current user, or one or a collection of users that must exist in Argilla. Each response corresponds to one of the FeedbackDataset questions, so the values should match the question type. Defaults to None.

  • suggestions – A list of SuggestionSchema that contains the suggestions for the current record. Every suggestion is linked to only one question. Defaults to an empty list.

  • external_id – The external ID of the record, which means that the user can specify this ID to identify the record no matter what the Argilla ID is. Defaults to None.

Examples

>>> from argilla.client.feedback.schemas.records import FeedbackRecord, ResponseSchema, SuggestionSchema, ValueSchema
>>> FeedbackRecord(
...     fields={"text": "This is the first record", "label": "positive"},
...     metadata={"first": True, "nested": {"more": "stuff"}},
...     responses=[ # optional
...         ResponseSchema(
...             user_id="user-1",
...             values={
...                 "question-1": ValueSchema(value="This is the first answer"),
...                 "question-2": ValueSchema(value=5),
...             },
...             status="submitted",
...         ),
...     ],
...     suggestions=[ # optional
...         SuggestionSchema(
...            question_name="question-1",
...            type="model",
...            score=0.9,
...            value="This is the first suggestion",
...            agent="agent-1",
...         ),
...     external_id="entry-1",
... )
class argilla.client.feedback.schemas.records.RankingValueSchema(*, value: StrictStr, rank: ConstrainedIntValue)#

Schema for the RankingQuestion response value for a RankingQuestion. Note that we may have more than one record in the same rank.

Parameters:

  • value – The value of the record.

  • rank – The rank of the record.

class argilla.client.feedback.schemas.records.RemoteFeedbackRecord(*, fields: Dict[str, str], metadata: Dict[str, Any] = None, responses: List[ResponseSchema] = None, suggestions: Union[Tuple[RemoteSuggestionSchema], List[RemoteSuggestionSchema]] = None, external_id: Optional[str] = None, client: Client, name2id: Dict[str, UUID], id: UUID)#

Schema for the records of a RemoteFeedbackDataset.

Note this schema shouldn’t be instantiated directly, but just internally by the RemoteFeedbackDataset class when fetching records from Argilla.

Parameters:

  • client – The Argilla client to use to push the record to Argilla. Is shared with the RemoteFeedbackDataset that created this record.

  • name2id – A dictionary that maps the question names to their corresponding IDs.

  • id – The ID of the record in Argilla. Defaults to None, and is automatically fulfilled internally once the record is pushed to Argilla.

  • suggestions – A list of RemoteSuggestionSchema that contains the suggestions for the current record in Argilla. Every suggestion is linked to only one question. Defaults to an empty list.

delete() → FeedbackRecord#

Deletes the RemoteFeedbackRecord from Argilla.

Returns:

The deleted record formatted as a FeedbackRecord.

Raises:

PermissionError – if the user does not have either owner or admin role.

delete_suggestions(suggestions: Union[RemoteSuggestionSchema, List[RemoteSuggestionSchema]]) → None#

Deletes the provided suggestions from the record in Argilla. Note that the suggestions must exist in Argilla to be removed from the record.

Parameters:

suggestions – can be a single RemoteSuggestionSchema or a list of RemoteSuggestionSchema.

Raises:

PermissionError – if the user does not have either owner or admin role.

update(suggestions: Union[RemoteSuggestionSchema, List[RemoteSuggestionSchema], SuggestionSchema, List[SuggestionSchema], Dict[str, Any], List[Dict[str, Any]]]) → None#

Update a RemoteFeedbackRecord. Currently just suggestions are supported.

Note that this method will update the record in Argilla directly.

Parameters:

suggestions – can be a single RemoteSuggestionSchema or SuggestionSchema, a list of RemoteSuggestionSchema or SuggestionSchema, a single dictionary, or a list of dictionaries. If a dictionary is provided, it will be converted to a RemoteSuggestionSchema internally.

Raises:

PermissionError – if the user does not have either owner or admin role.

class argilla.client.feedback.schemas.records.ResponseSchema(*, user_id: Optional[UUID] = None, values: Dict[str, ValueSchema], status: Literal['submitted', 'discarded'] = 'submitted')#

Schema for the FeedbackRecord response.

Parameters:

  • user_id – ID of the user that provided the response. Defaults to None, and is automatically fulfilled internally once the question is pushed to Argilla.

  • values – Values of the response, should match the questions in the record.

  • status – Status of the response. Defaults to submitted.

Examples

>>> from argilla.client.feedback.schemas.records import ResponseSchema
>>> ResponseSchema(
...     values={
...         "question_1": {"value": "answer_1"},
...         "question_2": {"value": "answer_2"},
...     }
... )
class argilla.client.feedback.schemas.records.SuggestionSchema(*, question_id: Optional[UUID] = None, question_name: str, type: Optional[Literal['human', 'model']] = None, score: Optional[float] = None, value: Any = None, agent: Optional[str] = None)#

Schema for the suggestions for the questions related to the record.

Parameters:

  • question_id – ID of the question in Argilla. Defaults to None, and is automatically fulfilled internally once the question is pushed to Argilla.

  • question_name – name of the question.

  • type – type of the question. Defaults to None. Possible values are model or human.

  • score – score of the suggestion. Defaults to None.

  • value – value of the suggestion, which should match the type of the question.

  • agent – agent that generated the suggestion. Defaults to None.

Examples

>>> from argilla.client.feedback.schemas.records import SuggestionSchema
>>> SuggestionSchema(
...     question_name="question-1",
...     type="model",
...     score=0.9,
...     value="This is the first suggestion",
...     agent="agent-1",
... )
class argilla.client.feedback.schemas.records.ValueSchema(*, value: Union[StrictStr, StrictInt, List[str], List[RankingValueSchema]])#

Schema for any FeedbackRecord response value.

Parameters:

value – The value of the record.