Importing and exporting datasets and records¶

This guide provides an overview of how to import and export your dataset or its records to Python, your local disk, or the Hugging Face Hub.

In Argilla, you can import/export two main components of a dataset:

The dataset's complete configuration is defined in rg.Settings. This is useful if you want to share your feedback task or restore it later in Argilla.
The records stored in the dataset, including Metadata, Vectors, Suggestions, and Responses. This is useful if you want to use your dataset's records outside of Argilla.

Check the Dataset - Python Reference to see the attributes, arguments, and methods of the export Dataset class in detail.

Main Classes

rg.Dataset.to_hubrg.Dataset.from_hubrg.Dataset.to_diskrg.Dataset.from_diskrg.Dataset.records.to_datasets()rg.Dataset.records.to_dict()rg.Dataset.records.to_list()

rg.Dataset.to_hub(
    repo_id="<my_org>/<my_dataset>",
    with_records=True,
    generate_card=True
)

rg.Dataset.from_hub(
    repo_id="<my_org>/<my_dataset>",
    name="my_dataset",
    workspace="my_workspace",
    client=rg.Client(),
    with_records=True
)

rg.Dataset.to_disk(
    path="<path-empty-directory>",
    with_records=True
)

rg.Dataset.from_disk(
    path="<path-dataset-directory>",
    name="my_dataset",
    workspace="my_workspace",
    client=rg.Client(),
    with_records=True
)

rg.Dataset.records.to_datasets()

rg.Dataset.records.to_dict()

rg.Dataset.records.to_list()

Check the Dataset - Python Reference to see the attributes, arguments, and methods of the export Dataset class in detail.

Check the Record - Python Reference to see the attributes, arguments, and methods of the Record class in detail.

Importing and exporting datasets¶

First, we will go through exporting a complete dataset from Argilla. This includes the dataset's settings and records. All of these methods use the rg.Dataset.from_* and rg.Dataset.to_* methods.

Hugging Face Hub¶

Export to Hub¶

You can push a dataset from Argilla to the Hugging Face Hub. This is useful if you want to share your dataset with the community or version control it. You can push the dataset to the Hugging Face Hub using the rg.Dataset.to_hub method.

import argilla as rg

client = rg.Argilla(api_url="<api_url>", api_key="<api_key>")

dataset = client.datasets(name="my_dataset")

dataset.to_hub(repo_id="<my_org>/<my_dataset>")

With or without records

The example above will push the dataset's Settings and records to the hub. If you only want to push the dataset's configuration, you can set the with_records parameter to False. This is useful if you're just interested in a specific dataset template or you want to make changes in the dataset settings and/or records.

dataset.to_hub(repo_id="<my_org>/<my_dataset>", with_records=False)

Import from Hub¶

You can pull a dataset from the Hugging Face Hub to Argilla. This is useful if you want to restore a dataset and its configuration. You can pull the dataset from the Hugging Face Hub using the rg.Dataset.from_hub method.

import argilla as rg

client = rg.Argilla(api_url="<api_url>", api_key="<api_key>")

dataset = rg.Dataset.from_hub(repo_id="<my_org>/<my_dataset>")

The rg.Dataset.from_hub method loads the configuration and records from the dataset repo. If you only want to load records, you can pass a datasets.Dataset object to the rg.Dataset.log method. This enables you to configure your own dataset and reuse existing Hub datasets. See the guide on records for more information.

With or without records

The example above will pull the dataset's Settings and records from the hub. If you only want to pull the dataset's configuration, you can set the with_records parameter to False. This is useful if you're just interested in a specific dataset template or you want to make changes in the dataset settings and/or records.

dataset = rg.Dataset.from_hub(repo_id="<my_org>/<my_dataset>", with_records=False)

With the dataset's configuration, you could then make changes to the dataset. For example, you could adapt the dataset's settings for a different task:

dataset.settings.questions = [rg.TextQuestion(name="answer")]
dataset.update()

You could then log the dataset's records using the load_dataset method of the datasets package and pass the dataset to the rg.Dataset.log method.

hf_dataset = load_dataset("<my_org>/<my_dataset>")
dataset.records.log(hf_dataset)

Local Disk¶

Export to Disk¶

You can save a dataset from Argilla to your local disk. This is useful if you want to back up your dataset. You can use the rg.Dataset.to_disk method. We recommend you to use an empty directory.

import argilla as rg

client = rg.Argilla(api_url="<api_url>", api_key="<api_key>")

dataset = client.datasets(name="my_dataset")

dataset.to_disk(path="<path-empty-directory>")

This will save the dataset's configuration and records to the specified path. If you only want to save the dataset's configuration, you can set the with_records parameter to False.

dataset.to_disk(path="<path-empty-directory>", with_records=False)

Import from Disk¶

You can load a dataset from your local disk to Argilla. This is useful if you want to restore a dataset's configuration. You can use the rg.Dataset.from_disk method.

import argilla as rg

dataset = rg.Dataset.from_disk(path="<path-dataset-directory>")

Directing the dataset to a name and workspace

You can also specify the name and workspace of the dataset when loading it from the disk.

dataset = rg.Dataset.from_disk(path="<path-dataset-directory>", name="my_dataset", workspace="my_workspace")

Importing and exporting records¶

The records alone can be exported from a dataset in Argilla. This is useful if you want to process the records in Python, export them to a different platform, or use them in model training. All of these methods use the rg.Dataset.records attribute.

Export records¶

The records can be exported as a dictionary, a list of dictionaries, or a Dataset of the datasets package.

With images

If your dataset includes images, the recommended approach for exporting records is to use the to_datasets method, which exports the images as rescaled PIL objects. With other methods, the images will be exported using the data URI schema.

To a python dictionaryTo a python listTo the datasets package

Records can be exported from Dataset.records as a dictionary. The to_dict method can be used to export records as a dictionary. You can specify the orientation of the dictionary output. You can also decide if to flatten or not the dictionary.

import argilla as rg

client = rg.Argilla(api_url="<api_url>", api_key="<api_key>")
dataset = client.datasets(name="my_dataset")

# Export records as a dictionary
exported_records = dataset.records.to_dict()
# {'fields': [{'text': 'Hello'},{'text': 'World'}], suggestions': [{'label': {'value': 'positive'}}, {'label': {'value': 'negative'}}]

# Export records as a dictionary with orient=index
exported_records = dataset.records.to_dict(orient="index")
# {"uuid": {'fields': {'text': 'Hello'}, 'suggestions': {'label': {'value': 'positive'}}}, {"uuid": {'fields': {'text': 'World'}, 'suggestions': {'label': {'value': 'negative'}}},

# Export records as a dictionary with flatten=True
exported_records = dataset.records.to_dict(flatten=True)
# {"text": ["Hello", "World"], "label.suggestion": ["greeting", "greeting"]}

Records can be exported from Dataset.records as a list of dictionaries. The to_list method can be used to export records as a list of dictionaries. You can decide if to flatten it or not.

import argilla as rg

client = rg.Argilla(api_url="<api_url>", api_key="<api_key>")

workspace = client.workspaces("my_workspace")

dataset = client.datasets(name="my_dataset", workspace=workspace)

# Export records as a list of dictionaries
exported_records = dataset.records.to_list()
# [{'fields': {'text': 'Hello'}, 'suggestion': {'label': {value: 'greeting'}}}, {'fields': {'text': 'World'}, 'suggestion': {'label': {value: 'greeting'}}}]

# Export records as a list of dictionaries with flatten=True
exported_records = dataset.records.to_list(flatten=True)
# [{"text": "Hello", "label": "greeting"}, {"text": "World", "label": "greeting"}]

Records can be exported from Dataset.records to the datasets package. The to_dataset method can be used to export records to the datasets package. You can specify the name of the dataset and the split to export the records.

import argilla as rg

client = rg.Argilla(api_url="<api_url>", api_key="<api_key>")
dataset = client.datasets(name="my_dataset")

# Export records as a dictionary
exported_dataset = dataset.records.to_datasets()

Import records¶

To import records to a dataset, use the rg.Datasets.records.log method. There is a guide on how to do this in How-to guides - Record, or you can check the Record - Python Reference.