rg.Suggestion

Class for interacting with Argilla Suggestions of records. Suggestions are typically created by a model prediction, unlike a Response which is typically created by a user in the UI or consumed from a data source as a label.

Usage Examples

Adding records with suggestions

Suggestions can be added to a record directly or via a dictionary structure. The following examples demonstrate how to add suggestions to a record object and how to access suggestions from a record object:

Add a response from a dictionary where key is the question name and value is the response:

dataset.records.log(
    [
        {
            "text": "Hello World, how are you?",
            "label": "negative", # this will be used as a suggestion
        },
    ]
)

If your data contains scores for suggestions you can add them as well via the mapping parameter. The following example demonstrates how to add a suggestion with a score to a record object:

dataset.records.log(
    [
        {
            "prompt": "Hello World, how are you?",
            "label": "negative",  # this will be used as a suggestion
            "score": 0.9,  # this will be used as the suggestion score
            "model": "model_name",  # this will be used as the suggestion agent
        },
    ],
    mapping={
        "score": "label.suggestion.score",
        "model": "label.suggestion.agent",
    },  # `label` is the question name in the dataset settings
)

Or, instantiate the Record and related Suggestions objects directly, like this:

dataset.records.log(
    [
        rg.Record(
            fields={"text": "Hello World, how are you?"},
            suggestions=[rg.Suggestion("negative", "label", score=0.9, agent="model_name")],
        )
    ]
)

Iterating over records with suggestions

Just like responses, suggestions can be accessed from a Record via their question name as an attribute of the record. So if a question is named label, the suggestion can be accessed as record.label. The following example demonstrates how to access suggestions from a record object:

for record in dataset.records(with_suggestions=True):
    print(record.suggestions["label"].value)

We can also add suggestions to records as we iterate over them using the add method:

for record in dataset.records(with_suggestions=True):
    if not record.suggestions["label"]: # (1)
        record.suggestions.add(
            rg.Suggestion("positive", "label", score=0.9, agent="model_name")
        ) # (2)

1. Validate that the record has a suggestion
2. Add a suggestion to the record if it does not already have one

Format per Question type

Depending on the Question type, responses might need to be formatted in a slightly different way.

For LabelQuestionFor MultiLabelQuestionFor RankingQuestionFor RatingQuestionFor SpanQuestionFor TextQuestion

rg.Suggestion(
    question_name="label",
    value="positive",
    score=0.9,
    agent="model_name"
)

rg.Suggestion(
    question_name="multi-label",
    value=["positive", "negative"],
    score=0.9,
    agent="model_name"
)

rg.Suggestion(
    question_name="rank",
    value=["1", "3", "2"],
    score=0.9,
    agent="model_name"
)

rg.Suggestion(
    question_name="rating",
    value=4,
    score=0.9,
    agent="model_name"
)

rg.Suggestion(
    question_name="span",
    value=[{"start": 0, "end": 9, "label": "MISC"}],
    score=0.9,
    agent="model_name"
)

rg.Suggestion(
    question_name="text",
    value="value",
    score=0.9,
    agent="model_name"
)

---

Suggestion

Bases: Resource

Class for interacting with Argilla Suggestions. Suggestions are typically model predictions for records. Suggestions are rendered in the user interfaces as 'hints' or 'suggestions' for the user to review and accept or reject.

Attributes:

Name	Type	Description
question_name	str	The name of the question that the suggestion is for.
value	str	The value of the suggestion
score	float	The score of the suggestion. For example, the probability of the model prediction.
agent	str	The agent that created the suggestion. For example, the model name.
type	str	The type of suggestion, either 'model' or 'human'.

Source code in src/argilla/suggestions.py

class Suggestion(Resource):
    """Class for interacting with Argilla Suggestions. Suggestions are typically model predictions for records.
    Suggestions are rendered in the user interfaces as 'hints' or 'suggestions' for the user to review and accept or reject.

    Attributes:
        question_name (str): The name of the question that the suggestion is for.
        value (str): The value of the suggestion
        score (float): The score of the suggestion. For example, the probability of the model prediction.
        agent (str): The agent that created the suggestion. For example, the model name.
        type (str): The type of suggestion, either 'model' or 'human'.
    """

    _model: SuggestionModel

    def __init__(
        self,
        question_name: str,
        value: Any,
        score: Union[float, List[float], None] = None,
        agent: Optional[str] = None,
        type: Optional[Literal["model", "human"]] = None,
        _record: Optional["Record"] = None,
    ) -> None:
        super().__init__()

        if question_name is None:
            raise ValueError("question_name is required")
        if value is None:
            raise ValueError("value is required")

        self.record = _record
        self._model = SuggestionModel(
            question_name=question_name,
            value=value,
            type=type,
            score=score,
            agent=agent,
        )

    ##############################
    # Properties
    ##############################

    @property
    def value(self) -> Any:
        """The value of the suggestion."""
        return self._model.value

    @property
    def question_name(self) -> Optional[str]:
        """The name of the question that the suggestion is for."""
        return self._model.question_name

    @question_name.setter
    def question_name(self, value: str) -> None:
        self._model.question_name = value

    @property
    def type(self) -> Optional[Literal["model", "human"]]:
        """The type of suggestion, either 'model' or 'human'."""
        return self._model.type

    @property
    def score(self) -> Optional[Union[float, List[float]]]:
        """The score of the suggestion."""
        return self._model.score

    @score.setter
    def score(self, value: float) -> None:
        self._model.score = value

    @property
    def agent(self) -> Optional[str]:
        """The agent that created the suggestion."""
        return self._model.agent

    @agent.setter
    def agent(self, value: str) -> None:
        self._model.agent = value

    @classmethod
    def from_model(cls, model: SuggestionModel, dataset: "Dataset") -> "Suggestion":
        question = dataset.settings.questions[model.question_id]
        model.question_name = question.name
        model.value = cls.__from_model_value(model.value, question)

        instance = cls(question.name, model.value)
        instance._model = model

        return instance

    def api_model(self) -> SuggestionModel:
        if self.record is None or self.record.dataset is None:
            return self._model

        question = self.record.dataset.settings.questions[self.question_name]
        if question:
            return SuggestionModel(
                value=self.__to_model_value(self.value, question),
                question_name=None if not question else question.name,
                question_id=None if not question else question.id,
                type=self._model.type,
                score=self._model.score,
                agent=self._model.agent,
                id=self._model.id,
            )
        else:
            raise RecordSuggestionsError(
                f"Record suggestion is invalid because question with name={self.question_name} does not exist in the dataset ({self.record.dataset.name}). Available questions are: {list(self.record.dataset.settings.questions._properties_by_name.keys())}"
            )

    @classmethod
    def __to_model_value(cls, value: Any, question: "QuestionType") -> Any:
        if isinstance(question, RankingQuestion):
            return cls.__ranking_to_model_value(value)
        return value

    @classmethod
    def __from_model_value(cls, value: Any, question: "QuestionType") -> Any:
        if isinstance(question, RankingQuestion):
            return cls.__ranking_from_model_value(value)
        return value

    @classmethod
    def __ranking_from_model_value(cls, value: List[Dict[str, Any]]) -> List[str]:
        return [v["value"] for v in value]

    @classmethod
    def __ranking_to_model_value(cls, value: List[str]) -> List[Dict[str, str]]:
        return [{"value": str(v)} for v in value]

value: Any property

The value of the suggestion.

question_name: Optional[str] property writable

The name of the question that the suggestion is for.

type: Optional[Literal['model', 'human']] property

The type of suggestion, either 'model' or 'human'.

score: Optional[Union[float, List[float]]] property writable

The score of the suggestion.

agent: Optional[str] property writable

The agent that created the suggestion.