String Evaluators

📄️ Criteria Evaluation

In scenarios where you wish to assess a model's output using a specific rubric or criteria set, the criteria evaluator proves to be a handy tool. It allows you to verify if an LLM or Chain's output complies with a defined set of criteria.

📄️ Custom String Evaluator

You can make your own custom string evaluators by inheriting from the StringEvaluator class and implementing the evaluatestrings (and aevaluatestrings for async support) methods.

📄️ Embedding Distance

To measure semantic similarity (or dissimilarity) between a prediction and a reference label string, you could use a vector vector distance metric the two embedded representations using the embeddingdistance evaluator.[1]

📄️ String Distance

One of the simplest ways to compare an LLM or chain's string output against a reference label is by using string distance measurements such as Levenshtein or postfix distance. This can be used alongside approximate/fuzzy matching criteria for very basic unit testing.