Evaluation from 2024-07-05 06:55:06

This report was generated by DevQualityEval benchmark in version 0.5.0.

REMARK: gemma-2-9b-it and gemma-2-27-it were originally evaluated together with the results then being split into separate folders. Therefore some logs might contain entries from "the other" gemma model.

Results

Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.

The results of all models have been divided into the following categories:

• category unknown: Models in this category could not be categorized.
• response error: Models in this category encountered an error.
• no code: Models in this category produced no code.
• invalid code: Models in this category produced invalid code.
• executable code: Models in this category produced executable code.
• statement coverage reached: Models in this category produced code that reached full statement coverage.
• no excess response: Models in this category did not respond with more content than requested.

The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.

Result category "category unknown"

Models in this category could not be categorized.

• custom-nvidia/google/gemma-2-27b-it
• custom-nvidia/google/gemma-2-9b-it