qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/2/README.md
This report was generated by DevQualityEval benchmark in version 0.4.0.
Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.
The results of all models have been divided into the following categories:
The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.
Models in this category encountered an error.
openrouter/anthropic/claude-2:betaopenrouter/anthropic/claude-2openrouter/anthropic/claude-instant-1:betaopenrouter/anthropic/claude-instant-1openrouter/haotian-liu/llava-13bopenrouter/jebcarter/psyfighter-13bopenrouter/jondurbin/bagel-34bopenrouter/nousresearch/nous-hermes-2-vision-7bopenrouter/openchat/openchat-7b:freeopenrouter/perplexity/pplx-7b-onlineopenrouter/recursal/rwkv-5-3b-ai-townopenrouter/undi95/remm-slerp-l2-13bModels in this category produced an empty response.
openrouter/01-ai/yi-34bopenrouter/databricks/dbrx-instructopenrouter/mistralai/mixtral-8x22bModels in this category produced no code.
openrouter/anthropic/claude-1.2openrouter/anthropic/claude-1openrouter/austism/chronos-hermes-13bopenrouter/codellama/codellama-70b-instructopenrouter/gryphe/mythomax-l2-13b:extendedopenrouter/gryphe/mythomax-l2-13bopenrouter/gryphe/mythomist-7b:freeopenrouter/gryphe/mythomist-7bopenrouter/huggingfaceh4/zephyr-orpo-141b-a35bopenrouter/intel/neural-chat-7bopenrouter/jondurbin/airoboros-l2-70bopenrouter/koboldai/psyfighter-13b-2openrouter/lynn/soliloquy-l3openrouter/mancer/weaveropenrouter/meta-llama/llama-3-8b-instruct:extendedopenrouter/mistralai/mistral-tinyopenrouter/mistralai/mixtral-8x7bopenrouter/neversleep/noromaid-20bopenrouter/neversleep/noromaid-mixtral-8x7b-instructopenrouter/nousresearch/nous-capybara-7b:freeopenrouter/open-orca/mistral-7b-openorcaopenrouter/openai/gpt-3.5-turbo-instructopenrouter/pygmalionai/mythalion-13bopenrouter/togethercomputer/stripedhyena-hessian-7bopenrouter/undi95/remm-slerp-l2-13b:extendedopenrouter/undi95/toppy-m-7b:nitroModels in this category produced invalid code.
openrouter/01-ai/yi-34b-chatopenrouter/01-ai/yi-6bopenrouter/alpindale/goliath-120bopenrouter/anthropic/claude-2.0:betaopenrouter/anthropic/claude-3-haiku:betaopenrouter/anthropic/claude-instant-1.1openrouter/anthropic/claude-instant-1.2openrouter/cognitivecomputations/dolphin-mixtral-8x7bopenrouter/cohere/command-r-plusopenrouter/cohere/commandopenrouter/google/gemini-pro-visionopenrouter/google/gemini-proopenrouter/google/gemma-7b-it:freeopenrouter/google/gemma-7b-it:nitroopenrouter/google/gemma-7b-itopenrouter/google/palm-2-chat-bison-32kopenrouter/google/palm-2-chat-bisonopenrouter/google/palm-2-codechat-bison-32kopenrouter/google/palm-2-codechat-bisonopenrouter/gryphe/mythomax-l2-13b:nitroopenrouter/huggingfaceh4/zephyr-7b-beta:freeopenrouter/huggingfaceh4/zephyr-7b-betaopenrouter/lizpreciatior/lzlv-70b-fp16-hfopenrouter/meta-llama/llama-2-70b-chatopenrouter/meta-llama/llama-3-70b-instruct:nitroopenrouter/meta-llama/llama-3-8b-instruct:nitroopenrouter/meta-llama/llama-3-8b-instructopenrouter/microsoft/wizardlm-2-7bopenrouter/mistralai/mistral-7b-instruct:freeopenrouter/mistralai/mistral-7b-instruct:nitroopenrouter/mistralai/mistral-7b-instructopenrouter/mistralai/mistral-largeopenrouter/mistralai/mistral-smallopenrouter/mistralai/mixtral-8x22b-instructopenrouter/nousresearch/nous-capybara-34bopenrouter/nousresearch/nous-capybara-7bopenrouter/nousresearch/nous-hermes-2-mistral-7b-dpoopenrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpoopenrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sftopenrouter/nousresearch/nous-hermes-llama2-13bopenrouter/nousresearch/nous-hermes-yi-34bopenrouter/openai/gpt-3.5-turbo-16kopenrouter/openchat/openchat-7bopenrouter/openrouter/cinematika-7b:freeopenrouter/openrouter/cinematika-7bopenrouter/perplexity/pplx-70b-onlineopenrouter/perplexity/pplx-7b-chatopenrouter/perplexity/sonar-medium-chatopenrouter/perplexity/sonar-medium-onlineopenrouter/perplexity/sonar-small-chatopenrouter/perplexity/sonar-small-onlineopenrouter/recursal/eagle-7bopenrouter/rwkv/rwkv-5-world-3bopenrouter/sao10k/fimbulvetr-11b-v2openrouter/sophosympatheia/midnight-rose-70bopenrouter/teknium/openhermes-2-mistral-7bopenrouter/teknium/openhermes-2.5-mistral-7bopenrouter/togethercomputer/stripedhyena-nous-7bopenrouter/undi95/toppy-m-7b:freeopenrouter/undi95/toppy-m-7bopenrouter/xwin-lm/xwin-lm-70bModels in this category produced code that reached full statement coverage.
openrouter/anthropic/claude-2.0openrouter/anthropic/claude-2.1:betaopenrouter/anthropic/claude-2.1openrouter/anthropic/claude-3-haikuopenrouter/anthropic/claude-3-opus:betaopenrouter/anthropic/claude-3-opusopenrouter/anthropic/claude-3-sonnet:betaopenrouter/anthropic/claude-3-sonnetopenrouter/anthropic/claude-instant-1.0openrouter/meta-llama/codellama-34b-instructopenrouter/meta-llama/llama-2-13b-chatopenrouter/meta-llama/llama-2-70b-chat:nitroopenrouter/microsoft/wizardlm-2-8x22b:nitroopenrouter/microsoft/wizardlm-2-8x22bopenrouter/mistralai/mistral-mediumopenrouter/mistralai/mixtral-8x7b-instruct:nitroopenrouter/mistralai/mixtral-8x7b-instructopenrouter/openai/gpt-3.5-turbo-0301openrouter/openai/gpt-3.5-turbo-1106openrouter/openai/gpt-4-1106-previewopenrouter/openai/gpt-4-32kopenrouter/openai/gpt-4openrouter/perplexity/pplx-70b-chatopenrouter/phind/phind-codellama-34bModels in this category did not respond with more content than requested.
openrouter/cohere/command-ropenrouter/google/gemini-pro-1.5openrouter/meta-llama/llama-3-70b-instructopenrouter/openai/gpt-3.5-turbo-0125openrouter/openai/gpt-3.5-turbo-0613openrouter/openai/gpt-3.5-turboopenrouter/openai/gpt-4-0314openrouter/openai/gpt-4-32k-0314openrouter/openai/gpt-4-turbo-previewopenrouter/openai/gpt-4-turboopenrouter/openai/gpt-4-vision-previewopenrouter/openrouter/autosymflower/symbolic-execution