qwencoder-eval/instruct/eval-dev-quality/docs/reports/v0.4.0/4/README.md
This report was generated by DevQualityEval benchmark in version 0.4.0.
Keep in mind that LLMs are nondeterministic. The following results just reflect a current snapshot.
The results of all models have been divided into the following categories:
The following sections list all models with their categories. The complete log of the evaluation with all outputs can be found here. Detailed scoring can be found here.
Models in this category encountered an error.
openrouter/jebcarter/psyfighter-13bopenrouter/anthropic/claude-instant-1openrouter/anthropic/claude-instant-1:betaopenrouter/nousresearch/nous-hermes-2-vision-7bopenrouter/haotian-liu/llava-13bopenrouter/jondurbin/bagel-34bopenrouter/anthropic/claude-2openrouter/anthropic/claude-2:betaModels in this category produced no code.
openrouter/gryphe/mythomist-7bopenrouter/meta-llama/llama-3-8b-instruct:extendedopenrouter/undi95/toppy-m-7b:nitroopenrouter/mancer/weaveropenrouter/nousresearch/nous-capybara-7b:freeopenrouter/anthropic/claude-instant-1.2openrouter/google/gemini-proopenrouter/anthropic/claude-1openrouter/gryphe/mythomax-l2-13b:nitroopenrouter/gryphe/mythomax-l2-13bopenrouter/lynn/soliloquy-l3openrouter/gryphe/mythomist-7b:freeopenrouter/neversleep/noromaid-20bopenrouter/openai/gpt-3.5-turbo-instructopenrouter/koboldai/psyfighter-13b-2openrouter/togethercomputer/stripedhyena-hessian-7bopenrouter/pygmalionai/mythalion-13bopenrouter/mistralai/mistral-tinyopenrouter/undi95/toppy-m-7bopenrouter/undi95/remm-slerp-l2-13bopenrouter/anthropic/claude-instant-1.0openrouter/intel/neural-chat-7bopenrouter/undi95/toppy-m-7b:freeopenrouter/neversleep/noromaid-mixtral-8x7b-instructopenrouter/undi95/remm-slerp-l2-13b:extendedModels in this category produced invalid code.
openrouter/mistralai/mistral-largeopenrouter/togethercomputer/stripedhyena-nous-7bopenrouter/gryphe/mythomax-l2-13b:extendedopenrouter/openai/gpt-3.5-turbo-0613openrouter/open-orca/mistral-7b-openorcaopenrouter/jondurbin/airoboros-l2-70bopenrouter/google/gemma-7b-it:nitroopenrouter/meta-llama/llama-3-8b-instructopenrouter/anthropic/claude-2.0openrouter/nousresearch/nous-hermes-2-mixtral-8x7b-sftopenrouter/google/palm-2-codechat-bison-32kopenrouter/google/gemini-pro-visionopenrouter/google/palm-2-codechat-bisonopenrouter/openai/gpt-3.5-turbo-16kopenrouter/perplexity/pplx-7b-onlineopenrouter/01-ai/yi-34bopenrouter/cognitivecomputations/dolphin-mixtral-8x7bopenrouter/openrouter/cinematika-7b:freeopenrouter/alpindale/goliath-120bopenrouter/teknium/openhermes-2.5-mistral-7bopenrouter/google/gemma-7b-itopenrouter/mistralai/mixtral-8x7b-instructopenrouter/meta-llama/llama-3-8b-instruct:nitroopenrouter/fireworks/firellava-13bopenrouter/mistralai/mistral-7b-instruct:nitroopenrouter/cohere/command-ropenrouter/nousresearch/nous-capybara-7bopenrouter/01-ai/yi-6bopenrouter/mistralai/mistral-7b-instruct:freeopenrouter/openchat/openchat-7bopenrouter/xwin-lm/xwin-lm-70bopenrouter/openchat/openchat-7b:freeopenrouter/perplexity/sonar-small-onlineopenrouter/phind/phind-codellama-34bopenrouter/mistralai/mixtral-8x22b-instructopenrouter/teknium/openhermes-2-mistral-7bopenrouter/huggingfaceh4/zephyr-7b-beta:freeopenrouter/microsoft/wizardlm-2-7bopenrouter/cohere/commandopenrouter/openrouter/cinematika-7bopenrouter/rwkv/rwkv-5-world-3bopenrouter/cohere/command-r-plusopenrouter/lizpreciatior/lzlv-70b-fp16-hfopenrouter/austism/chronos-hermes-13bopenrouter/mistralai/mixtral-8x7bopenrouter/perplexity/pplx-70b-onlineopenrouter/meta-llama/codellama-34b-instructopenrouter/nousresearch/nous-hermes-2-mistral-7b-dpoopenrouter/huggingfaceh4/zephyr-7b-betaopenrouter/meta-llama/llama-2-13b-chatopenrouter/nousresearch/nous-hermes-llama2-13bopenrouter/anthropic/claude-1.2openrouter/sao10k/fimbulvetr-11b-v2openrouter/nousresearch/nous-hermes-yi-34bopenrouter/google/palm-2-chat-bison-32kopenrouter/perplexity/sonar-medium-chatopenrouter/mistralai/mistral-smallopenrouter/mistralai/mixtral-8x22bopenrouter/openai/gpt-4-32kopenrouter/anthropic/claude-3-haiku:betaopenrouter/recursal/rwkv-5-3b-ai-townopenrouter/anthropic/claude-2.0:betaopenrouter/perplexity/sonar-medium-onlineopenrouter/nousresearch/nous-capybara-34bopenrouter/recursal/eagle-7bopenrouter/sophosympatheia/midnight-rose-70bopenrouter/huggingfaceh4/zephyr-orpo-141b-a35bopenrouter/mistralai/mixtral-8x7b-instruct:nitroopenrouter/openai/gpt-3.5-turboModels in this category produced executable code.
openrouter/codellama/codellama-70b-instructopenrouter/perplexity/pplx-7b-chatopenrouter/google/gemini-pro-1.5openrouter/perplexity/sonar-small-chatModels in this category produced code that reached full statement coverage.
openrouter/microsoft/wizardlm-2-8x22b:nitroopenrouter/anthropic/claude-3-sonnet:betaopenrouter/anthropic/claude-2.1openrouter/anthropic/claude-3-opusopenrouter/mistralai/mistral-mediumopenrouter/anthropic/claude-3-opus:betaopenrouter/openai/gpt-3.5-turbo-1106openrouter/openai/gpt-4-1106-previewopenrouter/01-ai/yi-34b-chatopenrouter/meta-llama/llama-2-70b-chat:nitroopenrouter/openai/gpt-3.5-turbo-0301openrouter/microsoft/wizardlm-2-8x22bopenrouter/anthropic/claude-3-sonnetopenrouter/nousresearch/nous-hermes-2-mixtral-8x7b-dpoopenrouter/anthropic/claude-2.1:betaopenrouter/perplexity/pplx-70b-chatopenrouter/anthropic/claude-instant-1.1openrouter/openai/gpt-4-32k-0314openrouter/anthropic/claude-3-haikuopenrouter/meta-llama/llama-2-70b-chatModels in this category did not respond with more content than requested.
openrouter/mistralai/mistral-7b-instructopenrouter/openai/gpt-4-vision-previewopenrouter/databricks/dbrx-instructopenrouter/openai/gpt-4-turboopenrouter/openai/gpt-4-0314openrouter/openai/gpt-3.5-turbo-0125openrouter/openai/gpt-4-turbo-previewsymflower/symbolic-executionopenrouter/meta-llama/llama-3-70b-instructopenrouter/google/gemma-7b-it:freeopenrouter/openai/gpt-4openrouter/openrouter/autoopenrouter/meta-llama/llama-3-70b-instruct:nitroopenrouter/google/palm-2-chat-bison