McEval: Massively Multilingual Code Evaluation

Repository for paper "McEval: Massively Multilingual Code Evaluation"

Data

Dataset	Download
McEval Evaluation Dataset	🤗 HuggingFace
McEval-Instruct	🤗 HuggingFace

</div>

Usage

Environment

Runtime environments for different programming languages could be found in Environments

We recommend using Docker for evaluation, we have created a Docker image with all the necessary environments pre-installed.

Directly pull the image from Docker Hub or Aliyun Docker Hub:

bash

# Docker hub:
docker pull multilingualnlp/mceval

# Aliyun docker hub:
docker pull registry.cn-hangzhou.aliyuncs.com/mceval/mceval:v1

docker run -it -d --restart=always --name mceval_dev --workdir  / <image-name>  /bin/bash
docker attach mceval_dev

Inference

We provide some model inference codes, including torch and vllm implementations.

Inference with torch

Take the evaluation generation task as an example.

bash

cd inference 
bash scripts/inference_torch.sh

Inference with vLLM(recommended)

Take the evaluation generation task as an example.

bash

cd inference 
bash scripts/run_generation_vllm.sh

Evaluation

Data Format

🛎️ Please prepare the inference results of the model in the following format and use them for the next evaluation step.

(1) Folder Structure Place the data in the following folder structure, each file corresponds to the test results of each language.

bash

\evaluate_model_name 
  - CPP.jsonl
  - Python.jsonl
  - Java.jsonl
  ...

You can use script split_result.py to split inference results.

bash

python split_result --split_file <inference_result> --save_dir <save_dir>

(2) File Format Each line in the file for each test language has the following format. The raw_generation field is the generated code. More examples can be found in Evualute Data Format Examples

bash

{
    "task_id": "Lang/1",
    "prompt": "",
    "canonical_solution": "",
    "test": "",
    "entry_point": "",
    "signature": "",
    "docstring": "",
    "instruction": "",
    "raw_generation": ["<Generated Code>"]
}

Evaluate Generation Task

Take the evaluation generation task as an example.

bash

cd eval 
bash scripts/eval_generation.sh