TIPC Linux端Benchmark测试文档

该文档为Benchmark测试说明，Benchmark预测功能测试的主程序为benchmark_train.sh，用于验证监控模型训练的性能。

1. 测试流程

1.1 准备数据和环境安装

运行test_tipc/prepare.sh，完成训练数据准备和安装环境流程。

shell

# 运行格式：bash test_tipc/prepare.sh  train_benchmark.txt  mode
bash test_tipc/prepare.sh test_tipc/configs/det_mv3_db_v2_0/train_infer_python.txt benchmark_train

1.2 功能测试

执行test_tipc/benchmark_train.sh，完成模型训练和日志解析

shell

# 运行格式：bash test_tipc/benchmark_train.sh train_benchmark.txt mode
bash test_tipc/benchmark_train.sh test_tipc/configs/det_mv3_db_v2_0/train_infer_python.txt benchmark_train

test_tipc/benchmark_train.sh支持根据传入的第三个参数实现只运行某一个训练配置，如下：

shell

# 运行格式：bash test_tipc/benchmark_train.sh train_benchmark.txt mode
bash test_tipc/benchmark_train.sh test_tipc/configs/det_mv3_db_v2_0/train_infer_python.txt benchmark_train  dynamic_bs8_fp32_DP_N1C1

dynamic_bs8_fp32_DP_N1C1为test_tipc/benchmark_train.sh传入的参数，格式如下： ${modeltype}_${batch_size}_${fp_item}_${run_mode}_${device_num} 包含的信息有：模型类型、batchsize大小、训练精度如fp32,fp16等、分布式运行模式以及分布式训练使用的机器信息如单机单卡（N1C1）。

2. 日志输出

运行后将保存模型的训练日志和解析日志，使用 test_tipc/configs/det_mv3_db_v2_0/train_infer_python.txt 参数文件的训练日志解析结果是：

{"model_branch": "dygaph", "model_commit": "7c39a1996b19087737c05d883fd346d2f39dbcc0", "model_name": "det_mv3_db_v2_0_bs8_fp32_SingleP_DP", "batch_size": 8, "fp_item": "fp32", "run_process_type": "SingleP", "run_mode": "DP", "convergence_value": "5.413110", "convergence_key": "loss:", "ips": 19.333, "speed_unit": "samples/s", "device_num": "N1C1", "model_run_time": "0", "frame_commit": "8cc09552473b842c651ead3b9848d41827a3dbab", "frame_version": "0.0.0"}

训练日志和日志解析结果保存在benchmark_log目录下，文件组织格式如下：

train_log/
├── index
│   ├── PaddleOCR_det_mv3_db_v2_0_bs8_fp32_SingleP_DP_N1C1_speed
│   └── PaddleOCR_det_mv3_db_v2_0_bs8_fp32_SingleP_DP_N1C4_speed
├── profiling_log
│   └── PaddleOCR_det_mv3_db_v2_0_bs8_fp32_SingleP_DP_N1C1_profiling
└── train_log
    ├── PaddleOCR_det_mv3_db_v2_0_bs8_fp32_SingleP_DP_N1C1_log
    └── PaddleOCR_det_mv3_db_v2_0_bs8_fp32_SingleP_DP_N1C4_log

3. 各模型单卡性能数据一览

*注：本节中的速度指标均使用单卡（1块Nvidia V100 16G GPU）测得。通常情况下。

模型名称	配置文件	大数据集 float32 fps	小数据集 float32 fps	diff	大数据集 float16 fps	小数据集 float16 fps	diff	大数据集大小	小数据集大小
ch_ppocr_mobile_v2.0_det	config	53.836	53.343 / 53.914 / 52.785	0.020940758	45.574	45.57 / 46.292 / 46.213	0.015596647	10,000	2,000
ch_ppocr_mobile_v2.0_rec	config	2083.311	2043.194 / 2066.372 / 2093.317	0.023944295	2153.261	2167.561 / 2165.726 / 2155.614	0.005511725	600,000	160,000
ch_ppocr_server_v2.0_det	config	20.716	20.739 / 20.807 / 20.755	0.003268131	20.592	20.498 / 20.993 / 20.75	0.023579288	10,000	2,000
ch_ppocr_server_v2.0_rec	config	528.56	528.386 / 528.991 / 528.391	0.001143687	1189.788	1190.007 / 1176.332 / 1192.084	0.013213834	600,000	160,000
ch_PP-OCRv2_det	config	13.87	13.386 / 13.529 / 13.428	0.010569887	17.847	17.746 / 17.908 / 17.96	0.011915367	10,000	2,000
ch_PP-OCRv2_rec	config	109.248	106.32 / 106.318 / 108.587	0.020895687	117.491	117.62 / 117.757 / 117.726	0.001163413	140,000	40,000
det_mv3_db_v2.0	config	61.802	62.078 / 61.802 / 62.008	0.00444602	82.947	84.294 / 84.457 / 84.005	0.005351836	10,000	2,000
det_r50_vd_db_v2.0	config	29.955	29.092 / 29.31 / 28.844	0.015899011	51.097	50.367 / 50.879 / 50.227	0.012814717	10,000	2,000
det_r50_vd_east_v2.0	config	42.485	42.624 / 42.663 / 42.561	0.00239083	67.61	67.825/ 68.299/ 68.51	0.00999854	10,000	2,000
det_r50_vd_pse_v2.0	config	16.455	16.517 / 16.555 / 16.353	0.012201752	27.02	27.288 / 27.152 / 27.408	0.009340339	10,000	2,000
rec_mv3_none_bilstm_ctc_v2.0	config	2288.358	2291.906 / 2293.725 / 2290.05	0.001602197	2336.17	2327.042 / 2328.093 / 2344.915	0.007622025	600,000	160,000
layoutxlm_ser	config	18.001	18.114 / 18.107 / 18.307	0.010924783	21.982	21.507 / 21.116 / 21.406	0.018180127	1490	1490
PP-Structure-table	config	14.151	14.077 / 14.23 / 14.25	0.012140351	16.285	16.595 / 16.878 / 16.531	0.020559308	20,000	5,000
det_r50_dcn_fce_ctw_v2.0	config	14.057	14.029 / 14.02 / 14.014	0.001069214	18.298	18.411 / 18.376 / 18.331	0.004345228	10,000	2,000
ch_PP-OCRv3_det	config	8.622	8.431 / 8.423 / 8.479	0.006604552	14.203	14.346 14.468 14.23	0.016450097	10,000	2,000
PP-OCRv3_mobile_rec	config	90.239	90.077 / 91.513 / 91.325	0.01569176				160,000	40,000