问题描述
线上碰到个问题，某个接口调用时间特别长，线上调用接口直接报gateway time out
分析处理
1、先关闭该功能 （该功能是非核心功能）
2、本地起服务连环境排查，发现本地正常。并且线上其他接口正常，只有这个接口异常。跳过gateway直接调用服务发现可以调通但是耗时140s。排除是gateway的问题，定位是服务接口问题。
3、排查代码
public List<OaClassScheduleChartVO> oaClassScheduleChart(OaClassScheduleQuery query, UserBaseDTO currentUser) {
log.info("OaClassScheduleService->oaClassSchedulChart query:{}", query);
query.setOrderBy("schedule_date desc");
List<OaClassScheduleChartVO> oaClassScheduleChartVOS = new ArrayList<>();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
List<OaClassSchedule> oaClassSchedules = this.queryList(query);
Map<String, OaClassScheduleChartVO> oaClassSchedulePictureVOMap = new HashMap<>();
List<String> dates = com.wpg.waterwork.common.util.DateUtil.getMonthFullDay(Integer.parseInt(query.getQueryDate().substring(0, 4)), Integer.parseInt(query.getQueryDate().substring(5, 7)));
//按名称分组存入map
oaClassSchedules.forEach(oaClassSchedule -> {
List<ClassGroupAndUserVO> classGroupAndUserVOS = JSON.parseArray(oaClassSchedule.getClassGroupUserVal(), ClassGroupAndUserVO.class);
log.info("OaClassScheduleService->oaClassSchedulChart->oaClassSchedules.forEach classGroupAndUserVOS:{}", JSON.toJSONString(classGroupAndUserVOS));
}
});
//返回数据
oaClassSchedulePictureVOMap.forEach((userId, pictureVO) -> {
//根据用户id和班组名称查询
boolean have = (StringUtils.isBlank(query.getUserId()) || userId.equals(query.getUserId())) &&
(StringUtils.isBlank(query.getClassGroupName()) || pictureVO.getClassGroupName().contains(query.getClassGroupName()));
if (have) {
//按班次顺序排序，便于前端处理
pictureVO.getDateClassList().forEach(dateClass -> {
if (dateClass.getClassValVOList() != null) {
dateClass.setClassValVOList(dateClass.getClassValVOList().stream().sorted(Comparator.comparing(ClassValVO::getSort)).collect(Collectors.toList()));
}
});
oaClassScheduleChartVOS.add(pictureVO);
}
});
log.info("OaClassScheduleService->oaClassSchedulChart->oaClassSchedulePictureVOMap.forEach oaClassSchedulePictureVOMap:{}", oaClassSchedulePictureVOMap);
return oaClassScheduleChartVOS;
}
代码非常简单就没有想太多，也没有外部接口调用，只看到查询数据库和对数据操作。直接考虑是线上内存问题
4、查看服务器内存（共64G）占用60%左右不算多，进入docker容器排查内存
jps
jcmd 1 VM.flags
查到堆空间256M,年轻代78M,from和to区各3.5M,新生代71M
查看从数据查到的数据
从数据库查到的对象特别大有2.6M,考虑是大对象导致的问题
排查调用时的jvm情况
jps -l
jstat -gcutil 1 1000
S0 — Heap上的 Survivor space 0 区已使用空间的百分比
S1 — Heap上的 Survivor space 1 区已使用空间的百分比
E — Heap上的 Eden space 区已使用空间的百分比
O — Heap上的 Old space 区已使用空间的百分比
P — Perm space 区已使用空间的百分比
YGC — 从应用程序启动到采样时发生 Young GC 的次数
YGCT– 从应用程序启动到采样时 Young GC 所用的时间(单位秒)
FGC — 从应用程序启动到采样时发生 Full GC 的次数
FGCT– 从应用程序启动到采样时 Full GC 所用的时间(单位秒)
GCT — 从应用程序启动到采样时用于垃圾回收的总时间(单位秒)
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
53.75 0.00 66.96 79.73 90.31 86.66 153136 1508.284 30787 6199.465 7707.749
53.75 0.00 67.02 79.73 90.31 86.66 153136 1508.284 30787 6199.465 7707.749
0.00 98.39 2.19 87.75 90.31 86.66 153147 1508.391 30787 6199.465 7707.856
28.41 0.00 0.00 90.10 90.31 86.66 153170 1508.587 30787 6199.465 7708.052
0.00 98.94 81.74 85.13 90.31 86.65 153188 1508.740 30788 6199.661 7708.401
0.00 6.99 72.82 93.67 90.31 86.65 153213 1508.999 30788 6199.661 7708.660
0.00 17.23 0.00 97.21 90.31 86.65 153241 1509.245 30788 6199.661 7708.906
70.84 0.00 0.00 97.08 90.31 86.65 153259 1509.406 30790 6199.826 7709.231
5.57 45.17 97.56 90.30 90.31 86.65 153282 1509.640 30790 6200.007 7709.646
0.00 0.00 100.00 98.95 90.31 86.65 153292 1509.725 30793 6200.359 7710.084
0.00 0.00 98.44 99.58 90.31 86.65 153294 1509.759 30797 6201.100 7710.859
0.00 97.51 99.48 98.28 90.31 86.65 153296 1509.776 30801 6201.884 7711.661
0.00 0.00 98.71 99.58 90.31 86.65 153297 1509.784 30805 6202.638 7712.422
0.00 0.00 98.15 99.58 90.31 86.65 153297 1509.784 30809 6203.393 7713.177
0.00 0.00 13.59 83.53 90.31 86.65 153300 1509.811 30812 6204.198 7714.009
0.00 0.00 99.49 98.81 90.31 86.65 153301 1509.825 30816 6204.835 7714.659
0.00 40.83 98.37 99.59 90.31 86.65 153304 1509.866 30820 6205.574 7715.440
0.00 0.92 0.00 94.98 90.31 86.65 153310 1509.941 30823 6206.304 7716.245
0.00 0.00 0.00 85.23 90.31 86.65 153330 1510.130 30824 6206.523 7716.653
0.00 36.15 0.00 93.73 90.31 86.65 153355 1510.363 30824 6206.523 7716.886
0.00 40.50 0.00 88.64 90.31 86.65 153372 1510.548 30825 6206.733 7717.281
70.34 0.00 0.00 88.65 90.31 86.65 153391 1510.728 30826 6206.911 7717.639
0.00 8.70 0.00 90.35 90.31 86.65 153416 1510.964 30826 6206.911 7717.875
1.00 0.00 100.00 98.82 90.31 86.65 153437 1511.200 30827 6206.911 7718.110
0.00 52.46 50.21 85.29 90.31 86.65 153442 1511.242 30830 6207.645 7718.887
21.53 0.00 97.62 98.19 90.31 86.65 153447 1511.292 30834 6208.192 7719.485
0.00 0.00 32.20 85.29 90.31 86.65 153448 1511.317 30838 6209.003 7720.320
0.00 0.00 68.29 85.29 90.31 86.65 153449 1511.327 30842 6209.769 7721.096
89.31 0.00 94.45 98.44 90.31 86.65 153451 1511.362 30845 6210.355 7721.717
0.00 20.80 0.00 97.20 90.31 86.65 153456 1511.412 30848 6211.200 7722.612
0.00 0.00 2.67 83.60 90.31 86.65 153458 1511.430 30852 6211.941 7723.371
0.00 45.68 0.00 97.99 90.31 86.65 153466 1511.528 30855 6212.469 7723.997
34.73 0.00 0.00 88.32 90.31 86.65 153482 1511.683 30857 6212.874 7724.557
62.36 5.16 96.71 96.81 90.31 86.65 153504 1511.885 30858 6212.874 7724.759
0.00 21.65 98.46 90.36 90.31 86.65 153529 1512.106 30858 6213.052 7725.158
73.27 0.00 0.00 97.28 90.31 86.65 153553 1512.350 30858 6213.052 7725.402
0.00 37.09 53.81 90.35 90.31 86.65 153573 1512.522 30859 6213.242 7725.763
0.00 0.75 100.00 94.95 90.31 86.65 153593 1512.702 30860 6213.413 7726.116
76.46 93.71 98.18 98.84 90.31 86.65 153599 1512.771 30864 6213.962 7726.733
0.00 97.69 97.99 98.49 90.31 86.65 153600 1512.779 30868 6214.705 7727.485
0.00 98.78 90.35 90.72 90.31 86.65 153602 1512.799 30872 6215.449 7728.248
0.00 0.00 0.00 83.62 90.31 86.65 153605 1512.833 30875 6216.210 7729.043
0.00 0.00 97.90 98.79 90.31 86.65 153613 1512.922 30878 6216.610 7729.532
0.00 0.00 97.92 98.80 90.31 86.65 153613 1512.922 30882 6217.441 7730.363
0.00 0.00 97.25 99.64 90.31 86.65 153613 1512.922 30886 6218.287 7731.208
50.59 0.00 22.11 98.05 90.31 86.65 153621 1513.018 30889 6218.910 7731.928
0.00 38.80 0.00 81.64 90.31 86.65 153633 1513.151 30891 6219.465 7732.617
17.89 0.00 0.00 88.44 90.31 86.65 153662 1513.424 30891 6219.465 7732.890
0.00 5.57 0.00 91.86 90.31 86.65 153689 1513.650 30891 6219.465 7733.116
0.00 65.59 0.00 85.01 90.31 86.65 153709 1513.824 30892 6219.638 7733.462
0.00 98.73 0.00 96.85 90.31 86.65 153735 1514.052 30892 6219.638 7733.690
0.00 24.77 0.00 93.99 90.31 86.65 153754 1514.228 30893 6219.823 7734.051
0.00 38.29 67.46 86.72 90.31 86.65 153765 1514.316 30896 6220.376 7734.692
0.00 98.14 100.00 99.24 90.31 86.65 153774 1514.408 30899 6220.763 7735.170
22.51 68.29 100.00 98.95 90.31 86.65 153793 1514.602 30901 6221.131 7735.732
14.89 36.61 97.16 99.34 90.31 86.65 153812 1514.806 30903 6221.507 7736.313
0.00 13.84 98.90 99.47 90.31 86.65 153816 1514.877 30905 6222.251 7737.127
0.00 47.63 98.38 99.47 90.31 86.65 153822 1514.945 30909 6222.821 7737.766
0.00 39.96 0.00 98.62 90.31 86.65 153837 1515.088 30911 6223.401 7738.489
0.00 0.00 0.00 81.36 90.31 86.65 153855 1515.278 30913 6223.813 7739.091
0.00 16.92 0.00 92.57 90.31 86.65 153890 1515.626 30913 6223.813 7739.439
41.41 0.00 6.65 96.00 90.31 86.65 153915 1515.856 30913 6223.813 7739.669
71.23 0.00 0.00 85.09 90.31 86.65 153933 1516.024 30914 6224.028 7740.051
0.00 64.14 19.92 93.57 90.31 86.65 153958 1516.298 30914 6224.028 7740.325
0.00 95.82 51.30 83.42 90.31 86.65 153972 1516.429 30916 6224.395 7740.825
57.41 0.00 84.44 97.00 90.31 86.65 153980 1516.498 30919 6224.993 7741.491
0.00 56.23 75.79 97.20 90.31 86.65 153982 1516.523 30923 6225.714 7742.237
63.98 0.00 98.68 99.57 90.31 86.65 153986 1516.564 30927 6226.267 7742.830
0.00 0.00 18.69 83.54 90.31 86.65 153990 1516.600 30930 6227.009 7743.608
0.00 0.00 99.40 98.72 90.31 86.65 153995 1516.672 30934 6227.551 7744.222
0.00 0.00 0.00 85.20 90.31 86.65 154000 1516.717 30937 6228.296 7745.012
0.00 97.00 96.32 98.76 90.31 86.65 154013 1516.847 30939 6228.659 7745.506
0.00 26.48 48.61 91.58 90.31 86.65 154027 1517.001 30941 6229.022 7746.023
0.00 98.04 0.00 98.40 90.31 86.65 154049 1517.250 30942 6229.022 7746.272
3.86 0.00 82.27 86.85 90.31 86.65 154068 1517.440 30942 6229.235 7746.675
0.00 26.34 0.00 90.40 90.31 86.65 154093 1517.678 30942 6229.235 7746.913
65.59 0.00 0.00 94.12 90.31 86.65 154118 1517.910 30942 6229.235 7747.145
0.00 98.39 0.00 98.78 90.31 86.65 154138 1518.081 30943 6229.415 7747.495
0.00 98.93 97.82 98.80 90.31 86.65 154151 1518.211 30945 6229.599 7747.810
90.29 0.00 82.16 90.58 90.31 86.65 154154 1518.258 30949 6230.395 7748.653
95.25 72.81 97.51 97.46 90.31 86.65 154157 1518.306 30952 6230.977 7749.283
0.00 0.00 97.33 97.96 90.31 86.65 154157 1518.306 30957 6231.895 7750.201
0.00 97.34 99.56 99.01 90.31 86.65 154162 1518.347 30960 6232.451 7750.798
37.88 0.00 98.24 99.77 90.31 86.65 154169 1518.425 30963 6233.086 7751.511
85.23 66.06 93.56 98.79 90.31 86.65 154179 1518.528 30966 6233.626 7752.154
0.00 61.12 0.00 92.17 90.31 86.65 154200 1518.719 30966 6233.804 7752.523
19.80 0.00 0.00 95.76 90.31 86.65 154222 1518.949 30966 6233.804 7752.753
0.00 56.52 20.74 88.37 90.30 86.65 154240 1519.113 30967 6234.095 7753.208
41.50 0.00 0.00 84.99 90.30 86.65 154258 1519.322 30968 6234.296 7753.619
0.00 23.16 0.00 90.19 90.30 86.65 154279 1519.586 30968 6234.296 7753.882
76.18 0.00 62.22 90.43 90.30 86.65 154300 1519.867 30968 6234.296 7754.163
0.00 98.03 98.01 99.91 90.30 86.65 154305 1519.912 30972 6234.879 7754.792
0.00 0.00 0.00 80.13 90.30 86.65 154307 1519.936 30975 6235.647 7755.584
0.00 97.13 98.82 99.87 90.30 86.65 154315 1520.011 30978 6236.030 7756.041
0.00 0.00 98.47 99.46 90.30 86.65 154319 1520.053 30982 6236.793 7756.845
0.00 0.00 0.00 83.40 90.30 86.65 154321 1520.086 30985 6237.620 7757.707
0.00 0.00 99.77 98.62 90.30 86.65 154324 1520.122 30989 6238.181 7758.303
0.00 0.00 99.46 98.64 90.30 86.65 154324 1520.122 30994 6239.090 7759.212
0.00 0.00 97.99 99.47 90.30 86.65 154327 1520.159 30997 6239.679 7759.839
0.00 0.00 98.00 99.47 90.30 86.65 154327 1520.159 31001 6240.503 7760.663
13.17 0.00 0.00 80.06 90.30 86.65 154332 1520.216 31004 6241.282 7761.498
0.00 99.95 0.00 85.10 90.30 86.65 154349 1520.372 31006 6241.696 7762.067
0.00 17.89 0.00 86.79 90.30 86.65 154376 1520.743 31006 6241.696 7762.438
0.00 27.40 0.00 86.85 90.30 86.65 154400 1520.963 31006 6241.696 7762.658
2.08 74.58 97.07 97.01 90.30 86.65 154422 1521.188 31007 6241.696 7762.884
54.99 0.00 0.00 93.66 90.30 86.65 154444 1521.398 31007 6241.912 7763.310
0.00 2.53 0.00 98.27 90.30 86.65 154471 1521.634 31007 6241.912 7763.546
0.00 0.00 36.10 85.11 90.30 86.65 154481 1521.720 31010 6242.429 7764.149
71.37 18.08 98.30 99.40 90.30 86.65 154498 1521.886 31012 6242.618 7764.504
0.00 0.00 24.57 83.42 90.30 86.65 154510 1522.013 31014 6243.182 7765.195
81.06 98.70 93.97 99.14 90.30 86.65 154519 1522.109 31017 6243.597 7765.707
25.23 16.03 100.00 99.56 90.30 86.65 154542 1522.318 31018 6243.940 7766.258
0.00 97.53 0.00 90.19 90.30 86.65 154560 1522.492 31020 6244.296 7766.788
15.87 0.00 0.00 91.93 90.30 86.65 154589 1522.757 31020 6244.296 7767.053
0.00 37.35 0.00 95.32 90.30 86.65 154614 1522.996 31020 6244.296 7767.292
14.56 0.00 0.00 90.41 90.30 86.65 154635 1523.189 31021 6244.484 7767.672
46.73 0.00 18.87 93.84 90.30 86.65 154658 1523.475 31021 6244.484 7767.958
73.01 0.00 96.70 98.93 90.30 86.65 154671 1523.750 31022 6244.484 7768.233
0.00 0.00 0.00 80.12 90.30 86.65 154679 1523.826 31024 6245.109 7768.935
65.13 63.02 97.23 98.62 90.30 86.65 154688 1523.928 31027 6245.478 7769.405
0.00 0.00 98.87 99.46 90.30 86.65 154696 1524.033 31030 6246.069 7770.102
0.00 56.27 97.59 99.54 90.30 86.65 154698 1524.091 31033 6246.681 7770.772
36.43 77.00 94.89 98.63 90.30 86.65 154710 1524.203 31036 6247.253 7771.456
93.07 98.39 8.80 98.62 90.30 86.65 154716 1524.265 31039 6247.888 7772.153
41.68 0.00 47.40 98.65 90.30 86.65 154723 1524.330 31041 6248.468 7772.798
37.09 0.00 66.82 85.09 90.30 86.65 154738 1524.466 31043 6248.849 7773.315
0.00 60.86 0.00 93.59 90.30 86.65 154764 1524.707 31043 6248.849 7773.555
22.55 0.00 45.11 93.55 90.30 86.65 154783 1524.901 31044 6249.059 7773.960
1.30 73.53 96.24 96.97 90.30 86.65 154802 1525.099 31045 6249.059 7774.158
97.55 0.00 76.20 88.44 90.30 86.65 154810 1525.176 31045 6249.274 7774.449
97.55 0.00 76.32 88.44 90.30 86.65 154810 1525.176 31045 6249.274 7774.449
97.55 0.00 76.90 88.44 90.30 86.65 154810 1525.176 31045 6249.274 7774.449
97.55 0.00 76.96 88.44 90.30 86.65 154810 1525.176 31045 6249.274 7774.449
97.55 0.00 77.22 88.44 90.30 86.65 154810 1525.176 31045 6249.274 7774.449
可以看到老年代在疯狂的gc 用7774-7707，gc时间70s
这是不正常的，并且老年代长期占比90 即使gc之后也占比90左右，没有清理太多空间，调用完成之后开始下降。
5、在本地设置jvm参数和线上一致
-Xmx256m -Xms256m -Xmn78m -XX:SurvivorRatio=20
本地调用的jvm情况
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
58.41 0.00 84.56 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 84.57 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 84.58 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 86.61 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 86.72 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 86.77 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 86.80 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 86.80 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 86.81 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 86.82 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
58.41 0.00 86.83 26.21 93.96 91.76 1280 15.835 25 5.309 21.144
0.00 40.06 66.26 26.21 94.05 91.81 1281 15.844 25 5.309 21.153
1.49 0.00 27.64 26.84 94.05 91.81 1284 15.877 25 5.309 21.187
2.76 0.00 40.50 29.33 94.05 91.81 1286 15.908 25 5.309 21.217
4.65 0.00 77.50 31.92 94.05 91.81 1288 15.960 25 5.309 21.269
0.00 4.57 0.00 35.82 94.05 91.82 1291 15.988 25 5.309 21.297
0.00 98.38 80.18 35.82 94.05 91.82 1293 16.011 25 5.309 21.321
71.34 0.00 45.67 35.83 94.05 91.82 1296 16.046 25 5.309 21.355
0.00 69.84 22.24 35.83 94.05 91.82 1299 16.087 25 5.309 21.396
0.00 37.59 96.33 36.17 94.05 91.82 1301 16.104 25 5.309 21.414
1.09 0.00 43.51 39.61 94.05 91.82 1304 16.138 25 5.309 21.447
3.91 0.00 81.61 42.33 94.05 91.82 1306 16.165 25 5.309 21.474
0.00 3.67 7.38 45.88 94.05 91.82 1309 16.226 25 5.309 21.535
0.00 43.29 72.63 45.88 94.06 91.82 1311 16.255 25 5.309 21.565
87.92 0.00 38.35 45.89 94.06 91.82 1314 16.282 25 5.309 21.591
0.00 97.00 2.64 46.32 94.06 91.82 1317 16.337 25 5.309 21.646
0.00 93.98 64.06 46.54 94.06 91.82 1319 16.362 25 5.309 21.672
0.62 0.00 16.52 49.46 94.06 91.82 1322 16.408 25 5.309 21.718
0.99 0.00 34.07 51.86 94.06 91.82 1324 16.430 25 5.309 21.739
92.23 0.00 75.27 53.04 94.06 91.82 1326 16.471 25 5.309 21.780
0.00 97.35 35.03 53.47 94.06 91.82 1329 16.502 25 5.309 21.811
9.79 0.00 6.42 53.47 94.06 91.82 1332 16.540 25 5.309 21.849
26.21 0.00 76.94 53.69 94.06 91.82 1334 16.561 25 5.309 21.870
0.00 25.94 34.39 56.24 94.07 91.82 1337 16.594 25 5.309 21.904
0.00 0.78 94.30 57.01 94.07 91.82 1339 16.618 25 5.309 21.927
0.00 1.50 58.88 61.69 94.08 91.82 1343 16.650 25 5.309 21.959
10.65 0.00 7.98 61.70 94.08 91.82 1346 16.688 25 5.309 21.997
42.16 0.00 79.46 62.13 94.08 91.82 1348 16.714 25 5.309 22.023
0.00 24.11 37.41 62.35 94.08 91.82 1351 16.773 25 5.309 22.083
0.00 2.60 92.89 68.13 94.08 91.82 1353 16.800 25 5.309 22.109
0.00 1.20 32.21 73.67 94.08 91.82 1357 16.853 25 5.309 22.163
0.00 3.16 58.36 76.66 94.08 91.82 1359 16.876 25 5.309 22.186
70.59 0.00 13.87 79.99 94.08 91.82 1362 16.906 25 5.309 22.215
20.30 0.00 80.71 80.00 94.08 91.82 1364 16.935 25 5.309 22.244
0.00 23.86 44.47 80.00 94.08 91.82 1367 16.974 25 5.309 22.283
0.89 0.00 5.56 80.61 94.08 91.82 1370 16.999 25 5.309 22.308
1.74 0.00 58.97 83.29 94.08 91.82 1372 17.020 25 5.309 22.330
0.00 4.33 26.48 86.73 94.08 91.82 1375 17.075 25 5.309 22.384
0.00 1.84 88.47 88.72 94.08 91.82 1377 17.112 25 5.309 22.421
37.28 0.00 51.16 88.72 94.08 91.82 1380 17.148 25 5.309 22.457
0.00 9.51 3.29 88.72 94.08 91.82 1383 17.187 25 5.309 22.496
0.00 98.92 45.94 89.58 94.08 91.82 1385 17.242 25 5.309 22.551
0.00 4.83 90.43 92.60 94.08 91.82 1387 17.264 25 5.309 22.574
6.03 0.00 18.62 96.38 94.08 91.82 1390 17.294 25 5.309 22.603
0.00 84.79 78.89 98.79 94.08 91.82 1393 17.342 25 5.309 22.651
74.18 0.00 78.54 11.12 94.08 91.82 1394 17.350 26 5.523 22.873
0.00 58.74 51.49 11.34 94.08 91.82 1397 17.376 26 5.523 22.899
65.59 0.00 14.89 11.56 94.08 91.82 1400 17.410 26 5.523 22.933
0.74 0.00 58.64 15.73 94.08 91.82 1402 17.441 26 5.523 22.964
0.00 0.74 16.73 18.86 94.08 91.82 1405 17.462 26 5.523 22.985
0.00 5.53 96.17 20.50 94.08 91.82 1407 17.475 26 5.523 22.998
2.60 0.00 69.61 22.88 94.08 91.82 1410 17.495 26 5.523 23.018
0.00 34.18 19.42 23.10 94.08 91.82 1413 17.532 26 5.523 23.055
0.00 77.70 99.52 23.11 94.08 91.82 1415 17.553 26 5.523 23.076
27.64 0.00 55.97 23.13 94.08 91.82 1418 17.623 26 5.523 23.146
0.00 0.69 0.00 25.52 94.08 91.82 1421 17.677 26 5.523 23.200
0.00 1.84 30.09 27.28 94.08 91.82 1423 17.696 26 5.523 23.219
0.00 2.46 96.91 29.00 94.08 91.82 1426 17.715 26 5.523 23.238
39.77 0.00 28.65 29.87 94.08 91.82 1428 17.772 26 5.523 23.295
45.40 0.00 93.87 29.87 94.08 91.82 1430 17.791 26 5.523 23.314
0.00 90.83 43.78 30.51 94.08 91.82 1433 17.835 26 5.523 23.358
0.00 4.47 77.05 32.54 94.08 91.82 1435 17.856 26 5.523 23.379
98.80 0.00 25.08 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 28.14 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 28.57 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 29.64 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 29.66 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 29.66 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 29.67 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 29.68 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 29.69 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 29.70 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 29.89 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.10 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.14 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.16 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.22 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.38 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.38 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.57 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.58 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.59 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 32.88 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 34.99 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.03 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.07 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.10 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.11 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.11 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.13 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.13 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.14 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 35.15 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 37.23 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 37.43 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 37.45 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 37.47 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 37.47 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
98.80 0.00 83.89 34.37 94.08 91.82 1438 17.882 26 5.523 23.406
可以看到本地fullgc次数只有一次，调用时gc花费1s,最大的区别是
在本地初始老年代占比只有20%且gc一次就几乎已经全部清理干净，清理一次占比从98.79->11.12
0.00 84.79 78.89 98.79 94.08 91.82 1393 17.342 25 5.309 22.651
74.18 0.00 78.54 11.12 94.08 91.82 1394 17.350 26 5.523 22.873
在线上老年代频繁清理且清理出来的空间并不多。
6、考虑是线上环境其他大对象占用了老年代，出现了内存泄漏。但是本地jvm参数和线上一致却没有出现这个问题。
然后排查是否有定时任务之类的其他任务占用内存，发现也没有。
然后比较本地和线上的环境和配置的差异，发现线上和本地日志级别不一样，并且线上不断打日志
7、修改日志级别后正常，原来代码有一段在双重循环里面打大对象的info日志。
问题总结
1、打日志代码不应该在循环里面打印大对象，非常占用内存，导致老年代的对象无法及时回收，出现内存泄露。
2、线上配置的日志级别错误，线上应该配置日志较少的级别。
