今天服务器 cpu 直接到 100% 日志中发现/test.html 一直被访问这是被恶意访问了吗

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

现在注册

已注册用户请登录

› RANDOM.org 密码生成器

› uBlock

› Authy

› LastPass

› FreebuF.com

› Beebeeto

› Dashlane 密码管理器

这是一个创建于 2107 天前的主题，其中的信息可能已经有所发展或是发生改变。

113.96.109.157 - - [07/Mar/2019:08:07:03 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 119.147.39.151 - - [07/Mar/2019:08:07:03 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 121.12.109.39 - - [07/Mar/2019:08:07:03 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 47.106.50.155 - - [07/Mar/2019:08:07:03 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 202.108.249.153 - - [07/Mar/2019:08:07:04 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 60.221.194.35 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 202.99.114.204 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 117.27.235.150 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 122.156.57.161 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 123.6.31.154 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 113.6.227.203 - - [07/Mar/2019:08:07:05 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 121.22.229.26 - - [07/Mar/2019:08:07:06 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 39.96.119.23 - - [07/Mar/2019:08:07:06 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 183.214.130.150 - - [07/Mar/2019:08:07:07 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 120.223.240.35 - - [07/Mar/2019:08:07:07 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 39.96.135.227 - - [07/Mar/2019:08:07:07 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 120.221.154.209 - - [07/Mar/2019:08:07:08 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 118.31.194.149 - - [07/Mar/2019:08:07:08 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 61.168.101.24 - - [07/Mar/2019:08:07:08 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 58.20.147.25 - - [07/Mar/2019:08:07:09 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 118.112.13.205 - - [07/Mar/2019:08:07:10 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 39.96.151.171 - - [07/Mar/2019:08:07:11 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 119.167.151.155 - - [07/Mar/2019:08:07:11 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 27.221.56.150 - - [07/Mar/2019:08:07:11 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 106.60.80.28 - - [07/Mar/2019:08:07:12 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 120.201.253.38 - - [07/Mar/2019:08:07:12 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 111.6.251.48 - - [07/Mar/2019:08:07:12 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 223.111.105.160 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 183.213.20.27 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 119.23.169.196 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 112.29.216.161 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 124.239.234.163 - - [07/Mar/2019:08:07:13 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 61.163.8.22 - - [07/Mar/2019:08:07:14 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 118.190.214.147 - - [07/Mar/2019:08:07:15 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 140.205.253.144 - - [07/Mar/2019:08:07:15 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 111.48.30.40 - - [07/Mar/2019:08:07:15 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1" 27.221.92.164 - - [07/Mar/2019:08:07:15 +0800] "GET /test.html HTTP/1.1" 404 132 "-" "Go-http-client/1.1

第 1 条附言 · 2019-03-07 12:39:35 +08:00

这一定是恶意访问了吧，已经 UA 403 还有什么办法解决吗

36.104.139.38 - - [07/Mar/2019:12:36:22 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 36.104.139.3 "-" "-" "-" "0.000"sendfileon
47.105.29.90 - - [07/Mar/2019:12:36:23 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 47.105.29.67 "-" "-" "-" "0.000"sendfileon
27.221.92.73 - - [07/Mar/2019:12:36:23 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 27.221.92.117 "-" "-" "-" "0.000"sendfileon
101.200.28.193 - - [07/Mar/2019:12:36:24 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" - "-" "-" "-" "0.000"sendfileon
14.204.147.209 - - [07/Mar/2019:12:36:25 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 14.204.147.197 "-" "-" "-" "0.000"sendfileon
61.168.101.152 - - [07/Mar/2019:12:36:25 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 61.168.101.131 "-" "-" "-" "0.000"sendfileon
58.216.118.23 - - [07/Mar/2019:12:36:26 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 58.216.118.3 "-" "-" "-" "0.000"sendfileon
118.31.194.184 - - [07/Mar/2019:12:36:26 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" - "-" "-" "-" "0.000"sendfileon
121.42.17.201 - - [07/Mar/2019:12:36:26 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 121.42.17.197 "-" "-" "-" "0.000"sendfileon
111.6.243.37 - - [07/Mar/2019:12:36:26 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 111.6.243.3 "-" "-" "-" "0.000"sendfileon
117.91.192.201 - - [07/Mar/2019:12:36:27 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 117.91.192.197 "-" "-" "-" "0.000"sendfileon
120.223.242.31 - - [07/Mar/2019:12:36:28 +0800] "GET /test.html HTTP/1.1" 403 169 "-" "Go-http-client/1.1" 120.223.242.3 "-" "-" "-" "0.000"sendfileon

第 2 条附言 · 2019-03-07 12:51:55 +08:00

阿里云 TCP 监控
https://raw.githubusercontent.com/zhuzhiqiang18/attachment/master/imgs/2019.02.02.tcp.png
阿里云 CPU 监控
https://raw.githubusercontent.com/zhuzhiqiang18/attachment/master/imgs/2019-3-7-cpu.png

第 3 条附言 · 2019-03-07 18:54:19 +08:00

问题找到了，CPU 跑满并不是这个恶意访问造成的。 [逃跑]

38 条回复 • 2019-04-03 17:35:23 +08:00

Humorce

2019-03-07 11:00:33 +08:00

所以这个页面是干什么的呢

zhuzhiqiang

2019-03-07 11:01:50 +08:00

@Humorce 没这个页面返回的 404

Humorce

2019-03-07 11:07:41 +08:00

域名是新买的？

这个时间密度，先更改配置 Block 掉此 UA 吧

zhuzhiqiang

2019-03-07 12:40:58 +08:00

@Humorce 403 了还有什么好的解决办法吗还一直在访问 [捂脸]

blless

2019-03-07 12:41:55 +08:00 via Android

404 都能跑满 CPU 吗

fiht

2019-03-07 12:49:29 +08:00

Go-http-client 这个是 go 语言程序呀

zhuzhiqiang

2019-03-07 12:52:18 +08:00

@blless 我觉得也不至于啊大佬看下 CPU 和 TCP

Steps

2019-03-07 13:06:22 +08:00

是否使用了负载均衡？

我的站跟你一模一样的情况，我过滤了 UA 直接给 503 了

现在一共跑了一千多万次吧。。。

claysec

2019-03-07 13:16:54 +08:00

@zhuzhiqiang 接个 cdn 让他慢慢跑呗

zhuzhiqiang

2019-03-07 13:20:19 +08:00

@Steps 没有使用均衡负载服务就 Nginx 做了个热备

boris1993

2019-03-07 13:21:09 +08:00 via Android

返回个 gzip 炸弹？

zhuzhiqiang

2019-03-07 13:22:47 +08:00

@Steps 老哥你的也是这个 UA 吗

Vhc

2019-03-07 13:25:19 +08:00

1、这个访问频次并不高，CPU 占用和这一毛钱关系也没有。
2、千万不要屏蔽 "Go-http-client/1.1" 这个 UA

dbpe

2019-03-07 13:27:42 +08:00

新知识..GZIp Boom..

gamexg

2019-03-07 13:31:11 +08:00

跳转到 ubuntu iso ？

gamexg

2019-03-07 13:31:42 +08:00

@gamexg #15 额，开源社区钱不多，还是跳转到微软 iso 吧。

zhuzhiqiang

2019-03-07 13:37:24 +08:00

@Vhc 大佬怎么说

CallMeReznov

2019-03-07 13:46:55 +08:00

@Vhc 为什么不能屏蔽？

Steps

2019-03-07 13:53:06 +08:00

@Vhc #13 不屏蔽？你告诉我这样的请求量
已不间断运行: 61 天 18 小时 34 分钟

只是一个 503 的错误页面，61 天 867G 流量，该如何处理？

LanAiFaZuo

2019-03-07 13:54:17 +08:00

我昨天到今天也是 cpu 爆满，用的宝塔。不知道是不是被黑了。

Steps

2019-03-07 14:15:59 +08:00

求高人解决！

ryd994

2019-03-07 15:07:59 +08:00

直接让 web 服务器给他一个静态页面不就好了。Nginx 和 Apache 都可以轻松实现这个目的。
这点请求量，简单的静态请求不可能打满 CPU。要么有别的恶意请求混在里面。要么这些连接有鬼，比如 HTTP slow post 之类的。tcpdump 抓包分析吧。

Newbing

2019-03-07 15:30:32 +08:00

我也遇到了，每天几十万次的请求。我后面建立了一个默认的页面。

Steps

2019-03-07 15:59:45 +08:00

@ryd994 #22 就算是静态页面只有 2b 一天几十万次，也有几个 G 的流量啊

Steps

2019-03-07 16:22:04 +08:00

@zhuzhiqiang #12 我是姐姐，还有，我的问题和你一模一样的

ryd994

2019-03-07 17:20:53 +08:00 via Android

@Steps 从你贴的 log 估测每秒 5 次，往上加一个数量级，50*24*3600 几百 MB 顶天了。你看到的这些都是烟幕弹。每秒 5 次的请求是不可能打出 100%CPU 的。
你可以直接使用 limit_req 控制频率。也可以写个脚本查 404 的频率，再把恶意 IP 加到 ipset 里让 iptables 过滤。

你先把 log 过滤一遍，看还有其他什么鬼。然后还是抓包分析。

不必屏蔽 UA：1.已经是 404 了，屏蔽 UA 也减少不了多少压力 2.换个 UA 还不简单？

opengps

2019-03-07 17:25:57 +08:00

404 不应该导致 cpu 爆满吧

SakuraKuma

2019-03-07 18:18:43 +08:00 via Android

fail2ban 检查到这个 test 封 ip

Marsss

2019-03-07 18:24:56 +08:00 via iPhone

触发验证码给他

Steps

2019-03-07 18:32:55 +08:00

@ryd994 #26 我并没有 100% CPU，我只是跑了很多流量出来。。

这里有请求的次数 503 错误就是被 UA 封闭的

Steps

2019-03-07 18:33:54 +08:00

@ryd994 #26
@SakuraKuma #28 问题就是 IP 从不重复。。这个很害怕的。。

zhuzhiqiang

2019-03-07 18:57:37 +08:00

@CallMeReznov
@Humorce
@LanAiFaZuo
@Marsss
@Newbing
@SakuraKuma
@Steps
@Vhc
@blless
@boris1993
感谢各位问题已经找到了
不是恶意访问造成的 CPU 100 如果不是 cpu100 还真发现不了这个恶意访问 [捂脸]

Steps

2019-03-08 13:58:48 +08:00

@zhuzhiqiang #32 问题是什么？

恶意访问不管了吗？

zhuzhiqiang

2019-03-08 15:33:12 +08:00

@Steps 恶意访问我也没办法 IP 动态的
导致 CPU100 的问题是业务写的有问题 SQL 慢查询 [逃跑]

zhuzhiqiang

2019-03-08 15:40:37 +08:00

@Steps 姐姐我刚刚 awk 统计了下 nginx log /test.html 5 个小时跑了 269247 次[捂脸]

Steps

2019-03-08 21:06:47 +08:00

@zhuzhiqiang #35 是的，我也是 1 天差不多 40 多万次吧。。。

很可怕，不晓得这是怎么回事！

Steps

2019-03-12 09:10:15 +08:00

/t/543424

看下这个，是否你也开了？

WanJiJun

2019-04-03 17:35:23 +08:00

是阿里云动态 CDN 搞的吧，我之前也这样，后来看 https://www.v2ex.com/t/543424，关掉动态加速恢复正常了。

今天服务器 cpu 直接到 100% 日志中发现/test.html 一直被访问 这是被恶意访问了吗

今天服务器 cpu 直接到 100% 日志中发现/test.html 一直被访问这是被恶意访问了吗