从谷歌机器人获得我的网络服务器点击

Question

我不明白为什么我看到这些日志到我的nginx服务器。

66.249.79.115 - - [06/Oct/2015:18:50:17 +0000] "GET /profile/?Rohatgi.Nikhil HTTP/1.1" 404 1031 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.115 - - [06/Oct/2015:18:50:49 +0000] "GET /profile/?Mukherjee.PankajKumar HTTP/1.1" 404 1038 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.115 - - [06/Oct/2015:18:51:21 +0000] "GET /profile/?Khorana.Ashish HTTP/1.1" 404 1031 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.117 - - [06/Oct/2015:18:51:52 +0000] "GET /profile/?Mittal.AshokKumar HTTP/1.1" 404 1034 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.117 - - [06/Oct/2015:18:52:24 +0000] "GET /profile/?Suri.Divya HTTP/1.1" 404 1029 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.117 - - [06/Oct/2015:18:52:56 +0000] "GET /profile/?gupta.member) HTTP/1.1" 404 1030 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 

我最近做了我的网站直播。

从日志我可以理解他们来自谷歌机器人一点点，但我想明白为什么我得到这个，我怎么能阻止他们？ 如果我阻止，我的网站将不会受到任何伤害。

Answer 1

这种行为是正常的，如果你把你的网站最近生活。 Google Bots正在抓取网页并为新网页编制索引，因此可以使用Google搜索找到它们。正如你所看到的，谷歌机器人假装是iPhone（iPhone;像iPhone OS X的CPU iPhone OS 8_3）。

如果要阻止googlebot，您可以按照以下指南进行操作：Nginx + CDN + GoogleBot or how to avoid many useless Googlebot hits请注意，如果您屏蔽了谷歌爬虫机器人，则无法使用谷歌搜索找到您的网页。 如果你想阻止更多的蜘蛛/爬行机器人，请参考post用户GD-hussle。请参阅Crawling, indexing & ranking。