日志中有很多同IP段的IP，怎么快速判断是否蜘蛛、恶意机器人？

Question

求助，查看日志，有很多同IP段的IP，如下：

114.250.43.82
114.250.44.141
114.250.44.35
114.250.48.177
114.250.48.18
114.250.48.84
114.250.49.134
114.250.49.162
114.250.49.2
114.250.49.44
114.250.49.86
114.250.49.91
114.250.50.110
114.250.50.144
114.250.50.43
114.250.51.102
114.250.51.105
114.250.51.112
114.250.51.115
114.250.51.12
114.250.51.125
114.250.51.149
114.250.51.17
114.250.51.195
114.250.51.199
114.250.51.214
114.250.51.235
114.250.51.243
114.250.51.248
114.250.51.26
114.250.51.30
114.250.51.35
114.250.51.54
114.250.51.57
114.250.51.8
123.126.109.106
123.126.109.114
123.126.109.128
123.126.109.130
123.126.109.151
123.126.109.154
123.126.109.160
123.126.109.171
123.126.109.189
123.126.109.194
123.126.109.235
123.126.109.237
123.126.109.238
123.126.109.24
123.126.109.31
123.126.109.32
123.126.109.41
123.126.109.43
123.126.109.48
123.126.109.6
123.126.109.66
123.126.109.68
123.126.109.72
123.126.109.79
123.126.109.93
123.126.109.98
123.126.109.99

怎么快速判断是否蜘蛛、恶意IP？

目前通过1、nslookup反查；2、分析日志的访问频率；
请求支援

Answer 1

一种楼主用反向DNS查询和User-Agent分析。

可以用Python脚本

第一种使用 requests 库和 ipinfo API 来查询IP信息，并判断是否为搜索引擎蜘蛛：

import requests
import logging

# 配置日志记录
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def is_search_engine_bot(ip, access_token):
    try:
        # 使用ipinfo API查询IP信息
        response = requests.get(f'https://ipinfo.io/{ip}?token={access_token}', timeout=10)
        response.raise_for_status()  # 检查请求是否成功
        data = response.json()

        # 判断IP是否属于已知的搜索引擎
        search_engines = ['googlebot.com', 'baidu.com', 'bing.com', 'yahoo.com']
        hostname = data.get('hostname', '')
        if any(engine in hostname for engine in search_engines):
            return True
        return False
    except requests.RequestException as e:
        logging.error(f'请求IPinfo API时出错: {e}')
        return False
    except Exception as e:
        logging.error(f'处理IP {ip} 时出错: {e}')
        return False

def main():
    # 示例IP地址列表
    ip_list = [
        '114.250.43.82', '114.250.44.141', '114.250.44.35',
        '123.126.109.106', '123.126.109.114', '123.126.109.128'
    ]

    # IPinfo API访问令牌
    access_token = 'your_ipinfo_access_token'

    # 检查每个IP是否为搜索引擎蜘蛛
    for ip in ip_list:
        if is_search_engine_bot(ip, access_token):
            logging.info(f'{ip} 是搜索引擎蜘蛛')
        else:
            logging.info(f'{ip} 不是搜索引擎蜘蛛')

if __name__ == '__main__':
    main()

使用步骤
1.获取IPinfo API访问令牌：注册IPinfo账号并获取API访问令牌。
2.安装requests库：在命令行中运行 pip install requests。
3.运行脚本：将上述代码保存为Python文件（例如 check_bots.py），然后在命令行中运行 python check_bots.py。

第二种可以批量查询IP地址的反向DNS、访问频率和User-Agent信息，并判断是否为恶意IP。使用了 requests 和 ipwhois 库

import requests
from ipwhois import IPWhois
import logging

# 配置日志记录
logging.basicConfig(filename='ip_check.log', level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# 要检查的IP地址列表
ip_list = [
    "114.250.43.82", "114.250.44.141", "114.250.44.35", "114.250.48.177", "114.250.48.18",
    "114.250.48.84", "114.250.49.134", "114.250.49.162", "114.250.49.2", "114.250.49.44",
    # 添加更多IP地址...
]

# 检查IP地址是否为恶意IP
def check_ip(ip):
    try:
        # 反向DNS查询
        obj = IPWhois(ip)
        results = obj.lookup_rdap()
        domain = results['network']['name']
        
        # 模拟访问，获取User-Agent
        response = requests.get(f"http://{ip}", timeout=5)
        user_agent = response.headers.get('User-Agent', 'Unknown')
        
        # 判断是否为恶意IP（示例规则）
        if "baidu.com" in domain or "Googlebot" in user_agent:
            logging.info(f"IP {ip} is likely a search engine bot: {domain}, {user_agent}")
            return False
        else:
            logging.warning(f"IP {ip} might be malicious: {domain}, {user_agent}")
            return True
    except Exception as e:
        logging.error(f"Error checking IP {ip}: {e}")
        return False

# 批量检查IP地址
for ip in ip_list:
    is_malicious = check_ip(ip)
    if is_malicious:
        print(f"IP {ip} is potentially malicious.")
    else:
        print(f"IP {ip} is likely safe.")

print("IP检查完成。")