[技术讨论] [分享]gawk去除文本中IP地址前两段重复10次以上的行

DAIC

QQ群里面看到的一个问题。待处理的文本 1.txt 内容如下：
110.240.41.125
110.240.254.150
110.240.253.242
222.16.80.78
222.16.80.120
183.0.182.41
183.184.154.170
183.184.163.82
183.184.227.2
112.0.120.79
112.0.120.82
112.0.121.37
112.0.123.84
112.0.123.103
112.0.124.47
112.0.123.159
61.128.110.38
61.128.110.181
61.128.111.150
61.128.111.162
61.128.122.133
61.128.134.42
61.128.134.238
61.128.172.192
61.128.173.112
61.128.176.118
61.128.177.5
61.128.177.27
61.128.235.123
120.192.18.172
120.192.19.24
120.192.20.163
120.192.25.234
120.192.31.160
120.192.73.133
120.192.74.242
120.192.78.104
120.192.78.105
120.192.78.102
120.192.79.42
120.192.79.57
120.192.79.36
120.192.79.50
120.192.79.60
120.192.79.52
120.192.79.35
120.192.79.61
120.192.79.45
120.192.79.43
120.192.79.59
120.192.79.58
120.192.79.37
120.192.80.226
120.192.80.227
120.192.85.35
120.192.85.114

代码：

gawk "BEGIN{FS=OFS=\".\"}NR==FNR{a[$1.$2]++}NR>FNR{if(a[$1.$2]<10)print}" 1.txt 1.txt > 2.txt
复制代码