asnahu

replace pioneer是基于PERL的啊，如果直接用PERL不是更好？另外，也可以用awk处理：

gawk "!a[$0]++" FILE
复制代码

awk '!a[$0]++'
复制代码

This one-liner is very idiomatic. It registers the lines seen in the associative-array “a” (arrays are always associative in Awk) and at the same time tests if it had seen the line before. If it had seen the line before, then a[line] > 0 and !a[line] == 0. Any expression that evaluates to false is a no-op, and any expression that evals to true is equal to “{ print }”.

For example, suppose the input is:

foo
bar
foo
baz
复制代码

When Awk sees the first “foo”, it evaluates the expression “!a["foo"]++”. “a["foo"]” is false, but “!a["foo"]” is true - Awk prints out “foo”. Then it increments “a["foo"]” by one with “++” post-increment operator. Array “a” now contains one value “a["foo"] == 1″.

Next Awk sees “bar”, it does exactly the same what it did to “foo” and prints out “bar”. Array “a” now contains two values “a["foo"] == 1″ and “a["bar"] == 1″.

Now Awk sees the second “foo”. This time “a["foo"]” is true, “!a["foo"]” is false and Awk does not print anything! Array “a” still contains two values
“a["foo"] == 2″ and “a["bar"] == 1″.

Finally Awk sees “baz” and prints it out because “!a["baz"]” is true. Array “a” now contains three values “a["foo"] == 2″ and “a["bar"] == 1″ and “a["baz"] == 1″.

The output:

foo
bar
baz
复制代码

Here is another one-liner to do the same. Eric in his one-liners says it’s the most efficient way to do it.

awk '!($0 in a) { a[$0]; print }'
复制代码

It’s basically the same as previous one, except that it uses the ‘in’ operator. Given an array “a”, an expression “foo in a” tests if variable “foo” is in “a”.

Note that an empty statement “a[$0]” creates an element in the array.

可以从这里下载：http://unxutils.sourceforge.net/

asnahu

系统要有gawk啊，下面就是解释嘛，哪里看不懂？

asnahu

gawk "!a[$0]++"<a.txt>b.txt
复制代码

看这样行不行

asnahu

awk处理行是非常快的了

asnahu

回复 12# cm535

我2楼的回答里不是有更加高效的方法了吗，要认真看啊

asnahu

我这没法测试处理这么大的文件到底需要多长时间，需要注意的是机器的硬件环境是很重要的。