asnahu

replace pioneer是基于PERL的啊，如果直接用PERL不是更好？另外，也可以用awk处理：

gawk "!a[$0]++" FILE
复制代码

awk '!a[$0]++'
复制代码

This one-liner is very idiomatic. It registers the lines seen in the associative-array “a” (arrays are always associative in Awk) and at the same time tests if it had seen the line before. If it had seen the line before, then a[line] > 0 and !a[line] == 0. Any expression that evaluates to false is a no-op, and any expression that evals to true is equal to “{ print }”.

For example, suppose the input is:

foo
bar
foo
baz
复制代码

When Awk sees the first “foo”, it evaluates the expression “!a["foo"]++”. “a["foo"]” is false, but “!a["foo"]” is true - Awk prints out “foo”. Then it increments “a["foo"]” by one with “++” post-increment operator. Array “a” now contains one value “a["foo"] == 1″.

Next Awk sees “bar”, it does exactly the same what it did to “foo” and prints out “bar”. Array “a” now contains two values “a["foo"] == 1″ and “a["bar"] == 1″.

Now Awk sees the second “foo”. This time “a["foo"]” is true, “!a["foo"]” is false and Awk does not print anything! Array “a” still contains two values
“a["foo"] == 2″ and “a["bar"] == 1″.

Finally Awk sees “baz” and prints it out because “!a["baz"]” is true. Array “a” now contains three values “a["foo"] == 2″ and “a["bar"] == 1″ and “a["baz"] == 1″.

The output:

foo
bar
baz
复制代码

Here is another one-liner to do the same. Eric in his one-liners says it’s the most efficient way to do it.

awk '!($0 in a) { a[$0]; print }'
复制代码

It’s basically the same as previous one, except that it uses the ‘in’ operator. Given an array “a”, an expression “foo in a” tests if variable “foo” is in “a”.

Note that an empty statement “a[$0]” creates an element in the array.

可以从这里下载：http://unxutils.sourceforge.net/