标题: [文本处理] 批处理如何删除文本里的重复行(相同行只保留一条)? [打印本页]
作者: smallbat 时间: 2014-12-7 10:31 标题: 批处理如何删除文本里的重复行(相同行只保留一条)?
本帖最后由 pcl_test 于 2016-8-5 23:14 编辑
文本里的内容是按一行一行的来书写的,形如
1
2
3
4
5
6
7
1
3
很显然里面有重复行,想让删除重复行,比如上面1和3是重复的,把这2个重复了的删除掉
1
2
3
4
5
6
7
作者: pcl_test 时间: 2016-8-5 23:13
- @echo off
- for /f "delims=" %%a in ('dir /a-d/b "文本.txt"') do (
- (for /f "delims=" %%b in ('type "%%a"') do (
- if not defined #%%b (echo;%%b&set "#%%b=1")
- ))>"%%~dpaNew_%%~nxa"
- )
- pause
复制代码
作者: pcl_test 时间: 2016-8-5 23:35
- //&cls&dir /a-d/b "文本.txt"|cscript -nologo -e:jscript "%~f0"&pause&exit
-
- var fso = new ActiveXObject('Scripting.Filesystemobject');
- while(!WSH.StdIn.AtEndOfStream){
- try{
- var file = WSH.StdIn.ReadLine();
- var text = unique(file);
- var f = file.replace(/(.+\\)?([^\\]+)/, '$1New_$2');
- fso.CreateTextFile(f, 2).Write(text);
- }catch(e){}
- }
- WSH.echo('Done');
-
- function unique(file){
- var f = fso.OpenTextFile(file, 1), s='', map={};
- while(!f.AtEndOfStream){
- var line = f.ReadLine();
- if(!/^[ \s]*$/.test(line)&&!map[line]){s+=line+'\r\n';map[line]=1}
- }
- f.Close();
- return s
- }
复制代码
作者: pcl_test 时间: 2016-8-6 00:40
本帖最后由 pcl_test 于 2016-8-6 00:43 编辑
第三方gawk
http://www.bathome.net/s/tool/index.html?key=gawk- #*&cls&dir /a-d/b "文本.txt"|gawk -f "%~f0"&pause&exit
- BEGIN{
- while(getline file>0){
- delete map;delete a;s="";
- while(getline<file>0){
- if($0!~/^[ [:space:]]*$/){
- if(!map[$0]){s=s$0"\n";map[$0]++;}
- }
- }
- match(file, /(.+\\)?([^\\]+)/, a);
- print s>a[1]"New_"a[2]
- }
- }END{print "Done"}
复制代码
欢迎光临 批处理之家 (http://www.bathome.net/) |
Powered by Discuz! 7.2 |