Merge txt files and remove duplicate lines

[Kyuft]

Hey,
I have two files that i want to merge. I want to merge 1.txt and 2.txt, however they have duplicate lines, and I want those to be removed. Can anyone guide me in making a batch file that will merge those files into 1 file and remove the duplicate lines? (I have no experience with batch)

-Kyuft:shades:

EDIT: Solved using TextPipe

 

[camelCase]

You can do that with batch files? ;O
I always thought that was only feasible with perl or something similar =x

Is there any particular order you need the lines to be in when merged?

 

[Kyuft]

There is no particular order that the lines need to be in. I was hoping this could be done in batch, but if you can do it any other way, I would greatly appreciate it. I just don't want to have to download special software to run the code (unless, of course, the solution is a software that you coded).

-Kyuft:shades:

 

[Siretu]

I know you said you solved this, but I just wanted to mention some things.

I assume this might be related to localization issues in SC2? I had a similar problem a while ago with Cruiser Command. There are some people interested in translating Cruiser Command to Chinese and publishing it on the chinese servers. So I told them how to extract the game strings, and then how to import it back into the map.

However when I give them a new version, I don't want them to have to re-translate everything. I created a program that basically goes through the new text file and checks if that key exists in the old translated file. If it does, it appends the translated row into the new merged document. If it doesn't exist in the old translated file, it takes the row from the new file and appends it in the merged document. It also alerts the user of any new rows that appear in the new document but not in the old so the translator doesn't have to search through thousands of rows to find the new rows.

I've done some basic testing with it and it seems to work. It should work with your map as well even if you're probably working with two english versions.

Here's the code. It's written in Python 2.7.2 but any 2.x version should work. I know you mentioned not wanting to download any extra software, but Python is awesome and nice for learning how to program.

Spoiler

print "Siretu's Text Merge Program"
print "Please put the old and new text files in the same folder as this program."
old = open(raw_input("Enter the name of the OLD text file. This is the one that contains all the translations: ")).read().split("\n")
new = open(raw_input("Enter the name of the NEW text file. This is the one that contains all the english strings: ")).read().split("\n")
merge = []

for s in new:
    if s:
        notFound = True
        key = s.split("=")[0]
        value = s.split("=")[1]
        for i in old:
            if i:
                key2 = i.split("=")[0]
                value2 = i.split("=")[1]
                if key == key2:
                    notFound = False
                    merge.append(key+"="+value2)
                    break
    if notFound:
        print "Found new key! Make sure you translate "+key
        merge.append(key+"="+value)

f = open("GameStringsMERGED.txt","w")
f.write("\n".join(merge))
f.close()

 

[Kyuft]

Well, I do want to learn how to program, and Python looks relatively simple, how would I go about learning Python code? Where should I start?

-Kyuft:shades:

 

[s3rius]

Here's about half a gazillion tutorials for non-programmers to learn phython:

http://wiki.python.org/moin/BeginnersGuide/NonProgrammers

Pick one or two that you find understandable and just hack away.

 

[Kyuft]

Are there advanced tutorials somewhere for when I'm done with the beginner ones?

-Kyuft:shades:

 

[Siretu]

Dive into python goes into some of the more advanced areas. When you're done with that, you just have to think of something you want to learn and search for it. You should be able to find a tutorial on it pretty easily.

 

[Kyuft]

Okay, I'm done with the beginner stuff, and its REALLY simple. I like it! One question about your program though. If I run this program on my files would it work? My old files has almost all of the trigger stuff, and the new file is mostly data. Therefore, there is new stuff in both files.

-Kyuft:shades:

 

[Siretu]

Yes, it should work anyway.

Since my program doesn't modify the original files, you could just try it out. You'll get a merged file in the same directory. Search for a string that is only in the first file and search for another one that is only in the second file. If you find both in the merged file, chances are that it worked.