Skip to content

How to remove duplicate emails in Opera M2

Having used Opera’s mail client on my laptop recently (I got sick of Mozilla), I was very pleased with its simplicity. It felt like setting up Thunderbird took forever,with options scattered in two places, but Opera’s defaults suited me perfectly.

I decided to take the plunge and switch to Opera for my primary email client. That means importing thousands of old emails from Thunderbird. The import process was easy and seemed to go by without problems. After it was complete, I noticed several messages had been inadvertently duplicated—seemingly at random, and on all the accounts I imported. About 14,000 of the almost 100,000 emails were duplicates.

I figured out a relatively convoluted and somewhat hacked-together way to remove them. I learned some things about Opera M2 in the process:

M2 keeps a database with some basic information from each email, but the messages themselves are in individual files which are neatly sorted by date. When you click or double-click on a message Opera loads it from these individual files. However, it only reads as much of the file as it knows its size to be—that is, adding text to the end of the body in the file will not show up in the mail client. The size, subject, (and probably sender and receiver) are stored in a separate database file—this is the one that is accessed to add the messages to the list. So ┬áchanging the subject line, for example, in the individual files does nothing to the subject as it appears in Opera. If you delete the message file, Opera does not remove it from its database. Instead, it looks as though it thinks it only downloaded the headers and the message is still on the server. So I figured there was no way to delete the emails other than from within Opera itself.

If you’re going to try this, don’t be stupid—backup your entire Opera mail directory like I did. Also, remember these scripts were developed by examples in my email collection, so it may not work for everyone. And like any good programmer, I only added checks for errors that actually occurred when I ran the scripts.

First, I wrote a Python script to calculate the MD5 checksum of the messages. It wasn’t as simple as calculating it for the entire file because Opera adds some “X” fields to the top of the header. By trial and error I arrived at the noted fields as being possibly different between otherwise identical emails. I think it’s safe to just discard the first 7 lines and calculate the checksum with the rest, but it didn’t seem to go significantly faster. Remember you are opening every single email you have, so expect this to take a while.

  1. #Walks a directory tree and calculates a checksum for each file.
  2.  
  3. import os, glob, shutil, string, sys, hashlib
  4.  
  5. dirroot = ‘S:\\UserProfile\\AppData\\Local\\Opera\\Opera\\mail’
  6.  
  7. fils =[‘mbs’]
  8.  
  9. chksumfile = open(‘checksums.txt’, ‘w’)
  10.  
  11. for dirpath, dirnames, filenames in os.walk(dirroot):
  12.     #For each directory, we look for the files.
  13.     #We never use dirnames or filenames
  14.     print dirpath
  15.     for fil in fils:
  16.         filext = ‘%s\\*.%s’ % (dirpath, fil)
  17.         for filename in glob.glob(filext):
  18.             thefile = open(filename, ‘rb’)
  19.             #There are a few things in the header that can change
  20.             #even for duplicate emails. We must discard these differences.
  21.             thefile.readline()
  22.             #look for the end of the first "X" lines.
  23.             while 1:
  24.                 filepos = thefile.tell()
  25.                 line = thefile.readline()
  26.                 if line[0:14] == ‘X-Opera-Status’:
  27.                     pass
  28.                 elif line[0:16] == ‘X-Opera-Location’:
  29.                     pass
  30.                 elif line[0:13] == ‘X-Account-Key’:
  31.                     pass
  32.                 elif line[0:6] == ‘X-UIDL’:
  33.                     pass
  34.                 elif line[0:16] == ‘X-Mozilla-Status’:
  35.                     pass
  36.                 elif line[0:17] == ‘X-Mozilla-Status2’:
  37.                     pass
  38.                 else:
  39.                     thefile.seek(filepos)
  40.                     break
  41.             datastr = thefile.read()
  42.             thefile.close()
  43.             m = hashlib.md5()
  44.             m.update(datastr)
  45.             chksumfile.write(‘%s\t%s\n % (filename, m.hexdigest()))
  46.  
  47. chksumfile.close()

Next I loaded the resulting message-checksum list into a spreadsheet program, sorted it by checksum, and saved the results.

The next script loads the sorted list and looks for messages with identical checksums. It assumes there is only a maximum of 2 identical messages, though I believe the entire process would still work with n duplicates. It then saves identical pairs in another file. Note that along with comparing the checksums, it also compares the messages’ directories. I ran into at least one case where two actually “distinct” messages had the same checksum. Note that if you have a lot of short messages on the same day and account, there’s a good chance they’ll end up being marked as duplicates, and all but one will be removed.

  1. chksumfile = open(‘checksums_sorted.txt’, ‘r’)
  2.  
  3. line = chksumfile.readline()
  4. if not line:
  5.     exit
  6. filesum = line.split()
  7. oldchksum = filesum[1]
  8. oldfile = filesum[0]
  9.  
  10. dupefile = open(‘dupes.txt’, ‘w’)
  11.  
  12. while 1:
  13.     line = chksumfile.readline()
  14.     if not line:
  15.         break
  16.     filesum = line.split()
  17.     if filesum[1] == oldchksum:
  18.         oldpath = oldfile.split(\\)
  19.         newpath = filesum[0].split(\\)
  20.         if len(oldpath) == len(newpath):
  21.             sameaccount = 1
  22.             for i in range(0, len(oldpath)1):
  23.                 if not (oldpath[i] == newpath[i]):
  24.                     sameaccount = 0
  25.                     break
  26.         if sameaccount:
  27.             dupefile.write(‘%s\t%s\n % (filesum[0], oldfile))
  28.     oldchksum = filesum[1]
  29.     oldfile = filesum[0]
  30.    
  31. chksumfile.close()
  32. dupefile.close()

Next I had to devise a way of marking the duplicate messages so that I could filter them in Opera and then delete them. This turned out to be more difficult than I had hoped for, because the subject line is not read from the message files (it is in the database) and because anything added to the end of the message file will not show up in Opera because it only reads up to the size that it has recorded for that message in the database. So the only way to achieve this was to find the beginning of the body and add some distinctive set of characters. Then the end of the message (or an attachment) would be cut off at the end, but we don’t care since we’re going to delete these guys anyway. I found that in my several years of email collection, going from (I think) MSN mail to Outlook Express to Outlook to Thunderbird to Opera, a few of the older guys got mangled into bare headers without a message. I noticed some of these had been duplicated, too, but my scripts do nothing to them.

Finding the beginning of the body is a bit tricky. The header is separated from the message by a blank line. But if there are attachments, a MIME header follows. The body is then separated by a blank line after that. This script takes that into account. You can uncomment the “raw_input” line to allow you to stop processing in case you want to see what it will do to the first duplicate. Also, I highly recommend you run the script once through with the line that adds the marker to the messages commented out to make sure no errors occur during processing. I suppose it’s no big deal, but if for some reason the script stops (or you have to stop it) before it’s done, the messages that have been processed will be reprocessed the next time you run the script, thus adding the marker twice. I don’t think that’s a problem, but beware.

  1. dupemarker = ‘DUPETASTIC20091231’
  2.  
  3. dupefile = open(‘dupes.txt’, ‘r’)
  4.  
  5. while 1:
  6.     line = dupefile.readline()
  7.     if not line:
  8.         break
  9.     files = line.split()
  10.     print files[0]
  11. ##    raw_input("Continue?")
  12.     duperead = open(files[0], ‘r’)
  13.     dupedata = duperead.readlines()
  14.     duperead.close()
  15.     dupewrite = open(files[0], ‘w’)
  16.     endofheader = 0
  17.     lookforboundary = 0
  18.     for i in range(len(dupedata)):
  19.         line = dupedata[i]
  20.         if lookforboundary == 1 and line[0:2] == ‘–‘:
  21.                 #We found the boundary. Let’s look for the next
  22.                 #blank line.
  23.                 lookforboundary = 0
  24.                 dupewrite.write(‘%s’ % line)
  25.                 continue
  26.         if len(line) == 1 and endofheader == 0:
  27.             if lookforboundary == 1:
  28.                 dupewrite.write(‘%s’ % line)
  29.                 continue
  30.             #This could be the end of the header, but only if
  31.             #the message has no attachments.
  32.             if i+1 == len(dupedata):
  33.                 dupewrite.write(‘%s’ % line)
  34.                 continue
  35.             line2 = dupedata[i+1]
  36.             if line2[0:44] == ‘This is a multi-part message in MIME format.’:
  37.                 lookforboundary = 1
  38.                 dupewrite.write(‘%s’ % line)
  39.                 continue
  40.             else:
  41.                 dupewrite.write(‘%s’ % line)
  42.                 #Comment the following line to simulate the process.
  43.                 dupewrite.write(‘%s\n % dupemarker)
  44.                 endofheader = 1
  45.         else:
  46.             dupewrite.write(‘%s’ % line)
  47.     dupewrite.close()
  48.  
  49. dupefile.close()

After all the messages have been tagged, you just need to set up a filter in Opera to search message bodies for whatever you set “dupemarker” to. Once the filter has gone through all the messages, you can check that the number of messages found is less than or equal to the number of pairs messages with the same checksum (it may be less than if some messages had the same checksum but were in different accounts, as happened at least once in my case, or if they are blank, which also happened at least a few times). In my case, I had 14550 duplicate pairs of which 14391 were marked by the script.

Once the filter is done, select all the messages, and delete! You can keep them in the trash for a while if you’d like. Or, you can just keep them in the filter until you’re sure nothing went wrong. Remember that although Opera will not show the end of the tagged messages, or claim that attachments cannot be loaded because of parsing errors, the information is still there. If you remove the marker from the messages, Opera will be able to read them properly. This is due to the fact that Opera keeps the size of the messages separate from the messages themselves.

Good luck.

Post a Comment

Your email is never published nor shared. Required fields are marked *
*
*