Large mbox file to import and make them searchable

Hello,

I have a large 20 GB Mbox file with emails since 2001, which I want to be searchable.
I wanted to import it into Mailinabox and I am still undecided if I want to run this big mailbox just as a searchable online backup or as daily account.

The server is with fast SSDs and CPU (it run an ethereum node without any problems).

My questions:

  • how to import the mbox from the command line?
  • is mailinbox even able to search fast enough (if the hardware is not the limit)?

thank you

A lot of people on this forum tend to use IMAP Sync → Mailbox Imapsync Online

I dont personally like that a 3rd party tools are moving my mail for me so I tend to just configure Mozilla Thunderbird with the old email server (gmail or whatever) and the new email server (MIAB) and then manually move messages with CTRL-A and drag and drop to the new miab location.

I’m not sure there is a direct tool to go from MBOX format to maildir

Mail in a Box should be able to search via roundcube web-ui however with a 20G Mailbox I think no matter which hardware is under the covers it will be challenging. Connecting it to Thunderbird or another mail client (like outlook) might provide a faster searching as some of these applications support indexing.

Hope this helps!

1 Like

As I understand it, your objective is to be able to search a large file of emails.

MIAB’s IMAP server is Dovecot. A default installation of Dovecot doesn’t support indexing of mail. Therefore when you search, the routine will have to examine every email in turn. Even with fast disk, that could take some time across 20GB of mail I suspect.

There is a email indexing add on available for Dovecot but as far as I know that is not configured in MIAB.

See here for some information on this with Cloudron’s email solution with regard to searching large email stores.

1 Like

I will look into it. Might be a good alternative.

I already saw that solr is not longer supported (a few years already)

I had used imapsync in the past. Actually pretty good. Thx

I also had a very good experience with
Mailjerry.com. I wouldn’t be surprised if they use imapsync under the hood.

GOOD idea, I did not thought of that!

and if you have the mbox file you can it also just import into local folder and upload from there to the new mailserver. but is kinda slow for 20gb :slight_smile:

Do you have what the process looks like for this?

Going from MBOX format to maildir?

I was thinking to use this, first in an temporary folder
https://manpages.ubuntu.com/manpages/focal/man1/mb2md.1.html

But I need to figure out the searchable (solr) or I just use it as an archive

In the past I have used GitHub - grosjo/fts-xapian: Dovecot FTS plugin based on Xapian
I’m not sure it actually works. I wonder what a good test case is. Maybe soon I’ll spend some more time on this :grin:

Actually now I remember.

Dovecot was bought/merged by open xchange in 2016 and they have a pro version.

Even the pro 2.3 version had full text search problems ( Heinlein consulting)

No idea about Dovecot pro 3.0

Community version is still 2.3
I doubt even with CV 3.0 there will be a solution for free for this problem.

I will move on to another solution

Importing a 20GB Mbox might be tricky for Mailinabox. Consider using a tool to split it into smaller chunks for easier import and searching. Mailinabox search functionality might struggle with a single massive mailbox.