Lou Plum­mer pub­lished a use­ful work­flow to send emails to Obsid­i­an using IFTTT, Drop­box, and Hazel. I liked the idea of cap­tur­ing and send­ing infor­ma­tion to an Obsid­i­an Vault via email. Since I use Obsid­i­an on all my devices, there are sit­u­a­tions where an email would be more effi­cient for cap­tur­ing infor­ma­tion and stor­ing it in Obsid­i­an. I have had a spe­cial email account for these kinds of things for years, but the emails end up in an inbox and not in Obsid­i­an. Addi­tion­al­ly, solv­ing this prob­lem presents an enjoy­able chal­lenge.

Here are my require­ments:

• Runs local­ly on my main computer—in my case, my Mac­Book
• Works with a ded­i­cat­ed email account
• Access­es emails via IMAP
• Down­loads all unread emails and con­verts them into Mark­down files
• Saves these files direct­ly into the Obsid­i­an Vault

Because I’m not yet an expert in Python pro­gram­ming, I asked my Obsid­i­an Co-Pilot using Ope­nAI GPT-4o for help. After three iter­a­tions, I have a ver­sion that works very well. In this post, I want to present this solu­tion, although I know it is only a quick and dirty first attempt. I will con­tin­ue to refine it and make it more sophis­ti­cat­ed.

Initial Prototype Implementation

As I men­tioned ear­li­er, I use a spe­cial email account sole­ly for this pur­pose, which serves as a basic secu­ri­ty fea­ture. I only use this account to send emails to myself, so I haven’t received any spam on it for years. How­ev­er, any account that uses IMAP can be used.

When­ev­er I don’t explic­it­ly request a spe­cif­ic lan­guage, Ope­nAI typ­i­cal­ly pro­vides a Python-based solu­tion. Let’s go through the steps:

Prepar­ing the Envi­ron­ment

The first step is to cre­ate a project fold­er and nav­i­gate into it:

mkdir /Users/leif/Documents/Projects/mail2obsidian
cd /Users/leif/Documents/Projects/mail2obsidian

Next, cre­ate and acti­vate a vir­tu­al Python envi­ron­ment inside your project fold­er. This ensures that all the nec­es­sary libraries are local to this project, and all paths are han­dled with­in this envi­ron­ment. This way, your stan­dard Python instal­la­tion remains unchanged.

python3 -m venv env
source env/bin/activate

After acti­vat­ing the envi­ron­ment, your shell prompt will change to some­thing like this:

(env) > $

You’ll need a few pack­ages to run the script:

pip install imapclient markdownify pyyaml

• imapclient is a library that enables the script to inter­act with email servers. It’s used to fetch all unread emails from your mail serv­er via IMAP.
• markdownify is a library that con­verts HTML con­tent into Mark­down for­mat. It’s used to con­vert HTML emails into Mark­down, though it’s not per­fect.
• pyYAML is a library for pars­ing and writ­ing YAML. In this case, it allows you to keep all con­fig­u­ra­tion para­me­ters out­side the code in a YAML file. This library can read and under­stand YAML.

Configuration

Next, we will cre­ate the con­fig­u­ra­tion file config.yml. You can use any text edi­tor; I typ­i­cal­ly use the CLI edi­tor nano. Enter your data and path as fol­lows:

email:
  server: "imap.example.com"
  user: "your-email@example.com"
  password: "your-password"

output:
  path: "/Users/leif/Documents/ObsidianVault/98 emails"

Make sure to replace the place­hold­ers with your actu­al email serv­er details and the desired out­put path.

The Main Code

Final­ly, cre­ate a new file and copy-paste the fol­low­ing Python script. I named mine mail2obsidian.py:

import imapclient
import email
from email.header import decode_header
import os
from markdownify import markdownify as md
import yaml

# Load configuration from YAML file
with open('config.yaml', 'r') as file:
    config = yaml.safe_load(file)

EMAIL = config['email']['user']
PASSWORD = config['email']['password']
IMAP_SERVER = config['email']['server']
OUTPUT_PATH = config['output']['path']

# Ensure the output directory exists
os.makedirs(OUTPUT_PATH, exist_ok=True)

# Connect to the server
with imapclient.IMAPClient(IMAP_SERVER) as server:
    server.login(EMAIL, PASSWORD)
    server.select_folder('INBOX')

    # Search for all unread emails
    messages = server.search(['UNSEEN'])

    for uid, message_data in server.fetch(messages, 'RFC822').items():
        email_message = email.message_from_bytes(message_data[b'RFC822'])

        # Decode email subject
        subject, encoding = decode_header(email_message['Subject'])[0]
        if isinstance(subject, bytes):
            subject = subject.decode(encoding if encoding else 'utf-8')

        # Create a safe filename
        filename = f"{subject}.md".replace('/', '_').replace('\\', '_')

        # Initialize email content
        email_content = f"# {subject}\n\n"

        # Extract email body
        for part in email_message.walk():
            if part.get_content_type() == "text/plain" or part.get_content_type() == "text/html":
                charset = part.get_content_charset() or 'utf-8'
                payload = part.get_payload(decode=True)
                if payload:
                    body = payload.decode(charset, errors='replace')
                    if part.get_content_type() == "text/html":
                        body = md(body)
                    email_content += body

        # Save the email content to a Markdown file
        file_path = os.path.join(OUTPUT_PATH, filename)
        with open(file_path, 'w', encoding='utf-8') as f:
            f.write(email_content)

        # Mark the email as read
        server.add_flags(uid, '\\Seen')

print("Unread emails have been converted to Markdown files.")

Here’s a brief expla­na­tion of what each part of the code does:

1. Imports:
   • imapclient: A library for inter­act­ing with email servers using the IMAP pro­to­col.
   • email: A mod­ule for han­dling email mes­sages.
   • decode_header: A func­tion to decode email head­ers.
   • os: A mod­ule for inter­act­ing with the oper­at­ing sys­tem, used here to han­dle file paths.
   • markdownify: A library to con­vert HTML con­tent to Mark­down.
   • yaml: A library to parse YAML con­fig­u­ra­tion files.
2. Con­fig­u­ra­tion Load­ing:
   • The script reads a con­fig­u­ra­tion file named config.yaml to get the email cre­den­tials (EMAIL, PASSWORD), the IMAP serv­er address (IMAP_SERVER), and the out­put direc­to­ry path (OUTPUT_PATH).
3. Ensure Out­put Direc­to­ry Exists:
   • os.makedirs(OUTPUT_PATH, exist_ok=True): Ensures that the direc­to­ry where the Mark­down files will be saved exists. If it does­n’t exist, it will be cre­at­ed.
4. Con­nect to the Email Serv­er:
   • The script uses IMAPClient to con­nect to the spec­i­fied IMAP serv­er and logs in using the pro­vid­ed cre­den­tials.
   • It selects the ‘INBOX’ fold­er to search for emails.
5. Search and Fetch Unread Emails:
   • server.search(['UNSEEN']): Search­es for all unread emails in the inbox.
   • server.fetch(messages, 'RFC822'): Fetch­es the full email data for each unread email.
6. Process Each Email:
   • For each email, it decodes the sub­ject line using decode_header.
   • It cre­ates a safe file­name by replac­ing any slash­es in the sub­ject with under­scores.
   • Ini­tial­izes a Mark­down string with the email sub­ject as a head­er.
7. Extract Email Body:
   • The script iter­ates over each part of the email to find text or HTML con­tent.
   • It decodes the con­tent using the appro­pri­ate charset.
   • If the con­tent is HTML, it con­verts it to Mark­down using markdownify.
   • The con­tent is append­ed to the Mark­down string.
8. Save Email as Mark­down File:
   • The email con­tent is saved to a file in the spec­i­fied out­put direc­to­ry with a .md exten­sion.
9. Mark Email as Read:
   • server.add_flags(uid, '\\Seen'): Marks the email as read on the serv­er.
10. Com­ple­tion Mes­sage:
    • Prints a mes­sage indi­cat­ing that the unread emails have been con­vert­ed to Mark­down files.

If you con­fig­ure every­thing cor­rect­ly and have a few unread emails in your inbox, you can run this script (while still in the active Python envi­ron­ment) with:

(env) > $ python mail2obsidian.py

If every­thing runs smooth­ly, you’ll see a suc­cess mes­sage in the ter­mi­nal, and all unread emails will be saved inside your Obsid­i­an vault at the defined loca­tion. Oth­er­wise, you might encounter some run­time errors.

Unread emails have been converted to Markdown files.

Outlook

This pro­posed solu­tion is far from com­plete. Although my account isn’t receiv­ing spam yet, it might be wise to ensure that not every email is auto­mat­i­cal­ly processed. For instance, requir­ing a spe­cif­ic string in the sub­ject line could be a use­ful fil­ter. Only emails con­tain­ing this string would be down­loaded.

This string could also indi­cate where to save the emails or serve as a tag. Dif­fer­ent codes could direct the email to dif­fer­ent fold­ers or rep­re­sent dif­fer­ent tags.

It might also be ben­e­fi­cial to add front­mat­ter for the sender’s email address or oth­er email prop­er­ties before sav­ing the file in the vault. In my vault, every note includes a foot­er area, so this could be added as well.

Cur­rent­ly, only plain text and HTML are being down­loaded. This is rel­a­tive­ly safe, as it reduces the risk of exe­cutable code being loaded. How­ev­er, I wel­come feed­back on how secure this approach is.

I believe the con­ver­sion of HTML emails could be improved, so there’s still a need to explore this fur­ther.

My plan is to com­pile the script once it’s fin­ished and then run it every 30 min­utes using a sched­uler like launchd. For this, all out­puts need to be redi­rect­ed to a log, and error han­dling should be exam­ined more close­ly.

Some peo­ple have asked for an Obsid­i­an plu­g­in to han­dle this, but I’m not sure a plu­g­in is always nec­es­sary. The great thing about Obsid­i­an is that all notes are Mark­down files, which can also be cre­at­ed with scripts or appli­ca­tions like this one. Not every­thing needs to be a plu­g­in, which could poten­tial­ly slow down Obsid­i­an. Scripts like these only con­sume resources when they are run­ning. I can use stan­dard sched­ulers like launchd or cron to run the script peri­od­i­cal­ly. It fetch­es emails even when Obsid­i­an is not run­ning, mak­ing it a per­fect way to use scripts out­side of Obsid­i­an to cre­ate work­flows for Obsid­i­an.

I hope this inspires you to come up with your own ideas. It would be great if you could share them here in the com­ments or leave a link.