Obsidian Publishing Workflow

Last updated Aug 22, 2022

• Type meta
• Status fluid

By utilizing all the great and publicly available open-source tools, meticulously designing, and persevering despite the hardships, technical limitations, and lack of expertise, I’ve been able to automate my Obsidian publishing workflow. After 3 whole days of constant (and quite frankly back-breaking) development, I’m now able to update my site from within Obsidian with a single Hotkey. I’ll now break down how I’ve achieved this and how it’s all connected.

• Obsidian: For note-taking and note-making.
• Obsidian Git: A plugin that sync my vault with the GitHub repo.
• obsidian-export: A CLI written in Rust that processes and exports notes from Obsidian.
• hugo-obsidian: A CLI written in Go that creates indexes that can later be used by Quartz to show graph etc.
• Quartz: A theme for Hugo SSG with extra features like graph view, backlinks, latex support, etc.
• Hugo: A static-site-generator written in Go.
• GitHub Actions: Allows us to automate our workflow based on GitHub events.
• Netlify: Deploys our website.

My obsidian vault is backed up in a private GitHub repository. This has been the case from the start. This allows me to have a reliable, free, and private backup of all my notes. I can also revert to a previous time, if need be.

Obsidian Git Plugin allows me to either set an interval to automatically back up my vault or do it manually using a Hotkey. On each backup, it first performs pull (download from repo if any changes have been made there) and then push (upload all the changes to the GitHub repo).

Here’s the link to the actual configuration file, GH Action configuration file to automatically publish my Obsidian Vault using Quartz (github.com). On each successful push to the repository, a GitHub Action gets triggered. It has all the steps listed one after another to build and deploy the website.

• First, it checks out my notes and the Quartz theme.
• It then downloads obsidian-export to process the notes.
• After that, it uses hugo-obsidian to generate link indices.
• It then builds the Hugo website using the Quartz theme and the processed notes.
• Lastly, it pushes the built website onto the gh-pages branch.

Here’s a detailed overview of what obsidian-export does. It is a rust CLI written by Nick Groenen to process notes from obsidian.

It has several helpful features by default, such as:

• Reference Embedding: If you have a reference to a file/section, it will embed that into the actual document.
  • It doesn’t currently work with Block-level References.
• Notes Filtering: You can specify all the files that you don’t want exported in a .export-ignore file.
  • This allows us to have more control over which file and directories we want to make public.
• Markdown Links: It will convert all the WikiLinks ([[]]) into Markdown Links ([]()). It will also make those links relative to the root path.

To make it work for my particular workflow, I modified it to include the following features:

• Retain WikiLinks: I’ve added an option to retain the WikiLinks format, as Quartz and many others are able to parse those easily.
• Hugo Frontmatter: Add new or rename existing yaml properties, so they work with Hugo.
  • If there’s no title property, generate one from the file name.
  • Remove alias/aliases as they have a different functionality in Hugo.
  • If created or modified properties are missing, calculate them using file creation and modification time. Rename them to date and lastmod respectively.
  • If summary is empty, remove it.
  • If there’s a publish property, rename it to draft. This allows us to exclude certain pages from publishing.
  • Rename id property to url.
    • All of my notes have an id property in the YAML frontmatter.
    • It is a nanoid which ensures that each note will have a unique URL-friendly string.
• Flat Hierarchy: Add option to generate a flat hierarchy.
  • It replaces . and \ with -.
  • If the path beforehand is Some.Dir/Some.File.md, then it would become Some-Dir-Some-File.md.
  • Hugo doesn’t work well with files and directories having a . in their path.
  • This option won’t result in any name collisions.
• Embed Info: Add some info regarding the embedding in surrounding <div> tags.
  • By default, obsidian-export has no indication that the content was embedded or not.
  • I’ve added <div> tags around the embedded content, so we can later style it however we want.
  • I’ve also added a link to the file that was embedded.
  • This would allow us to create an experience like in Obsidian, where we can go to the original location of the embedded content.

It scrapes all the links from vault and generates link and content indexes. These indexes are later used by Quartz to generate graph and to enable full-text search.

I’ve modified it to my liking as well:

• URL based: Use url property from the frontmatter for links.
  • By default, it uses file names to create Source and Destination links.
  • I’ve modified it so that the Source and Destination point to the url values instead of the file names.
• Strip Comments: Remove obsidian comments.
  • This gives us more fine-grained control over what we want to share.
  • If there’s a section within a file that we don’t want to share, we can simply wrap it inside Obsidian Comments %%.
  • I use this around Dataview tables and properties, as they don’t provide much value when published.
    • In Live Preview mode in Obsidian, the Dataview tables will remain visible.

Whenever gh-pages branch is updated, a Cloudflare Pages deployment action gets triggered. It looks at the content in the gh-pages branch and deploys it to the web. I have to use this because GitHub Pages doesn’t work with Private Repo.

First, I used Netlify to publish my site in the end, instead of Cloudflare Pages. But soon, I discovered a problem with that.

Whenever I refreshed my page, the URL would instantly become lowercase. For example, if the URL was https://notes.aadam.dev/SBYNtPHqsTW9Ck1Kuoxsu, then after page reload, it would automatically turn into https://notes.aadam.dev/sbyntphqstw9ck1kuoxsu/.

This is a major issue, as this doesn’t guarantee that all the IDs will be unique now. Before, AbC and aBC would’ve been treated as two different entities, but with this bug, they would resort to a single entity, resulting in a naming collision.

Also, even though my page was being loaded, the graph at the end of the page wasn’t working. This is because it also expects the IDs to be case-sensitive. So, when the URL turns to lowercase, it isn’t able to find that ID.

I spent a couple of hours trying to fix this problem. I wasn’t even sure where to look. There wasn’t much information about this on Hugo Forum, Stackoverflow, Google, etc.

At first, I thought that there’s some issue with Hugo and how I deploy it. I changed my GitHub Action workflow and tried different things. I tried moving the building process to Netlify instead of GitHub, but that had its own issues. I also tried moving to Vercel, but that didn’t work out either. I tried changing Hugo configuration files but still, no luck.

In the end, I came across an article which pinpointed the issue for me: Gotcha: Netlify Makes All Your Filenames Case-Insensitive · Jamie Tanna | Software Engineer (jvt.me). Lo and behold, it was Netlify all along. I was surprised because the website was working fine locally on my PC, but it was acting up when published.

Because of this, I moved my deployment workflow to Cloudflare Pages. Everything else is just the same, only the last part of the website deployment is shifted.