Publishing to my website instantly with Dropbox

November 3, 2017

I like files.

When I was a kid just learning how to use the computer, I would organize all my files into folders by type: my Kid Pix drawings into one, my SimCity cities into another, etc.

I still have distinct memories of thinking through the best ways to organize my files so that everything important to me was tucked away nicely and easy to find whenever I wanted it. This may have been an early sign of being a clean freak (or put more nicely, a minimalist) but file organization helped me maintain a peace of mind in an otherwise disorienting and scattered world of computing. It provided me a sanctuary of sorts within a broader realm of adventure.

I remember when Dropbox came out in 2009 that I realized immediately how this digital sanctuary could be extended to the cloud by turning one's strictly “local” collection of files into one that synced to "remote" storage on the internet, accessible wherever one went and with whatever device one may have on hand. The power of Dropbox was that I could simply drag all of my organized files into it and they would be instantly transformed from an isolated digital homebase to an omnipresent one, linked and yet independent from the physical reality of any one personal device.

Meanwhile, however, I've built a career on developing websites and applications that are backed by all sorts of databases, not the so-called flat files I have come to love in my personal life – those that any normal human could open and read without special software. Databases have all sorts of advantages in making it easy to query and relate data, but fundamentally, they fall short in terms of transparency, transportation and transformation when compared to files. It's simply easier to see content, move it somewhere else, or change it when that content is stored regular files as opposed to databases.

The content we each produce as individuals in increasingly important to the world in which we live, and the nature of the Internet in particular. Each of us is a small publishing house and content producer, and every year we see the growing power of that role we play in the public realm. But we're terrible content managers and crafters, with either our public or private data.

We tend to throw our content out there to whatever distribution point makes it easiest to get it in front of other people, but that content ends up as rows in databases we hardly control and displayed on websites we hardly design. Even if you host your own blog on your own domain, chances are the content is stored in a database and on server for which you can't quite remember the password. Sure, it's out there somewhere, but it'd be a lot more powerful to put it all at the immediate control of your finger tips, just as you have control over the files on your laptop computer using a wide range of learned techniques.

About a year ago I started to attempt a convergence of my two digital worlds – my sanctuary of private files and my slew of poorly managed public (or semi-public) content online – by releasing a new version of my website, powered by some custom open-source software I created under the broader tent of Neotoma. The website is powered entirely by flat files that I edit directly on my MacBook, iMac or iPhone. These files are loaded by the software's server piece so that the website piece can load the data it needs to show without involving any sort of database. For example, if you ask the server directly for the content that makes up this post, you'll see it provides exactly the same content stored as a file on my computer:

Until this week, however, any of these files saved to my Dropbox that I wanted to publish to my website had to be copied manually to my website's server using a script, making the convergence I was looking for hardly seamless. My goal has been to hit the save button on my computer's text editor and immediately have any changes go live on my website, without any other extra step required. That way, there's never a question of my content being out of sync between my private and published worlds. I simply have to make the choice as to what content I want private versus public, and that decision is made simply by the organization of files into different folders on my computer. The rest is magic.

I met that goal this week after noticing that Dropbox has a handy Linux version of their sync application that I could install on my website's server and configure to selectively sync just the folders I want to make public. After setting the app up, I now just need to save any file into Dropbox that I want published and wait momentarily for my computer's Dropbox sync application to upload the changes to Dropbox then download them to my server where they'll be published instantly.

I'm hardly the first person to think of this as a technique for personal website publishing. However, I hope to push the principle a lot further by automatically copying my content from a variety of database-driven, corporate services (such as Facebook, Twitter, and Foursquare) to my Dropbox in addition to my independently created content.

That copied content will then get automatically synced not only to my “local” devices (e.g. laptop) for backup as files but also to my website for republishing any way I choose. Aggregated republishing onto one's own website is one of the main use cases for the Neotoma sync service, and I've decided recently that I want to focus on developing the sync service primarily for this use case, dogfooding it on my own website and setting the system up for friends who want to self-publish in the same way as well.

Copying data from a wide range of services into this system will take time and lots of work, so I'm looking to release support for them iteratively, starting with Foursquare since it's an app I use intensively. I'm currently focused on setting up Neotoma to sync my check-ins from Foursquare to my Dropbox and subsequently to my website where I can list them in reverse chronological order, creating a timeline of my life (with photos and other content related to my daily travels).

Then I plan to sync my tips content from Foursquare as well to create custom-designed city guides on my website. These guides will automatically incorporate both the tips and check-ins data I generate on an ongoing basis to display the most up-to-date information about my favorite places in a way that's beautifully packaged, easy to use, and won't go stale.

If you're interested in setting up your personal website in a similar way, I'd be happy to chat about your needs and how my efforts here may help you as well. And if you're a developer interested in helping with any of the software involved, let's talk about Neotoma and where you could make an impact.