UPDATE: I've temporarily taken the site down due to low usage (who knew people didn't want to replace their faces with melons?), but let me know if you want to check it out. 

melonface.me is a web app that replaces faces in your photos with watermelons. The functionality has also been extended into a Twitter bot, @melonfaceme

Why replace faces with melons? Well, one hot summer day in 2017 I decided that I wanted some watermelon, but couldn't find any. I thought that it would be fun/easy to try to turn people's faces into watermelons instead. What I assumed would be a few hours of putting something quick together quickly morphed into a more extensive endeavor, more on that below. 

Process

I wanted to extend the open source computer vision library OpenCV with a Node.js app that would provide the functionality described above. Installing OpenCV on my Mac turned out to be quite difficult as the binaries for the older version that's compatible with the Node bindings I was using don't continue to be well maintained, but after a few hours I was able to get it installed. I got a basic prototype working pretty quickly after that and decided to push it to the web, which is where the real problems started. 

For whatever reason, Heroku wasn't able to install the OpenCV dependency I was using. I tried for hours on end and scoured the internet for solutions; none of the custom buildpacks I tried were working for me either and I couldn't get past this point. I gave up on the DIY computer vision aspect and switched to using Google's Cloud Vision API. If you want to look at a comparison of face detection with the Google Vision API and OpenCV, check it out here (I guess this is a spoiler because it indicates I eventually got it working).

Memory issues

In my local testing, my RSS memory use hovered around 150 MB when processing a single image, so I didn't foresee any memory issues on the web. Boy did I have another thing coming for me. Once I got everything running on Heroku, I fired up a test image and started watching the memory use. It started out at 300MB, and quickly skyrocketed up all the way to 800MB. Not good. Not only that, but none of that memory would ever get cleared without rebooting the entire dyno. 

This led into a multi day investigation of the purported memory leak, which was nowhere to be found. I spent hours investigating this, and thought it could have to do with the image library I was using (Jimp), so I switched to sharp. Turns out this wasn't the issue, but sharp performed much better anyway so I stuck with it. Eventually I decided to just switch to a server with a ton of memory because I didn't care. 

One of the first options I tested was Now, which offered pretty competitive pricing to Heroku (oh all the money I've spent on useless web apps). I fired app the app on Now, and what would you know... memory use was hovering between 100MB and 300MB at all times, even when sending concurrent requests. Turns out, Heroku just sucks. 

Image rotation

When testing on iPhone, I found out the iPhones tend to upload images with an incorrect orientation. This doesn't make any sense to me, but I spent a while figuring out how to read the EXIF data to rotate them. I ended up using a server side library, but also experimented with a client side implementation. The problem with the latter was that the image it was generating with the corrected orientation didn't have any compression applied, and tended to be considerably larger than the original iPhone image. For example, a 600 KB selfie could go all the way up to 5.5 MB, which makes for a slow upload (on mobile), and is also above the acceptable size for the Vision API. 

Twitter bot

I wanted people to melon face themselves, but I couldn't figure out how to get them to do so. Initially I just melon faced my friends, but that quickly got boring. I decided to make a Twitter bot that would automatically melon face people, but for that I needed to get OpenCV working because I didn't want to pay for the Vision API credits. Gave it a go and was able to get OpenCV working on Now in about 15 minutes. Further confirmation that Heroku sucks.

twitter_analytics.png

The Twitter bot was pretty successful. I was basically grabbing images using the streaming API and looking for faces. I added a manual approval process because there's a lot of trash on Twitter and I didn't want to post anything crude, but after spending around 10 minutes posting 100 photos (which were replies to people's original tweets), I stepped back to see what the engagement would be. It was pretty good!

Though I'm not quite sure how to quantify the success of a Twitter bot, 4,000 impressions on 100 tweets over a period of a few hours seems satisfactory (and it's all relative, my own Twitter gets basically no engagement). 

twitter_samples.png

Initially my moderation of the photos was limited to 'accepting' or 'rejecting' each melonface, and the tweet content was randomly selected from 15 drafts I had written. I thought this would be enough avoid being blocked by Twitter, but that turned out not to be the case. I ended up also having to write content for each manually, which didn't add considerable time to the process but does make me wonder if this type of Twitterbot could ever be truly automated without anyone continuously writing content. 

Hit me up if you ever want to run the melonface Twitter account, or if you have something actually useful I might spend my free time on.