A little birdy told us that we’ve got some new Fondu Ambassadors…

Say hello and be sure to follow them!

View Fondu Ambassadors in a larger map

*Want to become a Fondu Ambassador too? Learn how at http://bit.ly/yWPwUh*

This one made us smile AND shared a protip. The Most Creative: @DamianDazz

Wondee Siam I - “Hello Secret Thai Menu, I Want!” #6wordreview @fondu

— Damian ‘Dazz’ White (@DamianDazz) April 27, 2012

And here’s The Most Retweeted: @maxmeyers

Fresh sammies, warm cookies, nice staff. #6wordreview of Pret a Manger @fondu

— Max Meyers (@maxmeyers) April 27, 2012

For those following along with the contest, this should come as no surprise. The Randomly Selected: @henrymclay

Props to all of our entrants—be sure to keep up with Fondu on Twitter and Facebook to hear about other upcoming events & contests!

[This is a guest post by Matt Silberstein (matt4c), a Fondu Ambassador in Denver]

We pulled into the strip mall’s narrow, pot hole ridden parking lot on a recent Denver Sunday afternoon.  After tediously maneuvering the car into one last remaining space, we could barely wait for the food explosion soon to come.  We hurriedly opened the door to Lao Wang Noodle House only to be told by a very harried grandma Wang that they were completely out of food.  The place was packed. Sounds of noodle slurping and loud talk filled the tiny, frill free space.

Lao Wang Noodle House is what many would call a hole in the wall.  It’s part of a vibrant authentic Asian Denver food scene found along southern Federal Avenue.  This is not your typical Americanized Chinese food establishment.  No rice, egg rolls or chow mien on Lao Wang’s well focused menu.  Chung Ming Wang and her husband Tse Ming Wang, at an age when many think about retirement, serve up soulful Taiwanese fare.  Hand prepared soup dumplings (xiao long bao), fried potstickers and several types of noodles form much of their Shanghai inspired traditional fare. Seafood is scarce. Expect instead dishes with lots of pork and beef.

Xiao long bao is the one reason you want to try this place.  It originated in Shanghai and I’ve learned it’s a big part of today’s Taiwanese street stall reality.  Steamed, folded, pinched and served piping hot, these bite sized soup dumplings give up a zesty broth explosion upon first bite.  As the hot liquid leaves its tender, smooth skin, a bite of minced pork remains.  Just be careful and avoid eating these whole.  Nothing like burning your mouth to ruin a meal.  Served in a traditional steamer basket, you may be tempted to order a second round of ten.  

Other dishes don’t disappoint. Chinese style spaghetti with savory meat sauce or a silky rendition of wonton soup make for perfect soup dumpling companions.  Order off menu and ask for a healthy serving of flavorsome bok choi.  Prepare to be blown away by the simply prepared pan fried potstickers.  They are the best I’ve had anywhere.  Sharply seasoned cold cucumbers and spiced beef soup are also local favorites.  The service might be a bit slow but I’ve yet to eat a disappointing dish.  

You can wash all this down with an enormous 32 ounce sized cup of water grandma Wang places on your table, standard variety of soft drinks or a cold glass of specially formulated Lao Wang lager from CAUTION Brewing Co, a new brew venture run by the Wang’s son in NE Denver.   Lao Wang lager includes special spice blends designed to pair well with chinese food. Craft beer designed for dumplings.

If you can, grab a table early because it’s always busy. We like to go when they open at 11 am (Tue-Sun, closed Mondays).  If you go later you may risk a severely long wait or worse, a shortage of food.

(Lao Wang Noodle House, 945 S Federal Blvd, Denver, CO 80219)

This blog post, we’re taking a detour from food and dining and will be diving into the back-end technology stack that makes Fondu work. It’s something that we spend a ton of time thinking about and are really happy to share.

I’ll try to cover the technologies in a semi-cohesive manner but forgive me if it jumps around. This post covers a pretty broad, but specific, topic. This post focuses on the server-side of things. The iOS app has enough meat to be it’s own post in the future.

But first, a picture of our datacenter:

Amazon Web Services

Like a lot of other tech companies, we host our infrastructure on Amazon Web Services. We probably use more of their offerings than most. Specifically, our servers are on EC2, in a Virtual Private Cloud (VPC). That lets us maintain different subnets for dev and production, and maintain subnet-wide routing rules. To the public, these servers are exposed via Elastic Load Balancers (ELBs).

Our DNS is powered by AWS’s Route 53 service, which has treated us well so far. It’s got a really easy interface, and integrates with ELB’s seamlessly.

All of our image and file data is stored on S3. Although the latency sucks, you can’t really beat it for simplicity, reliability, and throughput speed. When being served, all that data is run through CloudFront, Amazon’s plug-in CDN service. Sooooo easy for such a big performance gain. We even built a custom image manipulation library, prism, that ensures derivative/thumbnail images are generated and stored in S3 for later use.

Our worker queues are run on Amazon’s Simple Queueing Service (SQS) as well. Once again, latency sucks (and you have to dumb-poll them), but for the price and simplicity, it’s hard to beat if you don’t mind a second or two of latency on the output end.

Back when we were on MySQL, we used RDS as well and were quite happy with it. Would love for AWS to start supporting admin-free servers for other DBs (without locking yourself into Dynamo).

MongoDB & Elasticsearch

Fondu’s primary data source lives in MongoDB. All necessary data is also indexed into our elasticsearch cluster for when we need to get loose with our queries.

We’ve been using Mongo for quite a while now (prototyping for the last year and a half, fully-committed in production since the fall). It’s been a godsend to us for the fast reads, and the schema-less flexibility. A pretty common practice for us is to handle both sides of a data migration in code, deploy it to production, run the data migration script, and then clean out the “old” schema handling from the code. Zero-downtime migrations for most types of schema manipulation, without locking the db. Most excellent.

MongoDB and elasticsearch are both JSON (fine, Mongo is BSON, I know), based data stores, meaning there’s no painful transformations to make one format fit into the other. The only real downside is that there’s no established and reliable way to mirror/transform MongoDB data modifications into elasticsearch. As a result, we end up updating our search index from the application layer whenever db objects are updated.

We host MongoDB and Elasticsearch ourselves. One common trick that folks use to increase disk performance on EC2 instances is to store the database directory on an EBS raid array. Each of our production servers has a 12 disk RAID0 array. with an additional 2 spares each. MongoDB and Elasticsearch both run in their own version of a replica set, so each server holds all data in case one fails (Also nice for zero-downtime updates).

Python

Python is our language on the backend. It lets us iterate incredibly quickly, has good package support, and is really clean to read and write. It’s functional built-ins come in handy from time to time as well. Ruby vs. Python is just a religious war, but whatever you’re using, having a dynamic language at your disposal is crucial for keeping a small, resource-strapped team moving fast. Static languages are safer and faster, but lose the flexibility of “just make it work” judgement calls that sometimes need to be made.

Another perk of building our runtime in Python is being able to access code from Fabric, which we use for deployment and server management. Some of our datacenter configuration is programatic, so sharing that with our deployment tool is one less source for errors.

Pyramid

We use the Pyramid framework to handle web requests. We’ve played with a good 8 or 9 Python web frameworks (Django all the way to Bottle), and currently Pyramid feels like the right balance. It covers most aspects of web serving you could want, but avoids being monolithic by acting as a glue layer between best-of-breed standalone libraries. The fact that an ORM/model layer is option when you start up a project says something about how little it assumes.

Workers

We use Amazon’s SQS to handle queuing of worker tasks. Each queued request is simply a chunk of JSON that maps to a specific module, method and params for our python shell interface into the system. This also makes it really easy to run worker tasks from the command line for maintenance or testing.

Integration

Facebook. foursquare, Twitter, Tumblr. We try to integrate 3rd party services in ways that naturally suit the platform. Everything’s being integrated nowadays, as it’s generally a better experience for the user. On the other hand, we try to keep 3rd party specific knowledge out of our core library. Fondu’s core should only care about how Fondu works, and how to call the middleware modules that encapsulate foreign API calls. That lets us maintain pure API libraries, and pure core system libraries, while letting the middlewares get as hairy and ugly as they needs to. As long as the middleware serves primarily as a transformation layer, they general stay thin and wide, preventing over-complexity.

We even ended up writing our own foursquare library, after being dissatisfied with what was already out there. No Fondu code in there, that’s what our integration middleware is for. Foursquare recommends it if you’re running Python.

Monitoring

We use a combination of Munin and AWS’s CloudWatch to monitor our servers. Our logs are handled by Graylog2 and we’ve been quite happy with those party gorillas. We used to run our logs to Loggly, but do to threading problems in the Python library, and Loggly’s inability to stay up for more than a week at a time, they’ve been moved to “backup” status. Graylog2 has a really easy router to ship logs off to Loggly in the background.

Future

We’re quite happy with the flexibility that working in Pyramid and Python has allowed, but we’re seeing bigger picture for our in-browser experience. We’re currently building a node.js based web-app. Playing with backbone.js at the moment, but we’re still deciding on the client-side structure. Running javascript on the backend provides a lot of benefits that wouldn’t come as easy as if we had kept going down the Python path (for the web server). The API is staying Python, both because of the large body of work it represents, but also because an async, event-based logic core of that size is enough to keep a one-man backend team up at night in cold sweats. So much potential for failure, so much trickier to debug and follow flow.

We’re hiring that web developer right now. Somebody that owns everything that lives inside the browser, and a little more. Node.js, client js framework, CSS, the whole shebang. Think you can top Twitter’s website for usability and smoothness on a seal-sized team? Ya, here’s my number, so call me maybe.