Larry (@lhalff) and I have been recording a podcast for the past year called Citizen Garden that covers various topics related to the web, technology, and social networking.
Well, given Ma.gnolia’s recent catastrophe, we decided that episode 11 would dedicated to exactly what went down and why, and what lessons Larry has learned that others should heed in order to avoid facing a similar crisis.
I think the basic take-away is that, four years ago, when Larry started Ma.gnolia, your IT options were pretty much to use commodity shared hosting or to do it yourself. If you used Ruby on Rails — in which Ma.gnolia is written — your options were even more limited. And so Larry chose to do it himself.
Today, with services like Amazon S3 & EC2, Joyent Accelerators and Google AppEngine, reliable, scalable hosting is no longer as much a problem, as these services have risen to meet the needs of applications like Ma.gnolia. But these are services that Larry did not take complete advantage of and the burden of taking care of over half a terabyte of data eventually caught up with him.
All is not lost necessarily, and Larry hopes that Ma.gnolia will someday return, perhaps as an invite-only service to start, in order to give him time to earn back people’s trust and scale the service slowly. I’m also confident that he’s decided to completely outsource his IT, taking the lessons from this current situation deeply to heart.
This episode is also downloadable as an MP3.
Factory Joe 
On just 4 Mac minis amazing! 🙂
Make sure you employ a pessimist.
Wow. I like his honesty.
But then I use Delicious.
Hi guys – thanks for the discussion. Can you recommend outsourcing options for start-ups?
What should go out, what shouldn’t?
Cheers,
-Robin.
I think it’d be interesting to have a site–kind of like http://highscalability.com but focused on the smaller guys–where startups or web app creators could disclose the infrastructure they use to power their sites.
There could even be a sort of rating system to help potential users determine the reliability or disaster preparedness of sites before “investing” in them.
It might even serve as an incentive for some startups to get their stuff together so that they can “win over” new users.
Just an idea.
Great podcast, and I really appreciate Larry’s honesty and humility. I agree that it’s really amazing and telling that such great services can be built with (relatively) inexpensive off the shelf hardware. Good luck with Ma.gnolia 2.0.
I just wanted to take a moment to say thank you for doing this. I really miss ma.gnolia and would love for it to come back.
Great job putting this together.
This is a fabulous discussion, I really appreciate your candor and tips. I’m reminded of a case I read about the creator of MultiMaps, a UK start-up similar to Mapquest. He had the server in his bedroom. I’m so glad I have Amazon available to me nowadays.
thanks Chris and Larry for this post – very interesting. What the interview seems to say is that the hardware and disks were working fine and this corruption is a problem with MySQL 5. Can you say more about this? Larry says he made a ‘huge mistake’ but he also says he doesnt know what went wrong – he’s waiting on recovery experts.
Using cloud services is a great idea but self-hosting has its advantages. Storage is cheap and having it local will always be faster than on the other end of the Internet. Using four mac-minis as the webservers is awesome since pre-corruption, the site was running fine.
Also, whats the 2^8th doing on the wall?
Delicious isn’t the heavyweight because of Yahoo’s involvement, it’s there because it has legacy, credibility and stability. Those of us who used it when it was del.icio.us trusted it then and trust it now. I did use ma.gnolia but it never had ‘stickability’ for me.
If nothing else this frank and illuminating interview reveals the ‘smoke and mirrors’ nature of emerging social technology.
Better late than never. Thanks Larry for granting us insight into the loss of our bookmarks. It’s good to hear you take responsibility and provide details. I’m really looking forward to using ma.gnolia again!
Outsourcing is about the worst mistake you can make. Your best bet is to hire a unix geek, one who built a home network incorporating as many enterprise features as they could “for fun.” Even a few years ago when magnol.ia started there were VPS’s and managed hosting with automated backups. A good unix geek will know about those and be able to find one at a reasonable price.
Outsourcing means you pay more everytime you need something different. Having your own unix geek on staff means you have someone who has a vested interest in keeping things going at lower cost, who can identify when your current resources are sufficient to do that new thing you want to do, and most importantly can identify when you can turn things costing you money off.
This is excellent! Thank you for sharing, lessons learned are one of the rare gems that are not shared enough or listened to.
There are also great nuggets of information in this are a broad array of subjects.
I am missing Ma.gnolia, but I like the idea of an iTunes Genius for bookmarks to aggregate link lists (similar to DevonThink, but in my context) and other new related materials I have not seen.
I am with Don Park. Very curious as to what happened since you hinted in the video that it wasn’t hardware and it was due to some data corruption.
Larry, could you at least let us know if you felt it was due to some MySQL specific issues?
the burden of half a terabyte of data.
i understand that burden. i have half a terabyte of data in my macbook although soon i’ll nearly double that with a second drive to replace my optical drive. i use time machine to back it up onto 1.5 terabyte drives.
i am a professional, but if all my data was lost, it’d just be my ass, few people would scream at me. most everything important is somewhere out there. obviously, if i was taking care of other people’s data, i’d take pains to ensure it was safe.
i don’t get it. half a terabyte of data. no backups.
my own personal site uses a mysql database over 130mb in size. it is backed up every day.
i guess half a terabyte of data is just so much bigger… :p
come back magnolia!
I think it is inescapable – there was a level of negligence in developers rushing an application to production without any IT engineering expertise involved. IT engineers, the guy recommending the On Staff Unix geek, the guy talking about backing up his 150GB mysql DB daily, can simply see through this. Those of us who do this professionally for enterprise class organizations know, there are no excuses for failure to protect data – there is generally simply job termination. I suppose, in a bizzare sense, Larry terminated himself.
This is not “griefing” or being “unduly negative” – it is how IT engineering *operates*. There are certain skillsets you need to bring to the table when you’re dealing hands-on with production environments. This is why developers should stick to the scratchbox machines and not have production access.
I can see a challenge here – and it is complex. Who regulates who, how, and when a developer with a good idea makes it available to the public, and what is that developer’s liability? We don’t want to stifle innovation, but there needs to be responsibility and accountability as well. Outsourcing sucks, as an IT professional, I can imagine why you don’t want to outsource to the server version of a “puppy mill” where a bunch of nameless IT engineers babysit your server with no real involvement in what it is or what it does. But as a startup, you can’t afford a guy like me to be on staff, can you?
A good place to start, though, is knowing your limitations, and I believe Larry didn’t know his until TOO late. He might have wanted to partner up with an IT professional. Ultimately, that is the problem. Ma.gnolia was running in production but was only 75% thought out. This illustrates the basic failure of the cloud, and while fat, local computing will still remain the most viable, frequent model of computing as we move forward.
Very Informative and Knowledgeable Interview.and good tips for new born companies. i forward this interview for my all friends and my orkut list.
Regards
Nilma Azeem
I just wanted to take a moment to say thank you for doing this.
Thanks for keeping this up.. good video 🙂