Open Source to the Rescue
By Cathleen Moore
InfoWorld
04/25/05 5:00 AM PT
In founding Friendster, Chairman Jonathan Abrams sought to create an online
network through which friends could connect with friends. When it launched,
the service was powered by a Java back end running on Apache Tomcat servers
with a MySQL database. That original architecture was soon crushed by the
coming load of traffic.
Who says open source Latest News about open source can't measure up to
commercial software for mission-critical applications? Far from being a mere
quick fix or low-cost alternative, open-source software is helping
real-world companies solve their most pressing IT problems.
Perhaps no more dramatic example exists than pioneering social networking
site Friendster. When Friendster launched in March 2003, no one imagined
that within two years the site would reach 60 million page views per day.
Unfortunately, as the site's traffic increased, so did its performance
issues. The problem, in essence, was that Friendster had unexpectedly become
a phenomenon.
"When I arrived it was a crisis point -- absolutely, all day, every day,"
says Chris Lunt, Friendster's director of engineering, who joined the
company in the summer of 2003. At that time, he says, Friendster's
architecture was nearly breaking beneath the traffic load.
Fast Track
"[Friendster] had taken off much faster than anyone could anticipate," Lunt
says. "We had our millionth user [when] the site had been up only six
months. The thing was overwhelmed."
Friendster's performance problems needed to be solved, fast. Rather than
stick to the paved road of commercial software, however, the company's
engineers took a major risk by betting on the open-source application stack
known as LAMP, which consists of and is named for -- the Linux OS, Apache
Web server Get the IBM Storage Resource Management Whitepaper. Latest News
about Servers, MySQL database, and PHP (PHP: Hypertext Preprocessor)
scripting language.
Fortunately, that gamble paid off. LAMP not only allowed Friendster's
engineers to scale the site's architecture to address its unwieldy growth,
but along the way, they implemented creative configurations that brought the
LAMP technologies themselves to a new level.
Brought to Its Knees
In founding Friendster, Chairman Jonathan Abrams sought to create an online
network through which friends could connect with friends. When it launched,
the service was powered by a Java Latest News about Java back end running on
Apache Tomcat servers with a MySQL database. That original architecture was
soon crushed by the coming load of traffic.
During the summer of 2003, Friendster was plagued by performance issues.
Often, the millions of users pounding the site where unable to access it,
and when they could, results were inconsistent from page to page. User
profile changes failed to show up because of lags in the distributed
architecture, and messages were dropped.
"If you had a huge network [of friends], you couldn't search ePilot Search
Engine Advertising - Click here to add your site today. it because just
building your list and comparing to the network took longer than the browser
would allow you to wait," says Dathan Pattishall, senior database and
software engineer at Friendster. Pattishall joined the company in November
2003 to tackle the site's database issues.
Tomcat and Java weren't the problem so much as the fact that the site's back
end was not architected to accommodate millions of users. Friendster had
grown to such a huge extent that simply throwing more hardware at the
problem wasn't enough. The site had to be re-engineered to make better use
of the hardware and applications.
Of course, that was easier said than done. At the time, Friendster's IT team
consisted of two engineers, and the challenges they faced were daunting.
"Developing for your desktop is one thing, but when [your site needs to
support] millions of hits a day, it is a different story," Lunt says.
The Tomcat-based Java implementation was bulky, difficult to manage, and
couldn't scale to meet the surge in traffic. Moreover, the MySQL back end
was a bottleneck. According to Pattishall, "Every application and
subapplication had a problem. And we had to fix it."
Lighting the LAMP
Lunt had some prior job experience in the late 1990s with a variety of open
source combinations, including running MySQL and Linux on Intel (Nasdaq:
INTC) Latest News about Intel hardware and developing applications using
Perl as a scripting language. He had also experimented with PHP, a language
similar to Perl but optimized for Web development. After considering his
options, Lunt led the charge to completely re-architect Friendster using
Apache and PHP.
"We had to go back and rewrite the site," Lunt says. "That was not a popular
choice here. They had a couple Tomcat authors here who were pretty
threatened by that."
Still, Friendster pushed forward with LAMP because it was the only toolset
that would allow the company to create its vision, according to Pattishall.
"The one thing we kept the same was using open source software - using
MySQL, code we could look at, and stuff we could download off the Net. This
let us do things [our own] way," he says.
Pattishall says the benefits of open source include better visibility in the
architecture and more control of fixes. "With open source you can look
directly at what is causing a problem, write your own fix, and get up and
running in such a fast manner. Instead of waiting four weeks for a vendor to
provide a qualified patch, you provide the fix yourself," he says.
The transition to LAMP wasn't entirely smooth. "What we were doing was
challenging and hard. We were a top 40 Web site Get a Free E-Commerce
Start-up Kit from Verisign. We were dealing with a volume of traffic that
most sites don't dream of," Lunt says.
Late-night emergencies were common, and firefighting consumed the whole of
many days. The problems were so severe that discussions were had as to
whether it might be more prudent to go a more traditional engineering route.
"There were discussions of ditching open source. Should we just buy
off-the-shelf software that would do what we want?" Pattishall explains. "We
investigated the market but found there wasn't anything close to what we
wanted to do, and it was way too expensive. We wouldn't be able to execute
on any business Get the IBM Collaboration demo for mid-market. plan because
we spent too much just on the software."
What's more, although taking the open-source route may seem intimidating
because of the difficulty in obtaining support, Lunt says traditional vendor
support is not always a safety net, adding, "Commercial software support
contracts are often a support blanket."
Re-architecting the Data
Some of Friendster's biggest headaches came from the site's MySQL
implementation, according to Pattishall. "Everything was wrong with the
usage of MySQL when I came in," he says. "Everything was going through one
location, which was a huge bottleneck. All our writes were taking forever."
Rather than ditch MySQL for a large commercial RDBMS such as Oracle (Nasdaq:
ORCL) Latest News about Oracle, however, Friendster's IT team decided to
rebuild its MySQL implementation, this time engineering it the right way.
"We worked to customize MySQL. We had great support from [MySQL AB, the
parent company of the software]. We wouldn't be able to do this with
Oracle," Lunt says. "The licensing costs would kill us.... It's a good
product, but not that much better to justify the cost."
One problem Friendster's engineers had to address was optimizing the way
queries were performed against the database. "Imagine asking a question to a
database, and in that question, you ask 700,000 to 5 million questions to be
answered all at the same time. That was what was being done to produce the
social network part of site," Pattishall says.
A key design goal for the new system was to move away from maintaining
session state toward a stateless architecture that would clean up after each
request, Lunt says. In addition, according to Pattishall, the team rebuilt
the MySQL data in a more compact form, re-indexing and repairing tables in
order to stabilize the site, improve page load times, and better handle
traffic.
Another important step in stabilizing the site was getting more hardware in
place and setting up a system to automate the hardware selection process.
"Rather than buy big, centralized boxes, [our philosophy] was about buying a
lot of thin, cheap boxes. If one fails, you roll over to another box," Lunt
says.
Friendster changed to a different white-box vendor, Open Source Storage.
"They gave us rock-solid, stable machines so we could skip time in
qualifying the actual hardware itself," Pattishall says.
According to Lunt, another key step to stabilizing the service was to
install a monitoring system that could alert engineers when a system or
hardware piece failed. For that part of the puzzle, Lunt chose the open
source network monitoring program Nagios.
Creative Configurations
With site stability within reach, Pattishall developed some creative MySQL
tools of his own that wouldn't have been possible with off-the-shelf
technologies. To address the lack of management features available with
MySQL, Pattishall developed a management tool that provides a real-time view
of a cluster and that graphs database statistics across time. Pattishall is
releasing the tool as an open source project.
Another problem was the management of the hardware, which proved
labor-intensive; disks often failed, requiring manual replacement. Lunt and
Pattishall looked for a better way to manage the systems, eventually
creating a unique configuration consisting of 64-bit Get your complimentary
copy of the IBM 64-Bit Computing Decision-Maker's Guide. AMD (NYSE: AMD)
Latest News about AMD Opteron hardware running on Linux with a custom kernel
for talking to a SAN.
"The boxes connect over a fiber link, so if a box fails, we don't lose all
those disks and can get back up quickly. No one was using the configuration
we're using," Pattishall says.
Pattishall also developed a custom configuration to load balance MySQL
traffic using an appliance from NetScaler that was originally designed for
accelerating Web pages.
"When our Web server makes a database [request], from the application point
of view it looks like it is connecting to a single server. Because of the
load balancer, we can take one request and put on 20 boxes. We are able to
handle more load than if we stuck all those requests on a single box,"
Pattishall says.
Development on the new version of the Friendster site running Apache and PHP
was completed in June 2004. In the process, Friendster's development team
had grown to 20 engineers, primarily focused on rolling out new features to
the service.
"Open source makes it easier to do that. The majority of new features have
been built on open source," Pattishall says.
According to Tim Denike, senior Unix administrator at Friendster, open
source was the only solution to conquer the site's massive growth problems.
"Open-source tools allowed us to scale a massively complex application
across a system that required very little administrative overhead compared
to those of other companies," he says.
So the next time your company has to weigh its technology options for an IT
project, don't hesitate to throw open source into the mix. Mature
open-source technologies really are ready for prime time; the new and
improved Friendster is living proof.
![[Nokia N91]](https://lh3.googleusercontent.com/blogger_img_proxy/AEn0k_s2GRn1jsJhmeXezxUouCC5c5BJaQ2Zbnon-INdjW-XHy81qJP-i-v4cDkikPQgaw-FhT97DLbDWN9ZxY3WicxKFydCnRDcy6SOphyEBDBadqRoI26xr5mlYbIlSi3AYe4zQmelX10xaSnZgQ=s0-d)
