Amazing that it’s been half a decade now since we got the new blade servers that we are running on (got them in June of 2006).
Sadly, 12GB in a server just isn’t what it used to be and the database servers are really starting to feel strained as they beg for more memory. Handling ~25GB of databases on servers with 12GB is just no bueno. I thought of upgrading the RAM in the blades, but the maximum RAM the BIOS supports is 16GB, and all the DIMM slots are in-use… so it more or less would mean tossing all the existing RAM and buying 160GB of new RAM. In the end spending that much money to gain 4GB per server isn’t really worth it (especially since 16GB wouldn’t really be enough either).
I started to do a little research on what would be some good options for upgrades… I think I want to stay away from blades simply because we did have the daughter card in the chassis fail once which took down all 10 blades (the daughter card controls the power buttons). Buying 2 complete sets of blades/chassis is overkill just so you have stuff still up if one complete chassis goes down. The “must haves” on my list are hot swappable drives with hardware RAID, hot swappable redundant power supply, some sort of 10Gbit/sec (or higher) connectivity for communicating between servers. On top of it all, the servers need to be generally “dense” (I don’t want to take an entire rack of space).
PowerEdge C6100
The Dell C6100 actually looked really nice at first glace…
Pros
- Super dense (4 servers in a 2U package)
- Redundant power supplies (hot swap)
- 24 (!!) hot swap drives
- Supports 10GbE or even QDR Infiniband (40Gbit/sec
Cons
- Age – the server itself came out about 18 months ago without any refresh. That means you have hard drive options and CPU options that are a year and a half old
- Price – OMG… a single unit loaded up with 4 nodes, drives, RAM, Infiniband, etc. works out to $54,709 (before tax).
The age factor really becomes an issue when it comes to the disk drives… you can get 15k rpm drives in a 2.5″ form factor, but Dell only offers 146GB models (there are 300GB models now). The CPU isn’t really too bad… Dell’s fastest offering is the Xeon X5670… 2.93Ghz, 6 core @95 watts (wattage is important because of so much stuff crammed in there). There is a slightly faster 95 watt, 6 core processor these days… the Xeon X5675… the same thing basically, just 3.06Ghz. 0.13Ghz speed difference isn’t a huge deal… but the hard drive difference is a big deal.
I started to think… well maybe I could just order it stripped down and then I could just replace the CPU/hard drives with better stuff. Doing that, you still end up spending about $60,000 because you end up with 8 Xeon processors (valued at about $1,500 each that you just are going to throw away).
Then I started to think harder… Wait a minute… Dell doesn’t even make their own motherboards (at least I don’t think so)… so maybe I could find the source of these super dense motherboards and build my own systems (or just find the source of something similar… which ended up being the case).
SuperServer 2026TT-H6IBQRF
What do we have here??? The Supermicro SuperServer 2026TT-H6IBQRF is more or less the same thing… 4 servers per 2U, hot swap hard drives, same BIOS/chip/controllers… supports the same Xeon processor family (up to 95 watts)… And as a bonus, Infiniband QDR is built in (it’s a $2,596 add-on for the Dell server) as well as an LSI MegaRAID hardware RAID card (a $2,156 add-on for the Dell server).
So let’s add up the cost to build one of these things with the CPU/hard drives I actually would WANT…
- Chassis (includes 4 motherboards, 2 power supplies, hard drive carriers, etc. – $4,630.98
- Xeon X5675 3.06Ghz CPU – $1,347.84 each, so $10,782.72 for 8
- 8GB ECC/Reg DIMM – $124.26 each, so $5,964.48 for 48
- 600GB Seagate Savvio 10K.5 – $391 each, so $9,384 for 24 (also 410% more capacity, 21% faster and 20% more reliable than the 146GB options from Dell)
Add It All Up…
$30,761.88 would be the total cost (a savings of $24,138.82) and in the end you get slightly faster CPUs and *way* better hard drives. So in a single 2U package, you end up with 48 Xeon cores at 3.06Ghz, 384GB of 1333Mhz memory, 14.4TB of drive space (9.6TB usable after it’s configured as double redundant parity striping with RAID-6… which should be able to do more than 750MB/sec read/write). 8 gigabit ethernet ports and 4 Infiniband QDR (40Gbit) I/O.
Get 2 or 3 of those rigs, and you have some nasty (in a good way) servers that would be gloriously fun to run MySQL Cluster, web servers and whatever else you want.
We have several of those chassis, very nice. At your drive pricing you might want to consider the intel 320 300GB drives, the performance difference is nothing short of awesome.
We’ve also been using the new 9265-8i controllers, they do a reasonable job keeping up with the ssds. For the sequential case at 8 drives we are PCIe bandwdith bound.
I put some of the raw notes from my testing of 48 of the older x-25m model drives on a pair of the 9265 controllers here:
https://gist.github.com/936429
At wikia we are in the process of switching all of our mysqls to the 8×2.5″ 1U chassis with the 9265-8i and 8 300GB ssds.
Yeah… I actually looked at those exact SSD drives (and some others). The ones I *really* would want are about $1,100 each… which would have more than doubled the cost of everything.
For the Intel SSD 320 drives, they have a rated MTBF of 1.2M hours and sustained read of 270MB/sec and writes of 205MB/sec and use the 3.0Gb/sec interface.
While the Seagate drives I’m leaning towards have MTBF of 2M hours, sustained read/write of 202MB/sec and use the 6.0Gb/sec interface.
With either drive, 6 drives in a RAID-6 are going to get close to saturating what the RAID controller/PCI bus can realistically pass anyway… so that’s why I’ve been leaning more towards the “more reliable” and less expensive non-SSDs.
Hey Shawn,
Are these servers purely to power the sql requests for the keyword tracking tool or also to balance the forum? You could possibly do some caching and spriting as well.
The forum is a VERY tiny part of what our backend DB servers handle (the forum takes *maybe* 1% of the DB resources).
http://tools.digitalpoint.com/tracker.php
http://tools.digitalpoint.com/geovisitors.php
http://tools.digitalpoint.com/phonehome.php
http://www.digitalpoint.com/tools/counter/
All take FAR more resources…
But the biggest resource hog is Digital Point Ads. There is always going to be at least one SQL query performed for every page view of any site using it, and potentially up to 4.
http://www.digitalpointads.com/
When tuned properly, the new servers setup listed above should be able to easily handle serving ads (and calculating realtime bidding costs) for 1,000,000+ page view per second. It would also be setup to scale linearly simply by adding additional nodes.
For sake of theory/argument, let’s say we started approaching that level. At that point we would be serving more than 100 billion ads/day before we would need to start thinking about adding additional hardware (raw bandwidth would be more of an issue at that point anyway… you would need ~20Gbit of Internet connectivity. I don’t think even Google AdSense serves 100 billion ads/day.
The point is, it should allow some room for growth. 🙂
Wow, that’s quite an accomplishment. Only 1 person has bought ads from DP ads on my site but my site still serves 1M+ impressions per month from your site so does that mean it counts a page view even if it doesn’t show DP ads?
Yes… the system still need to do one backend query for each attempt to serve ads (page view) to see if there are any to serve.
As far as not getting many advertisers… are you promoting it on your site? The ones that have the greatest success with it, are the ones that tell their users that they can advertise. The people that are going to have the greatest interest in advertising are the ones that use your site.
I can’t wait till DP ads becomes the norm in online advertising. 😛
shawn I have question ?
how many bandwith do you need (digitalpoint forum ) when your concurrent visitor is 20.000 ?
could your server of SuperServer 2026TT handle 60.000 concurrent user of vbulletin ?
i mean , how many bandwith do you need (digitalpoint forum ) when your concurrent visitor is 20.000 ?
do you need dedicated 1 gbps for that ?
and where did you put your server ? san jose dc?
Shawn love reading about your server upgrades but seeing as MySQL Clusters are mainly memory based fast SAS 15K is more than suited. But why go Xeon X5675 at US$1,350 each when you can go Xeon E5645 2.4Ghz hexa core 6 core/12 threads at US$590 each ? On 24 cpu purchase, you’d save US$19,200 going Xeon E5645 money you can put towards future SSD or memory purchases 😀
Yeah, I didn’t realize the E5645 was so economical. Definitely worth looking into.