Xeon server build

Forum to discuss and compare Hardware profiles and Benchmarking
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#41 Re: Xeon server build

Post by pinhodecarlos »

Thank you Scole. Took me awhile to read this and need some time to digest it. I'll come back with some queries.
SETI.USA member
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#42 Re: Xeon server build

Post by noetus »

I would like to butt in here, as since last Sept I've been aquiring both ES and non-ES E5 v3 and v4 Xeons for various system builds. I'm by no means experienced in this, but I have built up a bit of experience in purchasing and trying out several of these. All together 10 xeons purchased, 2 sent back, for a total of 8 in my possession, only 6 are active in system builds right now but the other two I have tested and will build out soon.

- Early on I decided to go four dual socket setups, because there is so much expense in each system tied up in other resources (case, PSU, disks, cooling, etc) that it seemed to me worth doubling up on the processors and spending a bit more on a dual socket motherboard. Especially as the CPUs can actually be quite cheap if buying v2 or v3 versions. I haven't regretted this decision. You get double the firepower, obvoiusly, but without having to spend double the money.

- Motherboards. I tried out the ASRock EP2C612D8C first of all and had no issues with it at all. This is an ATX motherboard, and there are very few dual socket ATX motherboards around. Supermicro makes some too, which seem to be well regarded (I have never tried them) but if you are going for dual socket and ATX I think your choice is basically between ASRock and Supermicro. I bought a couple more of those, and later I bought an EP2C612D8 (without the 'C' at the end) which is slightly cheaper. I had a bit more issues setting this one up as it doesn't seem as flexible in the way the memory is configured, and I had to play around with various configurations before settling on one that worked (and it wasn't one of the configurations recommended in the user manual). I am not sure which of these two I'll get for the next build. One slight issue is that neither board is 100% compatible with the standard ATX mounting hold configuration; there were one or two places where I had to leave a screw out. If I remember correctly the motherboard is held down by only 6 screws, and only two at the top, where the big heavy cpu coolers hang. For that reason I would NEVER transport these boxes upright with the motherboard, CPUs, and coolers installed (they should lie flat with the coolers upright rather than cantilevered and putting stress on the board), but I think it is fine when they are just set up under your desk or whatever.

-Cases. I wanted the ATX because I like the Fractal Design Define R4 cases; good cases that are low price. I have also tried the Define R5 which is newer and updated. I actually prefer the R4 and will go back to that for the next build (it is 20 - 30 € cheaper). The R4 has one weak spot however that if you grasp it by the front to pick it up the top of the front panel tends to pop out, disconnecting the USB and/or audio connections there. It is very awkward to get the connections back in place because space there is limited. This seems to be fixed with the R5. However if you know about this weakness and are careful how you pick up the case it isn't an issue I've found.

-CPU cooler. I have used the Noctua NH-U12DX i4 on each build and I am extremely happy with it. It's a very high quality cooler. Obviously you'll need two of these for each dual socket build. You don't need extra fans; a single fan on each cooler (as supplied) is enough. I also use Noctua case fans on all builds and have been doing so for years without any issues. Some people don't like the look of the Noctua fans but their brown/tan colour doesn't bother me. You can get the black Noctua fans for the same price but these run at much higher RPM and are quite noisy (though push a lot of air).

-PSU. DON'T skimp on the PSU; buy a high quality one (note: Corsair does NOT qualify as high quality). These builds don't need a lot of amperage, and it is much better to go with a low-power high quality PSU than an average or poor quality one with a 'high' output rating. The thing is, if you are not confident of your PSU, and you run into some stability issues, it will be hard to know whether the issue is with the PSU or something else. I have the XFX XTS fanless Platinum 520 W which is plenty enough power for these builds, and the are completely silent (having no fans).

-Disks. Obviously get SSDs, beyond that, it doesn't really matter (though I like the Samsungs). I use a 128 GB one for the system drive, and an additional (usually bigger) SSD for data, though actually a separate data drive on these boxes is a bit of overkill as most data is not hosted on these boxes which are primarily for crunching.

-CPUS. So the bit you actually asked about. I'd add to the above post by saying you have three options: ES ('Engineering Sample'), QS ('Qualification Sample') and used retail. The QS chips are just very late edition ES, sometimes identical or almost identical to the retail release. I bought a pair of ES chips and ended up sending them back because the way they ramped up the frequency in my system was just downright weird (I could never get all the cores ramped up to max frequency, and there was a lot of jumping around.) Since then I have only gone with QS and that is what I would recommend if you are buying non-retail. I am happy with the performance of all the QS chips I bought. You have to be quite careful looking at the CPU details in each eBay listing to figure out exactly what chip it is and whether it is ES or QS, although many of the QS variants will have "QS" in the listing title. Also, I personally would not bid on any listing that does not have CPU-Z screenshots, HWiNFO screenshots, and preferable Cinebench ones as well. I have dealt with several eBay vendors and there is one in particular I like (and have dealt with off-eBay, where I got a 10% discount or something by asking for a PayPal invoice directly and it worked out very well). PM me for details.

Also, feel free to order from the US based eBay site, where you'll find a lot more choice than on the UK one. All the vendors ship internationally for low cost (and the ones I dealt with, speedily) and you can usually get the customs declaration for a fraction of the actual value. The most I paid on customs was 50 €. The two CPUs I had to send back because they were weird was not an issue with the seller I used. (By the way, those ES CPUs worked fine on Linux; it was on Windows they were weird.)

There is a performance difference between the ES, QS, and retail chips; don't believe the listings when they say they are all the same. The prices wouldn't be different if that were the case. I also did get a pair of used retail CPUs and they perform a bit better than the QS ones (probably because they clock slightly higher). And like I said, the ES ones I tried out were a bit strange and I could never get their theoretical performance out of them.

The main difference between v3 and v4 E5 Xeons is not in performance but in power consumption. There is very little difference in their performance in my experience (the v3 are 22 nanometre, the v4 are 14 nanometre lithography) except perhaps for some specialised tasks I have no experience with (anyone?). I have not tried v2 (by now really cheap CPUs); perhaps someone else on the forum can advise on those. They are definitely more power hungry than the v3s again, but I don't know about performance. If you get v4 and the ASRock motherboard, you will need the latest 2.10 BIOS which it does NOT ship with, and you cannot update the BIOS with the v4 CPUs installed (or no CPUs installed). So your choice is either to get a BIOS chip with the 2.10 version preinstalled (eBay vendors sell them; possibly the vendor of the motherboard could too, worth asking) and pop that in ourself (supposedly easy - I've never done it), or use a v3 CPU to update the BIOS, then install the v4 CPUs. Depending on how much you pay for electricity (for me it has been free since last Sept, though this will end in Aug and I will be paying regular grid prices again - albeit in Hungary, the cheapest EU country for electricity), it can be worth paying extra for v4 CPUs, but this is something you'll have to decide.

One possible advantage with some v3 chips is that by dint of a design mistake by Intel some models are actually overclockable. There are details on this forum about how to do that. I haven't yet but plan to when I get my last pair of CPUs built out.

Once you have your system set up, especially if you are using ES or QS CPUs, run prime95 for several days (on high core count systems you might have to run two instances of prime95 concurrently) to make sure the system is stable.

It is possible to get an almost completely silent system, even when using all cores at 100%. With the Fractal Design cases (supposedly optimised for silence, with sound-deadening linings), several Noctua case fans each running at low RPM, the Noctua i4 CPU cooler I mentioned, and the fanless PSU (and no graphics card installed - there is onboard VGA for setup and then I RDP to these boxes anyway), at 32 C ambient temp the CPU temps on the v4 28 core systems are in the mid 50s at 100% on all cores! That is pretty amazing. (Temps on the 44 core system is a bit higher but not too much.)

EDIT: I would be sceptical about the above claim by scole250 that the E5-2690 ES QHV5 will clock to 3.1 GHz on all cores simultaneously. I am pretty sure the 3.1 GHz figure is for one core when other cores are not turbo'ed, which is how Intel defines max turbo for Xeons. There is another listing of this very CPU by another eBay vendor that includes a lot more info (CPU-Z, HWiNFO, and Cinebench as I advised):

E5-2690 ES QHV5

From what I can tell from the data in the screenshots that CPU will max out at 2.5 GHz on all cores simultaneously. Maybe someone else can correct me if that's wrong? That is the CPU uncore listed. I have a similar CPU, the E5-2658 v4 14 core, and it maxes out at 2.5 GHz on all cores, and mine is a QS not ES version. I would verify with the seller what this ES (not QS) chip will turbo up to on all cores. Some eBay sellers are very explicit about this in their listings. Also, because that is an ES and not QS sample, I would avoid it (notice how in the HWiNFO screenshots the turbo is a lot less than expected - why?)

By the way, you cannot trust the official Intel specs for this chip, or any ES/QS samples, as their specs appliy to retail versions, not ES versions.
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#43 Re: Xeon server build

Post by scole of TSBT »

noetus wrote: Fri Jun 30, 2017 2:16 pm EDIT: I would be sceptical about the above claim by scole250 that the E5-2690 ES QHV5 will clock to 3.1 GHz on all cores simultaneously. I am pretty sure the 3.1 GHz figure is for one core when other cores are not turbo'ed, which is how Intel defines max turbo for Xeons. There is another listing of this very CPU by another eBay vendor that includes a lot more info (CPU-Z, HWiNFO, and Cinebench as I advised):
I agree that you need to be skeptical and make sure you research the CPUs thoroughly. I mis-stated the max turbo on all cores but only a little but. It's actually only 3Ghz. While researching I found more than one post with HWinfo pics of a QHV5 with all cores clocking at 3.0. Here's one forum post....https://forums.servethehome.com/index.p ... ost-104865
Now I don't actually have one so I can't guaranty they actually clock at 3.0Ghz but I'd be willing to take the chance. Not all Xeon V4 ES CPUs claiming to be a E5-2690 are QHV5 CPUs though.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#44 Re: Xeon server build

Post by pinhodecarlos »

My budget is £1,000. What should I get?
SETI.USA member
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#45 Re: Xeon server build

Post by Dirk Broer »

pinhodecarlos wrote: Sat Jul 01, 2017 6:59 am My budget is £1,000. What should I get?
I'd say try a 2nd hand Dual Socket motherboard and two likewise Xeon's. For a hypothetical 32 thread Dual Xeon system I advise at least 64GB of ECC RAM -with the option to install an extra 64GB later(cash register sounds, please), and those two puppies need to be cooled well too, to ensure good performance.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#46 Re: Xeon server build

Post by noetus »

scole250 wrote: Fri Jun 30, 2017 11:42 pm I agree that you need to be skeptical and make sure you research the CPUs thoroughly. I mis-stated the max turbo on all cores but only a little but. It's actually only 3Ghz. While researching I found more than one post with HWinfo pics of a QHV5 with all cores clocking at 3.0. Here's one forum post....https://forums.servethehome.com/index.p ... ost-104865
Now I don't actually have one so I can't guaranty they actually clock at 3.0Ghz but I'd be willing to take the chance. Not all Xeon V4 ES CPUs claiming to be a E5-2690 are QHV5 CPUs though.
That's nice; depending on what you do, the extra cycles can make a difference. When I was purchasing my 14 core CPUs I wasn't aware that there were any ES versions that would clock this high on all cores. Research, research, research! Is there a QS version of the QHV5 E5-2690 I wonder?

EDIT: Here's an eBay seller selling a QHV5-2690 also with HIiNFO showing 3.0 GHz on all cores simultaneously. I can vouch for this eBay seller incidentally.
E5-2690 QHV5 on eBay @ 3.0 GHz all cores
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#47 Re: Xeon server build

Post by noetus »

Now here's the thing, and personally this is what makes it difficult for me to figure out exactly what the best purchase options for the CPU are.

Here's a seller selling an E5-4669 v3 QS QH9B. Now this will only clock to 2.4 GHz but it's an 18 core CPU, and slightly cheaper than the 14 core we've been discussing. Performance should be very similar. Admittedly it is v3 not v4; on the other hand it is QS and not ES (which I would feel happier about). If one were shopping in this budget range, how to choose?
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#48 Re: Xeon server build

Post by noetus »

pinhodecarlos wrote: Sat Jul 01, 2017 6:59 am My budget is £1,000. What should I get?
For the CPU(s) only, or the entire system?
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#49 Re: Xeon server build

Post by pinhodecarlos »

Entire system.
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#50 Re: Xeon server build

Post by scole of TSBT »

noetus wrote: Sun Jul 02, 2017 3:06 amEDIT: Here's an eBay seller selling a QHV5-2690 also with HIiNFO showing 3.0 GHz on all cores simultaneously. I can vouch for this eBay seller incidentally.
E5-2690 QHV5 on eBay @ 3.0 GHz all cores
That HWiNFO pic looks strangely familiar :think:
Image
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#51 Re: Xeon server build

Post by Bryan »

Excellent vendor ... I've bought 3 pairs of CPUs from them.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#52 Re: Xeon server build

Post by noetus »

pinhodecarlos wrote: Sun Jul 02, 2017 4:04 am Entire system.
On that budget I am guessing you will go single-socket.

I would figure out how much it will cost to buy the system minus the CPU, then see what is left over for the CPU. For example, you might already have a computer case lying around, or other bits and pieces such as a PSU you can put into it without having to spend much. Perhaps you are going to cannabilise an old system of yours for parts. Or have a look on eBay and see if you can find a PC with bad motherboard someone is trying to flog, that could form the basis of your system. Think creatively!

Then try pricing the components for various types of motherboard you are thinking of For example, if you price a used LGA-2011 motherboard (for v2 or v3 E5 processors) on eBay for the kind of case you envisage (eg. ATX or ITX) then you can view the selection of processors for that board and try to figure out which will get you the best bang for the buck with what you still have left in your budget. Try also LGA-2011 v3 (for later v3 and v4 processors), but you might find that pops your budget when you add in other components.

When you have a possible configuration, post it here (with prices for each component) for some advice. At this stage it would just be an idea, a suggestion to start off some discussion, nothing written in stone.
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#53 Re: Xeon server build

Post by Dirk Broer »

Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#54 Re: Xeon server build

Post by noetus »

That's a great article - it's what got me into these Xeon server builds in the first place! Thanks for posting it.
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#55 Re: Xeon server build

Post by noetus »

How about the following 24 core / 48 thread build:

Component         Model & Source             Price (£)

Motherboard:      ASRock EP2C612D8 Dual Socket     300
CPU:          2 x Intel E5-2648L 1.8GHz 12 Core    270
Memory:       4 x Kingston KVR21R15D8/8 (32 GB)    280
CPU cooler:   2 x Noctua NH-U12DX i4               110
Power supply:     Seasonic G450 450W 80+ Gold       75
PWM Fans:     2 x Noctua NF-A14 PWM                 50
ULN Fan:          Noctua NF-A14 ULN                 20
System Drive:     Samsung 850 EVO 120 GB            50
Data Drive:       Western Digital Red 3 TB         100
Case:             Fractal Design Define R4          75
TOTAL:                                            1330
  • Obviously the total is a bit more than you proposed as a budget, but I just wanted to showcase what is possible, and there are various ways of saving costs. If you already have a case and drives lying about or that can be cannabilised, that could save over 200. You can also drop down to a single socket board (EPC612 series), and/or get a processor with fewer cores. That could save a couple of hundred. You didn't specify your memory needs, and memory is expensive for these servers. You could save another 140 by going for 16GB instead of 32GB. The above list is just a starting off point, and offers some flexibility.
  • The article cited by Dirk features the ESRock EP2C602 series of boards. I recommend avoiding these boards as they are the by now outdated LGA 2011 boards which only let you install CPUs that are v3 (less power efficient) and only up to 8 cores (they maxed out their build at 16 cores / 32 threads). By getting the EP2C612 series (LGA 2011 v3) you can use v3 or v4 (with a BIOS update) processors with up to 22 cores each. That gives you much more flexibility now and in the future if you ever want to upgrade (just pop in replacement processors) and I found the cost to be practically the same for the two board variants when I did a quick price check.
  • The processors I only gave as an example. I linked to a QS (Qualification Sample) listing on eBay, but you can go for the cheaper ES (Engineering Sample) processors although I do not recommend doing that. You can choose a lower clock or higher clock, or fewer cores or more cores, Haswell or Skylake, and so on. There is really a lot of flexibility here, and it will depend on your specific usage scenario. I do think that the 12 core CPUs offer a very good performance/price tradeoff, especially if you install two of them. Before pulling the trigger on any processor purchase, make sure you post about it here with a specific link. And always check with the vendor about motherboard compatibility before purchasing, and tell them the specific motherboard you have or are planning on buying so they can 100% check availability. This is a bit of a hornet's nest, with all the processor variants about.
  • The memory I suggested is on the motherboard manufacturer's QV (qualified vendor) list and is proper ECC server memory. I would recommend using only QVL memory for any server board; you can find the list for this board here. It is clocked at 2133 and you could go for 2400 but there is little point; it is more expensive and offers only fractionally better performance.
  • I like Seasonic as a PSU manufacturer. Corsair is better known, but Seasonic supply PSUs for a lot of the OEMs and the quality is good (better than Corsair imho which is just well-marketed).
  • I've suggested purchasing three case fans for this build, along with the Fractal Design R4 case. I really like this case; it is cheap but doesn't feel cheap, and it offers lots of flexibility. It already comes with two servicable front fans which you can just leave in place, and the additional 3 Noctua fans will provide plenty of air circulation. I would mount the two PWM fans on the top of the case as exhaust fans, and the one ULN fan at the rear as an inlet fan. You could consider the purchase of an additional ULN fan as an inlet for the case side panel, but it isn't necessary (just leave the cover off). It is better to run more fans each at lower RPM; if you run the PWM fans with Noctua's low noise adapters (supplied in the fan packaging) you will find that the build is extremely silent, silent enough to be in your living room with all cores maxed out.
  • With some components it is worth ordering from the USA. I would do this with the motherboard and the memory; these are both easy to pack and ship, and the US prices seems significantly better. The Newegg deal on that motherboard is especially good. To order from Newegg, you might have to use a US-based reshipper such as reship.com to get the thing sent to the UK (with minimal or no customs charges) as they won't ship internationally; I've factored in an extra $50 to cover this. If you can't do this, that motherboard is available from Germany on the German eBay site, though for a bit more.
  • Definitely install your OS on a separate drive and make sure it is an SSD.

    I'll finish off by saying I've had personal experience building using all the above components (except for that particular processor and that particular Seasonic PSU; I've built with other Seasonics though) and I have never had any issues.
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#56 Re: Xeon server build

Post by Alez »

Jeez, I've just finished upgrading /updating all my GPU's ..... now you've got me thinking of updating my servers from gen 6 / 7 HP's #sadface and depleted wallet.
Image
The best form of help from above is a sniper on the rooftop....
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#57 Re: Xeon server build

Post by pinhodecarlos »

Thank you all. I've read everything and basically I need to invest in all parts since I only own laptops. I'm considering those old HP Z200 but I need to read more.
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#58 Re: Xeon server build

Post by scole of TSBT »

See what it cost to for shipping and customs for these systems...
Z600, 2x X5675 3.06Ghz , 48GB RAM

Z800, 2x X5690 3.46Ghz
48GB RAM for this system

Or perhaps a series new LGA 2011 system...
Dell T7600, 1x E5-2687W 3.1/3.4Ghz, 32GB RAM
2nd E5-2687W CPU
CPU cooler
additional 32 GB RAM (assuming base system using 4x8GB sticks)

If the apps you want to run have more need for Ghz than newer chipsets, memory speed and other chip features such as AVX2, then those series systems/cpus are not bad for the money
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#59 Re: Xeon server build

Post by noetus »

Does it really make sense to purchase 6 year old server technology? I imagine the cost savings over more recent hardware gets erased within a year or two if running them constantly. Isn't it better to pay this extra cost for more recent hardware that will be more powerful as well as more energy efficient?
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#60 Re: Xeon server build

Post by scole of TSBT »

noetus wrote: Thu Jul 06, 2017 5:14 pm Does it really make sense to purchase 6 year old server technology? I imagine the cost savings over more recent hardware gets erased within a year or two if running them constantly. Isn't it better to pay this extra cost for more recent hardware that will be more powerful as well as more energy efficient?
Newer would be better. What do you recommend for £1,000?
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#61 Re: Xeon server build

Post by Dirk Broer »

noetus wrote: Thu Jul 06, 2017 5:14 pm Does it really make sense to purchase 6 year old server technology? I imagine the cost savings over more recent hardware gets erased within a year or two if running them constantly. Isn't it better to pay this extra cost for more recent hardware that will be more powerful as well as more energy efficient?
You have to carefully select your Xeons in each generation. TDP varies wildly, together with the MHz.
Socket 2011 (DDR3) doesn't suffer much compared to the newer Socket 2011-3 (DDR4).
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#62 Re: Xeon server build

Post by scole of TSBT »

From my testing running Carlos' app, 14 threads on a Xeon V4 clocking at 2.8Ghz with 28GB RAM reserved runs about the same as a 2x X5675 (24 threads) using 36GB RAM. A 10c/20t V4 ES system clocking at 2.4Ghz on all threads should do as much. I'd estimate $1,200 to build a single cpu system w/32GB RAM, maybe $100 more for 12c/24t system. That's if you buy it here in the US though. If the parts can be bought and shipped to the UK for £1,000, then it would be worth it. Would use 50%-66% of the lecky a Z600 2x X5675 system uses (~350 watts?). If can save 150 watts, then assuming your power rate is .15 per kwh, you could save around 197 per year.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#63 Re: Xeon server build

Post by noetus »

Mains electricity costs vary in the EU. In US dollars (using Eurostat figures for 2016), in the UK it is 0.20 per kWh, 33% more than what you were basing your calculations on, which is significant. So the payoff for using more recent hardware is considerably more. In Denmark and Germany the price is even higher - 0.35 and 0.34 respectively, more than double what you were assuming! Just using your figures over 2.5 years we are talking about a cost saving of almost the original cost of the system. No wonder the big tech companies turf out all their old systems after a few years and invest in the latest hardware.

Hungary (where I live) and Bulgaria are the cheapest EU countries for power, 0.13 and 0.11 US dollars respectively. So the payoff for newer hardware partly depends where you live. You also have to factor in the higher chance of hardware failure for older equipment, especially server equipment that has seen 24/7 use (after 6 years some equipment, such as SSDs, might be reaching the end of their design life).

The cheapest place in Europe is Kosovo, a mere .07.

A quick glance at electricity prices in the US (from the US EIA) reveals wide state-by-state differences there too, from .27 in Hawaii to .09 in Washington, with prices generally lower than in the EU.

Another factor to consider is that if you buy one of those old servers you're going to get yourself a pretty noisy system, in all likelihood. If you build a system yourself you can make it near silent. And don't forget - you'll have manufacturer's warranties on all those new components of your build (excluding the CPU if you get a used/ES/QS version). I don't think you get any warranty purchasing used server equipment on eBay, maybe DOA that's all.

What is Carlos' app? I wouldn't mind running some tests of my own.
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#64 Re: Xeon server build

Post by pinhodecarlos »

Guys,

I'm an energy consultant and I trying to understand what's the most efficient machine to run several things like prime hunting, factoring (msieve, nfs@home) and prime gaps.

For example, the new intel desktop cpu's throttle when you use avx512.
So prime hunting uses avx, avx2.0 and in the near future avx512. When using Prime95/LLR with lots of cores the memory is the bottleneck.

Factoring (msieve client, or NFS@Home) is pure GHz needed as well as the gaps prime application.

What I don't understand is the example of my ivybridge laptop when running prime gaps on 4 threads (30 W, 18-20e9 n/s, depends on the range being processed) is more efficient than these systems:

6e9 n/s i3-530 2.9GHz 3 thread (Windows/Cygwin)
19e9 n/s E5-2666v3 2.9GHz 8 thread
27e9 n/s i7-4770k 4.3GHz 8 thread
32e9 n/s i7-6700k 4.2GHz 8 thread
34e9 n/s E5-2666v3 2.9GHz 16 thread
59e9 n/s E5-2686v4 2.7GHz 48 thread

My laptop, sandybridge i7- 3630QM.
Untitled.jpg
Carlos
You do not have the required permissions to view the files attached to this post.
SETI.USA member
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#65 Re: Xeon server build

Post by pinhodecarlos »

Prime gaps application:
gap5_g_linux.zip
gap5_g.zip
cygwin1.zip



gap5_g -n1 9000e15 -n2 9001e15 -gap 1346 -delta 155 -sb 24 -bs 18 -t 4 -mem 12

Windows:

start /low /min gap5_g.exe -n1 9000e15 -n2 9001e15 -gap 1364 -delta 155 -sb 24 -bs 18 -mem 12 -t 4

-t number of threads
-mem memory allocated
You do not have the required permissions to view the files attached to this post.
SETI.USA member
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#66 Re: Xeon server build

Post by pinhodecarlos »

gap5_g_linux_static.zip
You do not have the required permissions to view the files attached to this post.
SETI.USA member
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#67 Re: Xeon server build

Post by pinhodecarlos »

I'm also considering the prime gaps is not well optimised for full threads or it has memory bottleneck. I suppose it will be quicker on a multi core machine run several instances of the client even set to a lower 1 thread.
SETI.USA member
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#68 Re: Xeon server build

Post by Dirk Broer »

pinhodecarlos wrote: Fri Jul 07, 2017 12:27 pmWhat I don't understand is the example of my ivybridge laptop when running prime gaps on 4 threads (30 W, 18-20e9 n/s, depends on the range being processed) is more efficient than these systems:

6e9 n/s i3-530 2.9GHz 3 thread (Windows/Cygwin)
19e9 n/s E5-2666v3 2.9GHz 8 thread
27e9 n/s i7-4770k 4.3GHz 8 thread
32e9 n/s i7-6700k 4.2GHz 8 thread
34e9 n/s E5-2666v3 2.9GHz 16 thread
59e9 n/s E5-2686v4 2.7GHz 48 thread
Make sure each thread has the same amount of RAM, when you make a comparison like this.
IMHO a 48 thread Xeon should have 192GB of RAM (4GB/thread), a 16 thread Xeon 64GB, a 8 thread Xeon or i7 32GB.
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#69 Re: Xeon server build

Post by scole of TSBT »

Verify how the different options affect performance...
Usage: gap [options]

Options:
-n1 x First number to check is x
-n2 y last number to check is y
-gap g searching for gap>=g (default=1346)
-delta d sieving on k*m+[0,d) intervals (default=196)
-sb u sieve uses 2^u bits of memory (default=25)
-bs v one bucket size is 2^v bytes (default=14)
-mem m the maximal memory usage is m GB (m can be any real number)
-t k use k threads
Maybe different values for -gap, -delta, -sb, and -bs give better performance on higher thread or memory systems.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#70 Re: Xeon server build

Post by pinhodecarlos »

We are all using the same gap, delta, sb and bs sometimes.
SETI.USA member
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#71 Re: Xeon server build

Post by pinhodecarlos »

Reviving and old thread. Soon my wife will start her PhD which will be based on model simulation. I hope she will have access to high end computers to run gprooms or aspen, otherwise we will have to purchase a machine. I suppose both softwares will run better with higher number of threads and lots of memory. Could anyone advise me on tower racks, budget £1200. Anyone here is an U.K. is seller?

This is much appreciated.

Carlos
SETI.USA member
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#72 Re: Xeon server build

Post by Dirk Broer »

Image
Post Reply Previous topicNext topic

Return to “Benchmarking and Hardware”