Xeon server build

Forum to discuss and compare Hardware profiles and Benchmarking
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#1 Xeon server build

Post by Alez »

Carlos,
Started this thread for you and let's see if we can thrash out what type of system you fancy.
First off a few notes ( most of which I'm sure you know anyway )
Xeons will always be better than the i7's etc of the same age for crunching. They are not as fast but have better caches, better thermal efficency, better power efficiency and most important they are far more stable. My Xeons are only ever down when I switch them off. I never have hardware problems on any project with them generally only Lib errors with linux.
Obviously the older xeons are less expensive to buy but more expensive to run.
Rack servers all have RAID and support multiple plug and play discs. No more dead HDD's and complete rebuilds.
Rack servers all have far better error correction build in and support far greater memory capacity. ( very important with multi core / thread systems )
Server ram is more expensive but again far more stable and lasts forever.
It is always cheaper to buy a base system and add the second CPU, ram, HDD's etc.
My preference is HP ( all my systems )

The first question you need to answer before looking for anything is where will it go ? If it is in your house then you need to get a work station. If it can go outside then you want rack servers. Rack servers are far cheaper to buy but they are big and loud. I have 3 rack servers ( don't laugh ) outside in a shed in my garage ( Silver will confirm the shed in a garage ) They are G5 and G6 models and are far cheaper to buy than the equivalent workstation. this is your best option if you can do it. I simply put them in a metal rack available from any DIY store with the shelves mounted upside down to allow airflow round the entire unit.

to give you an idea Anisia Tech is a company I have dealt with
HP prolient rack servers here
HP workstations here

There are others I can point you towards, cheaper options etc and also companies I use for drives, memory etc. This is not a recomendation, just a guide so you can get an idea of what you want and then we can go on from there.
Alez
Image
The best form of help from above is a sniper on the rooftop....
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#2 Re: Xeon server build

Post by Dirk Broer »

If I were to build a Xeon system -as AMD User-, I'd choose a dual socket 2011-3 workstation motherboard, and would fill it with two Xeons that fit my budget (meaning two 22-core Xeon E5-2699 V4s when I win the lottery of course!).

But brand loyalty may point me to build a system with as much 32-core/64-thread AMD Zen CPUs as can be fitted on a Zen mobo :D
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#3 Re: Xeon server build

Post by scole of TSBT »

If money were no problem sure, but a dual Xeon Xeon E5-2643 v3, 24 thread, system will run you over $4500 US compared to a 24 thread LGA 1366 X5650/X5670/X5675 that you can assemble for $600-$700 US.

A dual Xeon Xeon E5-2620 2.4Ghz, 24 thread, system will only run about half cost of a E5-2643 v3 system but it's slower, uses nearly as much leckly as a X5675 system and probably not much faster than a 3.06Ghz X5675.

A 72 thread dual socket Xeon E5-2699 v3 system would run you probably over $12,000 but to be honest when you have that many threads running I bet the memory would be a huge bottleneck and you'd never see the performance you expect out of a LGA 2011 V3 system.
Image
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#4 Re: Xeon server build

Post by Alez »

Cash wise, 2nd hand G6 servers with LGA 1366 X5650/X5670/X5675 are the best value for money. G5's with the older 8 cores are also good as they are rediculously cheap. Most companies have not started replacing the G7's yet so even 2nd hand they are eye watering.
The reason for Xeons rather than AMD is that they are far more common in the buisness market and so are far more commonly replaced. Opterons are still quite rare and tend to be a generation below the xeons. Maybe companies don't buy them as commonly or maybe they last longer than the equivelent Xeons, but they are available with a lot more searching.
Image
The best form of help from above is a sniper on the rooftop....
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#5 Re: Xeon server build

Post by pinhodecarlos »

Thank you. Need to think about all this.
SETI.USA member
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#6 Re: Xeon server build

Post by Alez »

Once you are a little bit more sure about what you want we can try and narrow the field down a bit.
Image
The best form of help from above is a sniper on the rooftop....
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#7 Re: Xeon server build

Post by scole of TSBT »

Z600 w/2x X5670, 24GB RAM, £555.00 Buy it now...
http://www.ebay.co.uk/itm/272120235158

Z600 w/single X5670, 256GB SSD, £177.25 Buy it now (would need another X5670 and all new RAM)...
http://www.ebay.co.uk/itm/281936621587

Z600 w/2x hex core E5645, 12 RAM, no HDD (likely has 6x 2GB DIMMS so would need to replace all the RAM and get HDD), £296.36 Buy it now...
http://www.ebay.co.uk/itm/381539239048
(BTW, E5646 is only 80W TDP while the X56?? are 95W TDP)

Z600 w/2x hex core X5650, 12 RAM, no HDD (likely has 6x 2GB DIMMS so would need to replace all the RAM and get HDD), £317.63 Buy it now...
http://www.ebay.co.uk/itm/381543487985

Just a few to look at.

BTW...Z600s only have 3 DIMM slots per CPU, so you need 6x 4GB DIMMS for 24 GB or 6x 8GB DIMMS for 48GB. Mine work with both ECC or non-ECC DIMMS but they must all be the same. I only have 4GB DIMMS in mine. Not sure if a certain BIOS version is required for that or 8GB DIMMS. Also, if you buy a single CPU system and want to add a 2nd CPU, along with the 2nd CPU, you'll need the proprietary heatsink unless you can verify another LGA 1366 heatsink will fit AND figure out how to wire up another 5 pin fan. It uses a 5 pin fan, not 3 or 4. And it uses a proprietary PSU, which ain't gonna push much more than GTX 750Ti. Even with all those limitations, I like mine.
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#8 Re: Xeon server build

Post by Dirk Broer »

Alez wrote:Cash wise, 2nd hand G6 servers with LGA 1366 X5650/X5670/X5675 are the best value for money. G5's with the older 8 cores are also good as they are rediculously cheap. Most companies have not started replacing the G7's yet so even 2nd hand they are eye watering.
The reason for Xeons rather than AMD is that they are far more common in the buisness market and so are far more commonly replaced. Opterons are still quite rare and tend to be a generation below the xeons. Maybe companies don't buy them as commonly or maybe they last longer than the equivelent Xeons, but they are available with a lot more searching.
Meaning that a 2nd hand Apple MacPro with e.g. two 3.06 GHz 6-core "Westmere-EP" (X5675) Intel Xeon processors is also an option (also LGA 1366).
Image
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#9 Re: Xeon server build

Post by Alez »

Yes I guess, but I will defer to your knowledge as I have zero experience of macs so I personally can't comment on if that is a viable option or not.
Image
The best form of help from above is a sniper on the rooftop....
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#10 Re: Xeon server build

Post by Dirk Broer »

You better try your luck with these:
Image
It's a freaking Dalek! No, Dr. Who, it's the latest MacPro.
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#11 Re: Xeon server build

Post by scole of TSBT »

$4,000 US for a 12 thread Xeon system w/16GB RAM. I'll leave those for the hipsters.
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#12 Re: Xeon server build

Post by Dirk Broer »

scole250 wrote:$4,000 US for a 12 thread Xeon system w/16GB RAM. I'll leave those for the hipsters.
I take my Macs usually from dumpsters... :D
Image
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#13 Re: Xeon server build

Post by Bryan »

Dirk Broer wrote:
scole250 wrote:$4,000 US for a 12 thread Xeon system w/16GB RAM. I'll leave those for the hipsters.
I take my Macs usually from dumpsters... :D
Now that's what I call a good bang for the buck :lol:
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#14 Re: Xeon server build

Post by pinhodecarlos »

How fast are these 2 x Intel® Xeon® Processor E5450 on Enigma?
SETI.USA member
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#15 Re: Xeon server build

Post by Alez »

Hmmm, maybe scole can answer as Enigma has already deleted my results from the database. My E5450 is named Saturn V and is open for viewing on any project it is running. ( Currently dDM ).
It is slower than the X5460 ( though not by very much ) and they are again slower than the L5640 although the L5640 is 24 cores. The 54xx cores do not have hyperthreading so 8 cores is exactly that.

If it helps her's my E5450 on wuprop by run times E5450
heres the X5460
and heres the L5640

You can also look at the credit menu and get credit / core / day as a comparision.
Image
The best form of help from above is a sniper on the rooftop....
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#16 Re: Xeon server build

Post by scole of TSBT »

I don't have an E5450 but here's one to look at. It isn't running the opti app though. Times range from ~1500 sec for the 30 credit WUs to ~3000 for the 60 credit WUs.
http://www.enigmaathome.net/show_host_d ... tid=158893

The E5450 only has 4 cores/threads (no hyperthreading), is LGA771 (which is even older than LGA1366), no turbo boost and only has a PassMark benchmark of 4311 compared to 6710 for a L5640, 7607 for a X5650 and 8585 of a X5675.

A X5675 running the std app times range from ~2500 sec for the 30 credit WUs to ~4900 sec for the 60 credit WU, which is slower due to hyperthreading but each CPU is running 12 threads compared to the E5450's 4 threads.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#17 Re: Xeon server build

Post by pinhodecarlos »

Let's imagine this configuration.

https://www.bargainhardware.co.uk/cart. ... 976ed2dae7

A few questions:

1) can a hard drive be added?
2) can the OS be shared (Ubuntu) and will it detect all cores of each node? Example, by running BOINC will it detect 128 cores?
SETI.USA member
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#18 Re: Xeon server build

Post by Bryan »

I don't get anything on that link ... it says the cart is empty
Image
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#19 Re: Xeon server build

Post by scole of TSBT »

We can't see what's in your cart. Try to copy and paste the configuration here.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#20 Re: Xeon server build

Post by pinhodecarlos »

Sorry guys.

Four of these:
Untitled.jpg
Cheap E5-2600 Eight Core Cloud Foxconn Server (Configure-To-Order)
Processor #1: 2.00 GHz Eight (octa) -Core Xeon (E5-2650) - 2....
Processor #2: 2.00 GHz Eight (octa) -Core Xeon (E5-2650) - 2....
Memory: 64GB - (8 x 8GB) - DDR3 ECC Reg - 1333Mhz + £10...
RAID: OnBoard SATA - (6 x Caddies Included) + £0.00
Rails: Included + £0.00

http://www.bargainhardware.co.uk/cheap- ... -to-order/
You do not have the required permissions to view the files attached to this post.
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#21 Re: Xeon server build

Post by scole of TSBT »

That looks like a 32 core setup, assuming you enable hyper-threading in the BIOS. It doesn't appear to have any disks in the configuration yet, just the RAID controller and drive bays/caddies, You'll need 2 drives and set them up as RAID 1, mirrored.

Not sure what you mean by the OS being shared. The OS will use all cores on both CPUs. Nothing special you need to do. Or do you you mean a dual boot setup so you can boot and run Win64 or linux? If so, that's possible.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#22 Re: Xeon server build

Post by pinhodecarlos »

The E2650 has 8 cores, each rack has 2 CPU's. 16 cores per rack.
Building with 8 racks, 128 cores.

So those those drivers installed can be installed anywhere on those 8 racks and the OS will always detect the 128 cores, is this correct?
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#23 Re: Xeon server build

Post by scole of TSBT »

Those are rack servers, meaning they are made to install in a rack. Each rack server has 2 CPUs in it and each rack server will need a minimum of 2 disks (compatible with that server). Each rack server is a separate system and will require it's own OS and other software to be installed. If you are going to setup 8 rack servers, then you will have 8 separate servers. You can't share CPUs or cores between the separate servers.

NOTE: The E5-2650 has 8 cores but if you enable hyper-threading, it will have really run 16 threads, so you can actually get 32 threads per rack server. That is unless you really don't want to enable hyper-threading.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#24 Re: Xeon server build

Post by pinhodecarlos »

Thank you Scole.
My questions follow the fact that msieve.exe, the client who does the post-processing for NFS integers, is compatible with MPI (https://www.open-mpi.org/). The large integers are ran on big blusters with this msieve version and therefore I was wondering if I can use it on these 8 rack servers. Overall we are talking about 8 nodes each one with 32 threads, which msieve can easily take advantage off. This rack solution is cheap but lack of InfiniBand connection between racks where it is needed for msieve, nevertheless still a cheap option.
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#25 Re: Xeon server build

Post by scole of TSBT »

It appears to have a SFP+ 10Gbit ethernet port. Not sure if that helps. And I can't tell but I wonder if that blank above the SFP+ port is a slot for a PCIe adapter? If so, could you add an InfiniBand adapter?
Image
User avatar
Janos (retired)
Still a Newbie
Posts: 1919
Joined: Thu Feb 23, 2012 8:58 am
Location: Aberdeenshire, Scotland

#26 Re: Xeon server build

Post by Janos (retired) »

If you do get MPI working across all machines I would be very interested in learning what you found. Are you planning Linux or Windows? Judging by the .exe I am guessing Windows.
"Happiness can be defined as: a geek with non-work related code to write, no distractions and no deadline." - Janos
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#27 Re: Xeon server build

Post by pinhodecarlos »

My plan is to use linux because of MPI being compatible with it and not with windows.
SETI.USA member
User avatar
Bryan
Boinc Brigadier
Boinc Brigadier
Posts: 2621
Joined: Thu May 21, 2015 6:18 pm

#28 Re: Xeon server build

Post by Bryan »

Pete, the 1st link to the Amandtech forum doesn't work. Is it correct?

Did you actually try it?
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#29 Re: Xeon server build

Post by Dirk Broer »

Bryan wrote: Fri May 26, 2017 8:44 pm Pete, the 1st link to the Amandtech forum doesn't work. Is it correct?

Did you actually try it?
Looks like the correct link is https://forums.anandtech.com/threads/what-controls-turbo-core-in-xeons.2496647/.
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#30 Re: Xeon server build

Post by Dirk Broer »

A wonder of space, the EPC612D4I
Image and just slightly over 300 Euro's here. Would actually fit into the tiniest of iTX casings.
Cooling-wise, I'd recommend a Bitfenix Prodigy or so: them Xeon's can turn out to be real hotheads, especially when all cores go full turbo.
The board takes ECC SoDIMMS of up to 16 GB each, up to a maximum of 64 GB. It will take Intel® Xeon® processors of the E5-1600/2600 v3/v4 series*
(for the v4 series a BIOS version 2.0 or above is required). Why are those 18-core Intel Xeon E5-2699 v3 CPUs around 5,000 Euro?
On the sunny side: 2nd hand 6-core E5-2620 v3's are a little over 200 Euro's.

Edit: I found a source that wants to let go hexa-core E5-2620 v3's for 55 Euro's, and octo-core E5-2650L v3's for 65 Euro's.... :shifty:
Image
User avatar
Alez
[ TSBT's Pirate ]
[ TSBT's Pirate ]
Posts: 10363
Joined: Thu Oct 04, 2012 1:22 pm
Location: roaming the planet

#31 Re: Xeon server build

Post by Alez »

McShane of TSBT wrote: Fri May 26, 2017 11:11 pm No I hav'nt tried it yet due to RL and I am still working on two of my machines that are down. It is good that you found the right URL, I actually copied and pasted the whole article. If you are going to try it let me know how you went.
I've edited the post to make the full link clickable. Good luck with the sick 'puters
Image
The best form of help from above is a sniper on the rooftop....
Monk0054

#32 Re: Xeon server build

Post by Monk0054 »

Would never recommend using Xeon
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#33 Re: Xeon server build

Post by Dirk Broer »

If you have a motherboard that supports Xeon's, they're the best CPUs for it. Even an AMD User like me can find no fault in that line of reasoning.
Bulldozer-based Opterons did not challenge the performance of their Xeon counterparts. And there are motherboards for data centers that support a whopping eight Xeon's.
Software-wise Xeon is more adaptable than Itanium, Intel's in-house adversary for 64-bit Xeons. In the big server market only IBM's Power9 seems left to challenge Xeon -at the moment.
Then again: how many BOINC projects run on Itanium and/or Power9?
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#34 Re: Xeon server build

Post by pinhodecarlos »

Can't understand why www.bargainhardware.co.uk has dual Octo Xeons (16C/32T) machines costing a lot more than a new 8C/16T AMD Ryzen machine.
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#35 Re: Xeon server build

Post by scole of TSBT »

What specific system and model Xeons?
Image
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#36 Re: Xeon server build

Post by Dirk Broer »

pinhodecarlos wrote: Thu Jun 29, 2017 11:23 am Can't understand why www.bargainhardware.co.uk has dual Octo Xeons (16C/32T) machines costing a lot more than a new 8C/16T AMD Ryzen machine.
I'd compare them to dual octo EPYCs, if I were you...the octo EPYC seemingly priced at $400 each :D
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#37 Re: Xeon server build

Post by pinhodecarlos »

scole250 wrote: Thu Jun 29, 2017 1:07 pm What specific system and model Xeons?
Try the HP620 here:

http://www.bargainhardware.co.uk/hp-z62 ... -to-order/
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#38 Re: Xeon server build

Post by scole of TSBT »

If you configure it with 2x E5-2650 V2 CPUs (which appears to boost all threads to 3.1Ghz), 64GB RAM, 750GB HDD, Win 10; it's £1,499.98 (inc Tax)£1,249.98 (exc Tax)

How much for a single CPU Ryzen 1700X system with 32GB RAM and 750GB HDD?

There are a couple drawbacks to the Ryzen 1700X. Clock speed isn't one of them though and it does use faster DDR4 RAM but it only supports 2 memory channels and it only supports a single 16x PCIe 3.0 slot. The Intel system only supporta DDR3 but it has 4 memory channels but it supports 40 PCIe 3.0 lanes. You can run at least two big GPUs in it without a bottleneck on the PCIe bus.

Intel systems have always cost more than AMD systems. It's based on demand. The Threadripper CPUs may change things.

If you're willing to build your own system and willing shop a while, the Xeon V4 Broadwell EP engineering sample CPUs are the shizzle. They won't clock as fast as the Ryzen but you can get more cores to make up the difference. And if you run an app that makes heavy use of AVX2, nothing can touch it. The Xeon V4's absolutely smoke it when running the SRBase LLR app.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#39 Re: Xeon server build

Post by pinhodecarlos »

Scole,

Can you point me to those Xeon V4 Broadwell EP engineering sample CPUs ?

TIA.

Carlos
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#40 Re: Xeon server build

Post by scole of TSBT »

pinhodecarlos wrote: Fri Jun 30, 2017 3:16 amCan you point me to those Xeon V4 Broadwell EP engineering sample CPUs ?
xeon v4 es on Ebay UK

Disclaimer: Some sellers don't list their ES CPUs accurately. Sometimes they really have no idea what they are selling. ES CPUs have different versions. Early run samples may have bugs in them. You have to look at the stepping code and research that run of ES CPU. For example, there is a E5-2650 v4 ES CPU 2.2Ghz listed as stepping QHV6. I think the last 1 or 2 digits define the particular model and the first 1-2 digits define how early or late of a sample it is. I have a set with QHV7. It's a 14c/28t 2.2Ghz CPU. Also, even if they list a CPU as E5-2650, the CPU won't have the same base or turbo clock speed as a retail CPU. Usually a couple tenths of Ghz lower.

If I had the $ for another system, I would snatch up a couple of these E5-2690 ES CPUs, QHV5 , which I think turbo all 28 threads to 3.1 Ghz E5-2690 ES 2.4Ghz QHV5

Finally, ES CPUs won't run in just any motherboard. They will only work in certain models. This is info from a Ebay post...
There is a high possibility of compatibility issues using this ES CPU (such as boot failure) onto branded computers or branded servers, because generally MOST BIOS of the branded servers and branded computers do NOT contain the Specific Microcode to support the engineering sample (ES) version of E5-2600 (v1, v2, v3 and v4) series CPUs.

The following is a Non-exclusive list of compatible products:

Compatible Servers & Workstations:

Dell: Precision T5810 (BIOS A12 or above)
Dell: T7810 (BIOS A12 or above)
Dell: T7910 (BIOS A12 or above)

Compatible Motherboards:

Supermicro: X10S series (all models with BIOS R2.0 or above)
Supermicro: X10D series (all models with BIOS R2.0 or above)
Supermicro: C7X99-OCE (BIOS R2.0 or above)

Tyan: S562 series (BIOS V2.00 or above)
Tyan: S707 series (BIOS V2.00 or above)
Tyan: S708 series (BIOS V2.00 or above)

Asus: Z10PE-D8 WS (BIOS 3204 or above)
Asus: Z10PE-D16 WS (BIOS 3204 or above)
Asus: Z10PA-U8 (BIOS 3202 or above)
Asus: Z10PA-D8 (BIOS 3107 or above)
Asus: Z10PC-D8 (BIOS 3104 or above)
Asus: Z10PE-D16 (BIOS 3104 or above)
Asus: Z10PR-D16 (BIOS 3104 or above)

ASRock: X99 series (all models with BIOS 3.00 or above or above)
ASRock: EPC612 series (all models with BIOS 2.10 or above)
ASRock: FH-C612NM (BIOS 1.10 or above)

MSI: X99 Godlike Gaming Carbon (BIOS 2.4 or above)
MSI: X99 Godlike Gaming (BIOS 1.6 or above)
MSI: X99A XPower Gaming Titanium
MSI: X99A Gaming Pro Carbon (BIOS 1.1 or above)
MSI: X99A Gaming 9 ACK (BIOS 3.4 or above)
MSI: X99S Gaming 9 ACK (BIOS 2.8 or above)
MSI: X99S Gaming 9 AC (BIOS 1.B or above)
MSI: X99A XPower AC (BIOS A.5 or above)
MSI: X99S XPower AC (BIOS 1.B or above)
MSI: X99A/S Gaming 7 (BIOS H.E or above)
MSI: X99A/S MPower (BIOS M.A or above)
MSI: X99A/S SLI Krait Edition (BIOS N.7 or above)
MSI: X99A/S SLI Plus (BIOS 1.C or above)
MSI: X99A Raider (BIOS P.4 or above)

EVGA: X99 Family (all models with BIOS 2.01 or above)

Note: A BIOS upgrade may be necessary for motherboards manufactured prior to the introduction of Xeon E5-2600 v4 series CPUs series.
Note: A firmware upgrade of an onboard chip other than BIOS may be necessary for some Asus Z10 motherboards to enable dual-processor support.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

INCOMPATIBLE OR UNSTABLE Servers & Workstations:

Dell Precision Rack Workstation, Dell PowerEdge Rack Servers: support v4 series CPUs with A1/L0/B0/M0/R0 stepping, v4 series CPUs QS version or retail version/ OEM version CPUs only.

HP servers and workstations: support v4 series CPUs with B0/M0/R0 stepping, v4 series CPUs QS version or retail version/ OEM version CPUs only.

INCOMPATIBLE OR UNSTABLE Motherboards:

Asus X99 Express Chipset motherboards (Only supports assembled or DIY desktop computers): some but not all stuck on Q-Code 19 after reboot; They support v4 series CPUs with B0/M0/R0 stepping, v4 series CPUs QS version or retail version/ OEM version CPUs only.

Gigabyte X99 Express Chipset motherboards (Only supports assembled or DIY desktop computers, with BIOS F22 or above): some but not all boot loop; They support v4 series CPUs with B0/M0/R0 stepping, v4 series CPUs QS version or retail version/ OEM version CPUs only.

*******************************************************************
The CPU-Z screenshots shows the Stepping, the Number of CPU cores, level 1 cache , level 2 cache and level 3 cache ,Max TDP, multiplier, and the Core speed (the maximum frequency) which is the maximum ALL cores frequency at which the processor is capable of operating using Intel® Turbo Boost Technology (Turbo Boost all of the cores simultaneously).
All of the engineering sample (ES) version of E5-2600 series CPUs are NOT UNLOCKED but will be able to turbo up ALL cores (Turbo Boost all of the cores simultaneously) to the maximum frequency which the CPU-Z screenshots uploaded shows.
Special Notice:
1.
Intel's definition of Max Turbo Frequency in Intel's official site is "Max turbo frequency is the maximum SINGLE core frequency at which the processor is capable of operating using Intel® Turbo Boost Technology." Therefore, the meaning of "Max Turbo Frequency" which shows on intel's official site, is different from the meaning of the Core speed (Turbo Boost all of the cores simultaneously) shows in the CPU-Z screenshots uploaded on this page.
About Intel Turbo Boost, the Calculation of Intel Turbo Boost shows an Example on the middle of this page of this link.
Please refer to
en.wikipedia.org/wiki/Intel_Turbo_Boost
2.
About C6/C7 low-power states problem( If you are experiencing stability problems or worse, the system shut down completely): Because most power supplies do NOT support Haswell's C6/C7 low-power states, so If you are experiencing stability problems or worse, the system shut down completely, please disable C6/C7 power states, which will not affect the performance and will just affect the low-power sleep states.
Read more: According to an Intel document viewed by the gang at The Tech Report, Haswell’s C6/C7 states requires a minimum load of 0.05 amps on the 12V2 line. Power supplies that are older or budget models simply aren’t capable of delivering such a small amount of juice. As a result, users with incompatible power supplies may experience stability problems or worse, the system may shut down completely if the power supply unit (PSU)'s under- or over-voltage protection kicks in.
3. About Intel Turbo Boost VS Overclocking:
In one simple word it is a dynamic overclocking rather than a static overclocking.

Because of the item is an ES (engineering sample) version CPU, for detailed specifications, Intel has not yet announced.
Please refer to the OEM or retail version of the detailed specifications and the test screenshots of the CPU-Z release
(ES detailed specifications for individual products may differ from the OEM or retail version).
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#41 Re: Xeon server build

Post by pinhodecarlos »

Thank you Scole. Took me awhile to read this and need some time to digest it. I'll come back with some queries.
SETI.USA member
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#42 Re: Xeon server build

Post by noetus »

I would like to butt in here, as since last Sept I've been aquiring both ES and non-ES E5 v3 and v4 Xeons for various system builds. I'm by no means experienced in this, but I have built up a bit of experience in purchasing and trying out several of these. All together 10 xeons purchased, 2 sent back, for a total of 8 in my possession, only 6 are active in system builds right now but the other two I have tested and will build out soon.

- Early on I decided to go four dual socket setups, because there is so much expense in each system tied up in other resources (case, PSU, disks, cooling, etc) that it seemed to me worth doubling up on the processors and spending a bit more on a dual socket motherboard. Especially as the CPUs can actually be quite cheap if buying v2 or v3 versions. I haven't regretted this decision. You get double the firepower, obvoiusly, but without having to spend double the money.

- Motherboards. I tried out the ASRock EP2C612D8C first of all and had no issues with it at all. This is an ATX motherboard, and there are very few dual socket ATX motherboards around. Supermicro makes some too, which seem to be well regarded (I have never tried them) but if you are going for dual socket and ATX I think your choice is basically between ASRock and Supermicro. I bought a couple more of those, and later I bought an EP2C612D8 (without the 'C' at the end) which is slightly cheaper. I had a bit more issues setting this one up as it doesn't seem as flexible in the way the memory is configured, and I had to play around with various configurations before settling on one that worked (and it wasn't one of the configurations recommended in the user manual). I am not sure which of these two I'll get for the next build. One slight issue is that neither board is 100% compatible with the standard ATX mounting hold configuration; there were one or two places where I had to leave a screw out. If I remember correctly the motherboard is held down by only 6 screws, and only two at the top, where the big heavy cpu coolers hang. For that reason I would NEVER transport these boxes upright with the motherboard, CPUs, and coolers installed (they should lie flat with the coolers upright rather than cantilevered and putting stress on the board), but I think it is fine when they are just set up under your desk or whatever.

-Cases. I wanted the ATX because I like the Fractal Design Define R4 cases; good cases that are low price. I have also tried the Define R5 which is newer and updated. I actually prefer the R4 and will go back to that for the next build (it is 20 - 30 € cheaper). The R4 has one weak spot however that if you grasp it by the front to pick it up the top of the front panel tends to pop out, disconnecting the USB and/or audio connections there. It is very awkward to get the connections back in place because space there is limited. This seems to be fixed with the R5. However if you know about this weakness and are careful how you pick up the case it isn't an issue I've found.

-CPU cooler. I have used the Noctua NH-U12DX i4 on each build and I am extremely happy with it. It's a very high quality cooler. Obviously you'll need two of these for each dual socket build. You don't need extra fans; a single fan on each cooler (as supplied) is enough. I also use Noctua case fans on all builds and have been doing so for years without any issues. Some people don't like the look of the Noctua fans but their brown/tan colour doesn't bother me. You can get the black Noctua fans for the same price but these run at much higher RPM and are quite noisy (though push a lot of air).

-PSU. DON'T skimp on the PSU; buy a high quality one (note: Corsair does NOT qualify as high quality). These builds don't need a lot of amperage, and it is much better to go with a low-power high quality PSU than an average or poor quality one with a 'high' output rating. The thing is, if you are not confident of your PSU, and you run into some stability issues, it will be hard to know whether the issue is with the PSU or something else. I have the XFX XTS fanless Platinum 520 W which is plenty enough power for these builds, and the are completely silent (having no fans).

-Disks. Obviously get SSDs, beyond that, it doesn't really matter (though I like the Samsungs). I use a 128 GB one for the system drive, and an additional (usually bigger) SSD for data, though actually a separate data drive on these boxes is a bit of overkill as most data is not hosted on these boxes which are primarily for crunching.

-CPUS. So the bit you actually asked about. I'd add to the above post by saying you have three options: ES ('Engineering Sample'), QS ('Qualification Sample') and used retail. The QS chips are just very late edition ES, sometimes identical or almost identical to the retail release. I bought a pair of ES chips and ended up sending them back because the way they ramped up the frequency in my system was just downright weird (I could never get all the cores ramped up to max frequency, and there was a lot of jumping around.) Since then I have only gone with QS and that is what I would recommend if you are buying non-retail. I am happy with the performance of all the QS chips I bought. You have to be quite careful looking at the CPU details in each eBay listing to figure out exactly what chip it is and whether it is ES or QS, although many of the QS variants will have "QS" in the listing title. Also, I personally would not bid on any listing that does not have CPU-Z screenshots, HWiNFO screenshots, and preferable Cinebench ones as well. I have dealt with several eBay vendors and there is one in particular I like (and have dealt with off-eBay, where I got a 10% discount or something by asking for a PayPal invoice directly and it worked out very well). PM me for details.

Also, feel free to order from the US based eBay site, where you'll find a lot more choice than on the UK one. All the vendors ship internationally for low cost (and the ones I dealt with, speedily) and you can usually get the customs declaration for a fraction of the actual value. The most I paid on customs was 50 €. The two CPUs I had to send back because they were weird was not an issue with the seller I used. (By the way, those ES CPUs worked fine on Linux; it was on Windows they were weird.)

There is a performance difference between the ES, QS, and retail chips; don't believe the listings when they say they are all the same. The prices wouldn't be different if that were the case. I also did get a pair of used retail CPUs and they perform a bit better than the QS ones (probably because they clock slightly higher). And like I said, the ES ones I tried out were a bit strange and I could never get their theoretical performance out of them.

The main difference between v3 and v4 E5 Xeons is not in performance but in power consumption. There is very little difference in their performance in my experience (the v3 are 22 nanometre, the v4 are 14 nanometre lithography) except perhaps for some specialised tasks I have no experience with (anyone?). I have not tried v2 (by now really cheap CPUs); perhaps someone else on the forum can advise on those. They are definitely more power hungry than the v3s again, but I don't know about performance. If you get v4 and the ASRock motherboard, you will need the latest 2.10 BIOS which it does NOT ship with, and you cannot update the BIOS with the v4 CPUs installed (or no CPUs installed). So your choice is either to get a BIOS chip with the 2.10 version preinstalled (eBay vendors sell them; possibly the vendor of the motherboard could too, worth asking) and pop that in ourself (supposedly easy - I've never done it), or use a v3 CPU to update the BIOS, then install the v4 CPUs. Depending on how much you pay for electricity (for me it has been free since last Sept, though this will end in Aug and I will be paying regular grid prices again - albeit in Hungary, the cheapest EU country for electricity), it can be worth paying extra for v4 CPUs, but this is something you'll have to decide.

One possible advantage with some v3 chips is that by dint of a design mistake by Intel some models are actually overclockable. There are details on this forum about how to do that. I haven't yet but plan to when I get my last pair of CPUs built out.

Once you have your system set up, especially if you are using ES or QS CPUs, run prime95 for several days (on high core count systems you might have to run two instances of prime95 concurrently) to make sure the system is stable.

It is possible to get an almost completely silent system, even when using all cores at 100%. With the Fractal Design cases (supposedly optimised for silence, with sound-deadening linings), several Noctua case fans each running at low RPM, the Noctua i4 CPU cooler I mentioned, and the fanless PSU (and no graphics card installed - there is onboard VGA for setup and then I RDP to these boxes anyway), at 32 C ambient temp the CPU temps on the v4 28 core systems are in the mid 50s at 100% on all cores! That is pretty amazing. (Temps on the 44 core system is a bit higher but not too much.)

EDIT: I would be sceptical about the above claim by scole250 that the E5-2690 ES QHV5 will clock to 3.1 GHz on all cores simultaneously. I am pretty sure the 3.1 GHz figure is for one core when other cores are not turbo'ed, which is how Intel defines max turbo for Xeons. There is another listing of this very CPU by another eBay vendor that includes a lot more info (CPU-Z, HWiNFO, and Cinebench as I advised):

E5-2690 ES QHV5

From what I can tell from the data in the screenshots that CPU will max out at 2.5 GHz on all cores simultaneously. Maybe someone else can correct me if that's wrong? That is the CPU uncore listed. I have a similar CPU, the E5-2658 v4 14 core, and it maxes out at 2.5 GHz on all cores, and mine is a QS not ES version. I would verify with the seller what this ES (not QS) chip will turbo up to on all cores. Some eBay sellers are very explicit about this in their listings. Also, because that is an ES and not QS sample, I would avoid it (notice how in the HWiNFO screenshots the turbo is a lot less than expected - why?)

By the way, you cannot trust the official Intel specs for this chip, or any ES/QS samples, as their specs appliy to retail versions, not ES versions.
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#43 Re: Xeon server build

Post by scole of TSBT »

noetus wrote: Fri Jun 30, 2017 2:16 pm EDIT: I would be sceptical about the above claim by scole250 that the E5-2690 ES QHV5 will clock to 3.1 GHz on all cores simultaneously. I am pretty sure the 3.1 GHz figure is for one core when other cores are not turbo'ed, which is how Intel defines max turbo for Xeons. There is another listing of this very CPU by another eBay vendor that includes a lot more info (CPU-Z, HWiNFO, and Cinebench as I advised):
I agree that you need to be skeptical and make sure you research the CPUs thoroughly. I mis-stated the max turbo on all cores but only a little but. It's actually only 3Ghz. While researching I found more than one post with HWinfo pics of a QHV5 with all cores clocking at 3.0. Here's one forum post....https://forums.servethehome.com/index.p ... ost-104865
Now I don't actually have one so I can't guaranty they actually clock at 3.0Ghz but I'd be willing to take the chance. Not all Xeon V4 ES CPUs claiming to be a E5-2690 are QHV5 CPUs though.
Image
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#44 Re: Xeon server build

Post by pinhodecarlos »

My budget is £1,000. What should I get?
SETI.USA member
User avatar
Dirk Broer
Corsair
Corsair
Posts: 1962
Joined: Thu Feb 20, 2014 11:24 pm
Location: Leiden, South Holland, Netherlands
Contact:

#45 Re: Xeon server build

Post by Dirk Broer »

pinhodecarlos wrote: Sat Jul 01, 2017 6:59 am My budget is £1,000. What should I get?
I'd say try a 2nd hand Dual Socket motherboard and two likewise Xeon's. For a hypothetical 32 thread Dual Xeon system I advise at least 64GB of ECC RAM -with the option to install an extra 64GB later(cash register sounds, please), and those two puppies need to be cooled well too, to ensure good performance.
Image
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#46 Re: Xeon server build

Post by noetus »

scole250 wrote: Fri Jun 30, 2017 11:42 pm I agree that you need to be skeptical and make sure you research the CPUs thoroughly. I mis-stated the max turbo on all cores but only a little but. It's actually only 3Ghz. While researching I found more than one post with HWinfo pics of a QHV5 with all cores clocking at 3.0. Here's one forum post....https://forums.servethehome.com/index.p ... ost-104865
Now I don't actually have one so I can't guaranty they actually clock at 3.0Ghz but I'd be willing to take the chance. Not all Xeon V4 ES CPUs claiming to be a E5-2690 are QHV5 CPUs though.
That's nice; depending on what you do, the extra cycles can make a difference. When I was purchasing my 14 core CPUs I wasn't aware that there were any ES versions that would clock this high on all cores. Research, research, research! Is there a QS version of the QHV5 E5-2690 I wonder?

EDIT: Here's an eBay seller selling a QHV5-2690 also with HIiNFO showing 3.0 GHz on all cores simultaneously. I can vouch for this eBay seller incidentally.
E5-2690 QHV5 on eBay @ 3.0 GHz all cores
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#47 Re: Xeon server build

Post by noetus »

Now here's the thing, and personally this is what makes it difficult for me to figure out exactly what the best purchase options for the CPU are.

Here's a seller selling an E5-4669 v3 QS QH9B. Now this will only clock to 2.4 GHz but it's an 18 core CPU, and slightly cheaper than the 14 core we've been discussing. Performance should be very similar. Admittedly it is v3 not v4; on the other hand it is QS and not ES (which I would feel happier about). If one were shopping in this budget range, how to choose?
noetus
Boinc Corporal
Boinc Corporal
Posts: 50
Joined: Tue May 30, 2017 3:15 am

#48 Re: Xeon server build

Post by noetus »

pinhodecarlos wrote: Sat Jul 01, 2017 6:59 am My budget is £1,000. What should I get?
For the CPU(s) only, or the entire system?
pinhodecarlos
Boinc Warrant Officer Class 2
Boinc Warrant Officer Class 2
Posts: 320
Joined: Sun Feb 21, 2016 11:22 am
Location: Milton Keynes, UK

#49 Re: Xeon server build

Post by pinhodecarlos »

Entire system.
SETI.USA member
User avatar
scole of TSBT
Boinc Major General
Boinc Major General
Posts: 5980
Joined: Mon Feb 03, 2014 2:38 pm
Location: Goldsboro, (Eastern) North Carolina, USA

#50 Re: Xeon server build

Post by scole of TSBT »

noetus wrote: Sun Jul 02, 2017 3:06 amEDIT: Here's an eBay seller selling a QHV5-2690 also with HIiNFO showing 3.0 GHz on all cores simultaneously. I can vouch for this eBay seller incidentally.
E5-2690 QHV5 on eBay @ 3.0 GHz all cores
That HWiNFO pic looks strangely familiar :think:
Image
Post Reply Previous topicNext topic

Return to “Benchmarking and Hardware”