SCCT – Hardware Speed Test 2015

By Benno Hartwig Date 2015-09-28 16:39

> As we see, Stockfish 2.01GHz 2 core performed better than Stockfish 4.02GHz 1 core

This is interesting.
What kN/s-value did you get for 2 cores with 2,01 GHz?

An Idea for expanation:
Access to memory is not faster if you use a faster core, so that superfast core may have to wait several times. Times it could use fpr many calculations.
Memory-acess-time an CPU-speed might fit better when using 2 slower cores. While one core is waiting, the other can wark most times.

On the other hand I belive that one fast core with a bit less kN/s than 2 slower cores might produce the stronger chess yet.

Benno

By Sedat Canbaz Date 2015-09-29 18:03

Benno Hartwig schrieb:

Hallo Benno,

The values were at 2521 kN/s, here are the full details:

Points kN/s Cores EXE  Hardware  Speed 
322    2521   2   x64  i7-980X  @2.01GHz 
308    2472   1   x64  i7-980X  @4.02GHz

About your explanations,
Interesting...this can be true...

Best,
Sedat

By Sedat Canbaz Date 2015-09-29 23:30

Dear Chess Friends,

Very soon I plan to run another speed testing !

In shortly,
I will test Stockfish 290815, 1 core, with same 600 positions, on i7-980X 3.33GHz
But this time, I will set Time Control: 11 seconds per move

Really I wonder a lot about what will be the results...

And what is your estimation ?)
SF 1 core 11 sec/move will be performed stronger than SF 4 core 3 sec/move ?)

Note: SF 4 core 3 sec/move i7 980X 3.33 GHz managed to get: 374 points

Have fun,
Sedat

By Sedat Canbaz Date 2015-09-30 10:00

The speed hardware test has been completed, so here is the final result:
Stockfish 290815 x64 1 core 11 sec/move i7 980X 3.33 GHz: 395 points

What I can say more,
This time, SF 1 core 11 sec/move performed clearly better than SF 4 core 3 sec/move
For example, SF 1 core 11 sec/move's performance is very close to SF 6 core 3 sec/move

Best,
Sedat

By Sedat Canbaz Date 2015-09-30 23:24

Next Speed Test:
I plan to test Stockfish 160715 with same 132 positions, on i7-980X 3.33GHz
But this time I will set 6 seconds per move + 6 cores

Note: Stockfish 160715 1 core, 30 seconds/move performed: 46/132
https://sites.google.com/site/computerschess/tts-30s

Of course this time, I am not going to ask any questions, estimations...
Because it's a new way of testing...and it is very hard to guess...

But however,
Personally I expect to see similar points...maybe 6 core 6 sec/move will be slightly weaker...let's see

Best,
Sedat

By Sedat Canbaz Date 2015-10-01 10:26 Edited 2015-10-01 10:37

The current test is completed

And here are the results:

Conditions:
Intel Core i7-980X 3.33GHz
Hyper-Threading OFF
Large Pages OFF
Arena 2.01 Chess GUI
512 MB hashtable size
132 Tactical positions

Houdini 4 Tactical x64 6 cores, 6 seconds/move performed: 83/132
Houdini 4 Default x64 1 core, 30 seconds/move performed: 52/132
Houdini 4 Default x64 6 cores, 6 seconds/move performed: 46/132

Stockfish 160715 x64 1 core, 30 seconds/move performed: 46/132
Stockfish 160715 x64 6 cores, 6 seconds/move performed: 44/132

More Details,
Used unsolved positions in 3 sec/move 1 core by the current chess engine versions
Note that (for better comparing) I decided to test Houdini 4 6 core 6 sec/move too
As I expected, SF 6 cores 6sec/move performed very close to 1 core 30 sec/move

Note also that,
I tested SF 6c 6sec/move with 128 MB hash too: 43/132
Also SF 6c 6sec/move tested with 1024 MB hash: 42/132

What I can say more,
We can not use any right 'formula' in case of comparing the hardware speeds !
It all depends on our used conditions...some engines perform better some worse...

But however (in case of measuring the hardware speeds)
Stockfish 290815's KN/s values seems to be not bad choice...
Of course, one of the best choices is that: running similar test suite positions
Or running engine matches in Auto232 mode or via LAN etc...

Hopes helps,
Sedat

By Sedat Canbaz Date 2015-10-02 06:57

Here is another new speed test !

Rybka 4.1 x64 1 core, 30 seconds/move performed: 26/132
Rybka 4.1 x64 6 cores, 6 seconds/move performed: 21/132

Crafty 24.1 x64 6 cores, 6 seconds/move performed: 6/128
Crafty 24.1 x64 1 core, 30 seconds/move performed: 5/128

More Details:
The current chess engines are tested with unsolved positions in 3sec/move 1 core
Crafty 1 core 3 sec/move managed to solve 4 positions, thats why the number is 128
Well, Rybka 4.1 6 core 6sec/move's performance is weaker than 1 core 30sec/move
But... Crafty 24.1 6 core 6sec/move performed stronger than 1 core 30sec/move !

I noticed also that (during all the time of analyzing),
Crafty 24.1's 6 cores CPU usage was at approx. 66%...not sure why...
If Crafty was with 100% CPU usage...probably then its points would be higher...

Best,
Sedat

By Sedat Canbaz Date 2015-10-03 01:43

One new test more !)

Rybka 4.1 x64 1 core, i7 980X @4.02GHz, 222 kN/s, 6 sec/move: 9/132
Rybka 4.1 x64 2 core, i7 980X @2.04GHz, 215 kN/s, 6 sec/move: 7/132

Crafty 24.1 x64 2 core, i7 980X @2.04GHz, 5142 kN/s, 6 sec/move: 4/128
Crafty 24.1 x64 1 core, i7 980X @4.02GHz, 5124 kN/s, 6 sec/move: 1/128

Some Notes:
Used unsolved positions in 3 seconds/move, 1 core on i7 980X 3.33GHz
Exception Stockfish and Houdini, Crafty surpassed the formula rules too !)
As we see, the above mentioned engines did not follow the old formula ^0.76
It looks like, Rybka is more near to this old formula, but again in case of comparing the speeds:
- We can not use the strict number ^0.76
or it does not mean if same kN/s values, the playing strength should be same !
But however, the kN/s values are more useful for majority of the chess programs !

Continuing...
For example, I can say the same thing in case of doubling the processor speeds,
It will be wrong, if we use any strict formula ...I mean some engines gain 50 Elo, some 60 Elo etc...
Also it is similar issue, in case of increasing the time controls on same hardware...
For example, as far as I remember in one of my testings:
Rybka 30m+1s started to be stronger over Stockfish 1m+1s

In other words,
As I mentioned before, it all depends on our used conditions
For example here are some main factors, which are playing a very BIG role:
Hardware speeds, engine versions, opening books, time controls, etc..

Greetings,
Sedat

By Ingo B. Date 2015-10-03 08:36

Sedat Canbaz schrieb:

...
It will be wrong, if we use any strict formula ...I mean some engines gain 50 Elo, some 60 Elo etc...
...

Years ago I told you that a single benchmark can not reflect the chess world reality, even worse, it is false for most engines because of exactly the above statement (and the Fritzbench was the worst possibility to test hardware on top of the bad decisions). Back then you started to hyperventilate as soon as I started to talk about it ... Sometimes one has to wait until sense comes to people!

Ingo

PS: Sorry for beeing a smartass!

PPS: Your web site domain expired on 18th of September.

By Sedat Canbaz Date 2015-10-03 22:00 Edited 2015-10-03 23:00

Ingo B. schrieb:

Sedat Canbaz schrieb:

...
It will be wrong, if we use any strict formula ...I mean some engines gain 50 Elo, some 60 Elo etc...
...

PPS: Your web site domain expired on 18th of September.

Oh yes...now I remembered a little bit...but this discussion was 300-500 years before the new era !)
Sorry that I can't remember the details...probably due to your info was not so useful for me ))

Btw,
That's right... my old domain is expired, but I have good news:
My alternative SedatChess site is active and recently updated !
And if you wish... you can put my site to your Favorite list )
https://sites.google.com/site/computerschess

Ah one thing more,
Which one of my current speed hardware testings do you like much ?)

Best,
Sedat

By Ingo B. Date 2015-10-04 08:48

Sedat Canbaz schrieb:

...
Which one of my current speed hardware testings do you like much ?)
...

None! All rubbish because of the given argument above. You test something for exactly one engine and there is a good chance that this changes with the next version of that engine ... waste of energy, except someone wants to buy a new computer and run exactly that particular engine ...
What do you want to test, hardware or software?. Running a test with ONE engine is testing software, running the 10 best engines and average the speed increase* on different CPUs might be a "Chess Benchmark" (As long as these 10 Engines are really used).

Testing something with an engine gives you a rough estimation of a speed difference, true, but the smaller the speed increase is, the bigger is the likelyhood that your are wrong for most engines. E.G. If you say CPU A is 100% (double) as fast as CPU B that is fine as most likely it is faster for all engines. If you claim that CPU A is 5% faster than CPU B that is rubbish as this is only good for the particular engine you tested - and the 5% are nearly impossible to prove Elowise. (so double useless

)

On top of all this, benchmarking is mainly for the one who bought a computer and want to assure themself that he has made a good decision. Guess what, most people test that way that they have made a good decision because noone wants to admit a wrong decision (same goes for cars, tellies, vacations ...) That is just human.

Ingo

PS: This "average the speed increase" is quite difficult. You can't do that just by adding nodes/second and you have to do that for one to X cores, but than you have to run time to depth** and that several times to average that ttd, and then you have to do it with different positions - many different positions - ... ... ... and when you are finished (if ever!) the TOP 1 enigne has a new release. I tried that for a while, it is an endless task. (And if it is done it is as useless as the single test if someone want to run something else - except that you can assume that "in average" it might be right. It is like a rating list - you can proclaim the superiority af A over B but never an individual result for a game of A vs B.)

** and it seems that even ttd is not even good enough for some engines (eg Komodo) nowadays.

By Sedat Canbaz Date 2015-10-05 22:45 Edited 2015-10-05 23:00

Ingo B. schrieb:

Sedat Canbaz schrieb:

...
Which one of my current speed hardware testings do you like much ?)
...

Hehe )) as I expected...none, what kind of man are you ?)
So far...you are always negative on my threads !

It looks like you can not see what is one of the goals of this test....

Once more I'd like to mention about,
This is a new way of hardware speed testing, never played before !)
Stockfish 290815 kN's values are good choice in case of measuring...

Of course the current system is not perfect, but very easy, fast, reliable...!!

Yes, the new idea belongs to me, but of course this project can be improved...

For example,
it will be great if there will be more hardwares, and to be included slower time controls to the current list
Maybe I can do that...or maybe we will see another person who can do that...

And it is not required to be included exactly the best 10 engines,
The strongest top 3 engines will be quite enough, because mostly of the people are interested in that...

Btw, if we will follow strictly your mentality...
We should not run even engine games, because the next versions will be stronger...hehe...
What about then...is not a waste of energy ?? ))

Sorry...that I have no much free time to talk over this issue...

Best,
Sedat

By Ingo B. Date 2015-10-06 08:47

Sedat Canbaz schrieb:

Hehe )) as I expected...none, what kind of man are you ?)
So far...you are always negative on my threads !

No I am negative regarding benchmarks. When you tested books (not my particular interest) I never wrote a single word.

Sedat Canbaz schrieb:

It looks like you can not see what is one of the goals of this test....

It looks like you are still can't grasp the consequences of your writen statement that every engine reacts different - and that was coming from you not me!

Sedat Canbaz schrieb:

...
This is a new way of hardware speed testing, never played before !)
Stockfish 290815 kN's values are good choice in case of measuring...
Yes, the new idea belongs to me, but of course this project can be improved...

With all respect, but there is nothing new to your idea at all, except the choise of the engine (which might be a different one in 1, 3, 6 month) - but then read my above long statement about small CPU differences and different engines.

Sedat Canbaz schrieb:

Of course the current system is not perfect, but very easy, fast, reliable...!!

Again, it seems you do not think the matter to the end. How can YOU write that every engine reacts different and then claim your test is reliable? Somehow those two argument doesn't fit at all! And "fast" is exactly the problem. You could do much better (not good) with a wider, more elaborated, carefull test.

Sedat Canbaz schrieb:

And it is not required to be included exactly the best 10 engines,
The strongest top 3 engines will be quite enough, because mostly of the people are interested in that...

10 was just an example to show that more engines produce a more reliable result. Better would be 50 Enignes as it is most likely correct for the 51st engine (in average)
You can do with 3 (or even 1 engine) but you fail when there is a new TOP 3 engine. Besides your limitet test is to short and the problem with the uncertanty the lower the differences between CPUs (my 5% example) is, gets bigger with less engines.

Sedat Canbaz schrieb:

Btw, if we will follow strictly your mentality...
We should not run even engine games, because the next versions will be stronger...hehe...
What about then...is not a waste of energy ?? ))

The picture doesn't quite fit* as it is the purpose to test new engines while you try to test a specific hardware. From the argument given above you can never be very precise with close CPUs (except for one particular engine - and SF290815 is allready outdated and will be even more in a month or year))

Sedat Canbaz schrieb:

Sorry...that I have no much free time to talk over this issue...

Looking at all the fora were you posted this and answered a lot it is hard to believe that!

In summary I doubt that you understand (or want to understand) the arguments. Therefore you come back with some minor insults with which you want to show some ironical superiority. One can do that, but the argument:

Sedat Canbaz schrieb:

I mean some engines gain 50 Elo, some 60 Elo etc.

is comming from you. All you do with the insults is showing that you do not get the consequences of your own argument!

Ingo

*PS: Nonetheless, for completly different reasons you are right that testing engines in general is a waste of energy!

By Sedat Canbaz Date 2015-10-06 12:02 Edited 2015-10-06 12:53

Ok...what I can say more (after reading your comments)

You should 'ignore' my chess activities (thanks in advance)!

Zitat:

PPS: Your web site domain expired on 18th of September.

It seems, you are regular visitor of my activities, it will be hard to ignore, but you should try...
And just because it is hard...it does not mean to give up

What I noticed more,
'Ignore' word is similar and perfectly suites to your name: Ingo

Btw,
My current testing method is similar also to your testing method (IPON does not share games) !

For example,
My used test positions are currently private, but I hope to publish them after the end of TTs testings

Do you know what is the biggest difference between you and me:
I share...you don't...

And to be honest,
I give 0 (zero) cent for those....who does not share...!!

And last,
I don't know it is waste of energy or not but I feel BIG proud:

- So far more than 700 hardware users participated in SedatChess benchmarks (since 2002-2015) !
- Also more than 1000 private openings books participated in SCCT (since 2007-2015) !
- Perfect books have been used by thousands of users (since 2002-2015) !
- And so far I managed to run more than 1.000.000 blitz/slow games (since 1997-2015) !

Of course, without the support, all these my works would not exist
Thanks again for all of the chess friends, which they've showed interest throughout the years

I wish you good luck !
Sedat

By Ingo B. Date 2015-10-06 12:30

It seems you have a problem with "arguments" but none with "insults". To me that seems a bit childish. Especially the "ignore - Ingo" analogie is something one excpects from Kindergarten ...

Anyhow, you have to live with the discrepance between your perception and your writing.

Ingo

By Sedat Canbaz Date 2015-10-06 20:26

Ingo B. schrieb:

Hallo Ingo )

Sorry if you are affected...

Let's finish this discussion in a friendly way by this this wise saying:
"An honest man is always a child." - Socrates

Best,
Sedat