Not logged inCSS-Forum
Forum CSS-Online Help Search Login
CSS-Shop Impressum Datenschutz
Up Topic Hauptforen / CSS-Forum / ICCR project is planning to be canceled
- - By Sedat Canbaz Date 2012-01-05 13:44
Dear Chess Friends,

Unfortunately i am sorry to announce:
-I am planning to cancel my new project ICCR

More details about why i am planning to stop working on the new ICCR project:
-I still strongly believe (in case of such list) that we will see high-quality games,played under super fast machines
-I plan to cancel this project is not due to there is no interest over it...
*For example,even in a very short time,ICCR has already 3 Testers,which are waiting instructions from me

The main reason of planning to not continue working over ICCR is that:
-There will be wrong Elo calculations,in case of using adapted time controls under different speed processors
-I mean,in case of creating a such adapted rating- all participant Elo standings will be effected...
-So far...I could not find any right solution for measuring the real Elo strength of the ICCR engines
-The biggest Elo calculation problem is appearing e.g 6c against 4c or 6c against 12c ...

*A simple example of wrong Elo calculation about in case of if we are not combing the mp engines in one version:
How can we calculate the game results played by:
Intel Core i7 920       @ 4.00 GHz    4 core   12454   50m+10s
and
AMD Phenom II X6 1035T    2.60 GHz    6 core   9283    70m+10s


-The right hardware Elo calculation (6c against 4c):should be done on two separate machines via Auto232 mode
*I mean for hardware speed test (6c against 4c):
-Engines should be tested with adapted time controls,on two separate machines via Auto232 mode
-Even if we combine ICCR played game results by Quads/Six-Cores... in one chess engine version
-Then is appearing another Elo calculation problem:as Clemens Keck stated...some buggy mp engines will be effected from that
In my opinion,combining in one chess engine version is the right way,but only for NON-buggy MP chess engines
-But in case of combining all mp participants,then it will lead to another wrong Elo calculation results

A little note more:
-For the first time in my Computerchess life, i could not complete a project with a success

And after all: I am sorry dear Chess Friends for all that i can not continue to work over this new project

But i have good news:
-I plan to start a new Rating list (15m+10s) based on only i7 Six-core machines 
Note:more info coming soon...

BTW,i stopped the current hardware Elo speed test and the results are here:

Games:
http://www.sedatcanbaz.com/chess/games/Hardware_Speed_Elo_Test.rar

Best Wishes,
Sedat Canbaz
Parent - - By Clemens Keck Date 2012-01-05 14:47
Hi Sedat

other tester groups also have 2, 4, 6 cpu in their list.
How do they do that? Could you not learn from them a good system?

Regards, Clemens
Parent - - By Sedat Canbaz Date 2012-01-05 15:38
[quote="Clemens Keck"]
Hi Sedat

other tester groups also have 2, 4, 6 cpu in their list.
How do they do that? Could you not learn from them a good system?

Regards, Clemens
[/quote]

Dear Clemens,

Probably you mean about CEGT/CCRL ?!

First of all,i'd like to say that i have big respect to their works-CEGT/CCRL

And the both great teams are putting a lot free efforts-for this i am so thankful for that...

But anyway,i think the both rating list (CEGT/CCRL) include misunderstanding results,especially i mean for 6CPU and 4CPU
Or maybe 4CPUs and 2CPU or 1CPU too

For example,as far as i know,they are owner of AMD Phenom II X6

And their testings with 6CPU against 4 CPU on same hardware
In my opinion a such test is wrong-it will not give us right Elo performance

BTW,strange results indeed,e.g CEGT has 15 ELO difference:
http://www.husvankempen.de/nunn/40_40%20Rating%20List/40_40%20All%20Versions/rangliste.html

1   Houdini 1.5a x64 6CPU   3290   23   23   600   73.0%   3117   36.0%
2   Houdini 1.5a x64 4CPU   3275   11   11   2641   73.0%   3102   34.9%


CCRL Rating has 45 ELO difference:
http://computerchess.org.uk/ccrl/404/rating_list_all.html

Houdini 2.0c 64-bit 6CPU  3404
Houdini 2.0 64-bit 4CPU    3359


And i strongly believe in reality,AMD Phenom II X6 is not stronger than Intel Quads
Or maybe there will be 5-10 Elo difference-no more no less (it depend on the clock speed)

Honestly this is one of the main reason of canceling my new project-ICCR

Once more i'd like to thank you for your useful note

Greetings,
Sedat
Parent - - By Ernest Bonnem Date 2012-01-07 01:44
[quote="Sedat Canbaz"]BTW,strange results indeed,e.g CEGT has 15 ELO difference:
http://www.husvankempen.de/nunn/40_40%20Rating%20List/40_40%20All%20Versions/rangliste.html

1   Houdini 1.5a x64 6CPU   3290   23   23  
2   Houdini 1.5a x64 4CPU   3275   11   11   2641   73.0%   3102   34.9%


CCRL Rating has 45 ELO difference:
http://computerchess.org.uk/ccrl/404/rating_list_all.html

Houdini 2.0c 64-bit 6CPU  3404
Houdini 2.0 64-bit 4CPU    3359
[/quote]
Hi,

If you look at the error bars (you did not show them for CCRL, I put them herebelow), the results appear not so strange...

Actually, going from 4 to 6 CPU (or threads) should gain something of the order of 27 Elo (see empiric formula at
http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?pid=165304#pid165304)

CEGT
1   Houdini 1.5a x64 6CPU   3290   23   23  
2   Houdini 1.5a x64 4CPU   3275   11   11  
CCRL
1  Houdini 2.0c 64-bit 6CPU  3404  +13  -12 
   Houdini 2.0c 64-bit 4CPU  3377  +21  -20
   Houdini 2.0  64-bit 4CPU  3359  +20  -20
Parent - - By Sedat Canbaz Date 2012-01-07 08:33 Edited 2012-01-07 08:42
[quote="Ernest Bonnem"]
[quote="Sedat Canbaz"]BTW,strange results indeed,e.g CEGT has 15 ELO difference:
http://www.husvankempen.de/nunn/40_40%20Rating%20List/40_40%20All%20Versions/rangliste.html

1   Houdini 1.5a x64 6CPU   3290   23   23  
2   Houdini 1.5a x64 4CPU   3275   11   11   2641   73.0%   3102   34.9%


CCRL Rating has 45 ELO difference:
http://computerchess.org.uk/ccrl/404/rating_list_all.html

Houdini 2.0c 64-bit 6CPU  3404
Houdini 2.0 64-bit 4CPU    3359
[/quote]
Hi,

If you look at the error bars (you did not show them for CCRL, I put them herebelow), the results appear not so strange...

Actually, going from 4 to 6 CPU (or threads) should gain something of the order of 27 Elo (see empiric formula at
http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?pid=165304#pid165304)

CEGT
1   Houdini 1.5a x64 6CPU   3290   23   23  
2   Houdini 1.5a x64 4CPU   3275   11   11  
CCRL
1  Houdini 2.0c 64-bit 6CPU  3404  +13  -12 
   Houdini 2.0c 64-bit 4CPU  3377  +21  -20
   Houdini 2.0  64-bit 4CPU  3359  +20  -20

[/quote]

Dear Ernest,

It seems you missed to read some of my previous notes or you are one of the people,who has have difficulties to understand me what i mean exactly
So again i will try to explain that the adapted ratings don't give accurate Elo results

1.Example:
-Lets say i am a CEGT/CCRL Tester
So...i tested same mp engine versions at 40/4 ;Ponder OFF (i7 920 against AMD Phenom II X6 in Auto232 mode)
Do you you expect that there will be 30-40 ELO difference between the above processors-i7 Quad against AMD Six-Core ?

2.Example:
-Lets say,i tested same mp engine versions  (AMD Phenom II X6 against i7 920 in SCCT Auto232 rating conditions)
Do you you expect that there will be 30-40 ELO difference between the above processors-i7 Quad 4 core against AMD Six-Core ?

In my opinion,in case of calculating the adapting ratings-all tested processors names should be mentioned too,i mean like SSDF Rating list

In other words,its completely wrong and its NOT recommended measurable system:
-in case of combining all played games by Intel/AMD 4CPUs in one engine version
Note:due to AMD chips are slower than Intel chips (for chess)

About your empirical formula 27 ELO,
Is there any proof,is there a such kind of hardware Elo test ?

For example,a long time ago i have done a such Elo hardware test:
http://sedatchess.110mb.com/index.php?p=1_31
Note:2xQX9775@4.0GHz has almost same kns values as Core i7 920@4.1GHz,but it performed 50 Elo better

BTW,maybe...we need a serious proof,before to say about empirical formula of 27 ELO,but anyway if this system is right,then
the empirical formula maybe will gain of a 6-core relative to a Quad,if are all games played by Intel Nehalem architecture

In other words....ICCR project will work accurate,if all games will be played by Intel Nehalem of Six-core architecture

I hope this time helps...

And last:if anybody disagree with me...i will respect that too,but instead of comments...i will prefer to see and check the games (with annotations if possible)

Greetings,
Sedat
Parent - - By Sedat Canbaz Date 2012-01-07 08:56 Edited 2012-01-07 09:01
One thing more,

In my opinion:
Actually combining all mp engines  in one version is better idea than combining all AMD 4CPUS and INTEL 4 CPUs in one engine version

At least this small test is proving a little bit that the Elo strength are almost identical
Parent - - By Sedat Canbaz Date 2012-01-07 09:17
Just i'd like to mention again:

Even if we combine ICCR played game results by Quads/Six-Cores... in one chess engine version
-Then is appearing another Elo calculation problem:as Clemens Keck stated...some buggy mp engines will be effected from that
Parent - - By Clemens Keck Date 2012-01-07 11:16
Dear Sedat

what I did for my private list:
I clocked all my systems between 3,3 and 3,4 GHz.( I think overclocked systems for ratings is a waste of energy and missleading too) I benched them with Fritzmark etc.. I watched and compared the KNs of many engines. So at least I can say for my tests that my hardware is +- 5% all the same.
If I start now games on quad or sixcore with MP engines, I have the same base hardware which I used for my 1cpu list, and this will give me a good rating for example 4 cpu engines. I will continue my quad vs quad 20+5 ponder ON games in the future when I have time for that. I hope I get enough games to start a Base T4 list . I estimate 0,5 - 1 year till I have enough games.

So if you only use 6 core machines(no hyperthreading!) for a list, all should be okay.
One thing what I would think about is the adaption of the different CPU (AMD / Intel) and the different clock speeds.
First of all you need a basics system, lets say it is AMD 1090T. If you compare KNs of 10 different MP engines you can calculate a factor for the adaption.
Yes, the factor would be +-5% or even +-8% but this should not affect ratings .

Also their would be a nice experiment:

Is the adaption really needed at all? Every system plays the matches / engines under the sam hardware. So when all play 20 +5 or 40 +10 maybe the ratings are correct / the same without adaption?

kind regards, Clemens
Parent - - By Sedat Canbaz Date 2012-01-07 13:23 Edited 2012-01-07 13:32
Hello dear Clemens

I can give you another example about the current issue:

Lets say...we started creating a two different adapting rating lists

1.Example-Adapted rating in Auto232 mode (based on the bellow hardwares)

Hardware-Processor        Speed    Cores   kN/s   Time Control
AMD Phenom II X6 1075T    3.00 GHz    6    10601   60m+10s
Intel Core i7 2600K       3.40 GHz    4    10473   61m+10s
AMD Phenom II X4 940      3.00 GHz    4    6898    99m+10s


Note:the results are expected to be ended with same Elo performance

----------------------------------------------------------------------

2.Example-Adapted rating list (played on same PC, 6c against 4c)
Hardware-Processor        Speed    Cores   kN/s   Time Control
AMD Phenom II X6 1075T    3.00 GHz    6    10601   60m+10s
AMD Phenom II X6 1075T    3.00 GHz    4     6898   60m+10s
---> normally the time control of 4 cores should be 99m+10s (not 60m+10s)

The most important:e.g what about other Testers,who will send me the results played by AMD/Quads, 4core, 99m+10s

Note:in case of such test,the results are expected to be ended with approx.30 Elo better performance in favor for AMD X6 1075T 3.00 GHz 6 core

*In my opinion this is a wrong test and it is the main reason why CEGT/CCRL have better results with AMD Six cores than Intel Quads

Thats why i deceded to cancel my ICCR project

Good luck with your tournaments...
Sedat
Parent - - By Clemens Keck Date 2012-01-07 13:40
Hello Sedat

just dont mix 4 cores and 6 cores. This should avoid your problems.

Regards, Clemens
Parent - - By Sedat Canbaz Date 2012-01-08 11:19
[quote="Clemens Keck"]
Hello Sedat

just dont mix 4 cores and 6 cores. This should avoid your problems.

Regards, Clemens
[/quote]

Dear Clemens,

Yes...i agree with you and be sure i will not waste my time of creating a such wrong adapted rating list

Best,
Sedat
Parent - - By Sedat Canbaz Date 2012-01-08 11:43 Edited 2012-01-08 11:46
Dear Friends,

What is your opinion about the bellow results ?


For example,in my opinion (under CEGT/CCRL adapted rating conditions-played on same PC) a such Elo calculation should be at least 20-30 Elo in favor for i7 980X 4.33 GHz 6 core

But i strongly believe in reality, its not ...and in my estimation both participant will have almost same Elo strength

I just wonder...creating adapting rating by different hardwares (running matches 6c against 4c on same PC ) is a good idea ?

Best Regards,
Sedat
Parent - - By Ernest Bonnem Date 2012-01-08 19:03
[quote="Sedat Canbaz"]For example,in my opinion (under CEGT/CCRL adapted rating conditions-played on same PC) a such Elo calculation should be at least 20-30 Elo in favor for i7 980X 4.33 GHz 6 core

But i strongly believe in reality, its not ...and in my estimation both participant will have almost same Elo strength[/quote]
Hi,

Reality is in agreement with the calculation I make:

I will assume average game is around 60 moves, so (9m+2s)=11m and (4m+2s)=6m
From my empirical formula, at the same GHz frequency, i7 980X (6-core) is x1.31 faster than i7 920 (Quad), note that both are Intel (so I am not comparing Intel and AMD)

So the game efficiency ratio between your i7 920 and i7 980X is:

(3.30 * 11) / (4.33 * 6 *1.31) = 1.07

which gives exactly your +6 Elo for your i7 920 !!! 

Of course such an accuracy is ridiculous, but it should be close...
Parent - - By Sedat Canbaz Date 2012-01-08 21:35 Edited 2012-01-08 21:44
[quote="Ernest Bonnem"]
[quote="Sedat Canbaz"]For example,in my opinion (under CEGT/CCRL adapted rating conditions-played on same PC) a such Elo calculation should be at least 20-30 Elo in favor for i7 980X 4.33 GHz 6 core

But i strongly believe in reality, its not ...and in my estimation both participant will have almost same Elo strength[/quote]
Hi,

Reality is in agreement with the calculation I make:

I will assume average game is around 60 moves, so (9m+2s)=11m and (4m+2s)=6m
From my empirical formula, at the same GHz frequency, i7 980X (6-core) is x1.31 faster than i7 920 (Quad), note that both are Intel (so I am not comparing Intel and AMD)

So the game efficiency ratio between your i7 920 and i7 980X is:

(3.30 * 11) / (4.33 * 6 *1.31) = 1.07

which gives exactly your +6 Elo for your i7 920 !!! 

Of course such an accuracy is ridiculous, but it should be close...
[/quote]

Hello Ernest,

Actually my question was about:
-Creating adapting rating by different hardwares (running also matches 6c against 4c on same PC ) is a good idea ?

In other words,e.g ICCR/CEGT/CCRL testing system (6c against 4c on same PC) is a good idea or not ?!

I mean,do you recommend to start a new ICCR adapting rating list (similar to CEGT/CCRL conditions) ?

*And so far...i noticed that many people are not well-experienced with adapting time controls and processor speeds

And in my opinion,ICCR project will work more accurately,if all games are played by i7 Six-core processors
Otherwise,its a waste of time...

Sometimes i am wondering too about all available rating lists (including SCCT too)

*For what we are testing the engines (what is our goal) ? :
-to produce many games
-engines should be tested with all openings or only with a few openings (it doesn't matter weak or strong)
-only 100 % original chess engine versions should be tested

*If the considering engines,which are 100 % original work,then i wonder:
-why many of the original engines repeat almost same openings (without books)?
Is that can be reason that mostly of the original engines have similar opening-related code ?!
For more details:
http://www.talkchess.com/forum/viewtopic.php?t=41764&highlight=sedat
...

*Who care and prefer more about:
-accurate/right Elo calculations
-the real strength of the engines
-the quality of the games
-the quality of the opening books
-mp engines should be tested with maximum strength
...

About the adapting time controls and processors speed,
Thank you,but i am not new in ComputerChess,almost 10 years i organize chess benchmark lists
So i know very well how to calculate the processors speed  (no help is needed for the current issue)
If you have any questions or need help,please feel free to ask...

For example,i7 980X @4.33GHz is approx. x1.8 faster than i7 920 @3.30GHz
SCCT Auoto232's average game is 74 (its not 60) moves per game,that means 4min+2sec = 388sec (on i7 980X @4.33GHz)
On i7 920 @3.33GHz,the right adapted time control should be 9min+2sec = 688sec
Note:in the database there is one game played against i7 920  @4.0GHz 4c (6min+2sec) and the winner is i7 980X @4.33GHz
So..in other words,no any Elo difference:


Finely:
-There are actually many unrecognized things in ComputerChess that requires more knowledge/experience/learning
*And i hope all of us will do our best for ComputerChess's progress

Best Wishes,
Sedat
Parent - - By Sedat Canbaz Date 2012-01-08 22:04
EDIT:i7 920  @4.0GHz 4c is played at 7min+2sec
Parent - By Sedat Canbaz Date 2012-01-10 08:01
ICCR project is already canceled-for more details:
http://www.sedatcanbaz.com/chess/iccr/

Best Regards,
Sedat Canbaz
Up Topic Hauptforen / CSS-Forum / ICCR project is planning to be canceled

Powered by mwForum 2.29.3 © 1999-2014 Markus Wichitill