New Project: Open Source Chess Engine Rating

By Sedat Canbaz Date 2014-06-18 17:55

Hello there,

I like a lot the current tournament and here are the 1st results:

Code:

   # PLAYER                  : RATING    POINTS  PLAYED    (%)
   1 Stockfish 140614 x64    : 3359.9     284.5     312   91.2%
   2 Gull 3 x64              : 3350.0     282.0     311   90.7%
   3 Texel 1.04 x64          : 3068.5     221.0     312   70.8%
   4 Senpai1.0 x64           : 3065.8     219.5     312   70.4%
   5 Protector 1.6.0 x64     : 3035.7     208.5     309   67.5%
   6 DiscoCheck 5.2 64       : 2966.9     186.0     310   60.0%
   7 Booot 5.2.0 x64         : 2918.1     170.0     312   54.5%
   8 Crafty 24 x64           : 2888.3     157.5     310   50.8%
   9 MinkoChess 1.3 x64      : 2841.9      70.0     155   45.2%
  10 EXchess_v7.26a          : 2822.8     135.0     312   43.3%
  11 Scorpio 2.7.7 x64       : 2802.8     127.5     312   40.9%
  12 Arasan 17.2 x64         : 2799.6     125.0     313   39.9%
  13 Cheng4 0.36c x64        : 2794.3     122.5     311   39.4%
  14 Rodent 1.4 x64          : 2731.1     100.5     311   32.3%
  15 Bison 9.11 x64          : 2725.4     100.0     313   31.9%
  16 Bobcat 3.25 x64         : 2725.3      98.5     312   31.6%
  17 Cyrano 0.6b17 x64       : 2696.3      45.5     156   29.2%
  18 Pawny 1.0 x64           : 2662.6      76.0     308   24.7%
  19 Redqueen 1.1.97 x64     : 2640.7      71.5     311   23.0%

More Details:
It is very interesting to note that,
Still is not clear which chess engine will be the champion of this competition: Gull or Stockfish ))!!

Also another very interesting note is that...
Which participant will take the 3rd place: Senpai or Protector or Texel ?! ))

Let's see after 1000 games per player, what will be the standings...

Btw,
Right now we have 51 games lost on time (based on 2801 games played out):

Scorpio = 24 (all Scorpio's games are played on Windows 8.1 64-bit)
Booot = 10
Bison = 5
Rodent = 4
Redqueen = 3
Cheng = 2
Protector= 1
Gull = 1
Bobcat = 1

Note: the time loses issue is very important, especially for creating ratings
I mean weaker participants (which never loose on time) are getting very high elo points
And those bugs lead to missundestanding Elo calculations...

As we see, Scorpio has the highest number of time loses
And now I've set the default hashtable size (64MB) and WB1 for Scorpio
I hope with this configuration to solve the time issue
Otherwise, Scorpio will be changed with another participant

Also Booot seems to be buggy too...

And probably I will replay those games, which are lost on time
But only those engine's time lost games, which forfeit not very often
For example,
if X engine has up to 20 games lost on time of 1000 game played out

Best,
Sedat

By Sedat Canbaz Date 2014-06-20 17:02

Hello there,

Last update:
Octochess (played as WB)

While we are talking about Stockfish... Gull is the new Leader of SCCT's Rating list !!
Probably Stockfish's performance suffers with one core or due to Arena's hashtable bug ?!

Note also that sometimes mostly of the UCI engines use default hashtables under Arena 3.0

So I have a wish from Arena team:
- I will be very glad if those hashtable bugs will be fixed in the next release
Note: I sent (informed about those hashtable bugs) several time mails to Arena team....

Or if not...then I have a wish from the open source Engine Authors:
- It will be great if you release your UCI engine, where the default hashtable size to be minimum 64 MB or 128 MB

BTW,
Well-done to Fabien's engine, currently Senpai is 3rd !

Standings:
http://www.sedatcanbaz.com/chess/?page_id=522

Conditions:
http://www.sedatcanbaz.com/chess/?page_id=527

Games:
http://www.mediafire.com/download/2qju9rltc9872a5/scct_open_3m2s.rar

Best,
Sedat

By Krug Peter Date 2014-06-20 18:25

Hallo Sedat,

why is Houdini not in your list?

Peter

By Thomas Müller Date 2014-06-20 18:35

weil es in der liste nur "open source" engines sind

s. general rules
http://www.sedatcanbaz.com/chess/?page_id=527

By Krug Peter Date 2014-06-20 20:16

...ja stimmt!
Das sind die Regel.
Was dazu wohl Houdini selbst meint?
Aber der wird natürlich nicht gefragt

G. Peter

By Thomas Müller Date 2014-06-20 20:55

der meint nix, weil die regeln macht in dem fall Sedat wie er es will

Komodo usw ist auch nicht dabei.

gruß
thomas

By Michael Scheidl Date 2014-06-23 12:57

Wahrscheinlich würde er sagen, na, aus dieser Fesselung werde ich mich auch noch herauswinden!

By Sedat Canbaz Date 2014-06-20 21:37

Krug Peter schrieb:

Hallo Sedat,

why is Houdini not in your list?

Peter

Hallo Peter,

Simply because Houdini is not an open source engine
And I hope Robert Houdart will release his engine as open source and then I will test it with pleasure ))

This is a new project, and we should give a credit to the authors of the open source engines too!

In other words, this time SCCT's main goal: testing only the open source engines!!

Best,
Sedat

By Sedat Canbaz Date 2014-06-21 14:47

With the useful recommendation by Antares:
http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=28946

What you can do is a) DISABLE the "Engines->Manage...:UCI->Common hashtable size" and b) ENABLE individually for all engines "Engines->Manage...etails->Special->Init-String" where you enter a UCI-command like "setoption name Hash value 128".

Right now all UCI engines are started to use 128 MB hashtable size and now Arena 3 is almost a perfect GUI !!

BTW,
The latest Stockfish 210614 x64 engine will be tested soon...

Greetings,
Sedat

By Sedat Canbaz Date 2014-06-22 03:54

First of all,
I'd like to say: many thanks to all Authors of the current participated engines!

New Update:
Stockfish 210614 x64

Really this tournament is going to be much more interesting hour by hour...

Now the latest Stockfish 210614 is very close to Gull 3's Elo points!
But however, it is still not clear who will be the Champion !?

Btw, Texel is currently 3rd, well- done to Peter Osterlund!
And everything can happen in the next hours/days...
Senpai still has big chances to take the 3rd place

   # PLAYER                  : RATING    POINTS  PLAYED    (%)
   1 Gull 3 x64              : 3380.0     498.5     555   89.8%
   2 Stockfish 210614 x64    : 3379.4     426.0     465   91.6%
   3 Stockfish 140614 x64    : 3363.5     482.0     530   90.9%
   4 Texel 1.04 x64          : 3084.8     385.5     556   69.3%
   5 Senpai 1.0 x64          : 3081.6     379.0     556   68.2%
   6 Protector 1.6.0 x64     : 3058.3     366.5     554   66.2%
   7 DiscoCheck 5.2 64       : 2987.1     326.5     554   58.9%
   8 Booot 5.2.0 x64         : 2945.7     299.0     552   54.2%
   9 MinkoChess 1.3 x64      : 2902.4     271.0     552   49.1%
  10 Crafty 24 x64           : 2887.5     261.5     554   47.2%
  11 Scorpio 2.7.7 x64       : 2844.2     237.5     552   43.0%
  12 EXchess v7.26a          : 2838.2     232.0     556   41.7%
  13 Arasan 17.2 x64         : 2832.3     226.0     559   40.4%
  14 Octochess r5190 x64     : 2825.4     220.5     553   39.9%
  15 Cheng4 0.36c x64        : 2813.6     215.0     554   38.8%
  16 Rodent 1.4 x64          : 2769.1     186.0     555   33.5%
  17 Bobcat 3.25 x64         : 2757.8     179.5     551   32.6%
  18 Bison 9.11 x64          : 2736.3     167.5     552   30.3%
  19 Redqueen 1.1.97 x64     : 2681.7     136.5     553   24.7%
  20 Cyrano 0.6b17 x64       : 2676.7     133.5     557   24.0%
  21 Pawny 1.0 x64           : 2676.0     132.5     554   23.9%

Good luck to all !!

By Sedat Canbaz Date 2014-06-23 11:12

I've just calculated the current played games with 4 different calculation programs
Note: Crafty 24 x64 is fixed to 2900 Elo points

And as we see not only the chess engines,
even the calculation programs are different to each other... ))

And now I have a question to all:
- Which is the most accurate rating list ??

And under the current conditions, which chess engine is stronger:
- Gull or Stockfish ??

Btw, all games are available for downloading:
http://www.sedatcanbaz.com/chess/?page_id=522

A) Ordo 0.8 Calculation:

  # PLAYER                  : RATING    POINTS  PLAYED    (%)

   1 Gull 3 x64              : 3387.1     593.5     660   89.9%
   2 Stockfish 210614 x64    : 3385.2     577.5     631   91.5%
   3 Stockfish 140614 x64    : 3375.0     482.0     530   90.9%
   4 Senpai 1.0 x64          : 3096.8     456.5     661   69.1%
   5 Texel 1.04 x64          : 3095.1     455.0     657   69.3%
   6 Protector 1.6.0 x64     : 3071.9     443.0     663   66.8%
   7 DiscoCheck 5.2 64       : 2996.3     386.5     662   58.4%
   8 Booot 5.2.0 x64         : 2956.8     356.0     660   53.9%
   9 MinkoChess 1.3 x64      : 2913.7     326.5     661   49.4%
  10 Crafty 24 x64           : 2900.0     314.5     661   47.6%
  11 Scorpio 2.7.7 x64       : 2863.5     285.5     658   43.4%
  12 EXchess_v7.26a          : 2848.5     272.5     660   41.3%
  13 Arasan 17.2 x64         : 2845.4     271.5     665   40.8%
  14 Octochess r5190 x64     : 2839.4     266.5     659   40.4%
  15 Cheng4 0.36c x64        : 2838.7     265.0     662   40.0%
  16 Rodent 1.4 x64          : 2789.6     228.5     660   34.6%
  17 Bobcat 3.25 x64         : 2762.5     210.5     658   32.0%
  18 Bison 9.11 x64          : 2752.7     201.5     661   30.5%
  19 Redqueen 1.1.97 x64     : 2691.9     161.0     662   24.3%
  20 Cyrano 0.6b17 x64       : 2681.6     153.5     660   23.3%
  21 Pawny 1.0 x64           : 2673.8     149.0     661   22.5%

--------------------------------------------------------------------

B) Bayeselo 0056 Calculation (mm 1 1):

Rank Name                   Elo    +    - games score oppo. draws 
   1 Gull 3 x64            3331   33   31   660   90%  2910   13% 
   2 Stockfish 210614 x64  3323   34   32   631   92%  2892   13% 
   3 Stockfish 140614 x64  3321   37   34   530   91%  2894   14% 
   4 Senpai 1.0 x64        3080   23   23   661   69%  2924   24% 
   5 Texel 1.04 x64        3080   23   23   657   69%  2921   26% 
   6 Protector 1.6.0 x64   3056   23   22   663   67%  2923   27% 
   7 DiscoCheck 5.2 64     2993   22   22   662   58%  2930   27% 
   8 Booot 5.2.0 x64       2954   22   22   660   54%  2932   28% 
   9 MinkoChess 1.3 x64    2914   22   22   661   49%  2931   30% 
  10 Crafty 24 x64         2900   22   22   661   48%  2934   31% 
  11 Scorpio 2.7.7 x64     2868   22   22   658   43%  2933   27% 
  12 EXchess_v7.26a        2853   22   22   660   41%  2937   27% 
  13 Arasan 17.2 x64       2849   22   22   665   41%  2939   29% 
  14 Octochess r5190 x64   2846   22   22   659   40%  2936   31% 
  15 Cheng4 0.36c x64      2846   22   22   662   40%  2939   29% 
  16 Rodent 1.4 x64        2798   22   23   660   35%  2938   26% 
  17 Bobcat 3.25 x64       2771   23   23   658   32%  2936   25% 
  18 Bison 9.11 x64        2758   23   23   661   30%  2944   22% 
  19 Redqueen 1.1.97 x64   2702   24   24   662   24%  2943   21% 
  20 Cyrano 0.6b17 x64     2694   24   24   660   23%  2944   22% 
  21 Pawny 1.0 x64         2687   24   25   661   23%  2943   21%

------------------------------------------------------------------------------------

C) Elostat 1.3 Calculation

    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Stockfish 210614 x64           : 3293   37  36   631    91.5 %   2879   13.5 %
  2 Stockfish 140614 x64           : 3281   40  39   530    90.9 %   2881   13.6 %
  3 Gull 3 x64                     : 3277   35  35   660    89.9 %   2896   13.5 %
  4 Senpai 1.0 x64                 : 3049   24  24   661    69.1 %   2909   23.8 %
  5 Texel 1.04 x64                 : 3047   24  24   657    69.3 %   2906   25.6 %
  6 Protector 1.6.0 x64            : 3030   24  23   663    66.8 %   2909   27.1 %
  7 DiscoCheck 5.2 64              : 2973   23  23   662    58.4 %   2914   27.3 %
  8 Booot 5.2.0 x64                : 2943   23  23   660    53.9 %   2916   27.9 %
  9 MinkoChess 1.3 x64             : 2910   22  22   661    49.4 %   2915   30.4 %
 10 Crafty 24 x64                  : 2900   22  22   661    47.6 %   2917   30.7 %
 11 Scorpio 2.7.7 x64              : 2870   23  23   658    43.4 %   2917   26.9 %
 12 EXchess_v7.26a                 : 2859   23  23   660    41.3 %   2921   27.4 %
 13 Arasan 17.2 x64                : 2857   22  22   665    40.8 %   2922   29.3 %
 14 Octochess r5190 x64            : 2852   22  22   659    40.4 %   2919   31.4 %
 15 Cheng4 0.36c x64               : 2851   23  23   662    40.0 %   2922   28.7 %
 16 Rodent 1.4 x64                 : 2810   24  24   660    34.6 %   2921   25.6 %
 17 Bobcat 3.25 x64                : 2788   24  24   658    32.0 %   2919   25.1 %
 18 Bison 9.11 x64                 : 2782   25  25   661    30.5 %   2926   22.2 %
 19 Redqueen 1.1.97 x64            : 2728   26  26   662    24.3 %   2925   20.8 %
 20 Cyrano 0.6b17 x64              : 2719   26  26   660    23.3 %   2927   21.7 %
 21 Pawny 1.0 x64                  : 2711   26  27   661    22.5 %   2925   20.6 %

----------------------------------------------------------------------------------

D) Fritz 12 GUI Calculation:


 1   Stockfish 210614 x64    3290   
 2   Stockfish 140614 x64    3279   
 3   Gull 3 x64              3274   
 4   Senpai 1.0 x64          3047   
 5   Texel 1.04 x64          3046   
 6   Protector 1.6.0 x64     3029   
 7   DiscoCheck 5.2 64       2972   
 8   Booot 5.2.0 x64         2942   
 9   MinkoChess 1.3 x64      2910   
10   Crafty 24 x64           2900   
11   Scorpio 2.7.7 x64       2870   
12   EXchess_v7.26a          2859   
13   Arasan 17.2 x64         2857   
14   Octochess r5190 x64     2852   
15   Cheng4 0.36c x64        2851   
16   Rodent 1.4 x64          2810   
17   Bobcat 3.25 x64         2788   
18   Bison 9.11 x64          2782   
19   Redqueen 1.1.97 x64     2728   
20   Cyrano 0.6b17 x64       2720   
21   Pawny 1.0 x64           2711

By Wolfgang Battig Date 2014-06-23 11:43

Or you can use Arena 3.5. HT-bug is fixed with this version.
Best
Wolfgang

By Sedat Canbaz Date 2014-06-23 12:26

Wolfgang Battig schrieb:

Or you can use Arena 3.5. HT-bug is fixed with this version.
Best
Wolfgang

Hallo Wolfgang,

I've tried Arena 3.5 during my scratch testing, but I noticed more forfeits on time
And as far as I remember, Arena 3.5's global hashtable setting is buggy too

But with the useful recommendation by Antares,
All UCI engines use 128 hashtable size, and now Arena 3.0 is my favorite GUI!

Best,
Sedat

By Michael Scheidl Date 2014-06-23 13:01

I do not run very big tests with Arena 3.5, but so far even with time controls like 1m+0.5s I did not yet find a single loss on time, or any interface bugs at all. Maybe it is related to ponder on, because usually I run ponder off only (incl. Booot as Wolfgang mentiones).

By Sedat Canbaz Date 2014-06-23 17:51 Edited 2014-06-23 17:57

Michael Scheidl schrieb:

I noticed time forfeits on time not only under Arena, those time bugs are appearing even under Shredder Classic and Fritz GUIs too...
And not only due to pondering, for example some of the scratch engines have time management bugs:
http://www.sedatcanbaz.com/chess/?page_id=293

Note: I am talking about 70 participants (which many of them are amateur engines, and those buggy engines still need a lot work and optimizations... !

By Michael Scheidl Date 2014-06-24 13:05

Zitat:

I am talking about 70 participants

Ok, I guess that explains it. Tough job!

By Jens Schmidt-Wilke Date 2014-06-25 07:01

Hello Sedat!
Nice idea, this list. I appreciate your work a lot.
Why don't you use Arena 3.5? Has it also the Hashtable-Bug? I found 3.0 a lot more buggy then 3.5.
nice greetings

Jens

By Jens Schmidt-Wilke Date 2014-06-25 07:54

...ok, it is answered already - i didn't see.

By Sedat Canbaz Date 2014-06-25 19:02

Jens Schmidt-Wilke schrieb:

Hello Sedat!
Nice idea, this list. I appreciate your work a lot.
Why don't you use Arena 3.5? Has it also the Hashtable-Bug? I found 3.0 a lot more buggy then 3.5.
nice greetings

Jens

Hello Jens,

Thanks...

Ok...I will try again Arena 3.5

But however I wonder,
What are those bugs under Arena 3 ?
And what are the advantages of Arena 3.5 ?

Best,
Sedat

By Sedat Canbaz Date 2014-06-25 19:03

New update

Especially after 500 games per player,
Mostly of the ranks are stabilized:
http://www.sedatcanbaz.com/chess/?page_id=522

Have fun,
Sedat

By Jens Schmidt-Wilke Date 2014-06-26 08:42

...its been a while - what I remember: it was under Windows 7 and roundrobin tourns were nearly unplayable because of engine-cfg-bugs: some engines lost the config, played with less cores and hash then configured. There were some other minor bugs, which I dont remember - some of them even 3.5 has still, but they are not critical. I play roundrobin and single matches, sometimes "Spießrutenlauf" and all is working fine and comfortable.
nice greetings

Jens

By Sedat Canbaz Date 2014-06-26 12:04

Jens Schmidt-Wilke schrieb:

Thanks for the info

Yesterday I tried again Arena 3.5

But I noticed Senpai 1.0 crashed (forfeited on time)...so I switched again to Arena 3 ))

Best,
Sedat

By Sedat Canbaz Date 2014-06-26 12:05

Breaking News!

For more accurate standings,
I plan to repeat Gull 3's games against Top 10 participants

The reasons:
- Gull 3's default hashtable size was higher during up to 400-500 games per player
- Yesterday I noticed also that Gull 3 sometimes is using 6 cores (due to Arena's bug)

I've seen those bugs are only with engines, which have the default MP threads is set 'auto'
And those MP bugs are appearing even when the engine's threads option is set: Threads=1

And for solution: I set 'Threads=1' in the Gull's Command Line Parameters
So I hope the current setting will fix those bugs...
Otherwise I will use Fritz or Shredder GUIs for Gull 3 test

Note also that I don't plan to replay all Gull 3's games, due to:
The standings will be not changed much against the engines, which are 600-800 Elo points weaker...

By Sedat Canbaz Date 2014-06-27 22:16

Latest News

Gull 3 test is over, now Gull 3 is approx. 50 Elo behind than Stockfish

After replaying the Gull's games (against the top engines),
The current Gull + Stockfish's Elo difference is almost same as SCCT (3m+2s) MP Rating !!

Note also that after replaying the Gull's games,
The Elo difference is not changed due to Elo error bar margins
We see such Elo difference, just because before Gull was played with advantage...

But with equal conditions,
Gull's Elo points are quite correct now !

More Details
* The current rating list has:
- No double games
- No games up to 15 moves
- No games lost on time

In other words,
Now the list is very clean...))!

For Full Standings:
http://www.sedatcanbaz.com/chess/?page_id=522

Have fun,
Sedat

By Sedat Canbaz Date 2014-06-30 11:08

New Update

Daydreamer 1.75 x64 is participated, for full standings:
http://www.sedatcanbaz.com/chess/?page_id=522

Note: the current open-source competition is going to be paused...
Due to very soon SCCT Book ChampionShip II will be scheduled !

Greetings,
Sedat

By Wolfgang Battig Date 2014-06-23 11:41

Sedat Canbaz schrieb:

...Also Booot seems to be buggy too...

Booot causes severe problems with Ponder=ON. We have the same problem with Booot at our 5'+3"-list with Ponder.

These problems are independant from GUI. I had crashes with Arena 2.01/3.0/3.5, Shredder Classic and Fritz 13.

No probs with Ponder=OFF

Best
Wolfgang (CEGT Team)

By Sedat Canbaz Date 2014-06-23 12:29

Wolfgang Battig schrieb:

Sedat Canbaz schrieb:

...Also Booot seems to be buggy too...

Agreed...

For more accurate Elo measurement,
Probably in the end (when all participants will have 1000 games per player),
I plan to repeate those games which are lost on time...

Best,
Sedat

By Sedat Canbaz Date 2014-06-24 15:25

Finely (after 736 games),
Stockfish started to lead...!, even with all calculation programs !
Well-done to Stockfish team!

Gull's performance is superior too (his Elo points are very close to Stockfish)
Due to many of the opponents of Gull and Stockfish are approx. 500-600 Elo weaker...

For Full Standings:
http://www.sedatcanbaz.com/chess/?page_id=522

Note: Latest played games (with annotations) are available too...

Best,
Sedat

By Sedat Canbaz Date 2014-07-03 13:02 Edited 2014-07-03 13:11

Hallo )

This time,
I decided to replay the rest played games by Gull 3 too
Note: before Gull is replayed only against the Top 9 participants

And once more I'd like to mention about the reason:

- Gull 3's default hashtable size was higher during up to 400-500 games per player
- Yesterday I noticed also that Gull 3 sometimes is using 6 cores (due to Arena's bug)

Note also,
Gull's Elo difference between wrong and accurate testing is approx. 70-80 Elo
I mean for before, Gull was rated almost on same level strength as Stockfish

And now, (after the useful recommendations by Anton),
All engines are started to play at full performance under Arena 3 GUI
And of course the current standings are become more accurate...!!))

For Latest Standings:
http://www.sedatcanbaz.com/chess/?page_id=522

Best,
Sedat

By Sedat Canbaz Date 2014-08-16 18:21

LAST UPDATE:
Vajolet2 1.45 x64

Well done to Marco Belli, currently Vajolet's rank is 8th place !

TOP 20:
http://www.sedatcanbaz.com/chess/?page_id=522

ALL VERSIONS:
http://www.sedatcanbaz.com/chess/?page_id=711

CONDITIONS:
http://www.sedatcanbaz.com/chess/?page_id=527

GAMES:
http://www.mediafire.com/download/2qju9rltc9872a5/scct_open_3m2s.rar

NOTE:
This current rating is one of most original rating lists that I have ever seen !

Have a nice weekend,
Sedat