Big transformer  5. Improvements over BT4 include removing biases from qkv and centering and biases from layer normalizations, in addition to a new relative position encoding replacing the previous "smolgen". Training started late June 2024 and is expected to take 6 months. It has 15 layers with 1024 embedding size, 32 heads per layer, and 4096 dff size, roughly tripling the dff size over BT4 and doubling the overall model size.

Training can be monitored here: https://leela.plutie.ca/?darkMode=true#scalars&runSelectionState=eyJCVDUtMTAyNHgxNXgzMmgtcnBlLXRyYWluIjpmYWxzZSwiQlQ1LTEwMjR4MTV4MzJoLXJwZS1zd2EtdGVzdCI6ZmFsc2UsIkJUNS0xMDI0eDE1eDMyaC1ycGUtc3dhLXZhbGlkYXRpb24iOnRydWUsIkJUNC0xMDI0eDE1eDMyaC1zd2EtdGVzdCI6ZmFsc2UsIkJUNC0xMDI0eDE1eDMyaC1zd2EtdmFsaWRhdGlvbiI6dHJ1ZSwiQlQ0LTEwMjR4MTV4MzJoLXRyYWluIjpmYWxzZX0%3D&_smoothingWeight=0