Elon Musk names his AI Grok. What is Grok?
In the context of AI, grokking refers to a phenomenon where a neural network, after appearing to overfit its training data (performing well only on the training set), suddenly and dramatically improves its ability to generalize to new, unseen data. This "delayed generalization" can occur after a period where the model's performance on the training data seems to have plateaued. Essentially, the model "groks" the underlying patterns in the data, not just memorizing it, and this understanding manifests as a sudden improvement in its ability to generalize and make accurate predictions on data that it had not previously seen. (something that humans do). Grokking is like the AI equivalent of Enlightenment when it “begins to see the Light”. The chart above illustrates the meaning of Grokking. A Neural Network is trained on data to predict. The Blue line depicts what normally happens as each training cycle of the neural network progresses. Here we see that accuracy o...