Tokenisation

Tokenisation divides text into manageable units such as words, subwords, or characters. These units become the basic input that many NLP models process. The choice of tokenisation can significantly affect model