https://complete-dope.github.io/codex/tags/tokenizer/ Recent content in Tokenizer on Mohit Dulani Hugo -- 0.146.0 en Fri, 24 Oct 2025 00:00:00 +0000 https://complete-dope.github.io/codex/posts/tokenizer/ Fri, 24 Oct 2025 00:00:00 +0000 https://complete-dope.github.io/codex/posts/tokenizer/ <h1 id="encoding">Encoding</h1> <p>Its a way to send / transmit information <br> The transfering of data across internet is done in byte stream and we have a defined format to encode those in a byte stream Like if its a text data then we have <code>utf-8</code> encoding , if its a audio file its <code>mp3</code> encoding, image file its <code>png</code> encoded like this we define encoding for all dataformats and on client side decoding happens to output it in a compatible way to the end user</p>