[blog/seoc] Tokenization is now done entirely with tokens and without characters

1 files changed, 4 insertions, 3 deletions

diff --git a/src/blog/sane-explanation-of-compilers/part-1.md b/src/blog/sane-explanation-of-compilers/part-1.md
index 1ddaab4..282d315 100644
--- a/src/blog/sane-explanation-of-compilers/part-1.md
+++ b/src/blog/sane-explanation-of-compilers/part-1.md
@@ -325,9 +325,10 @@ Let's think a bit more about grammars: they're really good at handling structure
 Our math expression grammar (from [here](#grammars)) shouldn't really care if we're trying to sum integers or floats or chars[^static-typing], it's important how everything is layed out and that our operations are done with numbers.
 
 This is why we're going to separate and generalize the code into [tokens](https://en.wikipedia.org/wiki/Lexical_analysis#Token), basically into building blocks.
-As a rough example, we want `1 + (2 * 3)` to be transformed into `NUMBER + (NUMBER * NUMBER)`, `int var = 13;` into `TYPE NAME = NUMBER;` and so on, where "TYPE", "NUMBER" and "NAME" are tokens.
-Now we're reducing the amount of "objects" our grammars need to work on: from all characters to many (but significantly fewer) tokens and some really small amount of characters.
-Of course, each token will contain it's name (NUMBER, TYPE, ...) and it's value, since we aren't trying to change the code, but our grammars will work with the token names directly and just output the values at the very end.
+As a rough example, we want `1 + (2 * 3)` to be transformed into `NUMBER MATHOP LBRACE NUMBER MATHOP NUMBER RBRACE`, `int var = 13;` into `TYPE NAME ASSIGNMENT NUMBER` and so on, where "TYPE", "NUMBER", "NAME", ... are tokens.
+Now we're reducing the amount of "objects" our grammars need to work on: from all characters to many (but significantly fewer) tokens.
+Also we're simplifying our work, since we're trying to figure out the overall structure, it doesn't matter if we're multiplying or summing numbers, it's important that we're doing a mathematical operation on them.
+Of course, each token will actually be a pair, containing it's name (NUMBER, TYPE, ...) and it's value, but our grammars will work with the names only and output the actual values at the very end.
 Usually we identify at least six token types:
 
 - identifiers: variable and function names for example