diff options
| Commit | 2b2cbc4eb7cc6499327bee6b3ec19f1544c4f842 (patch) | |
|---|---|---|
| Author | 2023-04-12 16:53:53 +0300 | |
| Committer | 2023-04-12 16:53:53 +0300 | |
| Files | 0eef34c4e93b8a5c9db797021afc6d2ed1cc60e1 | |
| syndamia.com-2b2cbc4eb7cc6499327bee6b3ec19f1544c4f842.tar syndamia.com-2b2cbc4eb7cc6499327bee6b3ec19f1544c4f842.tar.gz syndamia.com-2b2cbc4eb7cc6499327bee6b3ec19f1544c4f842.zip | ||
| Parent | 9442627e595bf67e17d1cdd6793bacb707b86207 (diff) | |
[blog/seoc] Restructuring, minor changes to LR parsers and added more stuff for AST
4 files changed, 409 insertions, 58 deletions
diff --git a/src/blog/sane-explanation-of-compilers/img/tree-cycle.svg b/src/blog/sane-explanation-of-compilers/img/tree-cycle.svg new file mode 100644 index 0000000..408177e --- /dev/null +++ b/src/blog/sane-explanation-of-compilers/img/tree-cycle.svg @@ -0,0 +1,55 @@ +<svg version="1.1" baseProfile="full" width="124.0" height="229.0" viewbox="0 0 124 229" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"> + <desc >[A]->[B] + [B]->[C] + [C]->[A]</desc> + <g stroke-width="1.0" text-align="left" font="12pt Helvetica, Arial, sans-serif" font-size="12pt" font-family="Helvetica" font-weight="bold" font-style="normal"> + <g font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" stroke-width="3.0" stroke-linejoin="round" stroke-linecap="round" stroke="#33322E"> + <g stroke="transparent" fill="transparent"> + <rect x="0.0" y="0.0" height="229.0" width="124.0" stroke="none"></rect> + </g> + <g transform="translate(8, 8)" fill="#33322E"> + <g transform="translate(20, 20)" fill="#33322E" font-family="Helvetica" font-size="12pt" font-weight="normal" font-style="normal"> + <path d="M29.2 31.0 L14 51 L14 64.33333333333333 L14.0 64.3 " fill="none"></path> + <path d="M8.7 57.7 L14.0 64.3 L19.3 57.7 L14.0 71.0 Z"></path> + <path d="M14.0 102.0 L14 122 L25.175484465105697 136.69369253745379 L25.2 136.7 " fill="none"></path> + <path d="M16.9 134.6 L25.2 136.7 L25.4 128.2 L29.2 142.0 Z"></path> + <path d="M52.8 142.0 L68 122 L68 86.5 L68 86.5 L68 51 L68 51 L56.8245155348943 36.30630746254621 L56.8 36.3 " fill="none"></path> + <path d="M65.1 38.4 L56.8 36.3 L56.6 44.8 L52.8 31.0 Z"></path> + <g data-name="A"> + <g fill="#eee8d5" stroke="#33322E" data-name="A"> + <rect x="27.0" y="0.0" height="31.0" width="28.0" data-name="A"></rect> + </g> + <g transform="translate(27, 0)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="A"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="A"> + <text x="6.0" y="13.5" stroke="none" text-anchor="middle" data-name="A">A</text> + + </g> + </g> + </g> + <g data-name="B"> + <g fill="#eee8d5" stroke="#33322E" data-name="B"> + <rect x="0.0" y="71.0" height="31.0" width="28.0" data-name="B"></rect> + </g> + <g transform="translate(0, 71)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="B"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="B"> + <text x="6.0" y="13.5" stroke="none" text-anchor="middle" data-name="B">B</text> + + </g> + </g> + </g> + <g data-name="C"> + <g fill="#eee8d5" stroke="#33322E" data-name="C"> + <rect x="27.0" y="142.0" height="31.0" width="28.0" data-name="C"></rect> + </g> + <g transform="translate(27, 142)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="C"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="C"> + <text x="6.0" y="13.5" stroke="none" text-anchor="middle" data-name="C">C</text> + + </g> + </g> + </g> + </g> + </g> + </g> + </g> + </svg>
\ No newline at end of file diff --git a/src/blog/sane-explanation-of-compilers/img/tree-if.svg b/src/blog/sane-explanation-of-compilers/img/tree-if.svg new file mode 100644 index 0000000..4b8ae1d --- /dev/null +++ b/src/blog/sane-explanation-of-compilers/img/tree-if.svg @@ -0,0 +1,178 @@ +<svg version="1.1" baseProfile="full" width="474.5" height="300.0" viewbox="0 0 474.5 300" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"> + <desc >[if]->[>] + [>]->[a] + [>]->[b] + [if]->[body when true] + [body when true]->[=] + [=]->[c] + [=]->[5] + [if]->[body when false] + [body when false]->[+=] + [+=]->[d] + [+=]->[10]</desc> + <g stroke-width="1.0" text-align="left" font="12pt Helvetica, Arial, sans-serif" font-size="12pt" font-family="Helvetica" font-weight="bold" font-style="normal"> + <g font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" stroke-width="3.0" stroke-linejoin="round" stroke-linecap="round" stroke="#33322E"> + <g stroke="transparent" fill="transparent"> + <rect x="0.0" y="0.0" height="300.0" width="474.5" stroke="none"></rect> + </g> + <g transform="translate(8, 8)" fill="#33322E"> + <g transform="translate(20, 20)" fill="#33322E" font-family="Helvetica" font-size="12pt" font-weight="normal" font-style="normal"> + <path d="M157.5 19.2 L45.25 51 L45.25 64.33333333333333 L45.3 64.3 " fill="none"></path> + <path d="M39.9 57.7 L45.3 64.3 L50.6 57.7 L45.3 71.0 Z"></path> + <path d="M32.8 100.0 L12.5 122 L12.5 135.33333333333334 L12.5 135.3 " fill="none"></path> + <path d="M7.2 128.7 L12.5 135.3 L17.8 128.7 L12.5 142.0 Z"></path> + <path d="M57.8 100.0 L78 122 L78 135.33333333333334 L78.0 135.3 " fill="none"></path> + <path d="M72.7 128.7 L78.0 135.3 L83.3 128.7 L78.0 142.0 Z"></path> + <path d="M170.5 31.0 L170.5 51 L170.5 64.33333333333333 L170.5 64.3 " fill="none"></path> + <path d="M165.2 57.7 L170.5 64.3 L175.8 57.7 L170.5 71.0 Z"></path> + <path d="M170.5 102.0 L170.5 122 L170.5 135.33333333333334 L170.5 135.3 " fill="none"></path> + <path d="M165.2 128.7 L170.5 135.3 L175.8 128.7 L170.5 142.0 Z"></path> + <path d="M158.0 171.2 L138 193 L138 206.33333333333334 L138.0 206.3 " fill="none"></path> + <path d="M132.7 199.7 L138.0 206.3 L143.3 199.7 L138.0 213.0 Z"></path> + <path d="M183.0 165.4 L226.5 193 L226.5 206.33333333333334 L226.5 206.3 " fill="none"></path> + <path d="M221.2 199.7 L226.5 206.3 L231.8 199.7 L226.5 213.0 Z"></path> + <path d="M183.5 18.1 L348 51 L348 64.33333333333333 L348.0 64.3 " fill="none"></path> + <path d="M342.7 57.7 L348.0 64.3 L353.3 57.7 L348.0 71.0 Z"></path> + <path d="M348.0 102.0 L348 122 L348 135.33333333333334 L348.0 135.3 " fill="none"></path> + <path d="M342.7 128.7 L348.0 135.3 L353.3 128.7 L348.0 142.0 Z"></path> + <path d="M330.5 168.6 L292 193 L292 206.33333333333334 L292.0 206.3 " fill="none"></path> + <path d="M286.7 199.7 L292.0 206.3 L297.3 199.7 L292.0 213.0 Z"></path> + <path d="M363.3 173.0 L383 193 L383 206.33333333333334 L383.0 206.3 " fill="none"></path> + <path d="M377.7 199.7 L383.0 206.3 L388.3 199.7 L383.0 213.0 Z"></path> + <g data-name="if"> + <g fill="#eee8d5" stroke="#33322E" data-name="if"> + <rect x="157.5" y="0.0" height="31.0" width="26.0" data-name="if"></rect> + </g> + <g transform="translate(157.5, 0)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="if"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="if"> + <text x="5.0" y="13.5" stroke="none" text-anchor="middle" data-name="if">if</text> + + </g> + </g> + </g> + <g data-name=">"> + <g fill="#eee8d5" stroke="#33322E" data-name=">"> + <rect x="32.8" y="71.0" height="31.0" width="25.0" data-name=">"></rect> + </g> + <g transform="translate(32.75, 71)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name=">"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name=">"> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name=">">></text> + + </g> + </g> + </g> + <g data-name="a"> + <g fill="#eee8d5" stroke="#33322E" data-name="a"> + <rect x="0.0" y="142.0" height="31.0" width="25.0" data-name="a"></rect> + </g> + <g transform="translate(0, 142)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="a"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="a"> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name="a">a</text> + + </g> + </g> + </g> + <g data-name="b"> + <g fill="#eee8d5" stroke="#33322E" data-name="b"> + <rect x="65.0" y="142.0" height="31.0" width="26.0" data-name="b"></rect> + </g> + <g transform="translate(65, 142)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="b"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="b"> + <text x="5.0" y="13.5" stroke="none" text-anchor="middle" data-name="b">b</text> + + </g> + </g> + </g> + <g data-name="body when true"> + <g fill="#eee8d5" stroke="#33322E" data-name="body when true"> + <rect x="103.5" y="71.0" height="31.0" width="134.0" data-name="body when true"></rect> + </g> + <g transform="translate(103.5, 71)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="body when true"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="body when true"> + <text x="59.0" y="13.5" stroke="none" text-anchor="middle" data-name="body when true">body when true</text> + + </g> + </g> + </g> + <g data-name="="> + <g fill="#eee8d5" stroke="#33322E" data-name="="> + <rect x="158.0" y="142.0" height="31.0" width="25.0" data-name="="></rect> + </g> + <g transform="translate(158, 142)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="="> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="="> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name="=">=</text> + + </g> + </g> + </g> + <g data-name="c"> + <g fill="#eee8d5" stroke="#33322E" data-name="c"> + <rect x="125.5" y="213.0" height="31.0" width="25.0" data-name="c"></rect> + </g> + <g transform="translate(125.5, 213)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="c"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="c"> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name="c">c</text> + + </g> + </g> + </g> + <g data-name="5"> + <g fill="#eee8d5" stroke="#33322E" data-name="5"> + <rect x="214.0" y="213.0" height="31.0" width="25.0" data-name="5"></rect> + </g> + <g transform="translate(214, 213)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="5"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="5"> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name="5">5</text> + + </g> + </g> + </g> + <g data-name="body when false"> + <g fill="#eee8d5" stroke="#33322E" data-name="body when false"> + <rect x="277.5" y="71.0" height="31.0" width="141.0" data-name="body when false"></rect> + </g> + <g transform="translate(277.5, 71)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="body when false"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="body when false"> + <text x="62.5" y="13.5" stroke="none" text-anchor="middle" data-name="body when false">body when false</text> + + </g> + </g> + </g> + <g data-name="+="> + <g fill="#eee8d5" stroke="#33322E" data-name="+="> + <rect x="330.5" y="142.0" height="31.0" width="35.0" data-name="+="></rect> + </g> + <g transform="translate(330.5, 142)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="+="> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="+="> + <text x="9.5" y="13.5" stroke="none" text-anchor="middle" data-name="+=">+=</text> + + </g> + </g> + </g> + <g data-name="d"> + <g fill="#eee8d5" stroke="#33322E" data-name="d"> + <rect x="279.0" y="213.0" height="31.0" width="26.0" data-name="d"></rect> + </g> + <g transform="translate(279, 213)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="d"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="d"> + <text x="5.0" y="13.5" stroke="none" text-anchor="middle" data-name="d">d</text> + + </g> + </g> + </g> + <g data-name="10"> + <g fill="#eee8d5" stroke="#33322E" data-name="10"> + <rect x="366.0" y="213.0" height="31.0" width="34.0" data-name="10"></rect> + </g> + <g transform="translate(366, 213)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="10"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="10"> + <text x="9.0" y="13.5" stroke="none" text-anchor="middle" data-name="10">10</text> + + </g> + </g> + </g> + </g> + </g> + </g> + </g> + </svg>
\ No newline at end of file diff --git a/src/blog/sane-explanation-of-compilers/img/tree-simple.svg b/src/blog/sane-explanation-of-compilers/img/tree-simple.svg new file mode 100644 index 0000000..172472c --- /dev/null +++ b/src/blog/sane-explanation-of-compilers/img/tree-simple.svg @@ -0,0 +1,80 @@ +<svg version="1.1" baseProfile="full" width="177.0" height="229.0" viewbox="0 0 177 229" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"> + <desc >[+]->[*] + [*]->[1] + [*]->[2] + [+]->[3]</desc> + <g stroke-width="1.0" text-align="left" font="12pt Helvetica, Arial, sans-serif" font-size="12pt" font-family="Helvetica" font-weight="bold" font-style="normal"> + <g font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" stroke-width="3.0" stroke-linejoin="round" stroke-linecap="round" stroke="#33322E"> + <g stroke="transparent" fill="transparent"> + <rect x="0.0" y="0.0" height="229.0" width="177.0" stroke="none"></rect> + </g> + <g transform="translate(8, 8)" fill="#33322E"> + <g transform="translate(20, 20)" fill="#33322E" font-family="Helvetica" font-size="12pt" font-weight="normal" font-style="normal"> + <path d="M56.8 29.5 L76 51 L76 64.33333333333333 L76.0 64.3 " fill="none"></path> + <path d="M70.7 57.7 L76.0 64.3 L81.3 57.7 L76.0 71.0 Z"></path> + <path d="M65.0 98.5 L43.5 122 L43.5 135.33333333333334 L43.5 135.3 " fill="none"></path> + <path d="M38.2 128.7 L43.5 135.3 L48.8 128.7 L43.5 142.0 Z"></path> + <path d="M87.0 98.5 L108.5 122 L108.5 135.33333333333334 L108.5 135.3 " fill="none"></path> + <path d="M103.2 128.7 L108.5 135.3 L113.8 128.7 L108.5 142.0 Z"></path> + <path d="M31.8 29.5 L12.5 51 L12.5 64.33333333333333 L12.5 64.3 " fill="none"></path> + <path d="M7.2 57.7 L12.5 64.3 L17.8 57.7 L12.5 71.0 Z"></path> + <g data-name="+"> + <g fill="#eee8d5" stroke="#33322E" data-name="+"> + <rect x="31.8" y="0.0" height="31.0" width="25.0" data-name="+"></rect> + </g> + <g transform="translate(31.75, 0)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="+"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="+"> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name="+">+</text> + + </g> + </g> + </g> + <g data-name="*"> + <g fill="#eee8d5" stroke="#33322E" data-name="*"> + <rect x="65.0" y="71.0" height="31.0" width="22.0" data-name="*"></rect> + </g> + <g transform="translate(65, 71)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="*"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="*"> + <text x="3.0" y="13.5" stroke="none" text-anchor="middle" data-name="*">*</text> + + </g> + </g> + </g> + <g data-name="1"> + <g fill="#eee8d5" stroke="#33322E" data-name="1"> + <rect x="31.0" y="142.0" height="31.0" width="25.0" data-name="1"></rect> + </g> + <g transform="translate(31, 142)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="1"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="1"> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name="1">1</text> + + </g> + </g> + </g> + <g data-name="2"> + <g fill="#eee8d5" stroke="#33322E" data-name="2"> + <rect x="96.0" y="142.0" height="31.0" width="25.0" data-name="2"></rect> + </g> + <g transform="translate(96, 142)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="2"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="2"> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name="2">2</text> + + </g> + </g> + </g> + <g data-name="3"> + <g fill="#eee8d5" stroke="#33322E" data-name="3"> + <rect x="0.0" y="71.0" height="31.0" width="25.0" data-name="3"></rect> + </g> + <g transform="translate(0, 71)" font-family="Helvetica" font-size="12pt" font-weight="bold" font-style="normal" data-name="3"> + <g transform="translate(8, 8)" fill="#33322E" text-align="center" data-name="3"> + <text x="4.5" y="13.5" stroke="none" text-anchor="middle" data-name="3">3</text> + + </g> + </g> + </g> + </g> + </g> + </g> + </g> + </svg>
\ No newline at end of file diff --git a/src/blog/sane-explanation-of-compilers/index.md b/src/blog/sane-explanation-of-compilers/index.md index eb65f9c..21a6b64 100644 --- a/src/blog/sane-explanation-of-compilers/index.md +++ b/src/blog/sane-explanation-of-compilers/index.md @@ -3,7 +3,7 @@ ## Contents 1. [Preface](#preface) -2. [1. How do we understand code?](#1.-how-do-we-understand-code) +2. [I. How do we understand code?](#i.-how-do-we-understand-code) 1. [Math](#math) 1. [Regular expressions](#regular-expressions) 2. [Grammars](#grammars) @@ -11,14 +11,15 @@ 1. [Implementational difficulties with grammars](#implementational-difficulties-with-grammars) 2. [Usage in the real world](#usage-in-the-real-world) 3. [Parsing algorithms](#parsing-algorithms) + 1. [LL](#ll) + 2. [LR](#lr) + 3. [LR's family](#lrs-family) 4. [Abstract syntax tree](#abstract-syntax-tree) -3. [Implementing a front-end](#implementing-a-front-end) - 1. [lex](#lex) - 1. [Example 1](#example-1) - 2. [Example 2](#example-2) - 2. [yacc](#yacc) -4. [2. How can we improve the code?] -5. [3. Virtual machines] + 3. [Implementing a front-end](#implementing-a-front-end) + 1. [lex](#lex) + 2. [yacc](#yacc) +4. [II. How can we improve the code?] +5. [III. Virtual machines] 6. [Compilation schemes] 1. [AOT] 2. [JIT] @@ -33,7 +34,7 @@ Before I've forgotten everything, I'll try to hit that middle spot, generally co ## Preface Processors work with numbers, every action (instruction) a processor can do is defined by numbers. -However, writing code only with numbers is tedious, error-prone and requires a lot of manhours for not a lot of functionality. +However, writing code only with numbers is tedious, error-prone and simple stuff sometimes take ages to implement. That is why, over time, programmers have tried to abstract sets of instructions into words and text constructions that are easier to work with. Thanks to abstraction, programmers have been able to create more and more complicated projects with less and less work, that is more portable and compatible with different machines. @@ -41,10 +42,10 @@ The tradeoff is that we need to translate our nice programmer code to numbers a This job is delegated to a special type of program, called a compiler[^how-did-we-get-here], which is separated into three main layers/components: 1. Front-end: it understands the program, what the code is trying to do -2. Middle: does optimizations and simplifications +2. Middle: does optimizations and simplifications, though this part is often inside the back-end 3. Back-end: generates the actual machine instructions -## 1. How do we understand code? +## I. How do we understand code? Syntax in current programming languages can be quite sophisticated and ambiguous. For example (taken from [wikipedia](https://en.wikipedia.org/wiki/Most_vexing_parse)) this C++ snippet @@ -72,7 +73,7 @@ A word of caution, I'll use a decent amount of fancy names, so I have a way to m Don't be scared and skip over them if they're confusing you. There is a special field in mathematics that deals with syntactical analysis: [formal language theory](https://en.wikipedia.org/wiki/Formal_language). -Although learning even the basics merits a whole university course, I'll still try to roughly explain what's important, for those that haven't studyied it. +Although even the basics merit a whole university course, I'll still try to roughly explain what's important, for those that haven't studied it. This field works primarily with strings via the help of two (relevant for our purposes) instruments: regular expressions and grammars. #### Regular expressions @@ -118,7 +119,7 @@ At their very base, such a pattern is created with the following rules: + "ac**hievem**en**ts**". - [Kleene star](https://en.wikipedia.org/wiki/Kleene_star) - repeat the preceding letter or group, 0 or more times. - An asterisk is used to mark this operation (hence the name, "Kleen **star**"). + An asterisk is used to mark this operation (hence the name, "Kleene **star**"). ```regex accents* @@ -320,8 +321,8 @@ How exactly we deal with these problems comes later, for now the takeaway is tha #### Usage in the real world -Let's think a bit more about what grammars are supposed to do: they're really good at handling structure, especially recursive structure, so it's best if we use them only for that. -What I'm trying to say is, our math expression grammar (from [here](#grammars)) shouldn't really care if we're trying to sum integers or floats or chars[^static-typing], it's important how everything is layed out and that our operations are done with numbers. +Let's think a bit more about grammars: they're really good at handling structure, especially recursive structure, so it's best if we use them only for that. +Our math expression grammar (from [here](#grammars)) shouldn't really care if we're trying to sum integers or floats or chars[^static-typing], it's important how everything is layed out and that our operations are done with numbers. This is why we're going to separate and generalize the code into [tokens](https://en.wikipedia.org/wiki/Lexical_analysis#Token), basically into building blocks. As a rough example, we want `1 + (2 * 3)` to be transformed into `NUMBER + (NUMBER * NUMBER)`, `int var = 13;` into `TYPE NAME = NUMBER;` and so on, where "TYPE", "NUMBER" and "NAME" are tokens. @@ -340,7 +341,7 @@ As you might've noticed, here we can use regular expressions, grouping consequti Additionally, we're going to be reading the file, a decently slow operation, only once and for a comparatively very little time. In the compiler front-end, the part that implementes our regular expressions is called the "lexer", and the part that implements our grammars - the "parser". -In the last section we already talked about how easy implementing regexes (lexers) is and how difficult implementing grammars (parsers) is, but we didn't talk about actual impelementations of parsers. +In the last section we already talked about how easy implementing regexes (lexers) is and how difficult implementing grammars (parsers) is, let's look at the latter. #### Parsing algorithms @@ -388,7 +389,7 @@ We check the first k character of all expressions, and if anyone of them is a na we start with `S`, we peek and get the `(`, in our production we're looking at a name, S. Let's check the first character of every expression of S. - In the first expression, the first "character" is `F`, a name, so we look at every expression for F. + In the first expression, the first "character" is `F`, a name, so we look at every expression for `F`. There is only one, which is "a", different than "(", no dice. Now we check the second expression for S, it's first character is "(", success! We replace: @@ -412,18 +413,17 @@ We check the first k character of all expressions, and if anyone of them is a na ``` we peek and get the `a`, in our production the second "character" is the name `S`, `S` has to become `a`. - We check and the second expression won't do, so we replace with the first + We do that same check from before, first expression for `S` is `F`, whose first (and only) expression is `a`, success! ``` I v Input: (a + a) I v - Production: (F + F) + Production: (a + F) ``` - We're still peaking at `a`, in the current production we're looking at the name `F`, which can be replaced with `a` (or a string that starts with `a`, we only check the first character of each rule). - We replace, then incremenet the symbols, matches the middle characters, when at the second `F` we do the same, replace with `a`, then match `)` and we're done. + The middle characters " + " are matched and at the second `F` we check it's expressions, it has only `a`, success, replace, end of input, we're done. Another LL(1) example can be found in [Karl R. Abrahamson notes](http://www.cs.ecu.edu/karl/5220/spr16/Notes/Top-down/LL1.html). @@ -449,7 +449,13 @@ Conceptually, LR works in the exact opposite way to LL: for every character in t Figuring out an approach isn't as intuitive and easy as with LL, but it still isn't too hard. It all boils down to using a whole bunch of conditional statements and tracing where we "came" from. -- Let's look at the first LL example. +- Let's look at the first LL example: + + ```grammar + F -> a + S -> F | (S + F) + ``` + We need to track "how the current string looks", because if we read an "a", it could either be from the left handside of the "a + a" or from the right handside. We're going to track where we came from with numbers, which I will define like this: @@ -466,7 +472,8 @@ It all boils down to using a whole bunch of conditional statements and tracing w |8 |"(S + F)"; this is the one before the final| You can think of every number as the valid, but "unfinished" state of the current string. - It's important to note that this table is the most compact one possible, you could make a table for literally every valid state: "(", "(a", "(F", "(S", ... + For now don't think about how I've defined those numbers, it'll make sense later. + But you could make a table for literally every valid state: "(", "(a", "(F", "(S", ..., there is no problem. Keep in mind that the way I'll show the algorithm step by step isn't really correct, but will suffice for the current moment. We'll be working with a string, where reading a character from the input means putting the latest one in the String: @@ -515,7 +522,7 @@ It all boils down to using a whole bunch of conditional statements and tracing w Number : 6 ``` - Now, our string ends with an "a", so we should change our number to 4, but we shouldn't, this would remove our progress so far! + Now, our string ends with an "a", it's tempting to change our number to 4, but we shouldn't, this would remove our progress so far! Better yet, this number will also be special, if we have a 6 and we end with "a", rather than going to 4, we'll substitute "a" with F now and change number to 7. ``` @@ -525,31 +532,29 @@ It all boils down to using a whole bunch of conditional statements and tracing w And now it's trivial, we read ")", so we go to number 8. But if we're at number 8, don't continue reading, replace with number 1. - Now that we're at number 1, if we read any character (including end of string), we'll say the string upto now matches. + Now that we're at number 1, if we read any character (including end of string), we'll say the string upto now matches and we're done! It's also helpful to put what we do in different situations in another table: |Number|If ends on|Change number to|Also do| |------|----------|----------------|-------| |0 |"(" |3 | | - |1 | | |Finished| + |1 | | |Nothing, we matched| |2 | |5 |Replace "F" with "S"| - |3 |"a" |4 |Decrement reading| + |3 |"a" |4 || |4 | |2 |Replace "a" with "F"| |5 |" + " |6 | | - |6 | |7 |Replace "a" with "F"| + |6 |"a" |7 |Replace "a" with "F"| |7 |")" |8 | | |8 | |1 |Replace "(S + F)" with "S"| -- What you might notice is that - - 1. this "isn't" the smallest amount of number that work, we could remove 2 and in rule 4 directly replace "a" with "S", and - 2. we're replacing "a" with "F" in two different rules, isn't there a way to do it in only one? + where, if there is a value in "If ends on", we read the next character, while otherwise we don't and we just execute "Change number to" and "Also do". - The reason for both is that the way I showed the algorithm isn't exactly the way it normally works. - Rather than tracing the current number in a variable, we trace it with a stack, essentially an array. +- You're probably wondering why the table works so weird, like why on number 3 we don't directly replace "a" with "S" and change to number 5 for example. + The reason is that the way I showed the algorithm isn't exactly how it normally works. + Rather than tracing the current number in a variable, we trace it with a [stack](https://en.wikipedia.org/wiki/Stack_%28abstract_data_type%29). - When you read the character and have to go to another number, the number is now added to the array (this operation is called "shift"). + When you read a character and have to go to another number, the number is added to the stack (this operation is called "shift"). When we have to change characters from String (without reading anything), with a string (expression) of length j, we remove the last j added numbers and do a normal shift from the last number and character in String (this operation is called "reduce"). - Let's look at the same example, done with shift and reduce operations (without me showing the table): @@ -592,9 +597,11 @@ It all boils down to using a whole bunch of conditional statements and tracing w And so on, we continue. As you can see, we go back and forth, from 3 to 4, back to 3, then to 2, back to 3, then to 5. - This is how we avoid doing the same thing on different numbers and why the initial amount of numbers is the smallest amount. + This way we avoid doing different things on different numbers. + Additionally, in the real world nobody makes this table by hand, it is generated by a program, and this method makes table generation easier. -- As for the complete table, you can view it on the [Context Free Grammar Checker](https://smlweb.cpsc.ucalgary.ca/lr0.php?grammar=S+-%3E+F+%7C+%28+S+%2B+F+%29.%0AF+-%3E+a.&substs=) by [](TODO), or with a more pretty styling on [Grammophone](https://mdaines.github.io/grammophone/#/), where you input on the left: +- As for the complete table, you can view it on the [Context Free Grammar Checker](https://smlweb.cpsc.ucalgary.ca/lr0.php?grammar=S+-%3E+F+%7C+%28+S+%2B+F+%29.%0AF+-%3E+a.&substs=) by [](TODO), + or with a more pretty styling on [Grammophone](https://mdaines.github.io/grammophone/#/), where you input on the left: ``` S -> F | "(" S "+" F ")". @@ -614,11 +621,13 @@ It all boils down to using a whole bunch of conditional statements and tracing w ##### LR's family -LR is more of a family of algorithms, because what I didn't touch on and is very important was generation of the table. -This subject is really big can of beans, but generally there are 3 often used types: [CLR](https://en.wikipedia.org/wiki/Canonical_LR_parser), [SLR](https://en.wikipedia.org/wiki/Simple_LR_parser) and [LALR](https://en.wikipedia.org/wiki/LALR_parser). +LR is more of a family of algorithms, because there are multiple different ways to generate the table, each with it's own benefits and drawbacks. +This subject is a really big can of beans, which I'm not knowlegeable enough to talk about. +What you need to know is that there are 3 often used types: [CLR](https://en.wikipedia.org/wiki/Canonical_LR_parser), [SLR](https://en.wikipedia.org/wiki/Simple_LR_parser) and [LALR](https://en.wikipedia.org/wiki/LALR_parser). -CLR is an LR parser with 1 character lookahead (LR(1)), following [Knuth](TODO)'s original theory. -It requires a lot of memory for table generation, so Frank DeRemer created two simplifications, the SLR and LALR, which use singificantly less memory but work with a bit less grammars. +- CLR is an LR parser with 1 character lookahead (LR(1)), following [Knuth](TODO)'s original theory. + It requires a lot of memory for table generation, so Frank DeRemer created two simplifications: +- the SLR and LALR, which use singificantly less memory but work with a bit less grammars LALR is the most commonly used LR parser, however most compilers are custom-tailored pieces of software that very loosely follow the methods of LL and LR. @@ -636,20 +645,42 @@ If you want to learn more, I would highly suggest: #### Abstract syntax tree We already decided that the lexer will output tokens (and characters) to the parser, but we didn't talk about what the parser is supposed to output. -It definetly has to be some sort of structural representation of the overall code, for example, for `1 * 2 + 3` we have to be able to easily find out what the left and right handsides of the `+` and `*` are. -Drawing out this relationship, where the left line connects to the left handside and the right accordingly: +It definetly has to be some sort of structural representation of the overall code, for example, for `3 + 1 * 2` we have to be able to easily find out what the left and right handsides of the `+` and `*` are. +Let's draw out[^nomnoml] this relationship, where the left arrow connects to the left handside and the right accordingly: + + + +Hey, that looks an awful lot like a [tree](https://en.wikipedia.org/wiki/Tree_%28data_structure%29)! +Trees are nodes, some sort of elements, where there is a connection from one element to some other elements, with the only rule being that cycles aren't allowed: + + + +Great, but can we represent everything else with a tree? +Why, of course, for example we can represent: + +```cpp +if (a > b) + c = 5; +else + d += 10; ``` - + - / \ - * 3 - / \ -1 2 -``` -We notice it looks awfully a lot like a tree! +as + + + +And trust me, we can represent the whole code structure with a tree. +Now, compilers are made in low-level languages like C++, so we need to think about the data type of each node (element) in our tree. +We'll definetly need [object polymorphism](https://en.wikipedia.org/wiki/Subtyping#Examples) to be able to represent different nodes. + +At a later stage we'll need to be running/evaluating the whole code with this tree, so how are we going to work with it? +When we navigate to the node `<`, we'll do "if left element is less than right, do body when true, otherwise do body when false" but are we going to implement every single operator and language feature by hand? +Or alteratively we could have a virtual method "evaluate" which changes + -## Implementing a front-end + +### Implementing a front-end If you want, you could implement your own lexer and parser manually, but as a start, it's better to try and use already created stuff. [Compiler-compilers](https://en.wikipedia.org/wiki/Compiler-compiler) are tools that create parsers (and more) from their normal formal definition. @@ -657,13 +688,13 @@ If you want, you could implement your own lexer and parser manually, but as a st The two popular ones from the old days are lex and yacc. lex is a lexer generator and yacc is a LALR parser generator, both of which generate a corresponding C implementation of the given regular expressions and grammars. -### lex +#### lex Lex is the original program, from 1975, in the modern day it's successor/implementation is the opensource [flex](https://github.com/westes/flex). A lex source file is divided into three parts: -``` +```lex %{ // C code that will be put in the program verbatim %} @@ -687,8 +718,6 @@ To simplify your life, you can write only the regex or a part of it inside the " Referencing those names in later regexes is done by surrounding them in curly braces. Usually the `C_code` in the regex would be a return (more commonly called "emit") of a token, which will be available in a header file, but more on that later. -#### Example 1 - Let's look at a very simple example program: every whole number is printed as "NUMBER", while everything else is printed as-is: ```lex @@ -727,8 +756,6 @@ will output The number NUMBER changed to NUMBER, NUMBER, and NUMBER sequentially, before becoming NUMBER. ``` -#### Example 2 - Now let's try something more sophisticated, let's surround numbers and strings in HTML tags, the stuff that power webpages. We would want numbers to be in the format: @@ -781,7 +808,7 @@ John said <span class="num">10</span> times <span class="str">"2 + 2 = 5"</span> You've probably already noticed that the number inside the strings aren't matched, because the string rule results in a longer match. In programming languages this is fine, since strings are quite often "self-contained" objects, there isn't anything in them that you would need to match. -### yacc +#### yacc Similarly to lex, yacc originates from 1975, and it's modern replacement is [GNU Bison](https://www.gnu.org/software/bison/). @@ -803,6 +830,15 @@ A yacc file looks very similar to a lex one: // C code ``` +however rather than definitions you have settings, and the rules are in a completely different format (because they're grammars). + + +## II. How can we improve the code? + + + +## III. Virtual machines + [^how-did-we-get-here]: Though it's obvious, I would like to point out that modern compilers are most often written in languages that are themselves compiled. You might ask yourself "where did the first compiler come from?" The answer is simple: it was written with "numbers", with direct instructions to the processor, from the time when computers were simple enough that *all* programming was done with CPU instructions. [^terminals-and-nonterminals]: Formally, we call the rule "names" nonterminals, the idea being that all normal characters, that could appear in our string, are called terminals, since they would terminate replacement and string generation. For the purposes of this post, I won't use the formal names, with the hope that using less terminology will increase understandability. @@ -814,3 +850,5 @@ A yacc file looks very similar to a lex one: [^context-free-grammars]: Be careful about cases that are too complicated, the grammar I'm showing is "context-free", but that topic is way beyond the scope of this post. [^static-typing]: Yeah yeah, in a statically typed language you might not be able to do operations on doubles and integers at the same time, but you get the point. + +[^nomnoml]: The graphics are made with [nomnoml](https://nomnoml.com/) by [Daniel Kallin](https://github.com/skanaar) |