src/common/tokens.txt


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

Primitive Tokens

   Token                Token Type      Flags

   identifier           IDENT            b e
   integer-literal      INTLIT           b e
   real-literal         REALLIT          b e
   string-literal       STRINGLIT        b e
   cset-literal         CSETLIT          b e
   end-of-file          EOFX

Reserved Words

   Token                Token Type      Flags

   break                BREAK            b e
   by                   BY
   case                 CASE             b
   create               CREATE           b
   default              DEFAULT          b
   do                   DO
   else                 ELSE
   end                  END              b
   every                EVERY            b
   fail                 FAIL             b e
   global               GLOBAL
   if                   IF               b
   initial              INITIAL          b
   invocable            INVOCABLE
   link                 LINK
   local                LOCAL            b
   next                 NEXT             b e
   not                  NOT              b
   of                   OF
   procedure            PROCEDURE
   record               RECORD
   repeat               REPEAT           b
   return               RETURN           b e
   static               STATIC           b
   suspend              SUSPEND          b e
   then                 THEN
   to                   TO
   until                UNTIL            b
   while                WHILE            b

``Operator'' tokens not used in standard unary/binary syntax, see op.txt.

   Token                Token Type      Flags

   (                    LPAREN           b
   )                    RPAREN             e
   +:                   PCOLON
   ,                    COMMA
   -:                   MCOLON
   :                    COLON
   ;                    SEMICOL
   [                    LBRACK           b
   ]                    RBRACK             e
   {                    LBRACE           b
   }                    RBRACE             e

tokens starting with $ are alternate spellings for some tokens

   $(                   LBRACE           b
   $)                   RBRACE           e
   $<                   LBRACK           b
   $>                   RBRACK           e

Explanation of Flags

   b    indicates that the token may begin an expression.
   e    indicates that the token may end an expression.

   These two flags are used for semicolon insertion.  If a line
   ends with an "e" token, and the next token is a "b" token,
   a semicolon is inserted between the two tokens.