Documentation
¶
Overview ¶
Package token defines the lexical tokens of the SQLite SQL dialect.
Kind mirrors the TK_* token codes SQLite's tokenizer produces (src/tokenize.c) and the grammar consumes (src/parse.y). Names match the SQLite spelling so grammar attributions in the parser read directly against parse.y; where SQLite's name differs from the keyword (TK_COLUMNKW for COLUMN, TK_AUTOINCR for AUTOINCREMENT) the SQLite name is kept.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CanFallback ¶
CanFallback reports whether k is in SQLite's %fallback ID set.
Types ¶
type Kind ¶
type Kind int
Kind identifies the lexical class of a token.
const ( EOF Kind = iota // end of input; also the synthetic tokens SQLite appends // Tokenizer-only kinds. SPACE and COMMENT never reach the parser. SPACE COMMENT ILLEGAL // "unrecognized token" // Literals and names. ID STRING INTEGER FLOAT BLOB VARIABLE QNUMBER // numeric literal containing '_' digit separators // Punctuation and operators. LP // ( RP // ) SEMI // ; COMMA // , DOT // . PLUS // + MINUS // - STAR // * SLASH // / REM // % EQ // = or == NE // != or <> LT // < LE // <= GT // > GE // >= LSHIFT // << RSHIFT // >> BITAND // & BITOR // | BITNOT // ~ CONCAT // || PTR // -> or ->> // Keywords, in the order of SQLite's keyword table. ABORT ACTION ADD AFTER ALL ALTER ALWAYS ANALYZE AND AS ASC ATTACH AUTOINCR BEFORE BEGIN BETWEEN BY CASCADE CASE CAST CHECK COLLATE COLUMNKW COMMIT CONFLICT CONSTRAINT CREATE CURRENT CTIME_KW // CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP DATABASE DEFAULT DEFERRED DEFERRABLE DELETE DESC DETACH DISTINCT DO DROP END EACH ELSE ESCAPE EXCEPT EXCLUSIVE EXCLUDE EXISTS EXPLAIN FAIL FILTER FIRST FOLLOWING FOR FOREIGN FROM GENERATED GROUP GROUPS HAVING IF IGNORE IMMEDIATE IN INDEX INDEXED INITIALLY INSERT INSTEAD INTERSECT INTO IS ISNULL JOIN JOIN_KW // CROSS, FULL, INNER, LEFT, NATURAL, OUTER, RIGHT KEY LAST LIKE_KW // GLOB, LIKE, REGEXP LIMIT MATCH MATERIALIZED NO NOT NOTHING NOTNULL NULL NULLS OF OFFSET ON OR ORDER OTHERS OVER PARTITION PLAN PRAGMA PRECEDING PRIMARY QUERY RAISE RANGE RECURSIVE REFERENCES REINDEX RELEASE RENAME REPLACE RESTRICT RETURNING ROLLBACK ROW ROWS SAVEPOINT SELECT SET TABLE TEMP THEN TIES TO TRANSACTION TRIGGER UNBOUNDED UNION UNIQUE UPDATE USING VACUUM VALUES VIEW VIRTUAL WHEN WHERE WINDOW WITH WITHOUT // KindCount is one past the last kind, so a table indexed by Kind can // be sized without a map. KindCount )
The token kinds. The ordering carries no meaning (unlike SQLite's generated codes, which are ordered to shrink the LALR tables).
func Lookup ¶
Lookup returns the keyword kind for an identifier spelling, or ID if it is not a keyword. Matching is ASCII case-insensitive, as in keywordCode().
This runs once per identifier token, so it neither allocates nor copies: the candidates are found from the length and first letter alone, and each is compared against the input a byte at a time, upper-casing as it goes.
type Token ¶
Token is one lexical token: a kind and the half-open byte range [Pos, End) of the input it was lexed from.
The spelling is deliberately not a field. It is exactly src[Pos:End], so storing it would duplicate what the offsets already say -- and it would put a pointer in the struct. That is not a small thing here: a token slice is the largest single allocation a parse makes, and one holding pointers costs a write barrier per token to build and has to be walked by the garbage collector afterwards. Deriving the text instead makes Token 24 bytes rather than 40 and the slice invisible to the collector, which is worth about 2.7x on the cost of producing one.
Source Files