7 Splay Tree for Tokenization

▼ |  | Syntax Color: Utilities

1 | Parenthesis Matching
2 | Lexer Contract and The Don’t-Stop Structure Type
3 | Racket Lexer
4 | Default Lexer
5 | Module Lexer
6 | Scribble Lexer
7 | Splay Tree for Tokenization
8 | Interface for Clients of Syntax Color Output
9 | Racket S-Expression Indentation and Navigation

On this page:

token-tree%
new
get-root
search!
node?
node-token-length
node-token-data
node-left-subtree-length
node-left
node-right
insert-first!
insert-last!
insert-last-spec!

top contents ← prev up next →

## 7 Splay Tree for Tokenization🔗ℹ

(require syntax-color/token-tree)
| package: syntax-color-lib

class token-tree% : class?
superclass: object%

A splay-tree class specifically geared for the task of on-the-fly tokenization. Instead of keying nodes on values, each node has a length, and they are found by finding a node that follows a certain total length of preceding nodes.
FIXME: many methods are not yet documented.

> > constructor (new token-tree% [len len] [data data])
> > → (is-a?/c token-tree%)
> > len : (or/c exact-nonnegative-integer? fasle/c)
> > data : any/c

> Creates a token tree with a single element.

> > method (send a-token-tree get-root) → (or/c node? #f)

> Returns the root node in the tree.

> > method (send a-token-tree search! key-position) → void?
> > key-position : natural-number/c

> Splays, setting the root node to be the closest node to offset key-position (i.e., making the total length of the left tree at least key-position, if possible).

> procedure (node? v) → boolean?
> v : any/c
> procedure (node-token-length n) → natural-number/c
> n : node?
> procedure (node-token-data n) → any/c
> n : node?
> procedure (node-left-subtree-length n) → natural-number/c
> n : node?
> procedure (node-left n) → (or/c node? #f)
> n : node?
> procedure (node-right n) → (or/c node? #f)
> n : node?

Functions for working with nodes in a token-tree%.

> procedure (insert-first! tree1 tree2) → void?
> tree1 : (is-a?/c token-tree%)
> tree2 : (is-a?/c token-tree%)

Inserts tree1 into tree2 as the first thing, setting tree2’s root to #f.

> procedure (insert-last! tree1 tree2) → void?
> tree1 : (is-a?/c token-tree%)
> tree2 : (is-a?/c token-tree%)

Inserts tree1 into tree2 as the last thing, setting tree2’s root to #f.

> procedure (insert-last-spec! tree n v) → void?
> tree : (is-a?/c token-tree%)
> n : natural-number/c
> v : any/c

Same as

> (insert-last! tree
> (new token-tree%
> [length n]
> [data v]))

This optimization is important for the colorer.

top contents ← prev up next →