Description
Scanning a document that arrives in pieces, under a declared bound.
The Notation3 and SPARQL parsers both read a whole document into tokens and then parse the tokens. They did it with the same fifty lines, differing only in the dialect they passed, and neither bounded anything. An unterminated token was retained and rescanned on every chunk, so a document that never closes its first quote cost time quadratic in what had arrived and memory linear in it, without limit.
Both scan through here now. A bound reached is an outcome the caller reports in its own words, not an exception raised from underneath it.
Scan_Chunk
procedure Scan_Chunk
(Text : String;
Ended : Boolean;
Dialect : Lexers.Dialect_Kind;
Limits : Scan_Limits;
State : in out Scan_State;
Tokens : in out Token_Vectors.Vector;
Consumed : out Natural;
Outcome : out Scan_Outcome;
Stopped : out Parser_Cursors.Cursor_State;
Error : out Lexers.Scan_Error_Kind)
Scan as far as the bytes allow, appending whole tokens and reporting where the unread remainder begins.
Parameters
- Text
The retained bytes followed by the new ones
- Ended
Whether more bytes can still arrive
- Dialect
Grammar whose tokens these are
- Limits
Bounds for this document
- State
Carried between calls
- Tokens
Extended with every token completed
- Consumed
Index in Text where the remainder begins
- Outcome
How this scan ended
- Stopped
Cursor to report a failure at
- Error
Why a Malformed outcome was reported
Scan_Limits
type Scan_Limits is record
Maximum_Bytes : Positive := 64 * 1_024 * 1_024;
Maximum_Tokens : Natural := 0;
Maximum_Token_Bytes : Positive := 1_048_576;
end record;
What one document may cost to scan.
Record fields
- Maximum_Bytes
Total input accepted
- Maximum_Tokens
Tokens retained; zero means no limit
- Maximum_Token_Bytes
Longest single token. It also bounds the buffer held across a chunk boundary, which is what stops an unfinished token from being retained and rescanned without end
Scan_Outcome
type Scan_Outcome is
(Scanned, Malformed, Byte_Limit, Token_Limit, Token_Bytes_Limit);
How a scan ended.
Enumeration literals
- Scanned
Every token the bytes completed was appended
- Malformed
The bytes cannot begin or continue a token
- Byte_Limit
The document is longer than Maximum_Bytes
- Token_Limit
It holds more tokens than Maximum_Tokens
- Token_Bytes_Limit
One token is longer than Maximum_Token_Bytes, finished or not
Scan_State
type Scan_State is record
Origin : Parser_Cursors.Cursor_State :=
Parser_Cursors.Initial_State;
Bytes_Seen : Natural := 0;
end record;
What one scan carries to the next.
Record fields
- Origin
Cursor where the retained bytes begin
- Bytes_Seen
Input accepted so far, against Maximum_Bytes
Token_Vectors
package Token_Vectors is new Ada.Containers.Vectors
(Index_Type => Positive,
Element_Type => Lexers.Token,
"=" => Lexers."=");