Description
Turtle and TriG token scanner.
Scanning is separated from the grammar so that token boundaries are decided in one place. Several classes of grammar bug come from deciding them twice: a scanner that recognises the PREFIX keyword by looking at the next few bytes will misread a prefixed name whose prefix happens to be "prefix", and one that requires whitespace after the "a" keyword will reject "a<http://example.org/C>", which is well formed.
Here a name-shaped run is always scanned to its end first, and only then classified. Keywords are what is left over when a run is not followed by a colon.
"="
function "=" (Left, Right : Token) return Boolean
Compare two tokens by class and content, ignoring where they were found. Two occurrences of the same name in a document are equal.
Parameters
- Left
First token
- Right
Second token
Return value
True when the tokens denote the same lexeme
Dialect_Kind
type Dialect_Kind is (RDF_Dialect, N3_Dialect, SPARQL_Dialect);
Which token vocabulary to recognise.
The RDF grammars must not see an N3 token: a scanner that always produced them would make "=>" a token that the Turtle parser has to reject by name rather than by not knowing it, and the difference shows up the first time someone writes a prefixed name that happens to contain one.
Enumeration literals
- RDF_Dialect
Turtle, TriG, N-Triples, N-Quads
- N3_Dialect
Adds variables, implication, paths, quantifiers
- SPARQL_Dialect
Adds variables, operators, and bare keywords
Direction
function Direction (Value : Token) return Direction_Value
Return the base direction of a language token.
Parameters
- Value
Token to inspect
Return value
The direction
Direction_Value
type Direction_Value is (Left_To_Right, Right_To_Left);
Direction carried by a Direction_Token.
Enumeration literals
- Left_To_Right
The "ltr" base direction
- Right_To_Left
The "rtl" base direction
End_Position
function End_Position (Value : Token) return Parser_Cursors.Cursor_State
Position just past the token's last byte.
Parameters
- Value
Token to inspect
Return value
Cursor position where the token ends
Form
function Form (Value : Token) return String_Form
Return the quoting form of a string token.
Parameters
- Value
Token to inspect
Return value
Short_Quoted or Long_Quoted
Has_Direction
function Has_Direction (Value : Token) return Boolean
Report whether a language token carries a base direction.
RDF 1.2 writes the direction as a suffix of the language tag rather than as its own token, so "@en--ltr" is one token whose text is "en".
Parameters
- Value
Token to inspect
Return value
True when Direction is meaningful
Kind
function Kind (Value : Token) return Token_Kind
Report the token's lexical class.
Parameters
- Value
Token to inspect
Return value
The token's kind
Null_Token
function Null_Token return Token
A token that carries no text, used where a Token is needed before one has been scanned.
Return value
A Dot_Token at the initial position
Prefix
function Prefix (Value : Token) return String
Return the prefix part of a prefixed name, without the colon.
Parameters
- Value
Token to inspect
Return value
The prefix, empty for a name written ":local"
Scan
procedure Scan
(Text : String;
Position : in out Parser_Cursors.Cursor_State;
Index : in out Positive;
Last_Chunk : Boolean;
Result : out Token;
Status : out Scan_Status;
Error : out Scan_Error_Kind;
Dialect : Dialect_Kind := RDF_Dialect)
Scan one token from Text, beginning at the byte named by Position.
Leading whitespace and comments are consumed regardless of the outcome, so a caller that receives Needs_More_Input can retain only the bytes from the returned Position onwards.
Last_Chunk decides what the end of the buffer means, and the answer is genuinely different for the two cases. Mid-stream, a name or number running up to the last byte might continue in the next chunk, so the only safe answer is Needs_More_Input. On the final chunk the same bytes are a complete token, and a string or IRI reference left open is Unterminated_Token rather than a request for input that will never arrive.
Parameters
- Text
Buffer to scan
- Position
Cursor at the first unscanned byte; advanced past the token on success, and past skipped whitespace otherwise
- Index
Index into Text corresponding to Position; advanced the same way
- Last_Chunk
True when no further input will follow this buffer
- Result
The scanned token, meaningful only when Token_Found
- Status
Outcome of the scan
- Error
Why the scan failed, No_Error otherwise
- Dialect
Token vocabulary to recognise
Scan_Error_Kind
type Scan_Error_Kind is
(No_Error,
Invalid_Encoding,
Unterminated_Token,
Malformed_Escape,
Malformed_Number,
Malformed_Language_Tag,
Forbidden_Character,
Unexpected_Character);
Why a scan failed.
Enumeration literals
- No_Error
The scan did not fail
- Invalid_Encoding
not canonical UTF-8
The bytes are not canonical UTF-8
- Unterminated_Token
input ended inside a token at Finish
Input ended inside a token
- Malformed_Escape
\q, \u12, a surrogate escape
An escape sequence that does not decode
- Malformed_Number
A numeric literal the grammar does not admit
- Malformed_Language_Tag
A language tag that is not well formed
- Forbidden_Character
raw control byte inside <...>
A raw control byte where none may appear
- Unexpected_Character
A character no token may begin or continue
Scan_Status
type Scan_Status is
(Token_Found, Needs_More_Input, End_Of_Input, Scan_Error);
Outcome of one scan attempt.
Enumeration literals
- Token_Found
A complete token was scanned
- Needs_More_Input
The buffer ends part-way through a token; the caller should append the next chunk and retry
- End_Of_Input
Only whitespace and comments remained
- Scan_Error
The bytes cannot begin or continue any token
Start_Position
function Start_Position (Value : Token) return Parser_Cursors.Cursor_State
Position of the token's first byte.
Parameters
- Value
Token to inspect
Return value
Cursor position where the token starts
String_Form
type String_Form is (Short_Quoted, Long_Quoted);
Which quoting form a String_Token was written with. The grammar needs this only to reject a long-quoted string where a short one is required; the value is identical either way.
Enumeration literals
- Short_Quoted
Written with one quote character at each end
- Long_Quoted
Written with three
Text
function Text (Value : Token) return String
Return the token's decoded value.
Escapes are already resolved: a String_Token carries the characters the literal denotes, and an IRI_Reference_Token carries the IRI text with its numeric escapes applied.
Parameters
- Value
Token to inspect
Return value
The decoded text, empty for punctuation
Token
type Token (<>) is private;
A scanned token.
Token_Kind
type Token_Kind is
IRI_Reference_Token,
Prefixed_Name_Token,
Blank_Label_Token,
String_Token,
Integer_Token,
Decimal_Token,
Double_Token,
Boolean_Token,
,
,
,
,
,
,
…,
Slash_Token);
A lexical class of the Turtle and TriG grammars.
Enumeration literals
- IRI_Reference_Token
Terms <http://example.org/>
- Prefixed_Name_Token
ex:local, ex:, :local
- Blank_Label_Token
_:label
- String_Token
"text", 'text', """text""", '''text'''
- Integer_Token
42
- Decimal_Token
4.2
- Double_Token
4.2e1
- Boolean_Token
true, false
- A_Token
a
- Language_Token
Modifiers attaching to a preceding token @en-us, @en--ltr
- Datatype_Token
^^
- Prefix_Directive_Token
Directives @prefix
- Base_Directive_Token
@base
- Sparql_Prefix_Token
PREFIX
- Sparql_Base_Token
BASE
- Version_Directive_Token
VERSION
- Graph_Token
GRAPH
- Dot_Token
Structure
- Semicolon_Token
; -- another predicate, same subject
- Comma_Token
, -- another object, same predicate
- Open_Paren_Token
(
- Close_Paren_Token
)
- Open_Bracket_Token
[
- Close_Bracket_Token
]
- Open_Brace_Token
{
- Close_Brace_Token
}
- Open_Quoted_Token
RDF 1.2 <<
- Close_Quoted_Token
>>
- Open_Triple_Term_Token
<<(
- Close_Triple_Term_Token
)>>
- Reifier_Token
~
- Open_Annotation_Token
{|
- Close_Annotation_Token
|}
- Variable_Token
Notation3, recognised only in N3_Dialect ?name
- Implies_Token
=>
- Implied_By_Token
<=
- Equals_Token
=
- Forward_Path_Token
!
- Backward_Path_Token
^
- Reverse_Arrow_Token
<-
- For_All_Token
@forAll
- For_Some_Token
@forSome
- Keyword_Token
SPARQL, recognised only in SPARQL_Dialect SELECT, WHERE, FILTER, ...
- Or_Token
||
- And_Token
&&
- Not_Equal_Token
!=
- Less_Token
<
- Greater_Token
>
- Less_Or_Equal_Token
<=
- Greater_Or_Equal_Token
>=
- Plus_Token
- Minus_Token
- Star_Token
- Slash_Token
/