The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead). More...

#include <fe/lexer.h>

Classes
struct	Ahead
	A decoded code point together with the byte range it occupies. More...

Public Member Functions
	Lexer (std::string_view buf)
	Lexer (const Src &src)

Protected Member Functions
	Lexer (std::string_view buf, const Src *src)
	Delegate here to funnel both of the above into a single ctor of your own.
char32_t	ahead (size_t i=0) const
Loc	peek () const
	Location of the next character to be consumed (Lexer::ahead()); empty once the buffer is exhausted.
void	start ()
	Invoke before assembling the next token.
Text
What has been lexed since Lexer::start. The whole source sits in Lexer::buf_, so Lexer::loc_ already is the token and Lexer::view costs nothing.
std::string_view	view () const
std::string	lower () const
	Lexer::view, case-folded - what a case-insensitive language like FORTRAN or SQL wants to intern.
std::string	upper () const
std::string	fold (char32_t(*f)(char32_t) noexcept) const
Accept
char32_t	next ()
	Get next char32_t in Lexer::buf_ and extend Lexer::loc_ to cover it.
bool	accept (auto pred)
	Accept next character in Lexer::buf_ and Lexer::next it, if `pred` holds.
bool	accept (char32_t c)
bool	accept (char c)
bool	accept (char8_t c)
std::string_view	accept_while (auto pred)
	Lexer::next as long as `pred` holds.
std::string_view	accept_while_none_of (char8_t a, char8_t b)
	Lexer::next up to - but not including - the next byte that is `a` or `b`.
std::string_view	accept_while_none_of (char8_t a)
	As Lexer::accept_while_none_of(char8_t, char8_t), but a single stop byte, which one memchr finds outright.
std::string_view	accept_while_none_of (char a)
std::string_view	accept_while_none_of (char a, char b)
std::string_view	accept_until (std::string_view seq)
	Lexer::next up to - but not including - the next occurrence of `seq`.
Recover
Lexer::next input that cannot be part of a token, report it, and keep the current lexer going. Invoke after Lexer::start, so Lexer::loc_ spans exactly what was discarded.
bool	recover_utf8 ()
void	recover_char ()
	One character, reported as S::char_err.
Diagnostics
The defaults `S` may replace with one of its own. Each yields the Error it reported into, so a Note can be chained.
fe::Error &	error ()
const fe::Error &	error () const
fe::Error &	utf8_err ()
	Lexer::recover_utf8 discarded the malformed bytes at Lexer::loc_.
fe::Error &	char_err (char32_t c)
	Lexer::recover_char discarded `c` at Lexer::loc_.

Protected Attributes
std::string_view	buf_
const Src *	src_
size_t	cursor_ = 0
	Byte offset of the first not yet decoded character.
Ring< Ahead, K >	ahead_
Loc	loc_
	Location of the token we are currently constructing - see Lexer::view.

Detailed Description

template<size_t K, class S>
class fe::Lexer< K, S >

The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead).

You can "override" Lexer::next via CRTP (S is the child). The whole source has to sit in buf: a Pos is an index into it, so there is nothing left to keep track of - Lexer::next just hands out the byte range the code point it consumed occupied. S must provide somewhere to report to:

class MyLexer : public fe::Lexer<K, MyLexer> {
    fe::Driver& driver();                ///< The default diagnostic below lands in its Driver::error.
 
    friend fe::Lexer<K, MyLexer>;        ///< Otherwise, this may be private.
};

Lexer::utf8_err and Lexer::char_err come with a default; declare either in S to word it differently.

Definition at line 30 of file lexer.h.

Class Documentation

◆ fe::Lexer::Ahead

struct fe::Lexer::Ahead

template<size_t K, class S>
struct fe::Lexer< K, S >::Ahead

A decoded code point together with the byte range it occupies.

Definition at line 53 of file lexer.h.

Class Members
Pos	begin
char32_t	c = utf8::EoF
Pos	end

Constructor & Destructor Documentation

◆ Lexer() [1/3]

template<size_t K, class S>

fe::Lexer< K, S >::Lexer ( std::string_view buf )

inline

Definition at line 36 of file lexer.h.

References Lexer().

Referenced by Lexer(), and Lexer().

◆ Lexer() [2/3]

template<size_t K, class S>

fe::Lexer< K, S >::Lexer ( const Src & src )

inline

Definition at line 38 of file lexer.h.

References Lexer().

◆ Lexer() [3/3]

template<size_t K, class S>

fe::Lexer< K, S >::Lexer	(	std::string_view	buf,
		const Src *	src )

inlineprotected

Delegate here to funnel both of the above into a single ctor of your own.

Definition at line 43 of file lexer.h.

References ahead_, buf_, cursor_, src_, and start().

Member Function Documentation

◆ accept() [1/4]

template<size_t K, class S>

bool fe::Lexer< K, S >::accept ( auto pred )

inlineprotected

Accept next character in Lexer::buf_ and Lexer::next it, if pred holds.

Definition at line 97 of file lexer.h.

References ahead().

Referenced by recover_utf8().

◆ accept() [2/4]

template<size_t K, class S>

bool fe::Lexer< K, S >::accept ( char c )

inlineprotected

Definition at line 105 of file lexer.h.

References accept().

Referenced by accept().

◆ accept() [3/4]

template<size_t K, class S>

bool fe::Lexer< K, S >::accept ( char32_t c )

inlineprotected

Definition at line 104 of file lexer.h.

References accept().

Referenced by accept().

◆ accept() [4/4]

template<size_t K, class S>

bool fe::Lexer< K, S >::accept ( char8_t c )

inlineprotected

Definition at line 106 of file lexer.h.

References accept().

Referenced by accept().

◆ accept_until()

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::accept_until ( std::string_view seq )

inlineprotected

Lexer::next up to - but not including - the next occurrence of seq.

Where Lexer::accept_while_none_of takes a set of bytes, this one takes a sequence: only seq, spelled in exactly that order, ends the run - which is what closes a /* comment. Stops at the end of Lexer::buf_ if seq never shows up.

Note: Nothing in the run is decoded or validated as UTF-8.

Returns: the run just consumed.

Definition at line 174 of file lexer.h.

References ahead_, and buf_.

◆ accept_while()

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::accept_while ( auto pred )

inlineprotected

Lexer::next as long as pred holds.

An ASCII run is taken straight out of Lexer::buf_ - no code point is decoded and the lookahead is re-primed once at the end, which is what makes scanning an identifier or a stretch of white space cost a compare per byte. A character beyond ASCII falls back to Lexer::next, so pred may match one.

Note: Only worth it if the lexed text is expected to be long such as identifiers or comments.

Returns: the run just consumed.

Definition at line 116 of file lexer.h.

References ahead(), ahead_, buf_, cursor_, fe::Loc::end, loc_, fe::Pos::off, and fe::Pos::Pos().

◆ accept_while_none_of() [1/4]

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::accept_while_none_of ( char a )

inlineprotected

Definition at line 165 of file lexer.h.

References accept_while_none_of().

Referenced by accept_while_none_of().

◆ accept_while_none_of() [2/4]

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::accept_while_none_of	(	char	a,
		char	b )

inlineprotected

Definition at line 166 of file lexer.h.

References accept_while_none_of().

Referenced by accept_while_none_of().

◆ accept_while_none_of() [3/4]

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::accept_while_none_of ( char8_t a )

inlineprotected

As Lexer::accept_while_none_of(char8_t, char8_t), but a single stop byte, which one memchr finds outright.

Definition at line 158 of file lexer.h.

References ahead_, and buf_.

◆ accept_while_none_of() [4/4]

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::accept_while_none_of	(	char8_t	a,
		char8_t	b )

inlineprotected

Lexer::next up to - but not including - the next byte that is a or b.

The stop bytes are a set: whichever comes first ends the run, and this is no substring search. Stops at the end of Lexer::buf_ if none of them ever shows up. No UTF-8 sequence spells an ASCII byte, so such a run needs no decoding at all: this is how to skip a comment or a string literal, where Lexer::accept_while pays a compare - and Lexer::next a whole utf8::decode - per character.

Note: Nothing in the run is validated as UTF-8 - malformed bytes cannot spell a or b either.

Returns: the run just consumed.

Definition at line 146 of file lexer.h.

References ahead_, and buf_.

◆ ahead()

template<size_t K, class S>

char32_t fe::Lexer< K, S >::ahead ( size_t i = 0 ) const

inlineprotected

Definition at line 58 of file lexer.h.

References ahead_.

Referenced by accept(), accept_while(), and recover_char().

◆ char_err()

template<size_t K, class S>

fe::Error & fe::Lexer< K, S >::char_err ( char32_t c )

inlineprotected

Lexer::recover_char discarded c at Lexer::loc_.

Definition at line 221 of file lexer.h.

References fe::Error::e(), error(), and loc_.

◆ error() [1/2]

template<size_t K, class S>

fe::Error & fe::Lexer< K, S >::error ( )

inlineprotected

Definition at line 209 of file lexer.h.

Referenced by char_err(), and utf8_err().

◆ error() [2/2]

template<size_t K, class S>

const fe::Error & fe::Lexer< K, S >::error ( ) const

inlineprotected

Definition at line 210 of file lexer.h.

◆ fold()

template<size_t K, class S>

std::string fe::Lexer< K, S >::fold ( char32_t(* f )(char32_t) noexcept ) const

inlineprotected

Definition at line 78 of file lexer.h.

References view().

Referenced by lower(), and upper().

◆ lower()

template<size_t K, class S>

std::string fe::Lexer< K, S >::lower ( ) const

inlineprotected

Lexer::view, case-folded - what a case-insensitive language like FORTRAN or SQL wants to intern.

Note: Byte-wise, which is all it takes: only ASCII folds, and no UTF-8 sequence spells it.

Definition at line 74 of file lexer.h.

References fold(), and fe::utf8::tolower().

◆ next()

template<size_t K, class S>

char32_t fe::Lexer< K, S >::next ( )

inlineprotected

Get next char32_t in Lexer::buf_ and extend Lexer::loc_ to cover it.

Returns: utf8::Invalid on an invalid UTF-8 sequence.

Definition at line 91 of file lexer.h.

References ahead_, fe::Loc::end, and loc_.

◆ peek()

template<size_t K, class S>

Loc fe::Lexer< K, S >::peek ( ) const

inlineprotected

Location of the next character to be consumed (Lexer::ahead()); empty once the buffer is exhausted.

Definition at line 61 of file lexer.h.

References ahead_, and src_.

Referenced by start().

◆ recover_char()

template<size_t K, class S>

void fe::Lexer< K, S >::recover_char ( )

inlineprotected

One character, reported as S::char_err.

This is the last resort of your token dispatch: nothing in your language starts with it.

Warning: Never at utf8::EoF - accept that first or your lexer will spin.

Definition at line 198 of file lexer.h.

References ahead().

◆ recover_utf8()

template<size_t K, class S>

bool fe::Lexer< K, S >::recover_utf8 ( )

inlineprotected

A whole run of malformed UTF-8, if any, reported as one S::utf8_err. Check this before your token dispatch: utf8::Invalid is no code point and matches no rule of yours.

Definition at line 188 of file lexer.h.

References accept().

◆ start()

template<size_t K, class S>

void fe::Lexer< K, S >::start ( )

inlineprotected

Invoke before assembling the next token.

Definition at line 64 of file lexer.h.

References fe::Loc::anew_begin(), loc_, and peek().

Referenced by Lexer().

◆ upper()

template<size_t K, class S>

std::string fe::Lexer< K, S >::upper ( ) const

inlineprotected

Definition at line 75 of file lexer.h.

References fold(), and fe::utf8::toupper().

◆ utf8_err()

template<size_t K, class S>

fe::Error & fe::Lexer< K, S >::utf8_err ( )

inlineprotected

Lexer::recover_utf8 discarded the malformed bytes at Lexer::loc_.

Definition at line 213 of file lexer.h.

References fe::Error::e(), error(), and loc_.

◆ view()

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::view ( ) const

inlineprotected

Definition at line 70 of file lexer.h.

References fe::Loc::begin, buf_, loc_, fe::Pos::off, and fe::Loc::size().

Referenced by fold().

Member Data Documentation

◆ ahead_

template<size_t K, class S>

Ring<Ahead, K> fe::Lexer< K, S >::ahead_

protected

Definition at line 232 of file lexer.h.

Referenced by accept_until(), accept_while(), accept_while_none_of(), accept_while_none_of(), ahead(), Lexer(), next(), and peek().

◆ buf_

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::buf_

protected

Definition at line 229 of file lexer.h.

Referenced by accept_until(), accept_while(), accept_while_none_of(), accept_while_none_of(), Lexer(), and view().

◆ cursor_

template<size_t K, class S>

size_t fe::Lexer< K, S >::cursor_ = 0

protected

Byte offset of the first not yet decoded character.

Definition at line 231 of file lexer.h.

Referenced by accept_while(), and Lexer().

◆ loc_

template<size_t K, class S>

Loc fe::Lexer< K, S >::loc_

protected

Location of the token we are currently constructing - see Lexer::view.

Definition at line 233 of file lexer.h.

Referenced by accept_while(), char_err(), next(), start(), utf8_err(), and view().

◆ src_

template<size_t K, class S>

const Src* fe::Lexer< K, S >::src_

protected

Definition at line 230 of file lexer.h.

Referenced by Lexer(), and peek().

The documentation for this class was generated from the following file:

fe/lexer.h

Classes

Public Member Functions

Protected Member Functions

Protected Attributes

Detailed Description

Class Documentation

◆ fe::Lexer::Ahead

Constructor & Destructor Documentation

◆ Lexer() [1/3]

◆ Lexer() [2/3]

◆ Lexer() [3/3]

Member Function Documentation

◆ accept() [1/4]

◆ accept() [2/4]

◆ accept() [3/4]

◆ accept() [4/4]

◆ accept_until()

◆ accept_while()

◆ accept_while_none_of() [1/4]

◆ accept_while_none_of() [2/4]

◆ accept_while_none_of() [3/4]

◆ accept_while_none_of() [4/4]

◆ ahead()

◆ char_err()

◆ error() [1/2]

◆ error() [2/2]

◆ fold()

◆ lower()

◆ next()

◆ peek()

◆ recover_char()

◆ recover_utf8()

◆ start()

◆ upper()

◆ utf8_err()

◆ view()

Member Data Documentation

◆ ahead_

◆ buf_

◆ cursor_

◆ loc_

◆ src_