The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead). More...

#include <fe/lexer.h>

Classes
struct	Ahead
	A decoded code point together with the byte range it occupies. More...

Public Member Functions
	Lexer (std::string_view buf)
	Lexer (const Src &src)

Protected Member Functions
	Lexer (std::string_view buf, const Src *src)
	Delegate here to funnel both of the above into a single ctor of your own.
char32_t	ahead (size_t i=0) const
Loc	peek () const
	Location of the next character to be consumed (Lexer::ahead()); empty once the buffer is exhausted.
void	start ()
	Invoke before assembling the next token.
char32_t	next ()
	Get next char32_t in Lexer::buf_ and extend Lexer::loc_ to cover it.
Recover
Lexer::next input that cannot be part of a token, report it, and keep the current lexer going. Invoke after Lexer::start, so Lexer::loc_ spans exactly what was discarded.
bool	recover_utf8 ()
void	recover_char ()
	One character, reported as S::char_err.
Diagnostics
The defaults `S` may replace with one of its own.
void	utf8_err ()
void	char_err (char32_t c)
	Lexer::recover_char discarded `c` at Lexer::loc_.

Protected Attributes
std::string_view	buf_
const Src *	src_
size_t	cursor_ = 0
	Byte offset of the first not yet decoded character.
Ring< Ahead, K >	ahead_
Loc	loc_
	Location of the token we are currently constructing within Lexer::str_,.
std::string	str_

Accept
Accept next character in Lexer::buf_, depending on some condition.
enum class	Append { Off , On , Lower , Upper }
template<Append append = Append::On, class Pred>
bool	accept (Pred pred)
template<Append append = Append::On>
bool	accept (char32_t c)
template<Append append = Append::On>
bool	accept (char c)
template<Append append = Append::On>
bool	accept (char8_t c)

Detailed Description

template<size_t K, class S>
class fe::Lexer< K, S >

The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead).

You can "override" Lexer::next via CRTP (S is the child). The whole source has to sit in buf: a Pos is an index into it, so there is nothing left to keep track of - Lexer::next just hands out the byte range the code point it consumed occupied. S must provide somewhere to report to:

class MyLexer : public fe::Lexer<K, MyLexer> {
    fe::Driver& driver();                ///< The default diagnostic below lands in its Driver::error.
 
    friend fe::Lexer<K, MyLexer>;        ///< Otherwise, this may be private.
};

Lexer::utf8_err and Lexer::char_err come with a default; declare either in S to word it differently.

Definition at line 30 of file lexer.h.

Class Documentation

◆ fe::Lexer::Ahead

struct fe::Lexer::Ahead

template<size_t K, class S>
struct fe::Lexer< K, S >::Ahead

A decoded code point together with the byte range it occupies.

Definition at line 53 of file lexer.h.

Class Members
Pos	begin
char32_t	c = utf8::EoF
Pos	end

Member Enumeration Documentation

◆ Append

template<size_t K, class S>

enum class fe::Lexer::Append

strongprotected

What should happen to the accepted char? Normalize identifiers via Append::Lower or Append::Upper for case-insensitive languages like FORTRAN or SQL.

Enumerator
Off	Do not append accepted char to Lexer::str_.
On	Append accepted char as is to Lexer::str_.
Lower	Append accepted char via fe::utf8::tolower` to Lexer::str_.
Upper	Append accepted char via fe::utf8::toupper` to Lexer::str_.

Definition at line 81 of file lexer.h.

Constructor & Destructor Documentation

◆ Lexer() [1/3]

template<size_t K, class S>

fe::Lexer< K, S >::Lexer ( std::string_view buf )

inline

Definition at line 36 of file lexer.h.

References Lexer().

Referenced by Lexer(), and Lexer().

◆ Lexer() [2/3]

template<size_t K, class S>

fe::Lexer< K, S >::Lexer ( const Src & src )

inline

Definition at line 38 of file lexer.h.

References Lexer().

◆ Lexer() [3/3]

template<size_t K, class S>

fe::Lexer< K, S >::Lexer	(	std::string_view	buf,
		const Src *	src )

inlineprotected

Delegate here to funnel both of the above into a single ctor of your own.

Definition at line 43 of file lexer.h.

References ahead_, buf_, cursor_, src_, and start().

Member Function Documentation

◆ accept() [1/4]

template<size_t K, class S>

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char c )

inlineprotected

Definition at line 106 of file lexer.h.

References accept().

◆ accept() [2/4]

template<size_t K, class S>

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char32_t c )

inlineprotected

Definition at line 105 of file lexer.h.

References accept().

◆ accept() [3/4]

template<size_t K, class S>

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char8_t c )

inlineprotected

Definition at line 107 of file lexer.h.

References accept().

◆ accept() [4/4]

template<size_t K, class S>

template<Append append = Append::On, class Pred>

bool fe::Lexer< K, S >::accept ( Pred pred )

inlineprotected

Returns: true if pred holds. In this case invoke Lexer::next() and append to Lexer::str_, if append.

Definition at line 91 of file lexer.h.

References ahead(), Lower, Off, str_, fe::utf8::tolower(), fe::utf8::toupper(), and Upper.

Referenced by accept(), accept(), accept(), and recover_utf8().

◆ ahead()

template<size_t K, class S>

char32_t fe::Lexer< K, S >::ahead ( size_t i = 0 ) const

inlineprotected

Definition at line 58 of file lexer.h.

References ahead_.

Referenced by accept(), and recover_char().

◆ char_err()

template<size_t K, class S>

void fe::Lexer< K, S >::char_err ( char32_t c )

inlineprotected

Lexer::recover_char discarded c at Lexer::loc_.

Definition at line 146 of file lexer.h.

References loc_.

◆ next()

template<size_t K, class S>

char32_t fe::Lexer< K, S >::next ( )

inlineprotected

Get next char32_t in Lexer::buf_ and extend Lexer::loc_ to cover it.

Returns: utf8::Invalid on an invalid UTF-8 sequence.

Definition at line 71 of file lexer.h.

References ahead_, fe::Loc::end, and loc_.

◆ peek()

template<size_t K, class S>

Loc fe::Lexer< K, S >::peek ( ) const

inlineprotected

Location of the next character to be consumed (Lexer::ahead()); empty once the buffer is exhausted.

Definition at line 61 of file lexer.h.

References ahead_, and src_.

Referenced by start().

◆ recover_char()

template<size_t K, class S>

void fe::Lexer< K, S >::recover_char ( )

inlineprotected

One character, reported as S::char_err.

This is the last resort of your token dispatch: nothing in your language starts with it.

Warning: Never at utf8::EoF - accept that first or your lexer will spin.

Definition at line 127 of file lexer.h.

References ahead().

◆ recover_utf8()

template<size_t K, class S>

bool fe::Lexer< K, S >::recover_utf8 ( )

inlineprotected

A whole run of malformed UTF-8, if any, reported as one S::utf8_err. Check this before your token dispatch: utf8::Invalid is no code point and matches no rule of yours.

Definition at line 117 of file lexer.h.

References accept().

◆ start()

template<size_t K, class S>

void fe::Lexer< K, S >::start ( )

inlineprotected

Invoke before assembling the next token.

Definition at line 64 of file lexer.h.

References fe::Loc::anew_begin(), loc_, peek(), and str_.

Referenced by Lexer().

◆ utf8_err()

template<size_t K, class S>

void fe::Lexer< K, S >::utf8_err ( )

inlineprotected

Lexer::recover_utf8 discarded the malformed bytes at Lexer::loc_.

Definition at line 138 of file lexer.h.

References loc_.

Member Data Documentation

◆ ahead_

template<size_t K, class S>

Ring<Ahead, K> fe::Lexer< K, S >::ahead_

protected

Definition at line 157 of file lexer.h.

Referenced by ahead(), Lexer(), next(), and peek().

◆ buf_

template<size_t K, class S>

std::string_view fe::Lexer< K, S >::buf_

protected

Definition at line 154 of file lexer.h.

Referenced by Lexer().

◆ cursor_

template<size_t K, class S>

size_t fe::Lexer< K, S >::cursor_ = 0

protected

Byte offset of the first not yet decoded character.

Definition at line 156 of file lexer.h.

Referenced by Lexer().

◆ loc_

template<size_t K, class S>

Loc fe::Lexer< K, S >::loc_

protected

Location of the token we are currently constructing within Lexer::str_,.

Definition at line 158 of file lexer.h.

Referenced by char_err(), next(), start(), and utf8_err().

◆ src_

template<size_t K, class S>

const Src* fe::Lexer< K, S >::src_

protected

Definition at line 155 of file lexer.h.

Referenced by Lexer(), and peek().

◆ str_

template<size_t K, class S>

std::string fe::Lexer< K, S >::str_

protected

Definition at line 159 of file lexer.h.

Referenced by accept(), and start().

The documentation for this class was generated from the following file:

fe/lexer.h

Classes

Public Member Functions

Protected Member Functions

Protected Attributes

Accept

Detailed Description

Class Documentation

◆ fe::Lexer::Ahead

Member Enumeration Documentation

◆ Append

Constructor & Destructor Documentation

◆ Lexer() [1/3]

◆ Lexer() [2/3]

◆ Lexer() [3/3]

Member Function Documentation

◆ accept() [1/4]

◆ accept() [2/4]

◆ accept() [3/4]

◆ accept() [4/4]

◆ ahead()

◆ char_err()

◆ next()

◆ peek()

◆ recover_char()

◆ recover_utf8()

◆ start()

◆ utf8_err()

Member Data Documentation

◆ ahead_

◆ buf_

◆ cursor_

◆ loc_

◆ src_

◆ str_