fe::Lexer< K, S > Class Template Reference

The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead). More...

#include <fe/lexer.h>

Public Member Functions
	Lexer (std::istream &istream, const std::filesystem::path *path=nullptr)

Protected Member Functions
char32_t	ahead (size_t i=0) const
void	start ()
	Invoke before assembling the next token.
char32_t	next ()
	Get next char32_t in Lexer::istream_ and increase Lexer::loc_.

Protected Attributes
std::istream &	istream_
Ring< char32_t, K >	ahead_
Loc	loc_
	Location of the token we are currently constructing within Lexer::str_,.
Pos	peek_
	Position of ahead_::first;.
std::string	str_

Accept
Accept next character in Lexer::istream_, depending on some condition.
enum class	Append {   Off ,   On ,   Lower ,   Upper }
template<Append append = Append::On, class Pred >
bool	accept (Pred pred)
template<Append append = Append::On>
bool	accept (char32_t c)
template<Append append = Append::On>
bool	accept (char c)
template<Append append = Append::On>
bool	accept (char8_t c)

Detailed Description

template<size_t K, class S>
class fe::Lexer< K, S >

The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead).

You can "overide" Lexer::next via CRTP (S is the child).

Definition at line 14 of file lexer.h.

Member Enumeration Documentation

Append

template<size_t K, class S >

enum class fe::Lexer::Append

strongprotected

What should happend to the accepted char? Normalize identifiers via Append::Lower or Append::Upper for case-insensitive languages like FORTRAN or SQL.

Enumerator
Off	Do not append accepted char to Lexer::str_.
On	Append accepted char as is to Lexer::str_.
Lower	Append accepted char via fe::utf8::tolower` to Lexer::str_.
Upper	Append accepted char via fe::utf8::toupper` to Lexer::str_.

Definition at line 62 of file lexer.h.

Constructor & Destructor Documentation

Lexer()

template<size_t K, class S >

fe::Lexer< K, S >::Lexer ( std::istream & istream, const std::filesystem::path * path = nullptr )

inline

Definition at line 20 of file lexer.h.

Member Function Documentation

accept() [1/4]

template<size_t K, class S >

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char c )

inlineprotected

Definition at line 86 of file lexer.h.

References fe::Lexer< K, S >::accept().

accept() [2/4]

template<size_t K, class S >

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char32_t c )

inlineprotected

Definition at line 85 of file lexer.h.

References fe::Lexer< K, S >::accept().

accept() [3/4]

template<size_t K, class S >

template<Append append = Append::On>

bool fe::Lexer< K, S >::accept ( char8_t c )

inlineprotected

Definition at line 87 of file lexer.h.

References fe::Lexer< K, S >::accept().

accept() [4/4]

template<size_t K, class S >

template<Append append = Append::On, class Pred >

bool fe::Lexer< K, S >::accept ( Pred pred )

inlineprotected

Returns

true if pred holds. In this case invoke Lexer::next() and append to Lexer::str_, if append.

Definition at line 71 of file lexer.h.

References fe::Lexer< K, S >::ahead(), fe::Lexer< K, S >::Lower, fe::Lexer< K, S >::Off, fe::Lexer< K, S >::str_, fe::utf8::tolower(), fe::utf8::toupper(), and fe::Lexer< K, S >::Upper.

Referenced by fe::Lexer< K, S >::accept(), fe::Lexer< K, S >::accept(), and fe::Lexer< K, S >::accept().

ahead()

template<size_t K, class S >

char32_t fe::Lexer< K, S >::ahead ( size_t i = 0 ) const

inlineprotected

Definition at line 30 of file lexer.h.

References fe::Lexer< K, S >::ahead_.

Referenced by fe::Lexer< K, S >::accept().

next()

template<size_t K, class S >

char32_t fe::Lexer< K, S >::next ( )

inlineprotected

Get next char32_t in Lexer::istream_ and increase Lexer::loc_.

Returns

Null on an invalid UTF-8 sequence.

Definition at line 40 of file lexer.h.

References fe::Lexer< K, S >::ahead_, fe::utf8::BOM, fe::Pos::col, fe::utf8::decode(), fe::utf8::EoF, fe::Loc::finis, fe::Ring< T, N >::front(), fe::Lexer< K, S >::istream_, fe::Lexer< K, S >::loc_, fe::Lexer< K, S >::peek_, fe::Ring< T, N >::put(), and fe::Pos::row.

start()

template<size_t K, class S >

void fe::Lexer< K, S >::start ( )

inlineprotected

Invoke before assembling the next token.

Definition at line 33 of file lexer.h.

References fe::Loc::begin, fe::Lexer< K, S >::loc_, fe::Lexer< K, S >::peek_, and fe::Lexer< K, S >::str_.

Member Data Documentation

ahead_

template<size_t K, class S >

Ring<char32_t, K> fe::Lexer< K, S >::ahead_

protected

Definition at line 92 of file lexer.h.

Referenced by fe::Lexer< K, S >::ahead(), and fe::Lexer< K, S >::next().

istream_

template<size_t K, class S >

std::istream& fe::Lexer< K, S >::istream_

protected

Definition at line 91 of file lexer.h.

Referenced by fe::Lexer< K, S >::next().

loc_

template<size_t K, class S >

Loc fe::Lexer< K, S >::loc_

protected

Location of the token we are currently constructing within Lexer::str_,.

Definition at line 93 of file lexer.h.

Referenced by fe::Lexer< K, S >::next(), and fe::Lexer< K, S >::start().

peek_

template<size_t K, class S >

Pos fe::Lexer< K, S >::peek_

protected

Position of ahead_::first;.

Definition at line 94 of file lexer.h.

Referenced by fe::Lexer< K, S >::next(), and fe::Lexer< K, S >::start().

str_

template<size_t K, class S >

std::string fe::Lexer< K, S >::str_

protected

Definition at line 95 of file lexer.h.

Referenced by fe::Lexer< K, S >::accept(), and fe::Lexer< K, S >::start().

---

The documentation for this class was generated from the following file:

• fe/lexer.h