FE 0.5.0
A header-only C++ library for writing frontends
Loading...
Searching...
No Matches
Public Member Functions | Protected Member Functions | Protected Attributes | List of all members
fe::Lexer< K, S > Class Template Reference

The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead). More...

#include <fe/lexer.h>

Public Member Functions

 Lexer (std::istream &istream, const std::filesystem::path *path=nullptr)
 

Protected Member Functions

char32_t ahead (size_t i=0) const
 
void start ()
 Invoke before assembling the next token.
 
char32_t next ()
 Get next char32_t in Lexer::istream_ and increase Lexer::loc_.
 

Protected Attributes

std::istream & istream_
 
Ring< char32_t, K > ahead_
 
Loc loc_
 Location of the token we are currently constructing within Lexer::str_,.
 
Pos peek_
 Position of ahead_::first;.
 
std::string str_
 

Accept

Accept next character in Lexer::istream_, depending on some condition.

enum class  Append {
  Off ,
  On ,
  Lower ,
  Upper
}
 What should happend to the accepted char? Normalize identifiers via Append::Lower or Append::Upper for case-insensitive languages like FORTRAN or SQL. More...
 
template<Append append = Append::On, class Pred >
bool accept (Pred pred)
 
template<Append append = Append::On>
bool accept (char32_t c)
 
template<Append append = Append::On>
bool accept (char c)
 
template<Append append = Append::On>
bool accept (char8_t c)
 

Detailed Description

template<size_t K, class S>
class fe::Lexer< K, S >

The blueprint for a lexer with a buffer of K tokens to peek into the future (Lexer::ahead).

You can "overide" Lexer::next via CRTP (S is the child).

Definition at line 14 of file lexer.h.

Member Enumeration Documentation

◆ Append

template<size_t K, class S >
enum class fe::Lexer::Append
strongprotected

What should happend to the accepted char? Normalize identifiers via Append::Lower or Append::Upper for case-insensitive languages like FORTRAN or SQL.

Enumerator
Off 

Do not append accepted char to Lexer::str_.

On 

Append accepted char as is to Lexer::str_.

Lower 

Append accepted char via fe::utf8::tolower` to Lexer::str_.

Upper 

Append accepted char via fe::utf8::toupper` to Lexer::str_.

Definition at line 62 of file lexer.h.

Constructor & Destructor Documentation

◆ Lexer()

template<size_t K, class S >
fe::Lexer< K, S >::Lexer ( std::istream &  istream,
const std::filesystem::path *  path = nullptr 
)
inline

Definition at line 20 of file lexer.h.

Member Function Documentation

◆ accept() [1/4]

template<size_t K, class S >
template<Append append = Append::On>
bool fe::Lexer< K, S >::accept ( char  c)
inlineprotected

Definition at line 86 of file lexer.h.

◆ accept() [2/4]

template<size_t K, class S >
template<Append append = Append::On>
bool fe::Lexer< K, S >::accept ( char32_t  c)
inlineprotected

Definition at line 85 of file lexer.h.

◆ accept() [3/4]

template<size_t K, class S >
template<Append append = Append::On>
bool fe::Lexer< K, S >::accept ( char8_t  c)
inlineprotected

Definition at line 87 of file lexer.h.

◆ accept() [4/4]

template<size_t K, class S >
template<Append append = Append::On, class Pred >
bool fe::Lexer< K, S >::accept ( Pred  pred)
inlineprotected
Returns
true if pred holds. In this case invoke Lexer::next() and append to Lexer::str_, if append.

Definition at line 71 of file lexer.h.

References fe::Lexer< K, S >::ahead(), fe::Lexer< K, S >::Lower, fe::Lexer< K, S >::Off, fe::Lexer< K, S >::str_, fe::utf8::tolower(), fe::utf8::toupper(), and fe::Lexer< K, S >::Upper.

◆ ahead()

template<size_t K, class S >
char32_t fe::Lexer< K, S >::ahead ( size_t  i = 0) const
inlineprotected

Definition at line 30 of file lexer.h.

References fe::Lexer< K, S >::ahead_.

Referenced by fe::Lexer< K, S >::accept().

◆ next()

template<size_t K, class S >
char32_t fe::Lexer< K, S >::next ( )
inlineprotected

Get next char32_t in Lexer::istream_ and increase Lexer::loc_.

Returns
Null on an invalid UTF-8 sequence.

Definition at line 40 of file lexer.h.

References fe::Lexer< K, S >::ahead_, fe::utf8::BOM, fe::Pos::col, fe::utf8::decode(), fe::utf8::EoF, fe::Loc::finis, fe::Ring< T, N >::front(), fe::Lexer< K, S >::istream_, fe::Lexer< K, S >::loc_, fe::Lexer< K, S >::peek_, fe::Ring< T, N >::put(), and fe::Pos::row.

◆ start()

template<size_t K, class S >
void fe::Lexer< K, S >::start ( )
inlineprotected

Invoke before assembling the next token.

Definition at line 33 of file lexer.h.

References fe::Loc::begin, fe::Lexer< K, S >::loc_, fe::Lexer< K, S >::peek_, and fe::Lexer< K, S >::str_.

Member Data Documentation

◆ ahead_

template<size_t K, class S >
Ring<char32_t, K> fe::Lexer< K, S >::ahead_
protected

Definition at line 92 of file lexer.h.

Referenced by fe::Lexer< K, S >::ahead(), and fe::Lexer< K, S >::next().

◆ istream_

template<size_t K, class S >
std::istream& fe::Lexer< K, S >::istream_
protected

Definition at line 91 of file lexer.h.

Referenced by fe::Lexer< K, S >::next().

◆ loc_

template<size_t K, class S >
Loc fe::Lexer< K, S >::loc_
protected

Location of the token we are currently constructing within Lexer::str_,.

Definition at line 93 of file lexer.h.

Referenced by fe::Lexer< K, S >::next(), and fe::Lexer< K, S >::start().

◆ peek_

template<size_t K, class S >
Pos fe::Lexer< K, S >::peek_
protected

Position of ahead_::first;.

Definition at line 94 of file lexer.h.

Referenced by fe::Lexer< K, S >::next(), and fe::Lexer< K, S >::start().

◆ str_

template<size_t K, class S >
std::string fe::Lexer< K, S >::str_
protected

Definition at line 95 of file lexer.h.

Referenced by fe::Lexer< K, S >::accept(), and fe::Lexer< K, S >::start().

The documentation for this class was generated from the following file: