Colab notebook : https://colab.research.google.com/drive/1ptASBrEgGXJPjBrZHd040qsWXFnPYYoC?usp=sharing

This we are doing in the GGUF model

Grammar

So as in english grammar tell whether the sentence is correct or not ..

I am eating = eating am I = this is incorrect as the grammar is not met

Similarly all coding languages have a grammar that they use to check syntax errors / parsing .. like python : https://docs.python.org/3/reference/grammar.html

BNF

Its a form of writting that grammar

<expr>::=<term><expr>"+"<term>

:= means ‘is defined as’

so this is how lexer in compiler breaks it up in tokens and then verifies its order

CFG ( context free grammar )

This means the value / expression on the left side does not depend on anything :

ETxeprrmE"x1p"r|+"T2e"rm|Term

Top line is read as expression equals to (expression + term) or (term) alone Here I am saying expr is independent of whatever comes on the right side

so this can become ‘1+2+1’

Example of context-dependent grammar would be aA → ab this A can only be replaced by b only

Json / response-format mode

In LLM’s we dont force on next token generation so that they can freely write it out … but to get a particular json output from a model we can enforce this CFG to it using : https://github.com/ggerganov/llama.cpp/tree/master/grammars

So we this format called as GBNF ( is a grammar format only that’s required by ggml models ) and for we can create an GBNF for any pydantic class that we want using https://grammar.intrinsiclabs.ai/ , just convert your python schema to typescript once

More details in this page : https://til.simonwillison.net/llms/llama-cpp-python-grammars

Learning to write GBNF (as no LLM / online editor is helping)

So everything start with root::= that is defines the root architecture and in this we can define all variables that will help in this and every expression begins with a inverted-commas "<single-expression>" , and each expression is in itself a wrapped in inverted commas

1st example

rstttooeeeorrrttmmmh--:e123:s=e:::"a"s"{r{t}er"\\i"3"nssgtetaxatptturuoses\s\se"s"tnird:o:innw"s"gitsbahtrsre}iadnkteg-hfdii"osn}we"ntdealslsmodeltobeginwiththis{"status":

String is defined as:

So lets do this step wise:

First I am defining the string variable name and initializing it empty with inverted commas

string::=""

Second, add internal value to it

string::="\"active\"|\"inactive\""

this means it can be only a literal value Its comparison in pydantic class is

class JSONSchema(BaseModel):
  status : Literal['active', 'passive']

2nd Example (

rather than literal it can be any string

string::="\""([ˆ"]*)"\""

so this is also made up of 3 things part-1 : “"” –> should start with single inverted comma " part-2 : ([ˆ"]*) –> here we can write any string value part-3 : “"” –> should end with single inverted comma "

3rd example (list)

string-tlhiisstm:e:a=nsstwreincgan("r,e"pesattritnhgi)s*bundleof(,"value")0ormultipletimes

Possible solutions for errors :

  1. You are running into segmentation fault, cause the buffer-size for parser stack is reached and you are now accessing out of allocated memory (increase the parser buffer size)
  2. You are not using raw-text string , always use raw string when passing

Best is to use vLLM as it also has support for grammar