What is G-Eval, and how does it use LLMs for evaluation?

G-Eval is a framework for more reliable LLM-as-a-judge scoring. Instead of just asking a model to output a number, it first has the model generate the evaluation steps (a chain-of-thought reasoning process from the criteria), then scores against those steps. It also produces a finer, probability-weighted score by considering the distribution over rating tokens rather than a single sampled number, which reduces the coarse, clumpy scores you get from naive prompting. The result correlates better with human judgement than plain prompting on tasks like summarisation quality. It still inherits the general limits of model-based evaluation — judge bias and domain blind spots — so you calibrate it against human labels rather than trusting it blindly.