How do you evaluate multi-modal LLMs that handle both text and image inputs?

Evaluating multi-modal LLMs involves a combination of text and image evaluation metrics. For text, traditional NLP metrics like BLEU or ROUGE may be used if the model generates text. For the image component, metrics such as FID (Fréchet Inception Distance) or IS (Inception Score) can measure how well generated images match training data. Moreover, human judgment may be integrated by asking annotators to assess the coherence between text and image outputs. It's crucial to ensure that evaluation covers effectivity in both modalities.