Multimodal hierarchical classification using cascade-of-thought.

Abstract

We propose Cascade-of-Thought (CSOT), a novel prompt-based method for multimodal hierarchical classification (MHC) that requires no training or labeled exemplars. Inspired by the LLM-as-a-Judge (LaaJ) paradigm, CSOT decomposes classification into rationale generation, confidence scoring, and decision ranking—each implemented via structured prompts to a vision–language model (VLM). Experiments on two public MHC benchmarks demonstrate that CSOT yields substantial performance gains, particularly for weaker VLMs, while also enhancing the output quality of near-ceiling models. CSOT offers a flexible, generalizable solution for real-world MHC tasks.