Automated MSME Bookkeeping Using Vision-Language Models and Hierarchical Multi-Agent Systems

Abstract

We present a bookkeeping automation system for micro, small, and medium enterprises (MSMEs) that converts receipt photographs into structured ledger entries without manual data transcription. Receipt-based automation is difficult because document layouts vary widely across retailers, image quality in real-world capture conditions is unreliable, and writing extracted data into a spreadsheet accounting system requires coordinated multi-step execution that a single model cannot handle end to end. The system fine-tunes Qwen 3.5-4B with Low-Rank Adaptation (LoRA) on a 1,000-image annotated receipt corpus and couples the extraction model to a LangGraph-based hierarchical multi-agent architecture whose agents access SQLite and Google Sheets through standardized Model Context Protocol (MCP) tool endpoints. When evaluated on 100 unseen test images, the fine-tuned model, configured at LoRA rank 32 and epoch 3, yields a 94.58% Digit Accuracy and an ANLS* score of 93.99 while maintaining 100% JSON validity, representing a 43.71-point and 17.09-point improvement in KIEVal Group F1 and KIEVal Entity F1, respectively, over the untuned Qwen 3.5-4B baseline and outperforming all four zero-shot baselines across every metric. Beyond individual model capabilities, the multi-agent framework, served through vLLM and exposed via a cross-platform mobile client, successfully satisfies all 22 behavioral test cases covering single-agent tool routing, cross-agent orchestration, human-in-the-loop confirmation, and input-output guardrail enforcement, demonstrating a deployable end-to-end pipeline from receipt capture to ledger update.