Image Captioning in Vietnamese

The Challenge

Image captioning—automatically generating natural language descriptions for images—has achieved significant progress in English. However, for low-resource languages like Vietnamese, the lack of labeled datasets poses a major hurdle. This project aimed to bridge that gap by constructing a dataset and exploring diverse methodologies to adapt English-centric models to Vietnamese.

Dataset Construction

To create a benchmark for Vietnamese image captioning, I employed a semi-automated approach to translate a subset of the MS-COCO dataset.

• Source: 4,000 images from the MS-COCO training set.
• Volume: 20,000 image-caption pairs (5 captions per image).
• Process:
  • Machine Translation: Used Google Translate for initial candidates.
  • Human Translation: Manually refined translations to ensure cultural nuances and grammatical correctness were preserved, correcting errors where the machine translation conflicted with the visual content.

Methodology

The core architecture utilized the Show and Tell model (CNN + LSTM). I experimented with four distinct strategies to determine the most effective approach for Vietnamese:

1. Translate-First: Generating English captions and translating them to Vietnamese (CIDEr: 0.827).
2. Machine-Translated Training: Training directly on Google-translated captions (CIDEr: 0.935).
3. Human-Translated Training: Training on manually quality-controlled captions (CIDEr: 1.095).
4. Tokenized Training: Applying Vietnamese word segmentation (using RDRsegmenter) to handle the space-delimiter ambiguity in Vietnamese before training.

Results

The experiments demonstrated that data quality and language-specific preprocessing are crucial.

• Impact of Human Translation: Training on high-quality, human-translated data significantly outperformed machine-translated baselines.
• Impact of Tokenization: Vietnamese words often consist of multiple space-separated syllables. Treats them as single tokens (e.g., "trưởng thành" vs "trưởng", "thành") improved the CIDEr score to 1.148, the best result in the study.

Conclusion

This thesis established a baseline for Vietnamese image captioning, showing that while transfer learning from English is viable, adapting to the specific linguistic characteristics of Vietnamese (like word segmentation) yield the best performance.