[Feature]后续会支持llava等多模态模型的训练吗

[Vincent131499]

Reminder

• I have read the README and searched the existing issues.

Reproduction

如题

Expected behavior

No response

System Info

No response

Others

No response

[Katehuuh]

They are more state-of-the-art multimodal and diverse like video, sound, 3D, all-in-one… but I’m interested in using the LLaVA-1.5-LoRA because of oobabooga UI compatibility 4bit.

[Katehuuh]

Are multiple images supported in same conversation? as we currently have

LLaMA-Factory/data/mllm_demo.json

Lines 47 to 54 in 031775a

	},
	{
	"messages": [
	{
	"content": "Please describe this image<image>",
	"role": "user"
	},
	{

As referred to

LLaMA-Factory/data/mllm_demo.json

Lines 65 to 71 in 031775a

	}
	],
	"images": [
	"images/3.jpg"
	]
	}
	]

For example, if we have in the same conversation "content": "Is this the same person?<image>[2]"", and set it:

   "images": [ 
     "images/3.jpg",
     "images/4.jpg" 
   ] 

Edit1: it seems like no:

LLaMA-Factory/src/llmtuner/data/preprocess.py

Lines 29 to 33 in 8e09e20

	def _preprocess_visual_inputs(images: Sequence["ImageObject"], processor: "ProcessorMixin") -> "NDArray":
	# process visual inputs (currently only supports a single image)
	image_processor: "BaseImageProcessor" = getattr(processor, "image_processor")
	image = images[0] if len(images) != 0 else Image.new("RGB", (100, 100), (255, 255, 255))
	return image_processor(image, return_tensors="pt")["pixel_values"][0]

---

• From arg:
  

  LLaMA-Factory/examples/lora_single_gpu/sft_mllm.sh

  Line 6 in 031775a

  --model_name_or_path llava-hf/llava-1.5-7b-hf \

hiyouga: llava + qlora, it now requires ~5GB to fine-tune llava1.5 -7b

It seems like it cannot be train of the base Llama-2/3 however, we have liuhaotian/llava-v1.5-13b-lora that can be applied to Llama-2.

Ignore this line as it is derived but I've seen llava-Phi-3/Llama-3 from here: InternLM/xtuner.
No LoRa for llava-v1.6 but unsure if v1.6 is supported?