Suggest architecture/pipeline for general object detection + VLM call afterwards

Hi, I am looking for the following model selection/inference pipeline. Goal is something like this: 1) detect human -> describe human 2) detect human -> detect objects in human hand -> describe objects 3) detect animal -> get specific animal type 4) detect general object (i.e package) --- What are the suggested architecture/pipelines for this? So obviously I can use yolov26 or another baseline yolo model, but I am trying to avoid having to annotate everything from the ground up. I do not mind fine-tuning, I just want to move past coco classes into something a little more broad. My current idea