Understanding symptom-image associations is crucial for clinical reasoning. However, existing medical multimodal models often rely on simple one-to-one hard labeling, oversimplifying clinical reality where symptoms relate to multiple organs. In addition, they mainly use single-slice 2D features without incorporating 3D information, limiting their ability to capture full anatomical context. In this study, we propose Med-SORA, a framework for symptom-to-organ reasoning in abdominal CT images. Med-SORA introduces RAG-based dataset construction, soft labeling with learnable organ anchors to capture one-to-many symptom–organ relationships, and a 2D–3D cross-attention architecture to fuse local and global image features. To our knowledge, this is the first work to address symptom-to-organ reasoning in medical multimodal learning. Experimental results show that Med-SORA outperforms existing medical multimodal models and enables accurate 3D clinical reasoning, serving as a cornerstone for future research in interpretable and clinically grounded multimodal reasoning.
The pipeline of Med-SORA, showing the optimization process for $o_1=\text{liver}$. It consists of (a) RAG-based dataset construction, (b) soft labeling-based text embedding learning, and (c) 3D-2D feature-based image embedding learning with text-image embedding alignment.
A symptom text is often related to multiple organs rather than just one.
For example, feeling uncomfortable after eating is intuitively associated with the stomach, but may also involve other organs, such as the pancreas and gallbladder, which contribute to the digestive process.
Therefore, we propose a learnable anchor-based soft labeling to model complex symptom–organ associations as shown in Figure 3.
We propose a feature extraction framework that jointly captures both 2D and 3D representations from abdominal CT scans. To enhance organ-level understanding, we extract slice-wise features using a 2D encoder and volume-level context using a 3D encoder, and integrate them through a cross-attention mechanism (Figure 4).
To verify that complex symptom-organ relationships are effectively learned, we analyze the symptom text embedding space trained with soft labeling.
Figure 8 shows a t-SNE visualization of text embeddings learned with soft labeling.
Unlike hard labeling, which creates completely separated clusters for each class~(Figure 9), soft labeling forms flexible clusters that reflect associations between symptoms and multiple organs.
Table 2 further shows that LGN performs best with three neighboring slices. Using too few neighbors (0 or 1) provides limited contextual benefit, whereas using too many neighbors (5 or all) can dilute useful context and degrade refinement performance.
We evaluate the effectiveness of soft labeling in a text-to-image matching task, where a symptom text is used to retrieve related organs from abdominal CT images.
Table 6 compares the retrieval performance of hard and soft labeling approaches.
Soft labeling consistently outperforms hard labeling across all metrics, with larger performance gaps in Rank-2 and Rank-3.
To evaluate the effectiveness of fusing 2D slice-level details with 3D volume-level global information, we compare various image feature fusion methods. Table 10 summarizes the performance comparison of different 2D-3D feature fusion approaches.
We evaluate the performance of the proposed Med-SORA against existing methods on the symptom text-to-medical image reasoning task, as shown in Tab. 11.
Figure 11 illustrates the reasoning results of Med-SORA when given symptom text queries.
The model infers the most relevant organ and highlights its 3D segmentation based on the symptom description.
As shown in Fig. 11(a), when a symptom query is related to multiple organs, the model assigns probabilities to multiple relevant organs.
In contrast, symptoms that are strongly associated with a specific organ, as in Fig. 11(b), result in a distinctly high probability for that organ.
When irrelevant symptom queries such as "knee joint pain" or "headache" are given as input, as shown in Fig. 11(c), all abdominal organs receive uniformly low probability scores.
These results demonstrate that Med-SORA effectively handles both single-organ and multi-organ symptom reasoning, while remaining robust to out-of-domain queries without producing false positives.
