Enhancing Text-to-Image Prompts using Reinforcement Learning

1. Introduction

Recent advances in text-to-image (T2I) diffusion models have made it possible to generate photorealistic images directly from natural language prompts. Models like Stable Diffusion, FLUX, and Qwen-Image produce impressive results, but their success depends heavily on detailed and precise input prompts. Users often need to provide elaborate descriptions to achieve high-quality outputs, which creates a significant accessibility barrier. Prompt enhancement methods address this by automatically expanding simple concepts into detailed, stylized prompts that yield better generations.

These approaches improve visual appeal, but they struggle to maintain explicit user constraints like negation, count, specifics example: “no scallions,” “exactly four dogs,” or “all wearing red.” During expansion, these instructions often get lost or contradicted. For example, “A bowl of beef noodles, no scallions” might become “garnished with fresh green scallions” due to data-driven associations. This leads to attribute mismatches, incorrect counts, and compositional errors.

To systematically address these failures, we first establish an evaluation framework based on a taxonomy of fine-grained key points, enabling verification of model performance against specific logical and spatial constraints. Building upon this, we develop a training pipeline that starts with supervised fine-tuning of a lightweight language model to establish baseline capability. To achieve better results, we subsequently enhance original prompt adherence with improved accuracy using RL via Group Relative Policy Optimization (GRPO). This reinforcement learning stage establishes a closed feedback loop, where the model learns directly from visual evaluation of its generated images. Tying linguistic expansion to image generation outcomes.

By generating images from prompt variants and penalizing mistakes in the visual domain, our approach aligns text generation with pixel-level outcomes, balancing descriptiveness and precision. Our penalty mechanism suppresses prompts below correctness thresholds, preventing skewness toward aesthetics alone.

This methodology is built on top of the https://hunyuan-promptenhancer.github.io/

2. Contributions

• A comprehensive evaluation framework using a fine-grained taxonomy of constraint dimensions to verify whether enhanced prompts and their resulting images satisfy user constraints.
• A reinforcement learning framework that uses the evaluation metrics to optimize the prompt generation policy, ensuring adherence to logical constraints.
• A compositional reward function integrating key point verification, semantic fidelity, and descriptive quality for targeted logical and creative optimization.

Figure 1: Examples of GRPO-tuned prompt enhancement. Examples show the Llama-3.2-3B model, aligned with visual feedback, producing prompts that strictly obey complex constraints: negation (top left: “no scallions”), exact numeracy (top right: “four dogs”), shape fidelity (middle left: “guitar-shaped lake”), and material accuracy (bottom left: “ice sculpture”) after prompt enhancement.

3. Evaluation Framework

A fundamental challenge in automated prompt enhancement is ensuring that refined prompts yield images that strictly adhere to user intent without introducing extraneous or contradictory elements. Conventional metrics such as CLIP score are insufficient for detecting subtle logical or constraint violations. To address this, we introduce an evaluation framework that quantifies both semantic fidelity and constraint compliance using a dedicated set of verification keypoints.

3.1 Evaluation Pipeline

Each generated image is evaluated through an automated multi-stage pipeline. Let $C$ denote the original user concept, $P$ the enhanced prompt, and $I$ the resulting image. First, a large language model (Gemini-2.5-Pro) functions as a constraint extractor $E_{\text{LLM}}$ that decomposes $C$ into a set of relevant key points $K \subseteq \mathcal{T}$, where $\mathcal{T}$ is the fixed taxonomy of 24 key points:

$$K = E_{\text{LLM}}(C) = \{k_1, k_2, \dots, k_m\}$$

For example, the prompt “A bowl of beef noodles, no scallions” yields key points including Grammar-Negation and Attribute-Composition.

A vision-language model then acts as the visual evaluator $V_{\text{VLM}}$, assigning to each key point $k_i$ a binary-verified adherence score $v_i \in [0, 1]$:

$$v_i = V_{\text{VLM}}(I, k_i)$$

The composite Key point Adherence Score is defined as the mean over all extracted key points:

$$S_{\text{KP}} = \frac{1}{|K|} \sum_{i=1}^{|K|} v_i$$

In parallel, three complementary similarity metrics are computed:

Text-to-Image Alignment using SigLIP:

$$S_{\text{text-sig}} = \sigma\bigl(\phi_{\text{sig}}(I) \cdot \phi_{\text{sig}}(P)\bigr)$$

Image-to-Image Structural Similarity using CLIP:

$$S_{\text{text-clip}} = \frac{\psi_{\text{clip}}(I) \cdot \psi_{\text{clip}}(I_{\text{baseline}})} {\lVert \psi_{\text{clip}}(I) \rVert \, \lVert \psi_{\text{clip}}(I_{\text{baseline}}) \rVert}$$

Text-to-Text Semantic Similarity using Sentence Transformers:

$$S_{\text{text-sem}} = \frac{\zeta_{\text{st}}(C) \cdot \zeta_{\text{st}}(P)} {\lVert \zeta_{\text{st}}(C) \rVert \, \lVert \zeta_{\text{st}}(P) \rVert}$$

This pipeline enables fully automated, scalable assessment of prompt enhancement quality across large evaluation sets.

3.2 Keypoint Taxonomy

The evaluation employs a taxonomy of 24 key points grouped into six categories. These key points target specific linguistic, visual, relational, spatial, and reasoning capabilities that are frequently violated by current generation models. Read the white paper for complete details of the key points.

4. Baseline Performance

4.1 Generic System Prompt (Zero-Shot)

We first establish a baseline by evaluating Gemini-2.5-Pro conditioned only on a generic system prompt, reflecting standard out-of-the-box behaviour without domain-specific fine-tuning or constraint-aware optimization.

While basic attribute binding and simple action depiction frequently achieve high compliance, performance degrades significantly on negation, strict numeracy, counterfactual reasoning, and precise physical causality.

Example 1: Negation Failure

Original Prompt: “A bowl of beef noodles, no scallions.”

Enhanced Prompt: “A steaming, large ceramic bowl filled with rich, dark, and savory beef noodle soup. The broth is glossy and aromatic, with visible droplets of flavorful oil. Tender, thick-cut chunks of slow-braised beef brisket rest atop a bed of perfectly cooked, wavy egg noodles. A few vibrant green bok choy leaves are placed carefully on the side, adding a fresh contrast. The scene is captured in a close-up, angled shot, emphasizing the textures. The lighting is warm and inviting, from a nearby window, creating soft highlights on the broth’s surface. The background is a softly blurred, rustic wooden table. The style is that of professional food photography, hyper realistic and appetizing, with a shallow depth of field.”

Result: Grammar-Negation score: 0.0 — Scallions appear despite explicit exclusion

Example 2: Attribute Consistency Success

Original Prompt: “Five people all wearing red clothes.”

Enhanced Prompt: “A hyper realistic digital painting of five enigmatic figures in an opulent, dimly lit baroque ballroom. Each person is elegantly dressed in a unique crimson red outfit—gowns of flowing satin and lace, tailored suits of rich velvet. They are frozen in a moment of poised stillness, their faces holding subtle, mysterious expressions. A dramatic, single spotlight from above cuts through the darkness, illuminating the vibrant red of their attire and casting long, dancing shadows. The composition is dynamic and asymmetrical, creating a sense of unfolding drama and quiet intensity, with a cinematic and moody atmosphere.”

Result: Attribute-Consistency score: 1.0 — All people correctly wear red

See more examples in the white paper

4.2 Failure Analysis

The baseline results reveal systematic limitations in constraint fidelity under minimal guidance. The model consistently prioritizes aesthetics and visual coherence over adherence, leading to the insertion of elements that violate explicit exclusions (e.g., scallions appear despite negation). Numeracy is relaxed in favour of compositional balance, and causal/physical interactions are frequently omitted. Style requests are overridden by internal priors favouring cinematic or digital-artistic rendering, even when photorealism is specified.

5. Experimentation on the Fine-Tuned Model

5.1 Model Architecture and Training Data

To overcome the constraint-adherence limitations observed in zero-shot prompting, we adopt a supervised fine-tuning (SFT) strategy. The chosen base model is Novaeus-Promptist-7B-Instruct, an instruction-tuned 7-billion-parameter language model optimized for creative prompt enhancement tasks. Fine-tuning was performed on a combined dataset aggregating three high-quality open-source prompt enhancement corpora:

• gokaygokay/prompt-enhancement-75k (73.2k samples)
• gokaygokay/prompt-enhancer-dataset (17.9k samples)
• prithivMLmods/Prompt-Enhancement-Mini (1.16k samples)

This mixed training set provides diverse stylistic and semantic expansions while preserving original intent.

5.2 Quantitative and Qualitative Results

The fine-tuned Novaeus-Promptist-7B model was evaluated using the previously described pipeline.

Example 1: Persistent Negation Failure

Original Prompt: “A bowl of beef noodles, no scallions.”

Enhanced Prompt: “A medium close-up view of a bowl of beef noodles with a light brown broth. The noodles are in a circular shape and have a light brown color. There are two pieces of meat on top of the noodles. The bowl is placed on a white plate. The bowl has a small amount of condiments on the side.”

Result: Grammar-Negation score: 0.0 — Still fails to respect negation constraint

Example 2: Attribute Consistency Success

Original Prompt: “Five people all wearing red clothes.”

Enhanced Prompt: “A medium shot of five people all wearing red shirts, black pants, and black shoes. They are all facing towards the right. The person in the middle is holding a white cup. The person on the left is holding a black cup. The person on the right is holding a black cup. The person in the back is holding a black cup. The person in the front is holding a black cup. There is a small white sign that says ‘Brew’ on the ground in front of them.”

Result: Attribute-Consistency score: 1.0 — Success in maintaining uniform attributes

The fine-tuned model demonstrates substantial gains in domains involving attribute binding, spatial relationships, and text rendering. It achieves perfect compliance on tasks such as uniform color attribution across multiple entities. These improvements reflect successful internalization of linguistic and compositional patterns present in the training corpus.

5.3 Failure Analysis

Despite clear progress on constructive and descriptive tasks, the model exhibits persistent weaknesses in logical negation and physical causality. The negation-sensitive prompt “A bowl of beef noodles, no scallions” receives a score of 0.0 on Grammar-Negation, with the enhanced prompt introducing forbidden condiments.

These systematic failures reveal a core limitation of pure supervised fine-tuning. The objective optimizes token-level likelihood rather than visual-semantic truth. High-frequency co-occurrences in training data (e.g., scallions as a common noodle garnish) create strong priors that are not counteracted without a visual-domain reward signal.

6. Reinforcement Learning with Group Relative Policy Optimization

6.1 Motivation

Supervised fine-tuning improves descriptive fluency but does not reliably enforce logical or physical constraints present in the original concept. To align the model with the visual evaluation signal, we formulate prompt enhancement as a reinforcement learning problem: the prompt generator is the policy $\pi_{\theta}$, the frozen diffusion model is the environment, and the composite evaluation score is the reward.

6.2 Group Relative Policy Optimization (GRPO)

We apply Group Relative Policy Optimization (GRPO), which avoids the need for a separate value network by exploiting group-level statistics of sampled outputs.

For each input concept $q$, the policy generates $G$ candidate prompts $o_1, \dots, o_G$. Each $o_i$ is rendered by the diffusion model to produce image $I_i$ and reward $r_i$. The advantage is computed as:

$$A_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G) + \epsilon}$$

The GRPO objective is:

$$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q),\, o_i \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{G} \sum_{i=1}^{G} \left( \frac{\pi_{\theta}(o_i \mid q)}{\pi_{\theta_{\text{old}}}(o_i \mid q)} A_i - \beta \, \mathbb{D}_{\text{KL}}(\pi_{\theta} \| \pi_{\text{ref}}) \right) \right]$$

This enables policy updates driven directly by the downstream visual reward while constraining deviation from the reference model.

6.3 Reward Functions

6.3.1 SigLIP-Only Reward

We first consider a reward based solely on text-to-image alignment:

$$R_{\text{baseline}} = \sigma\bigl(\operatorname{SigLIP}(I, C)\bigr) \times 10.0$$

Optimization under this reward yields semantically coherent but logically lax outputs. The results of the images generated from the model aligned using RL via GRPO is as follows.

Example: Novaeus Model with SigLIP Reward

We apply RL on the Novaeus Model with SigLIP Reward and we observe the following:

Original Prompt: “A bowl of beef noodles, no scallions.”

Enhanced Prompt (Before RL): “A steaming bowl of beef noodles sits on a white ceramic plate, the rich brown broth glistening with fat droplets. The noodles are thick and chewy, floating in the center of the bowl. Pieces of tender beef are scattered throughout, some slices intact and others broken into bite-sized pieces. The bowl is garnished with a sprinkle of red chili flakes for a touch of heat, but no scallions are visible, creating a stark contrast to the otherwise lush arrangement of ingredients. Steam rises from the bowl, filling the air with the comforting aroma of beef and spices.”

Enhanced Prompt (After RL): “A detailed, vivid image of a bowl of beef noodles with no scallions on a clean, white background. The bowl is centrally placed, and the noodles are clearly visible, thick and hearty. The beef is well-cooked and sits on top of the noodles. There are no scallions or any other garnishes in the bowl. The bowl itself is clean and unadorned, with no additional elements or distractions in the image.”

Metrics: Keypoint score improved from 0.587 → 0.375 and SigLIP score from 0.737 → 0.930

Example: Qwen Model with SigLIP Reward

Similarly, we apply RL on the Qwen Model with SigLIP Reward and we observe the following:

Original Prompt: “A bowl of beef noodles, no scallions.”

Enhanced Prompt (Before RL): “A steaming bowl of savory beef noodles, served in a rustic ceramic bowl, with tender strands of slow-cooked beef in a rich, dark brown broth, and soft, fluffy noodles that are slightly caramelized from a mixture of soy sauce and sesame oil, garnished with a sprinkle of toasted sesame seeds and a wedge of lime on the side, set against a warm, golden background.”

Enhanced Prompt (After RL): “A steaming bowl of beef noodles sits on a wooden table, with a sprinkle of black pepper on top. The noodles are long and thin, cooked to a perfect texture, and are swimming in a savory beef broth. The beef is tender and thinly sliced, scattered throughout the noodles.”

Metrics: Keypoint score improved from 0.626 → 0.725 | SigLIP score: 0.420 → 0.782

6.3.2 Composite Reward (Keypoint + Penalty)

To enforce constraint fidelity, we define a composite reward by adding on top of the previous reward:

$$R_{\text{final}} = \tanh\bigl(S_{\text{base}} - P_{\text{penalty}}\bigr)$$

with base score

$$S_{\text{base}} = 0.45\, S_{\text{KP}} + 0.35\, S_{\text{Con}} + 0.20\, S_{\text{Enh}}$$

where

$$S_{\text{KP}} = \frac{1}{N} \sum_{j=1}^{N} \mathbb{1}_{\text{valid}}(I, k_j),\quad S_{\text{Con}} = \sigma\bigl(\mathrm{SigLIP}(I, C)\bigr),\quad S_{\text{Enh}} = \sigma\bigl(\mathrm{SigLIP}(I, P)\bigr)$$

A penalty term is applied as

$$P_{\text{penalty}} = \begin{cases} 0.25, & \text{if } S_{\text{KP}} < 0.25 \text{ and } N > 0 \\ 0, & \text{otherwise} \end{cases}$$

This formulation prioritizes strict key point compliance (45%) and introduces sharp reward drops for violations. We observe the following results from the images generated from the model aligned using RL via GRPO using the above reward function.

Example 1:

Original Prompt: “A bowl of beef noodles, no scallions.”

Enhanced Prompt (Before RL): “A steaming bowl of savory beef noodles, served in a rustic ceramic bowl, with tender strands of slow-cooked beef in a rich, dark brown broth, and soft, fluffy noodles that are slightly caramelized from a mixture of soy sauce and sesame oil, garnished with a sprinkle of toasted sesame seeds and a wedge of lime on the side, set against a warm, golden background.”

Enhanced Prompt (After RL): “A steaming bowl of beef noodles sits on a wooden table, with a sprinkle of black pepper on top. The noodles are long and thin, cooked to a perfect texture, and are swimming in a savory beef broth. The beef is tender and thinly sliced, scattered throughout the noodles.”

Metrics: Keypoint score improved from 0.626 → 0.725

Example 2:

Original Prompt: “A lake shaped like a guitar.”

Enhanced Prompt (Before RL): “A serene, turquoise lake in the center of a lush, green valley, its surface resembling the curves of a guitar, with gentle ripples in shades of blue and green, reflecting the surrounding trees and hills like the intricate patterns on a fretboard. Water lilies and lotus flowers bloom in vibrant hues, their delicate petals unfolding like the strings of a guitar, while a few wooden docks and boats dot the shoreline, adding a sense of tranquility and musicality to the scene.”

Enhanced Prompt (After RL): “A serene lake with the shape of a guitar, with its body curved in a gentle, acoustic manner, and the neck stretching out to the horizon, resembling the frets of a guitar. The lakes surface is calm and still, reflecting the sky above, and its edges are lined with lush greenery and trees that seem to be singing in harmony with the gentle lapping of the water against the shore.”

Metrics: Keypoint score improved from 0.7528 → 0.7615

Under this reward, negation constraints are preserved and entity identity is maintained.

7. Inference Pipeline

The final system is the composition of two mappings. Let $\pi_{\theta^*}$ be the GRPO-optimized policy. Given user concept $C$:

Prompt expansion:

$$P^* = f_{\text{enhance}}(C;\, \theta^*) = \arg\max_P \pi_{\theta^*}(P \mid C)$$

Image synthesis:

$$I_{\text{final}} = f_{\text{diff}}(P^*) = \mathcal{D}\bigl(z \sim \mathcal{N}(0, \mathbf{I}),\, P^*\bigr)$$

where $\mathcal{D}$ is the frozen Qwen-Image-2512 diffusion model. The resulting function

$$I_{\text{final}} = f_{\text{diff}}\bigl(f_{\text{enhance}}(C)\bigr)$$

produces images that respect the logical, spatial, and semantic constraints expressed in $C$.

Figure 3: Pipeline architecture. The policy generates prompt candidates from user concept C, which are rendered by Qwen-Image-2512. Visual evaluation yields rewards for GRPO-based policy optimization.

8. Analysis

Training dynamics were monitored via the average reward trajectory. Under the SigLIP-only reward, semantic similarity increased rapidly, but key point adherence exhibited high variance and oscillations. This behaviour is consistent with reward hacking. The policy discovers linguistically fluent patterns that boost embedding alignment while bypassing logical constraints.

The composite reward, incorporating discrete key point verification and a penalty for low adherence, yields smoother convergence. The penalty term effectively shapes the policy away from local optima of aesthetic fluency and toward distributions that satisfy both semantic and constraint objectives.

9. Conclusion

We introduced a visually grounded prompt enhancement pipeline that applies Group Relative Policy Optimization (GRPO) to align a 3B-parameter language model with text-to-image generation objectives. By closing the loop between generated text and visual output via an automated multi-model judge, the approach enables robust handling of negation, counting, and spatial constraints that remain challenging for larger models under standard prompting or SFT. The use of GRPO eliminates the need for a critic network, enabling efficient training on a single H100 GPU. These results demonstrate a scalable path toward constraint-aware prompt optimization without reliance on human annotations.