elem2design

Model Details#

Model Description#

This model aims to compose user-provided graphic elements into a pleasing graphical design. It takes graphic elements (i.e., the images and texts) from users as input and generates the position, color and font information of each element as output.

• Developed by: Jiawei Lin, Shizhao Sun, Danqing Huang, Ting Liu, Ji Li and Jiang Bian
• Model type: Large Language Models
• Language(s): Python
• License: MIT
• Finetuned from model: Llama-3.1-8B

Model Sources#

• Repository: https://github.com/microsoft/elem2design
• Paper: https://arxiv.org/abs/2412.19712

Uses#

Direct Use#

Compose user-provided graphic elements (i.e., images and texts) into a pleasing graphic design.

Elem2Design is being shared with the research community to facilitate reproduction of our results and foster further research in this area.

Out-of-Scope Use#

We do not recommend using Elem2Design in commercial or real-world applications without further testing and development. It is being released for research purposes.

Use in any manner that violates applicable laws or regulations.

Risks and Limitations#

Elem2Design inherits any biases, errors, or omissions produced by its base model. Developers are advised to choose an appropriate base LLM/MLLM carefully, depending on the intended use case. 

Elem2Design uses the Llama model. See https://huggingface.co/meta-llama/Llama-3.1-8B to understand the capabilities and limitations of this model.  

As the model is fine-tuned on very specific data about design composition, it is unlikely to generate information other than position, color and font. However, this is possible. It is more likely to happen when instructions unrelated to graphic design composition, e.g., how has the social media influenced our daily life, are fed into the model.

Graphic designs generated by Elem2Design may not be technically accurate or meet user specifications in all cases. Users are responsible for assessing the acceptability of generated content for each intended use case.

Elem2Design was developed for research and experimental purposes. Further testing and validation are needed before considering its application in commercial or real-world scenarios.

Recommendations#

Please only provide the images and texts that you want to show on the graphic design to the model.

Users are responsible for sourcing their content legally and ethically. This could include securing appropriate copy rights, ensuring consent for use of images of people, and/or the anonymization of data prior to use in research.  

How to Get Started with the Model#

python llava/infer/infer.py \
      --model_name_or_path /path/to/model/checkpoint-xxxx \
      --data_path /path/to/data/test.json \
      --image_folder /path/to/crello_images \
      --output_dir /path/to/output_dir \
      --start_layer_index 0 \ 
      --end_layer_index 4

For more information, please visit our GitHub repo: https://github.com/microsoft/elem2design.

Training Details#

Training Data#

The training data is from an open-source dataset (https://huggingface.co/datasets/cyberagent/crello).

Training Procedure#

Preprocessing#

The training samples with more than 25 design elements are filtered out to maintain a limited sequence length and thereby improve training efficiency.

Training Hyperparameters#

• Learning rate: 2e-4
• Global batch size: 128
• Number of training steps: 7000
• Rank and alpha of LoRA: 32 and 64

Speeds, Sizes, Times#

• Llama-3.1-8B: 8B parameters
• CLIP ViT-Large-Patch14: 428M parameters

Evaluation#

Testing Data#

The testing data is from an open-source dataset (https://huggingface.co/datasets/cyberagent/crello).

Metrics#

• Overall metrics. We use a robust proxy model (https://huggingface.co/llava-hf/llava-onevision-qwen2-7b-ov-hf) for comprehensive evaluation from five aspects: (i) design and layout, (ii) content relevance, (iii) typography and color, (iv) graphics and images, and (v) innovation and originality. We use the same prompts as presented in COLE [1].
• Geometry-related metrics. These metrics focus purely on the geometric attributes of elements without considering their content, including element validity (Val), Overlap (Ove), Alignment (Ali) and underlay effectiveness (Undl, Unds) [2][3].

Results#

We use prior work FlexDM [1] and prompting GPT-4o [2] as baselines. In comparison, Elem2Design demonstrates superior performance across nearly all metrics. For example, on overall metrics, Elem2Design achieves 8.08, 7.92, 8.00, 7.82 and 6.98 on the evaluated five aspects. For another example, regarding geometry-related metrics, Elem2Design and FlexDM achieve Ove score of 0.0865 and 0.3242 respectively, indicting that Elem2Design effectively addresses the overlap issue whereas FlexDM encounters difficulties in this area. See Table 1 for the complete evaluation in our paper (https://arxiv.org/pdf/2412.19712)

Environmental Impact#

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Citation#

@InProceedings{lin2024elements,
  title={From Elements to Design: A Layered Approach for Automatic Graphic Design Composition},
  author={Lin, Jiawei and Sun, Shizhao and Huang, Danqing and Liu, Ting and Li, Ji and Bian, Jiang},
  booktitle={CVPR},
  year={2025}
}

Model Card Contact#

We welcome feedback and collaboration from our audience. If you have suggestions, questions, or observe unexpected/offensive behavior in our technology, please contact us at Shizhao Sun, shizsu@microsoft.com.

If the team receives reports of undesired behavior or identifies issues independently, we will update this repository with appropriate mitigations.