聊聊Spring AI的Image Model

序

本文主要研究一下Spring AI的Image Model

ImageModel

org/springframework/ai/image/ImageModel.java

@FunctionalInterface
public interface ImageModel extends Model<ImagePrompt, ImageResponse> {

    ImageResponse call(ImagePrompt request);

}

ImageModel继承了Model接口，其call方法入参为ImagePrompt，返回ImageResponse

ImagePrompt

org/springframework/ai/image/ImagePrompt.java

public class ImagePrompt implements ModelRequest<List<ImageMessage>> {

    private final List<ImageMessage> messages;

    private ImageOptions imageModelOptions;

    public ImagePrompt(List<ImageMessage> messages) {
        this.messages = messages;
    }

    public ImagePrompt(List<ImageMessage> messages, ImageOptions imageModelOptions) {
        this.messages = messages;
        this.imageModelOptions = imageModelOptions;
    }

    public ImagePrompt(ImageMessage imageMessage, ImageOptions imageOptions) {
        this(Collections.singletonList(imageMessage), imageOptions);
    }

    public ImagePrompt(String instructions, ImageOptions imageOptions) {
        this(new ImageMessage(instructions), imageOptions);
    }

    public ImagePrompt(String instructions) {
        this(new ImageMessage(instructions), ImageOptionsBuilder.builder().build());
    }

    @Override
    public List<ImageMessage> getInstructions() {
        return this.messages;
    }

    @Override
    public ImageOptions getOptions() {
        return this.imageModelOptions;
    }

    //......
}    

ImagePrompt实现了ModelRequest接口，它定义了messages及imageModelOptions属性，其getInstructions返回的是List<ImageMessage>类型

ImageMessage

org/springframework/ai/image/ImageMessage.java

public class ImageMessage {

    private String text;

    private Float weight;

    public ImageMessage(String text) {
        this.text = text;
    }

    public ImageMessage(String text, Float weight) {
        this.text = text;
        this.weight = weight;
    }

    //......
}    

ImageMessage定义了text及weight属性

ImageOptions

org/springframework/ai/image/ImageOptions.java

public interface ImageOptions extends ModelOptions {

    @Nullable
    Integer getN();

    @Nullable
    String getModel();

    @Nullable
    Integer getWidth();

    @Nullable
    Integer getHeight();

    @Nullable
    String getResponseFormat();

    @Nullable
    String getStyle();

}

ImageOptions继承了ModelOptions，它定义了getN、getModel、getWidth、getHeight、getResponseFormat、getStyle方法

ImageResponse

org/springframework/ai/image/ImageResponse.java

public class ImageResponse implements ModelResponse<ImageGeneration> {

    private final ImageResponseMetadata imageResponseMetadata;

    /**
     * List of generate images returned by the AI provider.
     */
    private final List<ImageGeneration> imageGenerations;

    /**
     * Construct a new {@link ImageResponse} instance without metadata.
     * @param generations the {@link List} of {@link ImageGeneration} returned by the AI
     * provider.
     */
    public ImageResponse(List<ImageGeneration> generations) {
        this(generations, new ImageResponseMetadata());
    }

    /**
     * Construct a new {@link ImageResponse} instance.
     * @param generations the {@link List} of {@link ImageGeneration} returned by the AI
     * provider.
     * @param imageResponseMetadata {@link ImageResponseMetadata} containing information
     * about the use of the AI provider's API.
     */
    public ImageResponse(List<ImageGeneration> generations, ImageResponseMetadata imageResponseMetadata) {
        this.imageResponseMetadata = imageResponseMetadata;
        this.imageGenerations = List.copyOf(generations);
    }

    //......
}    

ImageResponse实现了ModelResponse接口，其getResult返回的是ImageGeneration类型

ImageGeneration

org/springframework/ai/image/ImageGeneration.java

public class ImageGeneration implements ModelResult<Image> {

    private ImageGenerationMetadata imageGenerationMetadata;

    private Image image;

    public ImageGeneration(Image image) {
        this.image = image;
    }

    public ImageGeneration(Image image, ImageGenerationMetadata imageGenerationMetadata) {
        this.image = image;
        this.imageGenerationMetadata = imageGenerationMetadata;
    }

    @Override
    public Image getOutput() {
        return this.image;
    }

    @Override
    public ImageGenerationMetadata getMetadata() {
        return this.imageGenerationMetadata;
    }

    @Override
    public String toString() {
        return "ImageGeneration{" + "imageGenerationMetadata=" + this.imageGenerationMetadata + ", image=" + this.image
                + '}';
    }

}

ImageGeneration实现了ModelResult接口，其getOutput返回的是Image类型，其getMetadata返回ImageGenerationMetadata

Image

org/springframework/ai/image/Image.java

public class Image {

    /**
     * The URL where the image can be accessed.
     */
    private String url;

    /**
     * Base64 encoded image string.
     */
    private String b64Json;

    //......
}    

Image定义了url与b64Json属性

ImageGenerationMetadata

org/springframework/ai/image/ImageGenerationMetadata.java

public interface ImageGenerationMetadata extends ResultMetadata {

}

ImageGenerationMetadata目前仅仅是继承了ResultMetadata

示例

pom.xml

        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-starter-model-zhipuai</artifactId>
        </dependency>

配置

spring:
  ai:
    model:
      image: zhipuai
    zhipuai:
      api-key: xxxx
      base-url: https://open.bigmodel.cn/api/paas
      images:
        options:
          model: cogview-3

代码

    @Test
    public void testGenImage() {
        ImagePrompt imagePrompt = new ImagePrompt("A light cream colored mini golden doodle",
                ZhiPuAiImageOptions.builder()
                        .model("cogview-3").build());
        ImageResponse response = imageModel.call(imagePrompt);
        log.info("resp:{}", JSON.toJSONString(response));
    }

输出示例

resp:{"metadata":{"created":1743578654097,"empty":true,"rawMap":{}},"result":{"output":{"url":"https://sfile.chatglm.cn/testpath/xxx.png"}},"results":[{"$ref":"$.result"}]}

源码

ZhiPuAiImageAutoConfiguration

org/springframework/ai/model/zhipuai/autoconfigure/ZhiPuAiImageAutoConfiguration.java

@AutoConfiguration(after = { RestClientAutoConfiguration.class, SpringAiRetryAutoConfiguration.class })
@ConditionalOnClass(ZhiPuAiApi.class)
@ConditionalOnProperty(name = SpringAIModelProperties.IMAGE_MODEL, havingValue = SpringAIModels.ZHIPUAI,
        matchIfMissing = true)
@EnableConfigurationProperties({ ZhiPuAiConnectionProperties.class, ZhiPuAiImageProperties.class })
public class ZhiPuAiImageAutoConfiguration {

    @Bean
    @ConditionalOnMissingBean
    public ZhiPuAiImageModel zhiPuAiImageModel(ZhiPuAiConnectionProperties commonProperties,
            ZhiPuAiImageProperties imageProperties, RestClient.Builder restClientBuilder, RetryTemplate retryTemplate,
            ResponseErrorHandler responseErrorHandler) {

        String apiKey = StringUtils.hasText(imageProperties.getApiKey()) ? imageProperties.getApiKey()
                : commonProperties.getApiKey();

        String baseUrl = StringUtils.hasText(imageProperties.getBaseUrl()) ? imageProperties.getBaseUrl()
                : commonProperties.getBaseUrl();

        Assert.hasText(apiKey, "ZhiPuAI API key must be set");
        Assert.hasText(baseUrl, "ZhiPuAI base URL must be set");

        var zhiPuAiImageApi = new ZhiPuAiImageApi(baseUrl, apiKey, restClientBuilder, responseErrorHandler);

        return new ZhiPuAiImageModel(zhiPuAiImageApi, imageProperties.getOptions(), retryTemplate);
    }

}

ZhiPuAiImageAutoConfiguration在spring.ai.model.image为zhipuai时启用，它会根据ZhiPuAiConnectionProperties及ZhiPuAiImageProperties自动配置ZhiPuAiImageModel

ZhiPuAiConnectionProperties

@ConfigurationProperties(ZhiPuAiConnectionProperties.CONFIG_PREFIX)
public class ZhiPuAiConnectionProperties extends ZhiPuAiParentProperties {

    public static final String CONFIG_PREFIX = "spring.ai.zhipuai";

    public static final String DEFAULT_BASE_URL = "https://open.bigmodel.cn/api/paas";

    public ZhiPuAiConnectionProperties() {
        super.setBaseUrl(DEFAULT_BASE_URL);
    }

}

class ZhiPuAiParentProperties {

    private String apiKey;

    private String baseUrl;

    public String getApiKey() {
        return this.apiKey;
    }

    public void setApiKey(String apiKey) {
        this.apiKey = apiKey;
    }

    public String getBaseUrl() {
        return this.baseUrl;
    }

    public void setBaseUrl(String baseUrl) {
        this.baseUrl = baseUrl;
    }

}

ZhiPuAiConnectionProperties继承了ZhiPuAiParentProperties属性，ZhiPuAiParentProperties定义了apiKey、baseUrl属性

ZhiPuAiImageProperties

org/springframework/ai/model/zhipuai/autoconfigure/ZhiPuAiImageProperties.java

@ConfigurationProperties(ZhiPuAiImageProperties.CONFIG_PREFIX)
public class ZhiPuAiImageProperties extends ZhiPuAiParentProperties {

    public static final String CONFIG_PREFIX = "spring.ai.zhipuai.image";

    /**
     * Options for ZhiPuAI Image API.
     */
    @NestedConfigurationProperty
    private ZhiPuAiImageOptions options = ZhiPuAiImageOptions.builder().build();

    public ZhiPuAiImageOptions getOptions() {
        return this.options;
    }

    public void setOptions(ZhiPuAiImageOptions options) {
        this.options = options;
    }

}

ZhiPuAiImageProperties继承了ZhiPuAiParentProperties，它主要配置spring.ai.zhipuai.image.options

ZhiPuAiImageOptions

org/springframework/ai/zhipuai/ZhiPuAiImageOptions.java

@JsonInclude(JsonInclude.Include.NON_NULL)
public class ZhiPuAiImageOptions implements ImageOptions {

    /**
     * The model to use for image generation.
     */
    @JsonProperty("model")
    private String model = ZhiPuAiImageApi.DEFAULT_IMAGE_MODEL;

    /**
     * A unique identifier representing your end-user, which can help ZhiPuAI to monitor
     * and detect abuse. User ID length requirement: minimum of 6 characters, maximum of
     * 128 characters
     */
    @JsonProperty("user_id")
    private String user;

    public static Builder builder() {
        return new Builder();
    }

    @Override
    @JsonIgnore
    public Integer getN() {
        return null;
    }

    @Override
    public String getModel() {
        return this.model;
    }

    public void setModel(String model) {
        this.model = model;
    }

    @Override
    @JsonIgnore
    public Integer getWidth() {
        return null;
    }

    @Override
    @JsonIgnore
    public Integer getHeight() {
        return null;
    }

    @Override
    @JsonIgnore
    public String getResponseFormat() {
        return null;
    }

    @Override
    @JsonIgnore
    public String getStyle() {
        return null;
    }

    public String getUser() {
        return this.user;
    }

    public void setUser(String user) {
        this.user = user;
    }

    //......
}    

ZhiPuAiImageOptions实现了ImageOptions接口，主要支持了getModel、getUser方法

小结

Spring AI提供了ImageModel接口，它继承了Model接口，其call方法入参为ImagePrompt，返回ImageResponse。spring-ai-starter-model-openai提供了OpenAiImageModel，spring-ai-starter-model-azure-openai提供了AzureOpenAiImageModel，spring-ai-starter-model-qianfan提供了QianFanImageModel，spring-ai-starter-model-stability-ai提供了StabilityAiImageModel，spring-ai-starter-model-zhipuai提供了ZhiPuAiImageModel。

doc

• Image Model API
• OpenAI Image Generation
• Azure OpenAI Image Generation
• QianFan Image Generation
• Stability AI Image Generation
• ZhiPuAI Image Generation