Distributed file upload causes the service to freeze?

introduction

I remember that when I first started learning web projects, I often needed to upload pictures. At that time, pictures were uploaded to the current project folder, and the pictures were lost every time the project was restarted. Although you can modify /tomcat/conf/server.xml configuration file to configure a local folder for uploading pictures, that is, configure a project configuration virtual path, which can avoid the loss of pictures when the project is restarted. Since I started working, I haven't encountered this way to store pictures. Generally, either build your own file server or use a paid file service. Such as Qiniu Cloud, Alibaba Cloud, Tencent Cloud, etc. Today, let's talk about how to upload files using Alibaba Cloud OSS .

oss file upload

To upload files using OSS, Alibaba Cloud provides the following methods. You can choose the method that suits you.

Web upload

A common upload method on the web side is that users upload files to the application server on the browser or app side, and the application server uploads the files to OSS. The specific process is shown in the figure below.

This method is definitely not advisable. It has the following disadvantages:

• Slow upload: User data needs to be uploaded to the application server first, and then uploaded to the OSS. The network transmission time is twice as long as the direct upload to the OSS. If the user data is not transferred through the application server, but directly transferred to the OSS, the speed will be greatly improved. Moreover, OSS adopts BGP bandwidth, which can ensure the transmission speed between operators in various places.
• Poor scalability: If the number of subsequent users gradually increases, the application server will become a bottleneck. Originally, OSS has been used to upload, and then I have to occupy my own server.

• High cost: multiple application servers need to be prepared. Since the OSS upstream traffic is free, if the data is directly transmitted to the OSS, the cost of multiple application servers will be saved.

  JavaScript client-side signature direct pass

  This method uses pure front-end upload directly, without going through the application server, but this method provides some core parameters about OSS upload from Alibaba Cloud (AccesssKey ID and AccessKey Secret are equivalent to the account and password we applied for on Alibaba Cloud). ) also needs to be written in the front-end code, which will easily cause our core parameters to be leaked. There are security risks. This method is also not recommended.

  Server-side signature and direct transmission

  Signing and uploading directly on the front end will cause security risks and parameter leakage. We can put the parameters on the server side, and then the server side interacts with Alibaba Cloud, so that there is no leakage of core parameters.
  

How to access

import dependencies

• Because I am engaged in java development, I directly introduce the latest maven dependencies provided by the official.

  <!-- https://mvnrepository.com/artifact/com.aliyun.oss/aliyun-sdk-oss -->
  <dependency>
    <groupId>com.aliyun.oss</groupId>
    <artifactId>aliyun-sdk-oss</artifactId>
    <version>3.14.0</version>
  </dependency>

  Why bring in the latest dependencies. Because if you encounter a problem and need to ask someone from Alibaba Cloud to help you solve it, most of them will ask you what version of sdk you have, and if you encounter a problem that is difficult to solve for a while, they will recommend you to upgrade to the latest version try. Because the bug you encountered may be fixed in the latest version. Some people may say, isn't the introduction of the latest version to help others step into the pit? What if a bug is resolved and two bug columns are introduced? This situation is not impossible.

  Server build signature

  
  The above picture is an introductory example provided by the official website. The code is a big lump. We can look at the slightly optimized code:
  Create a singleton of ossClient , which can reuse threads without going to new ossClient() every time.

        String host = String.format("https://%s.%s", ossPropertoooies.getBucketName(), ossPropertoooies.getEndpoint());
        long expiredTime = System.currentTimeMillis() + fileOssProperties.getUploadSignatureTtl();
        Date expiration = new Date(expiredTime);
  
        // 根据文件名和文件类型设置存储路径，可以按照文件类型+日期格式+UUID文件名 进行分割
        String filepath = getFilePath(request.getCategory(), request.getFilename());
  
        PolicyConditions policyConditions = new PolicyConditions();
        policyConditions.addConditionItem(PolicyConditions.COND_CONTENT_LENGTH_RANGE, 0, fileOssProperties.getUploadSizeLimit());
        policyConditions.addConditionItem(MatchMode.StartWith, PolicyConditions.COND_KEY, filepath);
  
        String postPolicy = ossClient.generatePostPolicy(expiration, policyConditions);
        byte[] binaryData = postPolicy.getBytes(StandardCharsets.UTF_8);
        String encodedPolicy = BinaryUtil.toBase64String(binaryData);
        String postSignature = ossClient.calculatePostSignature(postPolicy);
  
        SignatureDTO signature = new SignatureDTO();
        signature.setAccessId(ossPropertoooies.getAccessKeyId());
        signature.setPolicy(encodedPolicy);
        signature.setSignature(postSignature);
        signature.setFilepath(filepath);
        signature.setHost(host);
        signature.setExpire(fileOssProperties.getUploadSignatureTtl() / 1000);
        signature.setReqFilename(request.getFilename());

  It is very simple to access, a back-end signature, and the separate file upload before and after the front-end upload has been completed. Here we use postman simulate the front-end upload. Of course, we can change the front-end to use ajax , or any other way. The uploaded url is composed of bucketname and endpoint applied by ourselves
  
  But in fact, there are many pits in it, we still need to pay a little attention.

  Bandwidth limit

  There will be bandwidth limitations for uploading and downloading. If we use the external network to directly upload to Alibaba Cloud oss, we need to pay attention to whether our external network bandwidth is sufficient, and whether the upload of large files will fill up the bandwidth. If the bandwidth is full, we will upload it. The same download also has bandwidth limitations, and it is necessary to avoid downloading large files. If we encounter such large file downloads, we can use other methods, such as using the oss client. So we need to reasonably consider the bandwidth of our server. If our application is directly deployed on Alibaba Cloud, we can use intranet upload and download. This way there will be no bandwidth limitations.

  Points to note when using the API

  When we use some OSSclient provided by api , we need to take a closer look at how it is implemented, or see if its documentation has any special explanations.
  For example, using the OSSclient method provided by processObject , we finally need to close the input stream. If the stream is not closed, the link will not be released. The application link will be filled immediately, and then the service will become a state of suspended animation. We encountered this problem once in the production environment. As shown in the figure below, the thread has not been released.
  
  Why do we need to manually close the stream like this? Why don't api help us close it directly? Alibaba Cloud's reply is because the stream returned here may need to be copied or read by the business side itself. Therefore, the caller needs to actively close it. We also found the answer in this very secret document.
  

  end

• Due to my lack of knowledge and knowledge, it is inevitable that there will be mistakes. If you find any mistakes, please leave a message and point them out to me, and I will correct them.
• If you think the article is not bad, your forwarding, sharing, appreciation, likes, and comments are the greatest encouragement to me.
• Thank you for reading, you are very welcome and appreciate your attention.
  Picking apples while standing on the shoulders of giants:
  https://help.aliyun.com/document_detail/31924.html
  https://gosspublic.alicdn.com/AliyunJavaSDK/3.10.2/javadoc/index.html?spm=a2c4g.11186623.0.0.14c240c20nrAKm