With the substantial improvement of the network environment and the significant reduction of network tariffs, people's interaction on the network is no longer limited to traditional text content. Pictures, voices, videos, etc. have joined the ranks of interaction.
Compared with text content, multimedia has advantages in ease of use and emotional expression. Of course, compared with text, the volume is much larger. For example, a text message is less than 100 bytes, but a video message may exceed 100 MB. This results in multimedia being more difficult than text content in terms of network transmission and real-time access.
In websites or programs, in addition to interactive messages such as pictures and videos exchanged by users, there are also more and more large files in other forms. Ensuring the stability of uploading large files is a key point that requires long-term investment and breakthroughs. Today, let’s take a look at what methods are available to ensure that users have a good upload experience.
How to make resource delivery fast and stable
You may have noticed that when using QQ, the pictures and videos sent to you by your friends will load for a period of time before they are displayed. This is because the program does not send files to you, but sends the file path. When we access the path, The file will be loaded. This is the current mainstream communication method for large files, namely:
- Upload files to file server
- Push file path
- Received file path
- Load file
This is done because the transmission of large files takes too long, which will affect the immediacy of the message, so the file will not be transmitted directly.
Then, if we want to send multimedia messages such as pictures, videos, and voices quickly and steadily, we can start from the step of "uploading multimedia files to the file server".
upload access point
The first method of optimizing uploaded files is to optimize the upload access point.
The upload request is sent to the file server, the server responds to the establishment of the connection, and the uploader continues to transmit data until the upload is completed. The shorter this process takes, the faster the internet speed will be.
However, the current fixed-line broadband operators in our country are complex, and the high latency and instability of cross-operator network access has always been a difficult problem. For multimedia files such as videos and pictures with more data to be transmitted, the experience of multiple RTT (Round-Trip-Time, round-trip delay) will be worse if the files are uploaded across operators.
The current mainstream approach for this kind of network situation is to provide different upload access point IPs for different network operators, and then resolve them through the operator's DNS so that users can quickly upload files through the upload access point of the operator. At the same time, the back-end image upload and storage service will be deployed in the multi-line computer room to easily and quickly submit the file stream to the storage layer, thus avoiding the cross-network overhead from the access point to the storage service, and solving the problem of downloading pictures by users of other operators Need to cross-network issues.
However, although multiple operator access point IP can solve the problem of cross-operator network access, it will be more complicated in management and use. Therefore, most service providers will use multi-line BGP (Border Gateway Protocol, Border Gateway Protocol) computer room hosting, so that only IP can be provided, allowing users to choose the best route independently.
upload in
In addition to the optimization of the upload access point, block upload can also improve the upload performance and success rate of multimedia files.
Uploading in blocks means that on the upload client, the file to be uploaded is divided into multiple data blocks according to certain rules, and the serial numbers are marked according to the rules, and then these data blocks are uploaded separately. After the server receives it, it will re-write the data according to the serial number. Assemble multiple data blocks into a complete file.
For larger files such as pictures and videos, uploading in chunks allows the client to use a concurrent method to make full use of network bandwidth to upload multiple chunks at the same time after the chunks are completed, which effectively improves upload efficiency. In addition, in some scenarios where the network environment is poor and the signal is unstable, the block method is adopted. When retrying after the upload fails, only one of the failed blocks needs to be retransmitted instead of re-uploading the entire file, which is a significant improvement The success rate of resending.
In multi-part upload, the size of the part is also an important issue. If the block size is too large, the number of pieces is small, and the upload concurrency is insufficient, the upload efficiency may be reduced, and the cost of retransmitting each large block after failure will be higher. However, if the blocks are too small, the number of slices is too large, and the number of connections required for concurrency is too much, multiple connections will reduce the overall throughput of the network, and the overhead of splitting and merging the blocks between the client uploading side and the service receiving side will increase accordingly, and The extra traffic during transmission will be more.
Therefore, how to choose an appropriate block size under different network environments is very important. Generally speaking, in environments with good network conditions, such as WiFi and 4G, the corresponding block size should be set larger; and in the case of weak 2G and 3G networks, the block size can be set to be smaller.
breakpoint
Because the uploading of large files takes a long time, if the user needs to suspend the upload task when the upload is not completed due to network disconnection, device restart, etc. Of course, users will definitely not want the next upload to start from scratch. Therefore, the function of how to start uploading from the paused position becomes necessary. This is what we often call "resumed upload"
Resumable uploading is done by assigning a unique task ID to each upload behavior. In addition to carrying its own serial number when uploading each block, it also needs to bring this task ID. The server targets the same task ID it receives. If the upload is suspended for some reason, these temporary blocks will not be cleaned up immediately, but will be kept for a certain period of time.
When the uploader needs to continue uploading, it will continue to carry the previous task identifier while requesting. After receiving it, the client will check the status of the existing segment on the server, and if it has not expired, it will continue to upload from the last disconnected segment, otherwise it will re-upload.
When uploading large files, the breakpoint resume function can solve the pain point of having to start all over again due to network errors in the process of transferring large files, which is obvious for improving the upload experience of users.
How to quickly have these upload advantages
If you want to complete the above-mentioned large file upload optimization, you need to consider the architecture, design, operation and maintenance, etc., not only the optimization cost is high, but the transformation time is relatively long. If you want to have the advantages mentioned above, you can choose to shoot cloud storage again.
Another shot cloud storage can allow large files to be uploaded nearby through CDN, upload accelerated transmission, block upload and resumable transmission, and can also use image processing, video processing, file compression and decompression to process files. When files are accessed, large file segment caching, video dragging, link optimization and other functions are also used for network-wide distribution through CDN. Ensure the transmission, processing, storage, distribution, and disaster tolerance requirements of the enterprise's massive data.
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。