How to Use Serverless for Architecture and Project Management - Three Years of Full Stack Experience

This article is explained from the perspective of project engineering. From a technical perspective, please refer to another article "Real project code teaches you four steps to throw away traditional servers, allowing you to use Serverless for full-stack development gracefully" ( https://zhuanlan.zhihu.com /p/493447018 )

This article summarizes my experience in the actual development of multiple serverless full-stack projects, mainly for small and medium-sized projects. You can use my experience directly, or adjust it according to the actual situation.

Step 1: Decide on Serverless

The art of project management is reflected in how to use advantages and avoid disadvantages under limited conditions. Therefore, we need to first determine what are the advantages and disadvantages of serverless in small and medium-sized projects.

(1) Advantages of Serverless

• Low cost: For small projects, the cost of serverless can be said to be extremely low. Although our technical personnel are not very concerned about and responsible for the cost, for the company-level decision-making of small and medium projects, the cost is an unavoidable problem.
• Low personnel requirements (focus on development): It is no longer necessary to configure, manage and optimize the server environment, and the code can be used directly after uploading.
• Subversive performance optimization (performance is so convenient): This is the advantage I value the most, and it is also an advantage that must be used. The tuning configuration of traditional servers is for the whole project level. If an interface has a performance bottleneck, the configuration of the entire project needs to be pulled. Serverless can perform performance optimization at the level of a single API interface. For example, we can only improve the performance configuration of the home page and popular activities. This is something that other traditional projects need to pay a high development cost to do, but Serverless can achieve it extremely easily.
• Security: Due to its unique characteristics, Serverless is destined to be very different from traditional security. At present, mainstream network attacks such as system intrusion and DDos attack are mostly ineffective or even directly ineffective against serverless, and no attack tools have been found specifically for serverless. In addition, serverless itself provides some security mechanisms, such as Tencent Cloud SCF's single-function Exclusive quotas, shared quotas for all functions, API gateway authentication, etc., can be used reasonably to achieve high security at low cost.

Function's Exclusive Quota Settings Page

(2) Disadvantages of Serverless: His disadvantages need to be solved ecologically

• Less practice: Although Serverless came out in 2014, it really feels like it has been used in the past three or four years. It is mainly used by high-level projects or by individuals who play tickets by themselves, and there are even fewer full-stack projects. At the same time, those with practical experience lack the time and channels to preach, and the books on the market are mainly theoretical. Therefore, there is no mature and authoritative engineering theory in the industry to guide serverless DevSecOps, which puts forward higher requirements on project managers themselves, and even puts forward additional requirements on whether project managers can accurately assess risks and make bold decisions.
• Fewer frameworks: There is no serverless native framework yet.
• Deployment trouble: Needless to say, this is a flaw, but it is also the best solution. Like jQuery and other frameworks that emerged after the sudden rise of JavaScript more than 10 years ago, the drawbacks of Serverless are opportunities for developers, and those who boldly try to find a solution first will also lead Serverless.

The second step: project plan selection

(1) Selection of cloud service providers

Before the project starts, we have to choose a service provider. There are many serverless service providers on the market. Which service provider to choose is the first choice for all project leaders. In fact, for large service providers, you can choose whichever one you choose. I am an in-depth user of Tencent Cloud's cloud functions (so don't ask me, ask me to recommend Tencent Cloud), so the experience in this article is also based on Tencent Cloud. Of course, there are certain differences in the tuning of each platform, but the same goal can be used for reference.

(2) Cloud function selection

Tencent Cloud's cloud functions are divided into two categories: "WEB functions" and "event functions", and each category of deployment methods includes "image deployment" and "code deployment". For small and medium-sized projects, I recommend "event functions" + "Code Deployment".

WEB function: In my opinion, the "WEB function" was created to expand the market share of cloud functions. Even at the beginning, cloud functions did not have this "WEB function" at all. of.

Zhang Guo (zhihu iGuo), founder and CEO of Xiamen Liangchao Technology Co., Ltd., also mentioned that "WEB function" is actually a compromise.

The WEB function is to integrate the entire project into a cloud function for deployment, which will prevent us from taking advantage of the "interface-level management" and "security" advantages of Serverless, lose the biggest feature of Serverless, and only get "framework support". "This feature is very uneconomical. In fact, framework support is not important for small and medium-sized projects in Serverless, or even dispensable. For details, please refer to the following description.

Choice of deployment method:

Code deployment is simply deploying code directly to the cloud. Image deployment is to create an image locally and upload it to Tencent Cloud's "Container Image Service", and then push it to the cloud function.

In fact, the built-in extensions of Tencent Cloud have met the technical requirements of most projects. Some of the extensions that are not directly built-in can also be implemented directly through third-party packages. Except for projects with very complex requirements, code deployment has met the needs of most projects. .

The "code deployment" method maintains a high degree of consistency between the cloud and the local area. Even problems on the cloud can be quickly and easily restored locally. At the same time, deployment operations are simple, maintenance is convenient, speed is fast, performance is good, and debugging locally or on the cloud is convenient.

Therefore, it is recommended: try to use "mirror deployment" when "code deployment" really cannot meet the project requirements.

I think: The ultimate experience of serverless applications is local hello world development, which is equivalent to project development.

Step 3: Frame Selection

Taking PHP as an example, small and medium-sized projects use Serverless, and the traditional framework is likely to be a drag. Do you think it is strange? The purpose of the project adoption framework is to develop and maintain projects more efficiently, conveniently, and with higher performance from an engineering perspective. The reason why a framework is needed is that compared with server development, the development of local programs has fewer considerations, such as less need to worry about performance, no need to consider large concurrency, and so on. Frameworks do a lot of this work for us to help us address the needs of our online projects. But in Serverless, for ordinary projects, the framework has lost most of its value and meaning.

• Weakening performance: In serverless, what we actually use is not the server but the computing power. Since the computing power provided by the cloud function itself is quite strong, even the code of poor quality can withstand it, and the high performance of the framework has actually been weakened. .
• Weakening of concurrency: For small and medium-sized projects, most of the business execution time is very short. The pressure of high concurrency does not come from the pressure of efficient algorithms, but from the instantaneous growth of traffic and computing resources exceeding the capacity provided by the server. Each call of cloud functions can be said to be physically isolated, so cloud functions are actually equivalent to "automatic infinite expansion" of computing and network resources, so they are not afraid of high concurrency. The need for optimization in the high concurrency aspect of the framework is actually dispensable.
• Function failure: In Serverless, the traditional connection pool is invalid. We need to create an original connection pool. The databases of major cloud service providers are also optimized for the large number of link requirements of cloud functions, and directly provide support for more than 2,000 simultaneous links.
• Learning cost: Small and medium-sized projects do not require advanced technology, and there is no huge business chain. Quick and simple is the demand. Any framework, no matter how simple it is, requires learning costs, and most technical developers just start using it under Baidu, and do not study it in depth. Therefore, the smaller the project, when the framework itself is not satisfied or there are problems, the framework itself becomes the biggest obstacle. For small and medium projects, some people use frameworks for the sake of frameworks.
• Framework Not Supported: As mentioned above, the framework "event functions" for most languages are not supported.
• There is no native framework for Serverless.

Serverless technical framework description:

In order to take full advantage of Serverless, multi-function deployment must be adopted, that is, the microservice architecture pattern. The ultimate and best destination of Serverless must be the microservice architecture model. The current traditional frameworks are just temporary transitions before the native frameworks.

Traditional microservice frameworks are generally aimed at large projects and have high learning and entry costs. The microservices of small and medium-sized projects are actually very simple. We do not really use the microservice framework, but develop the project like writing a local hello world, and deploy an API interface and a cloud function.

We want the benefits of microservices, not the hassles. Therefore, the model we should adopt is: traditional architecture + traditional development + microservice deployment.

The easiest way is to use a simple framework and deploy each API that interacts with the client as a separate function. An efficient serverless development framework must support local unit testing, and the effect of unit testing must be equivalent to the execution effect after deployment. Because the framework that is inconvenient to test will make developers lazy and wait for the function to be deployed before testing, which is likely to cause a lot of rework problems. The implementation logic of unit testing can refer to my last article.

Therefore, in my opinion, for small and medium-sized projects, when not using a framework does not affect team collaboration and efficiency, there is no need to use a framework (traditional frameworks are not very suitable for cloud functions), or write a simple common project by yourself. structure or frame. It is deployed in units of API interfaces, and microservices are formed directly after deployment.

When developing, you only need to pay special attention to the fact that there is no session in the whole project. Each API interface is physically isolated from each other during execution, and the data is not interoperable.

The framework used in my project is a self-developed framework for Serverless, and I have used multiple projects. While the framework makes good use of the features of cloud functions, it not only does not increase the difficulty of development, but reduces it, so low that I feel a little LOW. As shown in the figure:

• xxxxapi.php: This is the interface file, located in the project root directory. It is the execution entry of all cloud functions divided by function, such as the sendadminsms function under adminapi.php. The cloud function name is sendadminsms when deployed, and the API gateway is: /admin/sendadminsms. Note: The function names in multiple interface files cannot be repeated, because the names of cloud functions on the cloud cannot be the same.
• ctrl directory: similar to the control module of other projects, responsible for business code. For example: AdminCtrl.php.
• model directory: database module, responsible for database operations, such as: AdminModel.php.
• Simplescf in vendor: core library I have developed a set of core libraries specifically for Tencent Cloud functions, which implements dynamic cloud function invocation, dynamic control, unit test execution, database encapsulation, log processing and a very convenient ORM Database encapsulation (support TCB free database), etc., similar to the SCF version developed by WeChat Yunyun. The cloud development of WeChat Cloud is very restrictive. This framework can support WEB, applet, and APP at the same time.
• The overall calling sequence: After the deployment is completed, the client-->API gateway-->xxxapi.php entry function-->ctrl processes business logic-->model processes data.

Screenshots of non-sensitive parts of the project

Step 4: Project division of labor schedule

In serverless project management, the division of tasks cannot simply be based on traditional functional modules. Instead, APIs are used as units to designate responsible persons, and workload assessments are performed. Testing is also performed in API units.

If the project involves collaboration, the complete process is recommended:

PM pre-creates each API, and formulates input and output parameters. Light creation is to predefine the calling entry function; deep creation not only creates entry functions, but also deploys the DEMO code to the cloud. Deep creation can effectively make front-end and back-end development asynchronous.
Arrange the person in charge of the API interface and make a development plan.
As far as possible, the API tasks that each person is responsible for are common, for example, the functions in the interface file raceapi.php are basically responsible for this person.
Distill the public functions of the project into public modules.
It is very important to establish a good unit test and let the localized test (unit test TDD) simulate cloud execution with high fidelity.

Step 5: Project Deployment

Currently, there are basically three types of serverless deployments: management background, Serverless Framework CLI, and SDK.

• Management background: You have to open the web page, you have to give the member account and password, you have to log in, and multiple people may need it at the same time, which is the word "trouble".
• Serverless Framework CLI: yaml configuration file is required, and you need to modify the configuration yourself. If the configuration is wrong, your project may be rushed. You need to give the key, or you can scan the QR code. This is really a Chinese characteristic. , I really want to rant. I, a heavy user, occasionally get misconfigured and overwrite the existing configuration, so I almost smashed the table.
• SDK: Forget it, do you research the SDK yourself for deployment? A trigger may require the cooperation of the SDKs of 2 products. Because I am a devops developer of Tencent Cloud Functions, I am a heavy user of the SDK, but I don't think a single project developer would do this! Not enough time wasted to hire someone to deploy it for you!

If the project is deployed in the microservice mode, there may be hundreds of APIs, that is, hundreds of cloud functions. At most, I have deployed nearly 300 cloud functions in one project. If CLI is deployed, do I have to toss 300 yamls? , if I want to update a certain function code in the middle, I have to configure the yaml of the update code. What if I want to update multiple function codes at the same time? You can't go wrong with these three hundred. Thinking about it this way, CLI is still very embarrassing for our small projects.

So what are the advantages of CLI? It can be perfectly integrated with CI, and the automatic deployment of the project can be realized through CI, so it is very useful for large projects, but for small projects, unit tests are not written, and you still tell me about CI?

So overall, do you feel that small and medium-sized projects will encounter big problems in the deployment of this work? My solution is to create a GUI deployment tool, one-click update (second-level deployment, update function code), and also implicitly do the guiding function of project management work. After about 2 months, the tool will be open sourced for everyone to use for free.

Author: Sheldor, Zhihu account: Xiaozhujun, welcome to pay attention to learn more about the practical experience of Serverless.