How to evaluate serverless service capabilities, this report gives 40 standards

Introduction to Nowadays, evaluation agencies have given 40 standards to evaluate the service capabilities of Serverless. These evaluation rules are not only a manifestation of the prosperity and development of the technology ecosystem, but also a way for new entrants to evaluate the effectiveness of the implementation of Serverless. Kind of reference basis.

Editor's note: Two years ago, we were still discussing what serverless is and how to implement serverless. Nowadays, some evaluation agencies have given 40 standards to evaluate the service capabilities of Serverless. These evaluation rules are not only a manifestation of the prosperity and development of the technology ecosystem, but also a reference basis for new entrants to evaluate the effectiveness of the implementation of Serverless.

In this function-as-a-service (FaaS) platform evaluation report of Forrester, we chose Alibaba, Amazon, Google, Huawei, IBM, Microsoft, Nimbella, Oracle and Tencent, the 9 most influential providers, and based 40 standards have been studied, analyzed and scored. The report shows the performance of each provider in various aspects, and aims to help professionals engaged in application development and delivery (AD&D) find the provider that best meets their needs.

Forrester Wave™: Function as a Service (FaaS) Platform Report for the First Quarter of 2021

FaaS platform helps developers quickly create cloud-native services

The abstraction of the FaaS platform allows developers to no longer need to pay attention to complex container or virtual machine cluster management and expansion, so that cloud-native microservices can be quickly created. After handing over the management of the underlying infrastructure to the FaaS provider, developers can use familiar languages such as Java, C#, JavaScript, or Python in the programming environment to write microservices into simple small functions. Then, FaaS providers will automatically expand or shrink these microservices based on service requirements. Developers using the FaaS platform said that by avoiding infrastructure management and using abstraction to eliminate complex operations related to this, they can quickly push new ideas into the deployment phase, and at the same time, they can implement microservices based on actual resource requirements. Determine infrastructure costs. When selecting a FaaS provider, developers should analyze whether the provider has the following conditions:

• Support function and container packaging. As developers deploy more and more types of workloads to the FaaS platform, the FaaS platform should allow developers to simply package a function into a ZIP or JAR file and deploy it, or package custom code to conform to open container standards (OCI) container and deploy the corresponding framework. The FaaS platform should support both options at the same time to provide developers with the greatest flexibility in developing and deploying Web, content, and event-driven workloads.
• Provide robust security features. As developers continue to expand the scope of application of the FaaS platform, it has become very important to ensure that relevant personnel can access the data and application program interfaces (APIs) encapsulated in the virtual private network or virtual private cloud (VPC) in a secure manner. important. In addition, while function expansion and contraction, relevant personnel also need to be able to quickly access these resources without waiting for a time-consuming "cold start".
• Support third-party ecosystems and open standards. Unless you are willing to completely rely on a public cloud provider, you should choose a FaaS provider with more convenient platform integration. The features you need to pay attention to include third-party observability, event binding, and message protocol.

Assessment summary

This Forrester Wave™ evaluation report classifies the objects to be evaluated into "Leaders", "Strong Performers", "Contenders" and "Challengers". Quadrant. This is an assessment of the leading manufacturers in the market and does not represent the overall situation of the market. You can check our report on serverless technology for more information about this market. We hope that this evaluation report is just a starting point, and we recommend that customers use an Excel-based vendor comparison tool to view product evaluations and adjust standard weights (see Figure 1 and Figure 2). Click the link at the beginning of the web version of the report on Forrester.com to download the above tools.

Figure 1 Evaluation results of "Forrester Wave™: Function as a Service (FaaS) Platform, Q1 2021"

* Gray circles indicate vendors that have not participated in the Forrester assessment

Figure 2 "Forrester Wave™: Function as a Service (FaaS) Platform, Q1 2021" score

All ratings are on a scale of 0 (weak) to 5 (strong)

* Indicates manufacturers that did not participate in the evaluation

each manufacturer

Forrester evaluated the products of nine vendors: Alibaba, Amazon, Google, Huawei, IBM, Microsoft, Nimbella, Oracle and Tencent (see Figure 3).

Figure 3 Manufacturers and products evaluated this time

Vendor profile

Through analysis, we found that each manufacturer has the following advantages and disadvantages.

leader

• Amazon is in the lead in scoring items such as geographic coverage, ecosystem breadth, and execution stability. In the 2020 Forrester Wave™ report, we mentioned that Lambda has outstanding performance in terms of geographic availability, customer adoption rate, developer experience of deploying functions, and support for the ecosystem. These advantages will continue into 2021. In addition, features such as support for Amazon Elastic File System (EFS), the ability to run OCI-compliant containers, and billing granularity accurate to 1 millisecond have enabled Lambda to continue to advance on the road of innovation, allowing more types of Work load is applicable. The strong support for serverless architecture security and special workloads allows Lambda to maintain its leading position in 2021. Amazon believes that the life cycle of FaaS workloads should be short. Developers who agree with this view will find that Lambda provides them with a powerful FaaS platform. The platform is gradually supporting third-party tools and a container-centric deployment process. The platform also provides unique services for detailed functions such as state management, database operations, and APIs.

Reference customers gave a high evaluation of Amazon Web Services (Amazon Cloud Computing Services, AWS) Lambda, believing that the platform can provide first-class customer support, excellent performance, and integrate various AWS services and third-party services. In maintaining close cooperation with experienced industry insiders, the product has also been unanimously appreciated by customers. Reference customers believe that the platform has room for improvement in observability, and expect it to provide developers with more simplified tools. AWS Lambda is best suited for temporary workloads that run asynchronously, including integration with other AWS services, the Internet of Things, batch processing, new web applications, and event-driven integration.

• Alibaba uses containers and open standards to accelerate web and content workloads. In the 2020 report, we described Alibaba Function Compute as "a robust universal platform." In 2021, we found that the platform's support for containers and open source projects and standards, including OpenTracing, OpenTelemetry, Grafana, and Jaeger, has further improved its capabilities. In addition to the platform's investment in supporting mainstream web frameworks, content-centric workloads, 1 millisecond billing granularity, and runtime, we have included Alibaba's functional computing in the "leader" quadrant of the FaaS market this time. Alibaba function computing is particularly suitable for workloads that need to process a large number of videos, and uses a container-centric model to achieve function concurrency. For customers with such needs, this platform is the best choice.

Reference customers highly appreciated the event integration capabilities of Alibaba's function computing in the Chinese ISV ecosystem, and spoke highly of their efforts to strengthen customer support during festivals such as Christmas and New Year. In terms of technology, customers are deeply satisfied with the ability to scale to tens of thousands of instances within a few minutes. Reference customers hope that the platform can provide better programming language support in remote debugging and improve support for GPU computing. Although the platform is currently most suitable for companies deploying workloads in Hong Kong or Mainland China, its international coverage is expanding and currently supports 10 regions and a total of 21 availability zones.

• Microsoft Azure Functions retains important enterprise-level features for premium plans. The consumption plan of Azure Functions supports multiple programming languages and stateful durable functions (Durable Functions), but if you want to use reserved instances, support for Docker containers, and powerful security features, developers need to upgrade to advanced dedicated The plan is the Azure App Service plan. Persistent functions provide stateful functions and support binding to Azure Event Hub, while Azure Event Grid can help developers build event-driven microservices. This year, by increasing investment, Microsoft has improved new user guidance, provided developers with more tools, and supported development and debugging on local computers. It has performed outstandingly on the standard of developer experience and received the highest score.

Reference customers highly recognized the ability of Azure Functions to integrate with other Azure services such as Key Vault to achieve confidential storage, and specifically mentioned its powerful built-in observability. The performance and service quality of Azure Functions also make customers generally satisfied. However, if customers want to use important enterprise functions such as VPC services and reserved instances, they have to choose a higher-cost premium service plan. This is where customers concentrate on complaining. For customers who need other Azure services or a FaaS platform covering the world, Azure Functions is the most suitable solution. But if your team wants to build or integrate enterprise-level workloads, you need to upgrade to the premium version of the application service plan.

Strong performer

• Tencent Cloud Function (SCF) platform optimizes cloud-centric development. Tencent has built an excellent set of web-based tools for function development, and combined it with a reliable runtime, supporting multiple programming languages, complex workflows, and function instances with longer running times. Developers can reserve function instances, and Tencent also provides professional support for artificial intelligence (AI) workloads and specific industry clouds. SCF can also be used as an extension mechanism of the WeChat ecosystem, effectively helping companies interact with customers through the extension of WeChat. But for developers who want to develop and debug local functions, Tencent's cloud-centric development philosophy does not fit them well.

Reference customers have spoken highly of SCF's performance, customer support, and low cost of workload execution. The customer also expressed concerns about the stability of the platform and hoped to use more cloud services from Tencent and third parties through more built-in event triggers. Tencent SCF is a good choice for customers who need to deploy FaaS services in China, integrate and expand with WeChat to attract more customers and accept cloud-centric development methods.

• Nimbella fully supports hybrid cloud through the "FaaS everywhere" approach. Like IBM Cloud Functions, Nimbella is based on the open source Apache OpenWhisk project. Nimbella has a simple and fast start-up process for developers. At the same time, its runtime can be used under a pure FaaS model, can also be deployed as an installable function platform to other public clouds, and can be deployed locally. Nimbella achieves "FaaS everywhere" through powerful container support, multiple programming languages, and configurable function maximum runtime settings. Such a FaaS solution is very suitable for stateful functions that need to run for a long time. Despite the high flexibility of the platform, some issues still need attention. Its hosting solution provides tenants with a built-in isolation function, while the installable solution supports hybrid deployment. The team can make specific adjustments according to different organizational needs and security policies. Therefore, a team that chooses a hybrid deployment may spend more time configuring the infrastructure of the production environment.

Reference customers spoke highly of Nimbella's performance in customer support and helping to implement best practices. Customers also consider minimizing the risk of vendor lock-in as a reason for choosing Nimbella. The customer's concerns include security and observability, and they hope to increase the speed of getting started by adding code examples and digital content. Nimbella is a very suitable solution for development teams who adopt a hybrid FaaS solution and want to provide their own network and integration functions on the basis of a serverless architecture platform. In addition, if you prefer to pay monthly or choose an enterprise agreement compared to the usage-based pricing model, Nimbella's licensing model is also the most attractive solution.

• Google uses Cloud Functions and Cloud Run to create "all-round" solutions. From Google’s FaaS products, it can be seen that as public cloud providers use containers and Kubernetes platforms as key platform engines for FaaS runtimes, multiple FaaS platforms have begun to transform. Google's Cloud Functions supports common function packaging and deployment, and its runtime can be well expanded and slowly cooled, and the Google Cloud Ops integration function also ensures excellent observability. Using Google Cloud Run, developers can package functions with Google's Functions Framework, deploy them using Knative, and run them in a mixed deployment scenario. Compared with the evaluation results in 2020, the platform's support for programming languages has also been strengthened.

Reference customers gave Google a high score for performance and simplicity, and praised the security and scalability of Cloud Functions. In addition, the direct participation and support of the product management and customer support teams have also been unanimously recognized by customers. Areas for improvement include: better support for packaging third-party libraries and functions (for example, for processing Kafka messages), and the transparency of the dependencies of the Cloud Functions runtime needs to be strengthened. If you want to deploy applications on a global scale and smoothly transition from functions to containers in hybrid FaaS deployment scenarios, Google Cloud Run should be your right choice.

• IBM completes Cloud Functions and prepares to launch Code Engine. IBM Cloud Functions is based on the open source Apache OpenWhisk project. It adds a built-in API gateway, good event stream support, and powerful container-level function isolation. Judging from the results of this evaluation, IBM is superior to any other manufacturer in terms of service level agreement terms and availability. However, Cloud Functions still needs improvement in some aspects, such as limited network isolation options and support for reserved function instances. In the 2020 evaluation, we pointed out that after IBM acquired Red Hat, it needed to establish a unified function runtime strategy. It seems that IBM Cloud Code Engine will take on this task. Although it has not yet entered the general availability (GA) stage (so it is not included in our analysis), Code Engine has added support for Kubernetes and Knative to IBM's FaaS strategy.

The rapid automatic scaling of IBM's function instances and the high degree of concurrency of computational workloads have been unanimously recognized by reference customers. Other customers pointed out that support for OpenWhisk enables portability, thereby reducing the risk of vendor lock-in. The areas that need improvement include improved orchestration, observability, and support for reserved function instances. Old IBM customers will find Cloud Functions very suitable for integrating their workloads, and its extensive support for multiple programming languages can also effectively help most developers to smoothly transition to a serverless architecture. After entering the GA stage, IBM Cloud Code Engine is also worth looking forward to, especially for customers who are investing in a Kubernetes-based infrastructure and want to incorporate functions into package objects.

competitor

• Oracle puts security, containers, and event-driven integration first. When we analyzed Oracle Cloud Functions in 2020, we believed that the platform was relatively new, and the evaluation results also reflected this. In 2021, Oracle's FaaS platform has been upgraded from the "Challenger" quadrant to the "Competitor" quadrant. Oracle Cloud Functions is based on the open source Fn project, and functions are packaged into OCI containers. When Oracle built this FaaS platform, it mainly focused on the "secure-by-design" model, which also meets the needs of many enterprise development teams. By default, function instances are only available in the subnet of the customer's virtual cloud network. Unlike some other FaaS vendors, Oracle's private cloud support features do not charge additional fees. As a result, Oracle's adoption rate in North America, Europe, Asia and Latin America is steadily increasing.

Reference customers gave high scores on Oracle Cloud Functions' performance, ease of use, development time, depth of documentation, and overall execution cost. The shortcomings are reflected in the need to come up with alternative solutions to solve the occasional delay problem. The possible reason is the current lack of support for reserved instances. Oracle Cloud Functions is very suitable for enterprises that use other OCI services. In addition, the platform has currently opened 23 regions, which can help customers implement deployment on a global scale.

Challenger

• Huawei's FunctionGraph has just entered the market, but it has a pretty good prospect. The functional workflow platform is still in the stage of gradual development, but during the evaluation process, we found that the platform has reached the market average level with its strong support for development tools, local development and event-driven integration. The platform can execute functions with a maximum running time of 15 minutes, and supports multiple programming languages such as Node.js, Java, Python, Go, C#, and PHP. As a new face in the market, this platform still has some shortcomings. For example, the documentation is not yet complete, the support for special workloads is poor, and the support for integration is insufficient. Availability in some areas is also limited. Therefore, we believe that Huawei's functional workflow platform is most suitable for developers who want to deploy workloads in China, or in international markets where Huawei Cloud provides regional support such as Brazil, Chile, Mexico, Singapore, South Africa, or Thailand. Huawei refused to participate in the complete Forrester Wave™ evaluation process.

Overview of the assessment report

We have evaluated manufacturers strictly in accordance with 40 standards, which can be divided into three categories:

• current product. each manufacturer on the vertical axis of the Forrester Wave™ quadrant chart represents the performance of its current product features. The key criteria for evaluating these solutions include the developer's experience, programming model, runtime execution environment, and security features.
• strategic vision. represents the strategic performance of the manufacturer. We evaluated the FaaS platform provider's vision, execution capabilities, ecosystem, participation, and pricing strategy.
• market share. by the size of the circle on the figure. Our market performance score reflects the geographic distribution, customer base size and product revenue of each manufacturer.

Vendor selection criteria

Forrester evaluated 9 vendors this time: Alibaba, Amazon, Google, Huawei, IBM, Microsoft, Nimbella, Oracle and Tencent. They all meet the following conditions:

• has a FaaS platform . Each manufacturer has a FaaS platform that customers can use and run productive workloads at any time.
• participated in Forrester's Now Tech architecture provider evaluation 160cc447a87b1d. The FaaS platform is a key component of building cloud native applications through a serverless architecture. In Forrester's "Now Tech: Serverless Architecture, Q1 2021" report, we detail the serverless architectures developers can choose.

Supplementary materials

Online resources

We organize all Forrester Wave™ scores and weights into an Excel file, which contains detailed product evaluations and supports custom sorting. Click the link at the beginning of the web version of the report on Forrester.com to download the file. We hope that these scores and default weights are only an entry point, and readers are advised to adjust the weights according to their personal needs.

Forrester Wave™ evaluation method

The Forrester Wave™ report provides guidelines for organizations to purchase products in the technology market. To ensure fairness to all participants, Forrester follows the "Forrester Wave™ Methodological Guide" to evaluate participating vendors. In the evaluation process, we first conduct preliminary research and select vendors for later evaluation. Then according to the selection criteria, further screening and determining the final list. Then collect detailed information about the product and strategy through detailed questionnaire surveys, demonstrations/briefings, and reference customer surveys/interviews. Finally, this information, combined with the analyst's industry experience and expertise, is used to score manufacturers through a relative scoring system that compares a manufacturer with other manufacturers participating in the evaluation. We clearly indicate the release date (year and quarter) of the Forrester Wave™ in the title of each Forrester Wave™ report. In the evaluation process of this Forrester Wave™ report, we used the materials provided to us by the manufacturer before January 21, 2021, and do not accept other information provided after this date. We hope that readers can analyze how the market and manufacturer's products have changed over time. According to the "Forrester Wave™ and New Wave™ Vendor Review Policy", Forrester requires vendors to review our evaluation results before the report is released to ensure accuracy. Vendors marked as not participating in the evaluation in the Forrester Wave™ quadrant chart meet the selection criteria defined by us, but refuse to participate or only partially participate in this evaluation. We scored these vendors according to the "Forrester Wave™ and Forrester New Wave™ Non-Participation and Incomplete Participation Vendor Policy" and included them in the quadrant chart along with other vendors participating in the evaluation.

Integrity Code

All our research, including Forrester Wave™ assessments, follow the "Integrity Code" published on our website.

Link to the original report:

https://reprints2.forrester.com/#/assets/2/1948/RES161673/report

Copyright Statement: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.