KubeVela: How to Quickly Introduce AWS's 50 Most Popular Cloud Resources with 100 Lines of Code

Author: KubeVela Community

KubeVela currently supports cloud vendors such as AWS, Azure, GCP, Alibaba Cloud, Tencent Cloud, Baidu Cloud, and UCloud, and also provides simple and quick command-line tools [1] to introduce cloud resources from cloud service providers, but in KubeVela one Supporting the cloud resources of cloud service providers alone is not conducive to quickly meeting users' needs for cloud resources. This article provides a solution to quickly introduce the top 50 most popular cloud resources in AWS with less than 100 lines of code.

At the same time, we also expect users to be inspired by this article and contribute cloud resources from other cloud service providers.

Where are AWS's most popular cloud resources?

Terraform's official website provides Terraform modules of various cloud service providers, such as AWS's cloud resources Terraform modules[2]. Among them, cloud resources are sorted by popularity (downloads), for example, AWS VPC has 18.7 million downloads.

Through simple analysis, we found that the data of AWS top 50 Terraform modules can be accessed through request https://registry.terraform.io/v2/modules?filter%5Bprovider%5D=aws&include=latest-version&page%5Bsize%5D=50&page%5Bnumber %5D=1 get.

before the start

The code accepts two user-passed parameters:
• the name of the provider
• The URL of the Terraform Modules corresponding to this provider

For AWS, the provider name is "aws", and the corresponding Terraform modules are the Terraform Modules json format interface [3] (that is, search the Terraform Registry [4] for the most popular 50 cloud resources when the provider is aws).
Before executing the code, you need to confirm that the providerName(aws) and Modules links are correct.

execute code

Then you can quickly introduce the top 50 most popular AWS cloud resources in batches through the following 100 or so lines of code (file name gen.go).

import (
  "encoding/json"
  "fmt"
  "io"
  "log"
  "net/http"
  "os"
  "os/exec"
  "path/filepath"
  "strings"
  "github.com/pkg/errors"
)
type TFDownload struct {
  Data     []DataItem     `json:"data"`
  Included []IncludedItem `json:"included"`
}
type IncludedItem struct {
  Id         string     `json:"id"`
  Attributes Attributes `json:"attributes"`
}
type DataItem struct {
  Attributes    Attributes    `json:"attributes"`
  Relationships Relationships `json:"relationships"`
}
type Relationships struct {
  LatestVersion RelationshipLatestVersion `json:"latest-version"`
}
type RelationshipLatestVersion struct {
  Data RelationshipData `json:"data"`
}
type RelationshipData struct {
  Id string `json:"id"`
}
var errNoVariables = errors.New("failed to find main.tf or variables.tf in Terraform configurations")
type Attributes struct {
  Name        string `json:"name"`
  Downloads   int    `json:"downloads"`
  Source      string `json:"source"`
  Description string `json:"description"`
  Verified    bool   `json:"verified"`
}
func main() {
  if len(os.Args) < 2 {
     fmt.Println("Please provide the cloud provider name and an official Terraform modules URL")
     os.Exit(1)
  }
  providerName := os.Args[1]
  terraformModulesUrl := os.Args[2]
  resp, err := http.Get(terraformModulesUrl)
  if err != nil {
     log.Fatal(err)
  }
  defer resp.Body.Close()
  body, err := io.ReadAll(resp.Body)
  if err != nil {
     log.Fatal(err)
  }
  var modules TFDownload
  if err := json.Unmarshal(body, &modules); err != nil {
     fmt.Println(err.Error())
     os.Exit(1)
  }
  if _, err = os.Stat(providerName); err == nil {
     if err := os.RemoveAll(providerName); err != nil {
        log.Fatal(err)
     }
     fmt.Printf("Successfully deleted existed directory %s\n", providerName)
  }
  if _, err = os.Stat(providerName); os.IsNotExist(err) {
     if err := os.Mkdir(providerName, 0755); err != nil {
        if !os.IsExist(err) {
           log.Fatal(err)
        }
        fmt.Printf("Successfully created directory %s\n", providerName)
     }
  }
  for _, module := range modules.Data {
     var description string
     for _, attr := range modules.Included {
        if module.Relationships.LatestVersion.Data.Id == attr.Id {
           description = attr.Attributes.Description
        }
     }
     if description == "" {
        description = strings.ToUpper(providerName) + " " + strings.Title(module.Attributes.Name)
     }
     outputFile := fmt.Sprintf("%s/terraform-%s-%s.yaml", providerName, providerName, module.Attributes.Name)
     if _, err := os.Stat(outputFile); !os.IsNotExist(err) {
        continue
     }
     if providerName == "aws" && (module.Attributes.Name == "rds" || module.Attributes.Name == "s3-bucket" ||
        module.Attributes.Name == "subnet" || module.Attributes.Name == "vpc") {
        continue
     }
     if err := generateDefinition(providerName, module.Attributes.Name, module.Attributes.Source, "", description); err != nil {
        fmt.Println(err.Error())
        os.Exit(1)
     }
  }
}
func generateDefinition(provider, name, gitURL, path, description string) error {
  defYaml := filepath.Join(provider, fmt.Sprintf("terraform-%s-%s.yaml", provider, name))
  cmd := fmt.Sprintf("vela def init %s --type component --provider %s --git %s.git --desc \"%s\" -o %s",
     name, provider, gitURL, description, defYaml)
  if path != "" {
     cmd = fmt.Sprintf("%s --path %s", cmd, path)
  }
  fmt.Println(cmd)
  stdout, err := exec.Command("bash", "-c", cmd).CombinedOutput()
  if err != nil {
     return errors.Wrap(err, string(stdout))
  }
  fmt.Println(string(stdout))
  return nil

Excuting an order:

go run gen.go aws "https://registry.terraform.io/v2/modules?filter%5Bprovider%5D=aws&include=latest-version&page%5Bsize%5D=50&page%5Bnumber%5D=1"

Code Brief Description

Parse cloud resource data

Access the URL passed in by the user, and parse the returned json data into a structure in Go.

The json format corresponding to the resource is as follows:

{
  "data": [
    {
      "type": "modules",
      "id": "23",
      "attributes": {
        "downloads": 18440513,
        "full-name": "terraform-aws-modules/vpc/aws",
        "name": "vpc",
        "namespace": "terraform-aws-modules",
        "owner-name": "",
        "provider-logo-url": "/images/providers/aws.png",
        "provider-name": "aws",
        "source": "https://github.com/terraform-aws-modules/terraform-aws-vpc",
        "verified": true
      },
      "relationships": {
        "latest-version": {
          "data": {
            "id": "142143",
            "type": "module-versions"
          }
        }
      },
      "links": {
        "self": "/v2/modules/23"
      }
    },
    ...
  ],
  "included": [
    {
      "type": "module-versions",
      "id": "36806",
      "attributes": {
        "created-at": "2020-01-03T11:35:36Z",
        "description": "Terraform module Terraform module for creating AWS IAM Roles with heredocs",
        "downloads": 260030,
        "published-at": "2020-02-06T06:26:08Z",
        "source": "",
        "tag": "v2.0.0",
        "updated-at": "2022-02-22T00:45:44Z",
        "version": "2.0.0"
      },
      "links": {
        "self": "/v2/module-versions/36806"
      }
    },
    ...
  ],
  ...
}

In the json data corresponding to Modules, we only care about two key-value pairs, namely:

• data: a list containing the Modules names and properties
• Included: Filtered out specific version of Modules specific information

Among them, for each Module element in data, parse its attributes, Id and the id corresponding to latest-version in relationship; for each Module version element in Included, parse its attributes and Id.

The attributes are further parsed into the following five items:

• Name
• Downloads
• Source
• Description
• Verified

The structure is defined in the structure TFDownload, the json data is obtained through the http library, and the structure of Terraform modules is parsed through json.Unmarshal.

Mass production of cloud resources

1. Create a new directory and generate the required files for the resource

After parsing, create a new folder in the current directory, and name the folder the provider name.
Traverse the parsed data, and for each Module element, perform the following operations to generate corresponding configuration files, definitions and corresponding documents for it.

2. Generate definition file

Use the following vela instructions to read the corresponding information from the github repository corresponding to the module to generate a definition file.

vela def init {ModuleName} --type component --provider {providerName} --git {gitURL} --desc {description} -o {yamlFileName}

Several items that need to be filled in the instruction are passed in by the parsed Module structure.

• gitURL: {Module.Attributes.Source}.git

• description: If the element ID in Included is the same as the corresponding ID of latest-version in the module relationship, then description is the description of the corresponding element attribute in Included; otherwise, description is the concatenation of providerName and module name

• yamlFileName：terraform-{providerName}-{Module.Attributes.Name}.yaml

Are you going to try it too?

There are also many cloud service providers that provide rich Terraform modules, such as

GCP：
https://registry.terraform.io/namespaces/terraform-google-modules

Ali Cloud:
https://registry.terraform.io/namespaces/terraform-alicloud-modules

Do you want to also introduce the cloud resources of the cloud service provider you are using or like to KubeVela?

Related Links

[1] Simple and fast command line tool

https://kubevela.io/docs/next/platform-engineers/components/component-terraform

[2] AWS's cloud resources Terraform modules

https://registry.terraform.io/namespaces/terraform-aws-modules

[3] Terraform Modules json format interface

https://registry.terraform.io/v2/modules?filter%5Bprovider%5D=aws&include=latest-version&page%5Bsize%5D=50&page%5Bnumber%5D=1

[4] Terraform Registry

https://aregistry.terraform.io/