Amazon EC2 Serial Console - Simplified and Efficient Troubleshooting of Network Issues

Solving production problems is one of the primary responsibilities of system and network administrators. In fact, I've always found this to be one of the most interesting things about infrastructure engineering. Dig deep into the problem you encounter as needed, and you'll not only be able to (eventually) solve it satisfactorily, but you'll also learn a lot in the process that you wouldn't normally be able to touch.

The operating system certainly has such an opportunity. Operating systems become more complex over time, and administrators must master countless configuration files and settings. While infrastructure-as-code and automation have greatly improved server provisioning and management, errors and failures are always inevitable, preventing systems from starting properly. There are countless types of problems: missing hardware drivers, misconfigured file systems, invalid network configurations, incorrect permissions, and more. To make matters worse, many issues can actually lock out administrators, preventing them from logging in, diagnosing problems, and performing appropriate fixes. The only option is to establish an out-of-band connection to the server. Additionally, while customers can view the console output Amazon EC2 instance, they cannot interact with it until then.

• Console output:
  https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_GetConsoleOutput.html

Recently, Amazon Cloud Technologies is pleased to announce the Amazon EC2 Serial Console, which establishes a serial connection to your Amazon Elastic Compute Cloud (EC2) instance for easy and secure troubleshooting of startup and network connectivity issues .

• Amazon EC2 Serial Console:
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html
• Amazon Elastic Compute Cloud (EC2)：
  https://aws.amazon.com/ec2/

📢 To learn more about the latest technology releases and practical innovations of Amazon Cloud Technology, please pay attention to the 2021 Amazon Cloud Technology China Summit! Click on the image to register now~

Amazon EC2 Serial Console

Amazon EC2 Serial Console Amazon EC2 instances based on Amazon Nitro system It supports all major Linux distributions, FreeBSD, NetBSD, Microsoft Windows and VMWare.

using a browser-based shell in the Amazon Cloud Management Console or via an SSH connection to a hosted console server. No need to run an sshd server on your instance: just root  account as you will need this password to log in. You can then enter commands as if you were using a keyboard and monitor connected directly to the instance's serial port.

• Amazon EC2 Serial Console:
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html
• Amazon Nitro System:
  https://aws.amazon.com/ec2/nitro/

Additionally, you can trigger OS-specific processes:

• On Linux, you can trigger the Magic SysRq command to generate crash dumps, kill processes, etc.
• On Windows, you can use Emergency Management Services (EMS) and Special Administration Console (SAC) to interrupt the startup process and then start in safe mode.

• Magic SysRq：
  https://en.wikipedia.org/wiki/Magic_SysRq_key

Gaining access to the instance console is a privileged operation and should be tightly controlled, so account-level access to Amazon EC2 Serial Console is not allowed by default. After you allow this access in your account, it applies to all instances in this account. With the service control policies and Amazon Identity and Access Management (IAM) permissions, administrators can also exercise control at the organization level and instance level, respectively. As you would expect, Amazon EC2 serial console are encrypted, and we generate a unique key for each session.

• Amazon EC2 Serial Console:
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html
• Service Control Policy:
  https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html
• AWS Identity and Access Management (IAM)：
  https://aws.amazon.com/iam/
• Amazon EC2 Serial Console:
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html

Let's use Linux as a quick demonstration. The process is similar when using other operating systems.

through the Amazon Cloud Technology Management Console

Amazon EC2 Serial Console

First, I launched an Amazon Linux 2 instance. After logging into the instance, I decided to manage the network configuration of its ethernet interface (
/etc/sysconfig/network-scripts/ifcfg-eth0
) to set a completely fictitious static IP address. Please don't try this on a production instance!

Then, I restart the instance. After a few seconds, I can't connect to the instance using SSH, although the instance is up and running in the Amazon EC2 console, and port 22 is open in its security group.

1$ ssh -i ~/.ssh/mykey.pem [e-mail]
2ssh: connect to host ec2-3-238-8-46.compute-1.amazonaws.com port 22: Operation timed out

161e8bb2076d0b Amazon EC2 serial console

• Amazon EC2 Serial Console:
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html

First, I need to allow access to the console in my account. This can be done by simply ticking a box in the Amazon EC2 settings.

Then right-click the instance's name in the Amazon EC2 console, choose Monitor and troubleshoot , then choose Amazon EC2 Serial Console (EC2 Serial Console).

A new window will open, confirm the instance ID and the serial port number to connect to. I just click Connect (connect).

A new tab will open in my browser. Hit Enter and I'll see the familiar login prompt.

1Amazon Linux 2
2Kernel 4.14.225-168.357.amzn2.x86_64 on an x86_64
3ip-172-31-67-148 login:

root logging in as 061e8bb2076dd9 and entering the familiar shell prompt interface, I immediately felt relieved.

To enable Magic SysRq ( sysctl -w kernel.sysrq=1 ) for this session, I first list the available commands ( CTRL-0  + h ), then ask for a memory report ( CTRL-0 +  m ). You can click on the image below for a larger view.

marvelous! This will definitely come in handy when solving complex problems. Of course this is not needed here: I quickly restore the normal configuration of the network interface, then restart the network stack.

Try connecting to the instance again and I can see that the problem is resolved.

1$ ssh -i ~/.ssh/mykey.pem [e-mail]
2
3__|   __|_  )
4_|   (    / Amazon Linux 2 AMI
5___|\___|___|
6
7https://aws.amazon.com/amazon-linux-2/
8[ec2-user@ip-172-31-67-148 ~]$

Below I quickly demonstrate the equivalent command below when using the Amazon Cloud Technologies command line interface

• Command Line Interface:
  https://aws.amazon.com/cli/

via the Amazon CLI

Amazon EC2 Serial Console

It's just as easy. First, I send the SSH public key of the instance key pair to the serial console. Be sure to add the file:// prefix.

1$ aws ec2-instance-connect send-serial-console-ssh-public-key --instance-id i-003aecec198b537b0 --ssh-public-key file://~/.ssh/mykey.pub --serial-port 0 --region us-east-1

I then <instance id>.port<port number>  and got a login prompt.

1$ ssh -i ~/.ssh/mykey.pem [e-mail]
2
3Amazon Linux 2
4Kernel 4.14.225-168.357.amzn2.x86_64 on an x86_64
5ip-172-31-67-148 login:

After logging in, Magic SysRq is already available and I can trigger it  ~B + command I can also use ~. to kill the console session.

started with the Amazon EC2 serial console

As you can see, the Amazon EC2 serial console makes it easier for you to debug and fix complex launch and network issues on Amazon EC2 instances. You can start using this console immediately at no additional cost in the following Amazon cloud technology regions:

• US East (N. Virginia), US West (Oregon), US East (Ohio)
• Europe (Ireland), Europe (Frankfurt)
• Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Singapore)

Try it now and give us your feedback. We always look forward to your feedback. You can send feedback through your usual Amazon Support contacts, or through the Amazon Cloud Forum

• Amazon EC2 Serial Console:
  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console.html
• Amazon Cloud Tech Forum for Amazon EC2:
  https://forums.aws.amazon.com/forum.jspa?forumID=30