HowTo Migrate a managed AWS Elasticsearch cluster to graviton2

6-minute read
aws • elasticsearch • graviton2 • terraform

Amazon Web Services (AWS) offers managed Amazon Elasticsearch clusters (or OpenSearch as they call their fork of the open source Elasticsearch) for deploying and managing search applications. Lately, I’ve been managing several self-managed clusters and showed how I’ve migrated those to graviton2; for newer clusters, I’ve started exploring the AWS managed solution for Elasticsearch. This has several advantages:

  • Simplified management: With Amazon OpenSearch, AWS takes care of the infrastructure management tasks, such as deploying and configuring the Elasticsearch cluster, patching the underlying operating system, and handling backups and disaster recovery. This allows you to focus on using Elasticsearch to build your search applications, rather than worrying about the underlying infrastructure.
  • Scalability: Amazon OpenSearch makes it easy to scale your Elasticsearch cluster up or down to meet changing demands. You can add or remove nodes to the cluster with just a few clicks, or use features like auto-scaling to automatically adjust the size of the cluster based on workload patterns.
  • Availability: Amazon OpenSearch is designed for high availability, with built-in replication and failover capabilities. This helps ensure that your search application remains available even in the face of hardware failures or other issues.
  • Security: Amazon OpenSearch provides a range of security features to help protect your data, including encryption at rest and in transit, access controls, and integration with AWS Identity and Access Management (IAM) for fine-grained access control.
  • Integration with other AWS services: Amazon OpenSearch integrates with other AWS services, such as Amazon CloudWatch for monitoring, AWS CloudTrail for auditing, and AWS Identity and Access Management (IAM) for access control. This makes it easy to build end-to-end search applications on AWS.
  • Cost optimization: Using Amazon OpenSearch can help reduce costs by eliminating the need to manage and maintain your own Elasticsearch cluster infrastructure. Additionally, AWS offers Graviton2-based instances that are optimized for running Amazon OpenSearch and provide better performance and cost efficiency compared to traditional x86-based instances.

In this post, I’ll walk you through the process of upgrading an existing managed Amazon Elasticsearch cluster to Graviton2. We will use the same cluster to perform the upgrade, which means we will upgrade the cluster in place, without creating a new cluster.

Step 1: Change the instance types to Graviton2

This migration is much easier than the self-managed one; it requires only one step ;). We first need to figure out the graviton2 instance type we want to use for our migration. As like with regular ec2 instances AWS provides a wide range of instances for Elasticsearch. The main ones are the “m” instances which are general-purpose instances, and the “r” instances are memory-optimized. The number after the instance type (e.g. “2xlarge” or “16xlarge”) indicates the number of vCPUs and the amount of memory available on the instance:

The available Amazon OpenSearch-optimized Graviton2 instances are:

m6g.medium.elasticsearch
m6g.large.elasticsearch
m6g.xlarge.elasticsearch
m6g.2xlarge.elasticsearch
m6g.4xlarge.elasticsearch
m6g.8xlarge.elasticsearch
m6g.12xlarge.elasticsearch
m6g.16xlarge.elasticsearch
r6g.large.elasticsearch
r6g.xlarge.elasticsearch
r6g.2xlarge.elasticsearch
r6g.4xlarge.elasticsearch
r6g.8xlarge.elasticsearch
r6g.12xlarge.elasticsearch
r6g.16xlarge.elasticsearch

Note: we will also need to make sure we run a supported version of the managed AWS Elasticsearch/OpenSearch that supports graviton2 instances. For the older Elasticsearch anything newer than 7.8 should work, and if you are using the OpenSearch version then any version would work as this has been available since version 1.0.0. If you are running an older version you will need first to upgrade to a supported version before moving forward.

The actual migration only requires us to change the instance type. This can be done in the AWS console, using the AWS cli, or a tool like terraform. Since I use terraform to manage all the cloud assets I will show how this is done with terraform; this would look something like:

resource "aws_opensearch_domain" "elasticsearch_domain" {
  domain_name           = "search-domain"
  elasticsearch_version = "7.10"

  cluster_config {
    instance_type           = "m6g.large.elasticsearch" # this replaces the previous m5 type of instance we had
    instance_count          = 3
    dedicated_master_enabled = true
    dedicated_master_count   = 3
  }

  ebs_options {
    ebs_enabled = true
    volume_type = "gp3"
    volume_size = 1000
  }
... # other elasticsearch cluster configs
}

I want to point out also that we are now able to use gp3 for the ebs volume which allows for much better performance and increased size allowed per data node. This is great optimization that can make the cluster much faster and reduce the need for extra data nodes (we were able to cut our nodes in half from this combination: graviton2 for better performance and gp3 for higher storage capacity per node)

Once you run terraform apply with the new instance type this will kick in the automatic blue-green deployment from AWS managed Elasticsearch that will spin up a new set of nodes and migrate the data to the new nodes; once this is done the original nodes are automatically removed. Depending on the size of your data in the cluster this might take a long time and terraform might time out (60m by default). If this happens, you can use the AWS console or cli to monitor the status of the migration.

aws es describe-upgrade --domain-name <domain-name>

should show the status of the upgrade for the specified domain. You can also check the health of the cluster after the upgrade:

aws es describe-elasticsearch-domain --domain-name <domain-name> --query 'DomainStatus.ClusterStatus.Health'

This command will return the current health status of the Elasticsearch cluster. If the upgrade has been completed successfully, the cluster should have a green health status. If there are any issues with the upgrade, the cluster may have a yellow or red health status, indicating that there are problems that need to be addressed

Note: theoretically there should be no downtime during the process, but the performance might be slightly impacted during the blue-green migration.

As you can see there is a huge advantage while performing such a migration using a managed service compared with the self-managed solution where we had to handle and take care of everything ourselves.

Conclusion

Upgrading a managed Amazon Elasticsearch cluster to Graviton2 is a straightforward process that can provide significant benefits. By upgrading to Graviton2 instances, you can improve performance, reduce costs, and increase the efficiency of your infrastructure. AWS offers several Graviton2 instance types optimized for Amazon OpenSearch, each with its own set of advantages.

In this post, I have walked you through the process of upgrading an existing managed Amazon OpenSearch cluster to Graviton2 instances. We have used the same cluster to perform the upgrade, which means we have upgraded the cluster in place, without creating a new cluster. I have also provided examples and command-line steps to help you through the process.

Overall, upgrading your managed Amazon OpenSearch cluster to Graviton2 instances is a great way to take advantage of the latest technology and improve the performance and cost efficiency of your search application.

comments powered by Disqus