OpenVidu High Availability: AWS Configuration and Administration#

Warning

While in BETA this section is subject to changes. We are working to simplify the configuration and administration of OpenVidu High Availability.

The deployment of OpenVidu High Availability on AWS is automated using AWS CloudFormation, with Master and Media Nodes managed within an Auto Scaling Group. The Auto Scaling Group of Master Nodes is fixed to 4 instances while the Auto Scaling Group of Media Nodes is configured to scale based on the target average CPU utilization.

Internally, the AWS deployment mirrors the on-premises setup, allowing you to follow the same administration and configuration guidelines provided in the On Premises High Availability documentation. However, there are specific considerations unique to the AWS environment that are worth taking into account.

Cluster Shutdown and Startup#

You can start and stop the OpenVidu High Availability cluster at any time. The following sections detail the procedures.

Shutdown the ClusterStartup the Cluster

To shut down the cluster, you need to stop the Media Nodes first and then stop the Master Nodes; this way, any ongoing session will not be interrupted.

Navigate to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
In the "Resources" tab, locate the resource with the logical ID: OpenViduMediaNodeASG, and click on it to go to the Auto Scaling Group Dashboard with the Auto Scaling Group of the Media Nodes selected.
Click on "Actions > Edit".
Set the "Desired capacity", "Min desired capacity", and "Max desired capacity" to 0, and click on "Update".
Wait until the "Instance Management" tab shows that there are no instances in the Auto Scaling Group.

Warning

It may happen that some instances are still in the "Terminating:Wait" lifecycle state after setting the desired capacity to 0. This is because the Auto Scaling Group waits for the instances to finish processing any ongoing room, ingress, or egress operations before terminating them. This can take a few minutes. If you want to force the termination of the instances, you can manually terminate them from the EC2 Dashboard.
After confirming that all Media Node instances are terminated, go back to the CloudFormation Stack and locate the resource with the logical ID: OpenViduMasterNodeASG. Click on it to go to the Auto Scaling Group Dashboard with the Auto Scaling Group of the Master Nodes selected.
Click on "Actions > Edit".
Set the "Desired capacity", "Min desired capacity", and "Max desired capacity" to 0, and click on "Update".
Wait until the "Instance Management" tab shows that there are no instances in the Auto Scaling Group.

To start the cluster, we recommend starting the Master Node first and then the Media Nodes.

Navigate to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
Locate the resource with the logical ID: OpenViduMasterNodeASG. Click on it to go to the Auto Scaling Group Dashboard with the Auto Scaling Group of the Master Nodes selected.
Click on "Actions > Edit".
Set the "Desired capacity", "Min desired capacity", and "Max desired capacity" to the desired number of Media Nodes, and click on "Update". For the Master Nodes auto scaling group, the number of instances must be 4.
Wait until the "Instance Management" tab shows that there are the desired number of instances in the Auto Scaling Group.
Go back to the CloudFormation Stack and locate the resource with the logical ID: OpenViduMediaNodeASG. Click on it to go to the Auto Scaling Group Dashboard with the Auto Scaling Group of the Media Nodes selected.
Click on "Actions > Edit".
Set the "Desired capacity", "Min desired capacity", and "Max desired capacity" to the desired number of Media Nodes, and click on "Update". In this example, we set the desired capacity to 2.
Wait until the "Instance Management" tab shows that there are the desired number of instances in the Auto Scaling Group.

Change the instance type#

It is possible to change the instance type of both the Master Node and the Media Nodes. The following section details the procedures.

Master NodesMedia Nodes

Warning

This procedure requires downtime, as it involves stopping all the Master Nodes and starting them again with the new instance type.

Go to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
Locate the resource with the logical ID: OpenViduMasterLaunchTemplate. Click on it to go to the Launch Template Dashboard with the Launch Template of the Master Nodes selected.
Click on "Actions > Modify template (Create new version)".
In the "Instance type" section, select the new instance type and click on "Create template version".
Go to the CloudFormation Stack and locate the resource with the logical ID: OpenViduMasterNodeASG. Click on it to go to the Auto Scaling Group Dashboard with the Auto Scaling Group of the Master Nodes selected.
Click on "Actions > Edit".
In the Launch Template section, select the new version of the launch template we just created at step 5 which is the highest version number.

Then, click on "Update".

Info

By configuring "Latest" as the launch template version, you no longer need to update the Auto Scaling Group every time you modify the launch template. The Auto Scaling Group will automatically use the latest version of the launch template.
Terminate the old instances manually from the EC2 Dashboard if you want to force the termination of the instances. New instances will be launched with the new instance type.

Info

The information at /opt/openvidu is persisted as AWS Elastic Block Store (EBS) volumes. When you terminate an instance, the EBS volume is detached and preserved. When the Auto Scaling Group launches a new instance, the EC2 instance is attached to the EBS volume, and the data is retained. This means that the data stored in the /opt/openvidu directory is not lost when you terminate an instance.

Go to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
Locate the resource with the logical ID: OpenViduMediaNodeLaunchTemplate. Click on it to go to the Launch Template Dashboard with the Launch Template of the Media Nodes selected.
Click on "Actions > Modify template (Create new version)".
In the "Instance type" section, select the new instance type and click on "Create template version".
Go to the CloudFormation Stack and locate the resource with the logical ID: OpenViduMediaNodeASG. Click on it to go to the Auto Scaling Group Dashboard with the Auto Scaling Group of the Media Nodes selected.
Click on "Actions > Edit".
In the Launch Template section, select the new version of the launch template we just created at step 5 which is the highest version number.

Then, click on "Update".

Info

By configuring "Latest" as the launch template version, you no longer need to update the Auto Scaling Group every time you modify the launch template. The Auto Scaling Group will automatically use the latest version of the launch template.
Terminate the old instances manually from the EC2 Dashboard if you want to force the termination of the instances. New instances will be launched with the new instance type.

Info

If you want to avoid downtime, you can wait until the Auto Scaling Group replaces the old instances with the new ones. But you will need to increase the desired capacity to force the replacement of the instances and then decrease it to the desired number of instances.

Media Nodes Autoscaling Configuration#

To configure the Auto Scaling settings for the Media Nodes, follow the steps outlined below. This configuration allows you to set the parameters that control how the Auto Scaling Group will scale based on the target average CPU utilization.

Media Nodes Autoscaling Configuration

Navigate to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
In the "Resources" tab, locate the resource with the logical ID: OpenViduMediaNodeASG and click on it to go to the Auto Scaling Group Dashboard.
Click on "Actions > Edit".
To configure scaling policies, navigate to the "Automatic scaling" tab within the Auto Scaling Group Dashboard, select the unique "Target tracking scaling" autoscaling policy, and click on "Actions > Edit".
It will open a panel where you can configure multiple parameters. In this example, we set the target average CPU utilization to 30%. Then, click on "Update".

Info

OpenVidu High Availability is by default configured with a "Target tracking scaling" policy that scales based on the target average CPU utilization, however, you can configure different autoscaling policies according to your needs. For more information on the various types of autoscaling policies and how to implement them, refer to the AWS Auto Scaling documentation.

Fixed Number of Media Nodes#

If you need to maintain a fixed number of Media Nodes instead of allowing the Auto Scaling Group to dynamically adjust based on CPU utilization, you can configure the desired capacity settings accordingly. Follow the steps below to set a fixed number of Media Nodes:

Set Fixed Number of Media Nodes

Navigate to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
In the "Resources" tab, locate the resource with the logical ID: OpenViduMediaNodeASG and click on it to go to the Auto Scaling Group Dashboard.
Click on "Actions > Edit".
Set the "Desired capacity", "Min desired capacity", and "Max desired capacity" to the fixed number of Media Nodes you require, and click on "Update". In this example, we set the desired capacity to 2.
Wait until the "Instance Management" tab shows that the Auto Scaling Group has the fixed number of instances running.

Configuration Management#

This section explains how to manage and update the configuration settings for your OpenVidu High Availability deployment. It is divided into three parts:

Configuring CloudFormation YAML for Node Services Configuration: Details how to pre-configure settings in the CloudFormation template to avoid manual interventions post-deployment.
Global Configuration: Covers parameters that affect the entire cluster when the CloudFormation stack is already deployed.
Node Services Configuration: Focuses on settings specific to Master and Media Nodes services when the CloudFormation stack is already deployed.

Configuring CloudFormation YAML for Node Services Configuration#

To avoid manual intervention after deployment, you can pre-configure the node services configuration directly in the CloudFormation YAML template. This ensures that the necessary configurations are applied automatically upon deployment.

Master NodeMedia Nodes

Get the CloudFormation YAML template used to deploy OpenVidu High Availability on AWS.

Locate the section defining the Launch Template for the Master Node and update the UserData property with the required configuration parameters. The section looks like this:

OpenViduMasterLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
        LaunchTemplateData:
            UserData:
                Fn::Base64: !Sub |
                    #!/bin/bash
                    ...
                    ...
                    # Install OpenVidu
                    /usr/local/bin/install.sh || { /usr/local/bin/set-as-unhealthy.sh; exit 1; }

                    ######### APPLY CUSTOM CONFIGURATIONS #########
                    # If you want to apply any modification to the configuration files
                    # of the OpenVidu services at /opt/openvidu, you can do it here.
                    # Examples:
                    #
                    # - Change minio public port
                    # yq eval '.apps.http.servers.minio.listen[0] = ":9001"' -i /opt/openvidu/caddy.yaml
                    #
                    # - Disable dashboard access
                    # yq eval 'del(.apps.http.servers.public.routes[] | \
                    #  select(.handle[]?.handler == "subroute" and \
                    #  .handle[].routes[].handle[].strip_path_prefix == "/dashboard"))' \
                    #  -i /opt/openvidu/caddy.yaml



                    ######### END CUSTOM CONFIGURATIONS #########

                    # Start OpenVidu
                    systemctl start openvidu || { /usr/local/bin/set-as-unhealthy.sh; exit 1; }
                    ...
                    ...

The area between APPLY CUSTOM CONFIGURATIONS and END CUSTOM CONFIGURATIONS is where you can add your custom configuration commands. You can use yq to modify the configuration files of the OpenVidu services. The example shows how to change the minio public port and how to disable dashboard access in the caddy.yaml configuration file.

Save the YAML file and use it to deploy your CloudFormation stack. This will apply the Master Node configuration automatically during the deployment process.

Get the CloudFormation YAML template used to deploy OpenVidu High Availability on AWS.

Locate the section defining the Launch Template for the Media Nodes and update the UserData property with the required configuration parameters. The section looks like this:

OpenViduMediaNodeLaunchTemplate:
    Type: "AWS::EC2::LaunchTemplate"
    Properties:
        LaunchTemplateData:
            UserData:
                Fn::Base64: !Sub |
                    #!/bin/bash
                    ...
                    ...
                    # Install OpenVidu
                    /usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }

                    ######### APPLY CUSTOM CONFIGURATIONS #########
                    # If you want to apply any modification to the configuration files
                    # of the OpenVidu services at /opt/openvidu, you can do it in this section.
                    # Examples:
                    #
                    # - Announce specific IP address for the media node
                    # yq eval '.rtc.node_ip = 1.2.3.4' -i /opt/openvidu/livekit.yaml
                    #
                    # - Add webhooks to livekit
                    # yq eval '.webhook.urls += ["http://new-endpoint.example.com/webhook"]' -i /opt/openvidu/livekit.yaml



                    ######### END CUSTOM CONFIGURATIONS #########

                    # Start OpenVidu
                    systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }

The area between APPLY CUSTOM CONFIGURATIONS and END CUSTOM CONFIGURATIONS is where you can add your custom configuration commands. You can use yq to modify the configuration files of the OpenVidu services. The example shows how to change the rtc.node_ip parameter and how to add a webhook to the livekit.yaml configuration file.

Save the YAML file and use it to deploy your CloudFormation stack. This will apply the node services configuration automatically during the deployment process.

By pre-configuring the CloudFormation template, you streamline the deployment process and reduce the need for post-deployment manual configuration.

Global Configuration#

The global configuration parameters for the OpenVidu High Availability deployment are stored in a secret resource deployed by the CloudFormation stack. These parameters can affect the configuration of all the nodes in the cluster. To update any of these parameters, follow the steps below:

Update Global Configuration Parameters

Navigate to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
In the "Resources" tab, locate the resource with the logical ID: OpenViduSharedInfo and click on it to view the secret in AWS Secrets Manager.
Click on "Retrieve secret value" to view the parameters.
Edit the desired parameters within the secret. For example, you can change the RTC_ENGINE parameter to pion or mediasoup depending on the WebRTC engine you want to use. Just click on "Edit", modify the value, and click on "Save".
To apply the changes, stop the cluster and then start it again following the procedures outlined in the Cluster Shutdown and Startup section.

Node Services Configuration#

The node services configuration parameters for the OpenVidu High Availability deployment are stored in the configuration files of the services running on the Master and Media Nodes. These parameters can affect the configuration of the individual nodes in the cluster. To update any of these configuration files, follow the steps below:

Master NodeMedia Node

Master Node configurations require changes to be made in the Launch Template "User Data" section. To update the configuration:

Navigate to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
Locate the resource with the logical ID: OpenViduMasterLaunchTemplate. Click on it to go to the Launch Template Dashboard with the Launch Template of the Master Node selected.
Click on "Actions > Modify template (Create new version)".
Go to the "Advanced details" section and modify the "User data" field with the new configuration. You can modify the configuration parameters of the services running on the Master Node following the same script structure as the one used in the Automatic installation and configuration of nodes, for the Master Node. When you finish, click on "Create template version".
Go to the CloudFormation Stack and locate the resource with the logical ID: OpenViduMasterNodeASG. Click on it to go to the EC2 Dashboard with the Master Node instance selected.
Click on "Actions > Edit".
In the Launch Template section, select the new version of the launch template we just created at step 5 which is the highest version number. Then, click on "Update".

Info

By configuring "Latest" as the launch template version, you no longer need to update the Auto Scaling Group every time you modify the launch template. The Auto Scaling Group will automatically use the latest version of the launch template.
Terminate the old instances manually from the EC2 Dashboard. New instances will be launched with the new configuration.

Warning

This process requires downtime, as it involves terminating the old instances and launching new ones with the updated configuration.

Media Node configurations require changes to be made in the Launch Template "User Data" section. To update the configuration:

Navigate to the CloudFormation Dashboard on AWS.
Select the CloudFormation Stack that you used to deploy OpenVidu High Availability.
Locate the resource with the logical ID: OpenViduMediaNodeLaunchTemplate. Click on it to go to the Launch Template Dashboard with the Launch Template of the Media Nodes selected.
Click on "Actions > Modify template (Create new version)".
Go to the "Advanced details" section and modify the "User data" field with the new configuration. You can modify the configuration parameters of the services running on the Media Nodes following the same script structure as the one used in the Automatic installation and configuration of nodes, for the Media Nodes. When you finish, click on "Create template version".
Go to the CloudFormation Stack and locate the resource with the logical ID: OpenViduMediaNodeASG. Click on it to go to the Auto Scaling Group Dashboard with the Auto Scaling Group of the Media Nodes selected.
Click on "Actions > Edit".
In the Launch Template section, select the new version of the launch template we just created at step 5 which is the highest version number. Then, click on "Update".

Info

By configuring "Latest" as the launch template version, you no longer need to update the Auto Scaling Group every time you modify the launch template. The Auto Scaling Group will automatically use the latest version of the launch template.
Terminate the old instances manually from the EC2 Dashboard if you want to force the termination of the instances. New instances will be launched with the new configuration.

Warning

This process requires downtime, as it involves terminating the old instances and launching new ones with the updated configuration.

Enabling Webhooks#

A common use case for custom configuration is to enable webhooks. To enable webhooks you need to add the webhook URL to every Media Node. As the Media Nodes are managed by an Auto Scaling Group, you need to update the Launch Template to include the webhook URL. Just follow the instructions at the Node Services Configuration to update the Launch Template with the webhook URL, specifically add this command to the UserData section:

yq eval '.webhook.urls += ["<YOUR_WEBHOOK_URL"]' \
    -i /opt/openvidu/config/livekit.yaml

Where <YOUR_WEBHOOK_URL> is the URL of the webhook you want to enable. Remember to terminate the old instances manually from the EC2 Dashboard if you want to force the termination of the instances. New instances will be launched with the updated configuration.

In case you want to enable webhooks before the deployment, you can pre-configure the CloudFormation YAML template with the webhook URL. Just follow the instructions at the Configuring CloudFormation YAML for Node Services Configuration to update the Launch Template with the webhook URL.

Enabling OpenVidu v2 webhooks (v2compatibility)#

If you are using the v2compatibility module, if you want to enable webhooks, you need to add the webhook URL in the /opt/openvidu/.env file of every Master Node. As the Master Nodes are managed by an Auto Scaling Group, you need to update the Launch Template to include the webhook URL. Just follow the instructions at the Node Services Configuration to update the Launch Template with the webhook URL, specifically add this command to the UserData section:

sed -i \
    's/V2COMPAT_OPENVIDU_WEBHOOK_ENDPOINT=.*/V2COMPAT_OPENVIDU_WEBHOOK_ENDPOINT="<YOUR_WEBHOOK_URL>"/' \
    /opt/openvidu/.env

Where <YOUR_WEBHOOK_URL> is the URL of the webhook you want to enable. After that, you need to destroy the Master Node instances and let the Auto Scaling Group launch new instances with the updated configuration.

In case you want to enable webhooks before the deployment, you can pre-configure the CloudFormation YAML template with the webhook URL. Just follow the instructions at the Configuring CloudFormation YAML for Node Services Configuration to update the Master Node Launch Template with the webhook URL.