Planning and pre-requisites

Choosing the deployment environment type

Quobis communication platform supports on-premise or cloud scenarios, from local and private deployment to global and multi location deployments depending on your organization needs.

  • On premises: deployments use your own network, hardware and hypervisor. You will need to consider that compromised data will be stored in your private infrastructure with restricted access, providing a higher level of security. On premise deployments will need an appropriate server where to host the VMs and the proper network configuration. If you already have a mature IT network with servers in the appropriate locations, this may be a suitable option. If you want to install on-premises, you will need to install a hypervisor on your servers to manage the deployment and configure the VMs to setup the Kubernetes cluster. Quobis communication platform currently supports VMware hypervisor only.

  • Cloud deployments: the supported public cloud provider as of today is Amazon Web Services (AWS). This option does not require hardware infrastructures, you just have to reach an agreement with the cloud service provider, supplying an easy and fast way to set up the environment to deploy the platform.

Kubernetes cluster sizing

A Kubernetes cluster consists of a set of worker machines, called nodes, that run containerized applications. Every cluster has at least one worker node which host the Pods, that are the components of the application workload. The control plane manages the worker nodes and the Pods in the cluster. In production environments, the control plane usually runs across multiple computers and a cluster usually runs multiple nodes, providing fault-tolerance and high availability.

The performance of the Quobis communication platform is determined by the number of nodes and the amount of resources of each node, in terms of CPU and memory. Depending on the number of nodes and the resources used by each of them the number of calls managed at time by the system may vary significantly. Additionally, Kubernetes provides to the cluster the capability to support High Availability, which means a seamless service when a node breaks down. This scenario needs, at least, 3 master nodes and 2 worker nodes in order to balance the workload between the active nodes.

You also must take into account the deployment location to improve the conference’s performance. The server should be located as close as possible to the main base or your organization due to the time issues. For instance, choose the AWS EU-region if your organization base is located in Europe.

Depending on the type of deployment, the number of current calls and the number of provisioned users, the number of nodes and their resources must be chosen accordingly.

  • For a high-availability deployment, 3 master nodes and at least 2 worker nodes are required

  • For a standalone deployent, 1 master node and at least 2 worker nodes are required

Hardware requirements

There are no specific hardware or vendor constrains as Quobis Communication Platform can run on top of virtual machines (VMs). Just make sure that the hardware sizing is appropriate for the VW specified below.

On-premises deployments

  • The selected hypervisor must be VWMare

  • Supported OS: Debian 9

  • Master nodes: at least 4 vCPU, 8 GB RAM, at least 100GB HDD

  • Worker nodes: at least 8 vCPU, 16 GB RAM, at least 100GB HDD

  • Storage: a Network File System (NFS) server is provided within the product to store all the persistent data used by the cluster.

Cloud deployments

  • The selected hypervisor must be the AWS virtualization platform and EC2 instances

  • Supported OS: Debian 9 or higher, Ubuntu 16 or higher

  • Master nodes: AWS t3a.medium instances, 2 vCPU, 4 GB RAM, EBS storage, up to 5Gbps bandwidth

  • Worker nodes: AWS c5.2xlarge instances, 8 vCPU, 16 GB RAM, EBS storage, up to 10Gbps bandwidth, up to 4750Mbps EBS throughput

  • Storage: Amazon Elastic File System (EFS) standard is the solution used to provide persistent storage in Amazon Web Services deployments.

Sizing of media server nodes

The Quobis platform deals with video and audio in different ways, as explained in detail in the media management section:

  • The audio is managed in a MCU (Media Control Unit), also referred as audiomixer. The audio coming from all the participants is mixed here.

  • The video is managed in an SFU (Selective Forwarding Unit). The video coming from all the participants is relayed to the rest of participants.

There are two services which consume CPU cycles in the media server nodes:

  • Audiomixer: it consumes CPU mainly in two tasks:

    • Opus transcoding: the media coming from the WebRTC endpoints (browsers and mobile SDKs) needs to be transcoded before being mixed. Opus transcoding is a CPU-intensive task and the one which porcentually consumes more CPU cycles. If PCMU/PCMA (G711) is used instead of Opus, the CPU consumption is much lower.

    • Audiomixing: the mixing of audio, once transcoded, also requires CPU real-time processing.

  • SFU: it consumes CPU to keep the media connections open between browsers/mobile apps and the SFU. These connections are called PeerConnections in WebRTC terminology. The number of open PeerConnections increases linearly with the number of simultaneous media sessions and pseudo-exponentially with the number of participants in each session.

Taking into account the factors above and an average distribution of users in conferences with different numbers of participants, we can consider the following sizing rules:

  • One node (8vCPU) for each 200cc using Opus

  • One node (8vCPU) for each 400cc using PCMU/PCMA.

Sizing of application server nodes

Besides the media processing, the CPU consumption of the application pods must also be considered. The CPU processing will depend on several factors:

  • the number of simultaneous active sessions

  • the number of concurrent calls.

  • the number of calls per secon

  • use of the presence service. Presence can be an issue in large setups where each user receives an update with any presence change. So a huge average number of contacts per user will lead to a higher CPU utilization.

  • provisioning processes through the REST API may lead to peak CPU usage.

  • Sessions from browser-based apps that remain active while the web app is open.

  • Sessions from mobile apps that keep the session active only while the app is in foreground.

Taking into account the factors above, we can consider the following sizing rules as a rule of thumb:

  • one 8vCPU node per 1,000 simultaneous sessions when presence is enabled.

  • one 8vCPU node per 2,000 simultaneous sessions when presence is not enabled.

Please consider the rules above as references, since the load may differ depending on the use cases. An active monitoring of the platform may is quite userful to adjust the number of nodes.

Network requirements

Firewall rules and security considerations

Quobis Communication Platform needs communication between cluster nodes (master and workers) and also needs to expose different ports to allow voice and video calls. The following table summarizes the required network ports to configure:

Networking requirements

Req #

Origin

Destination

Destination IP

Destination ports

Protocol

Service

Encryption

Description

1

WebRTC clients

AS

AS IP

80,443

TCP

WSS

SSL

Application traffic

2

WebRTC clients

MS

IP GW-SFU

20000-29999

UDP

DTLS-SRTP

SSL

WebRTC media

3

WebRTC clients

MS

IP GW-TURN

3478

UDP

STUN

Unencrypted

TURN (NAT discovery)

4

WebRTC clients

MS

IP GW-TURN

443

TCP

COTURN

TLS

TURN (media relay)

There are other rules that need to be applied only when there is a SIP trunk connection with an external SIP entity such as a PBX, PSTN via SBC, etc. (rules 5,6,7) or when the deployment includes mobile clients (rules 8, 9 and 10) as per the following table:

Advanced networking requirements

Req #

Origin

Destination

Destination IP

Destination ports

Protocol

Service

Encryption

Description

5

SIP trunk

MS

IP GW-audio

5060

UDP

SIP

Unencrypted

SIP signalling

6

SIP trunk

MS

IP GW-audio

5061

TCP

SIP

TLS

SIP signaling

7

SIP trunk

MS

IP GW-audio

20000-29999

UPD

RTP/RTCP

Unencrypted

SIP audio (optionally encrypted using SRTP)

8

Quobis AS

AS

AS

5222

TCP

XMPP

TLS

Messaging from mobile apps

9

WebRTC client

Apple APNS

17.0.0.0/8

443

TCP

HTTP/TLS

SSL

Mobile push notifications

10

Quobis AS

Apple APNS

17.0.0.0/8

2195,2196

TCP

HTTP/TLS

SSL

Mobile push notifications

Note: rule #5 refers to the port used to exchange traffic between the SIP proxy and the core SIP network elements.

Note: rule #8 is needed because the messaging in mobile applications does not use Websocket but TLS over TCP connections to the server.

Note

Websocket connection timeouts: we minimize the amount of keepalive messages in order to optimize the connection setup time and the amount of traffic exchanged between our SDKs and the backend. To avoid connectivity problems, we recommend to setup traffic timeouts with high values (3600s) in the reverse proxies (e.g. F5 solutions) between the clients and the Quobis WAC cluster. This way we prevent websocket connections (RFC6455) from being abruptly closed.

Sizing ports for the SIP media connections

AS explained in the rule #7 above, a range of UDP ports need to be opened in order to allow media traffic from the SIP trunk connection. The following considerations must be taken into account:

  • Non-ParallelSIP scenarios: the number of ports can be obtained as AverageNumConcurrentConferences * 2 (one port for RTP, one port for RTCP)

  • ParallelSIP scenarios: the number of ports can be obtained as AverageNumConcurrentConferences * 4

Sizing ports for the WebRTC media connections

As explained in the rule #2 in the table above, a range of UDP ports need to be opened in order to allow WebRTC media traffic from/to the WebRTC endpoints. As explained in the media management section, Quobis WAC uses a SFU (Selective Forwarding Technology) to deliver the video flows to all the participants in a video conference. That means that the media servers will send an individual flow of the video published by each participant to each and one of the participants of the conference, simultaneously. Taking this fact into accoun, the number of used ports will depend on two variables:

  • the number of simultaneous users of the platform

  • the average number of participants at each conference

The formula used to calculate the ports used in a single conference can be computed from the following paramenters:

  • NumParticipants = Number of participants in the conference.

  • NumPortsPerConference = Total number of used UDP ports used in a conference.

  • NumPortsVideoSubscribers = Number of UDP ports used for video per conference to receive video from the SFU.

  • NumPortsVideoPublishers = Number of UDP ports used for video per conference to send video to the SFU.

  • NumPortsAudio = Number of UDP ports used for audio per conference. For audio the same port is used to publish and subscribe.

and applying this formula:

  • NumPortsVideoSubscribers = (NumParticipants - 1) * NumParticipants

  • NumPortsVideoPublishers = NumParticipants

  • NumPortsAudio = NumParticipants

  • NumPortsPerConference = NumPortsVideoSubscribers + NumPortsVideoPublishers + NumPortsAudio

Please note that this formula only applies if all the participants are publishing video and the rest of participants are subscribed to it. This represents the worst case in terms of port usage. If a user is neither subscribed nor publishing video those ports should not be considered in the formula. Please note that this refers to the number of open ports per media server, if your architecture includes several media servers then the number of ports must be distributed amongst them. As a reference, in the table below you can see the correspondence between number of participants in a conference and the number of ports used:

Number of ports used in a conference room

Participants in the conference

Ports used in the conference

2

6

3

12

4

20

8

72

16

272

32

1056

64

4160

As it can clearly seen, the figures are exponential and for conferences above 32 participants, with all of them publishing video at the same time, the number of ports is huge and so will be the used bandwidth. We can perform an estimation to calculate the number of open ports that we need which takes into account the number of simultaneous users using the service and the average number of participants at a conference. This will give a good estimation of the number of ports which need to be configured to avoid any service disruption if the limit is reached. So to calculate the right number of UDP ports we need to open please consider the following definitions:

  • SimultaneousUsers = number of users connected to a conference at the same time (do not confuse with the number of registered users).

  • AverageNumConferenceParticipants = average number of participants in a conference.

  • AverageNumConferences = average number of conferences.

  • TotalPorts = Total number of ports required by the system.

  • NumPortsPerConference = number of ports per conference calculated using the formula previously defined.

and compute the following formula:

  • AverageNumConferences = ceil (SimultaneousUsers / AverageNumConferenceParticipants)

  • TotalPorts = NumPortsPerConference(AverageNumConferenceParticipants) * AverageNumConferences

As a reference, the table below shows the number of estimated ports required depending on AverageNumConferences and SimultaneousUsers (calculations per media server):

Number of ports used in a conference room

SimultaneousUsers

Average Num Conference Participants

Total ports used

100

4

500

100

8

900

500

4

2500

500

8

4500

1000

4

5000

1000

8

9000

The figures above should give a conservative value. However if a shortage of ports is detected in high load conditions the range must be increased. If the use of such a wide range of ports can mean a problem then you should consider the use of a TURN server which can multiplex the media coming and going from/to different endpoints on a single UDP port. However using UDP would be the preferred option in terms of simplicity and performance.

Sizing of media servers required according to traffic load

In summary, according to the selected MS worker nodes referred above, each of them must open the indicated number of ports depending on the audio codec in order to cover the main available scenarios:

  • 2000 opened port per SFU + MCU (Opus) -> 200 cc per Media Server

  • 4000 opened port per SFU + MCU (G711)-> 400 cc per Media Server

Bandwith requirements

Administrators need to take into account how much bandwith is going to be required to cope with the application traffic, specially for video applicatons. This calculation is directly related with the codecs in use and the quality allowed for each video stream. Please check the media quality section to understand the codec choice and how much badwith will be required in order to provide a good quality of service.

Certificates

Digital certificates are needed in the system in order to complete the installation. You need to provide:

  • TLS certificates for the ingress controller (mandatory). More information can be found in the Nginx documentation site

  • Apple push notification services (only when mobile devices are included in the deployment).

Additional requirements

There are other additional requirements that need to be fulfilled before starting the installation process:

  • Remote access must be guaranteed to master and worker nodes during the installation process.

  • Internet access is needed in all the nodes during the installation procedure in order to download external dependencies and container images. Once the Quobis Communication Platform is installed and running, the internet connection can be blocked if needed.

  • Quobis Registry credentials (username and token) is needed to download the docker images during installation and update procedures from registry.quobis.com and registry.gitlab.com

  • Gitlab credentials or accees token to download the repository with the installation sofware located at https://gitlab.com/quobis/devops/sippo-k8s

  • Domain and valid certificates: a public domain and the SSL certificates have to be available to access the Quobis Communication Platform from a browser.

  • Public IP addresses: each Media Server (MS) worker needs a public IP address in order to manage video and audio of WebRTC calls. The IP traffic can be forwarded from an external firewall.