Planning and pre-requisites
Choosing the deployment environment type
Quobis communication platform supports on-premise or cloud scenarios, from local and private deployment to global and multi location deployments depending on your organization needs.
On premises: deployments use your own network, hardware and hypervisor. You will need to consider that compromised data will be stored in your private infrastructure with restricted access, providing a higher level of security. On premise deployments will need an appropriate server where to host the VMs and the proper network configuration. If you already have a mature IT network with servers in the appropriate locations, this may be a suitable option. If you want to install on-premises, you will need to install a hypervisor on your servers to manage the deployment and configure the VMs to setup the Kubernetes cluster. Quobis communication platform currently supports VMware hypervisor only.
Cloud deployments: the supported public cloud provider as of today is Amazon Web Services (AWS). This option does not require hardware infrastructures, you just have to reach an agreement with the cloud service provider, supplying an easy and fast way to set up the environment to deploy the platform.
Kubernetes cluster sizing
A Kubernetes cluster consists of a set of worker machines, called nodes, that run containerized applications. Every cluster has at least one worker node which host the Pods, that are the components of the application workload. The control plane manages the worker nodes and the Pods in the cluster. In production environments, the control plane usually runs across multiple computers and a cluster usually runs multiple nodes, providing fault-tolerance and high availability.
The performance of the Quobis communication platform is determined by the number of nodes and the amount of resources of each node, in terms of CPU and memory. Depending on the number of nodes and the resources used by each of them the number of calls managed at time by the system may vary significantly. Additionally, Kubernetes provides to the cluster the capability to support High Availability, which means a seamless service when a node breaks down. This scenario needs, at least, 3 master nodes and 2 worker nodes in order to balance the workload between the active nodes.
You also must take into account the deployment location to improve the conference’s performance. The server should be located as close as possible to the main base or your organization due to the time issues. For instance, choose the AWS EU-region if your organization base is located in Europe.
Depending on the type of deployment, the number of current calls and the number of provisioned users, the number of nodes and their resources must be chosen accordingly.
For a high-availability deployment, 3 master nodes and at least 2 worker nodes are required
For a standalone deployment, 1 master node and at least 2 worker nodes are required
Hardware requirements
There are no specific hardware or vendor constrains as Quobis Communication Platform can run on top of virtual machines (VMs). Just make sure that the hardware sizing is appropriate for the VW specified below.
On-premises deployments
The selected hypervisor must be VWMare
Supported OS: Debian 9 or higher
Master nodes: at least 4 vCPU, 8 GB RAM, at least 100GB HDD
Worker nodes: at least 8 vCPU, 16 GB RAM, at least 100GB HDD
Two option for storage:
Highly available persistent storage for Kubernetes through Longhorn
A Network File System (NFS) server provided within the product to store all the persistent data used by the cluster.
Cloud deployments
The selected hypervisor must be the AWS virtualization platform and EC2 instances
Supported OS: Debian 9 or higher, Ubuntu 16 or higher
Master nodes: AWS t3a.medium instances, 2 vCPU, 4 GB RAM, EBS storage, up to 5Gbps bandwidth
Worker nodes: AWS c5.2xlarge instances, 8 vCPU, 16 GB RAM, EBS storage, up to 10Gbps bandwidth, up to 4750Mbps EBS throughput
Storage: Amazon Elastic File System (EFS) standard is the solution used to provide persistent storage in Amazon Web Services deployments.
Sizing of media server nodes
The Quobis platform deals with video and audio in different ways, as explained in detail in the media management section:
The audio is managed in a MCU (Media Control Unit), also referred as audiomixer. The audio coming from all the participants is mixed here.
The video is managed in an SFU (Selective Forwarding Unit). The video coming from all the participants is relayed to the rest of participants.
There are two services which consume CPU cycles in the media server nodes:
Audiomixer: it consumes CPU mainly in two tasks:
Opus transcoding: the media coming from the WebRTC endpoints (browsers and mobile SDKs) needs to be transcoded before being mixed. Opus transcoding is a CPU-intensive task and the one which porcentually consumes more CPU cycles. If PCMU/PCMA (G711) is used instead of Opus, the CPU consumption is much lower.
Audiomixing: the mixing of audio, once transcoded, also requires CPU real-time processing.
SFU: it consumes CPU to keep the media connections open between browsers/mobile apps and the SFU. These connections are called PeerConnections in WebRTC terminology. The number of open PeerConnections increases linearly with the number of simultaneous media sessions and pseudo-exponentially with the number of participants in each session.
Taking into account the factors above and an average distribution of users in conferences with different numbers of participants, we can consider the following sizing rules:
One node (8vCPU) for each 200cc using Opus
One node (8vCPU) for each 400cc using PCMU/PCMA.
Sizing of application server nodes
Besides the media processing, the CPU consumption of the application pods must also be considered. The CPU processing will depend on several factors:
the number of simultaneous active sessions
the number of concurrent calls.
the number of calls per secon
use of the presence service. Presence can be an issue in large setups where each user receives an update with any presence change. So a huge average number of contacts per user will lead to a higher CPU utilization.
provisioning processes through the REST API may lead to peak CPU usage.
Sessions from browser-based apps that remain active while the web app is open.
Sessions from mobile apps that keep the session active only while the app is in foreground.
Taking into account the factors above, we can consider the following sizing rules as a rule of thumb:
one 8vCPU node per 1,000 simultaneous sessions when presence is enabled.
one 8vCPU node per 2,000 simultaneous sessions when presence is not enabled.
Please consider the rules above as references, since the load may differ depending on the use cases. An active monitoring of the platform may is quite userful to adjust the number of nodes.
Network requirements
Firewall rules and security considerations
Quobis Communication Platform needs communication between cluster nodes (master and workers) and also needs to expose different ports to allow voice and video calls. The following table summarizes the required network ports to configure:
Req # |
Origin |
Destination |
Destination IP |
Destination ports |
Protocol |
Service |
Encryption |
Description |
|---|---|---|---|---|---|---|---|---|
1 |
WebRTC clients |
AS |
AS IP |
80,443 |
TCP |
WSS |
SSL |
Application traffic |
2 |
WebRTC clients |
MS |
IP GW-SFU |
20000-29999 |
UDP |
DTLS-SRTP |
SSL |
WebRTC media |
3 |
WebRTC clients |
MS |
IP GW-TURN |
3478 |
UDP |
STUN |
Unencrypted |
TURN (NAT discovery) |
4 |
WebRTC clients |
MS |
IP GW-TURN |
443 |
TCP |
COTURN |
TLS |
TURN (media relay) |
There are other rules that need to be applied only when there is a SIP trunk connection with an external SIP entity such as a PBX, PSTN via SBC, etc. (rules 5,6,7) or when the deployment includes mobile clients (rules 8, 9 and 10) as per the following table:
Req # |
Origin |
Destination |
Destination IP |
Destination ports |
Protocol |
Service |
Encryption |
Description |
|---|---|---|---|---|---|---|---|---|
5 |
SIP trunk |
MS |
IP GW-audio |
5060 |
UDP |
SIP |
Unencrypted |
SIP signalling |
6 |
SIP trunk |
MS |
IP GW-audio |
5061 |
TCP |
SIP |
TLS |
SIP signaling |
7 |
SIP trunk |
MS |
IP GW-audio |
20000-29999 |
UPD |
RTP/RTCP |
Unencrypted |
SIP audio (optionally encrypted using SRTP) |
8 |
Quobis AS |
AS |
AS |
5222 |
TCP |
XMPP |
TLS |
Messaging from mobile apps |
9 |
WebRTC client |
Apple APNS |
17.0.0.0/8 |
443 |
TCP |
HTTP/TLS |
SSL |
Mobile push notifications |
10 |
Quobis AS |
Apple APNS |
17.0.0.0/8 |
2195,2196 |
TCP |
HTTP/TLS |
SSL |
Mobile push notifications |
Note: rule #5 refers to the port used to exchange traffic between the SIP proxy and the core SIP network elements.
Note: rule #8 is needed because the messaging in mobile applications does not use Websocket but TLS over TCP connections to the server.
Note
Websocket connection timeouts: we minimize the amount of keepalive messages in order to optimize the connection setup time and the amount of traffic exchanged between our SDKs and the backend. To avoid connectivity problems, we recommend to setup traffic timeouts with high values (3600s) in the reverse proxies (e.g. F5 solutions) between the clients and the Quobis WAC cluster. This way we prevent websocket connections (RFC6455) from being abruptly closed.
Sizing ports for the SIP media connections
AS explained in the rule #7 above, a range of UDP ports need to be opened in order to allow media traffic from the SIP trunk connection. The following considerations must be taken into account:
Non-ParallelSIP scenarios: the number of ports can be obtained as AverageNumConcurrentConferences * 2 (one port for RTP, one port for RTCP)
ParallelSIP scenarios: the number of ports can be obtained as AverageNumConcurrentConferences * 4
Sizing ports for the WebRTC media connections
As explained in the rule #2 in the table above, a range of UDP ports need to be opened in order to allow WebRTC media traffic from/to the WebRTC endpoints. As explained in the media management section, Quobis WAC uses a SFU (Selective Forwarding Technology) to deliver the video flows to all the participants in a video conference. That means that the media servers will send an individual flow of the video published by each participant to each and one of the participants of the conference, simultaneously. Taking this fact into accoun, the number of used ports will depend on two variables:
the number of simultaneous users of the platform
the average number of participants at each conference
The formula used to calculate the ports used in a single conference can be computed from the following paramenters:
NumParticipants = Number of participants in the conference.
NumPortsPerConference = Total number of used UDP ports used in a conference.
NumPortsVideoSubscribers = Number of UDP ports used for video per conference to receive video from the SFU.
NumPortsVideoPublishers = Number of UDP ports used for video per conference to send video to the SFU.
NumPortsAudio = Number of UDP ports used for audio per conference. For audio the same port is used to publish and subscribe.
and applying this formula:
NumPortsVideoSubscribers = (NumParticipants - 1) * NumParticipants
NumPortsVideoPublishers = NumParticipants
NumPortsAudio = NumParticipants
NumPortsPerConference = NumPortsVideoSubscribers + NumPortsVideoPublishers + NumPortsAudio
Please note that this formula only applies if all the participants are publishing video and the rest of participants are subscribed to it. This represents the worst case in terms of port usage. If a user is neither subscribed nor publishing video those ports should not be considered in the formula. Please note that this refers to the number of open ports per media server, if your architecture includes several media servers then the number of ports must be distributed amongst them. As a reference, in the table below you can see the correspondence between number of participants in a conference and the number of ports used:
Participants in the conference |
Ports used in the conference |
|---|---|
2 |
6 |
3 |
12 |
4 |
20 |
8 |
72 |
16 |
272 |
32 |
1056 |
64 |
4160 |
As it can clearly seen, the figures are exponential and for conferences above 32 participants, with all of them publishing video at the same time, the number of ports is huge and so will be the used bandwidth. We can perform an estimation to calculate the number of open ports that we need which takes into account the number of simultaneous users using the service and the average number of participants at a conference. This will give a good estimation of the number of ports which need to be configured to avoid any service disruption if the limit is reached. So to calculate the right number of UDP ports we need to open please consider the following definitions:
SimultaneousUsers = number of users connected to a conference at the same time (do not confuse with the number of registered users).
AverageNumConferenceParticipants = average number of participants in a conference.
AverageNumConferences = average number of conferences.
TotalPorts = Total number of ports required by the system.
NumPortsPerConference = number of ports per conference calculated using the formula previously defined.
and compute the following formula:
AverageNumConferences = ceil (SimultaneousUsers / AverageNumConferenceParticipants)
TotalPorts = NumPortsPerConference(AverageNumConferenceParticipants) * AverageNumConferences
As a reference, the table below shows the number of estimated ports required depending on AverageNumConferences and SimultaneousUsers (calculations per media server):
SimultaneousUsers |
Average Num Conference Participants |
Total ports used |
|---|---|---|
100 |
4 |
500 |
100 |
8 |
900 |
500 |
4 |
2500 |
500 |
8 |
4500 |
1000 |
4 |
5000 |
1000 |
8 |
9000 |
The figures above should give a conservative value. However if a shortage of ports is detected in high load conditions the range must be increased. If the use of such a wide range of ports can mean a problem then you should consider the use of a TURN server which can multiplex the media coming and going from/to different endpoints on a single UDP port. However using UDP would be the preferred option in terms of simplicity and performance.
Sizing of media servers required according to traffic load
In summary, according to the selected MS worker nodes referred above, each of them must open the indicated number of ports depending on the audio codec in order to cover the main available scenarios:
2000 opened port per SFU + MCU (Opus) -> 200 cc per Media Server
4000 opened port per SFU + MCU (G711)-> 400 cc per Media Server
Bandwith requirements
Administrators need to take into account how much bandwith is going to be required to cope with the application traffic, specially for video applicatons. This calculation is directly related with the codecs in use and the quality allowed for each video stream. Please check the media quality section to understand the codec choice and how much badwith will be required in order to provide a good quality of service.
Certificates
Digital certificates are needed in the system in order to complete the installation. You need to provide:
TLS certificates for the ingress controller (mandatory). More information can be found in the Nginx documentation site
Apple push notification services (only when mobile devices are included in the deployment).
Additional requirements
There are other additional requirements that need to be fulfilled before starting the installation process:
Remote access must be guaranteed to master and worker nodes during the installation process.
Internet access is needed in all the nodes during the installation procedure in order to download external dependencies and container images. Once the Quobis Communication Platform is installed and running, the internet connection can be blocked if needed.
Quobis Registry credentials (username and token) is needed to download the docker images during installation and update procedures from registry.quobis.com and registry.gitlab.com
Gitlab credentials or accees token to download the repository with the installation sofware located at https://gitlab.com/quobis/devops/sippo-k8s
Domain and valid certificates: a public domain and the SSL certificates have to be available to access the Quobis Communication Platform from a browser.
Public IP addresses: each Media Server (MS) worker needs a public IP address in order to manage video and audio of WebRTC calls. The IP traffic can be forwarded from an external firewall.