Voice and video filters

OpenVidu API offers a simple way of applying filters to video and audio streams in the server side by making use of Kurento Media Server capabilities. This is the current status of filter support in OpenVidu:

  • You can apply one filter at a time to a published Stream. Every user subscribed to it will receive the modified stream.
  • You can remove an applied filter.
  • You can call any remote method offered by an applied filter
  • You can add and remove event listeners to any event dispatched by an applied filter.
  • You must configure in the participant token the allowed filters the user can apply.
WARNING: experimental option. This feature may suffer breaking changes in the near future.

Step by step


1) Generate a token with the filters the user will be able to apply

This is a simple way of securing the ability of applying filters from OpenVidu Browser, so that not every user is able to apply any filter at any time.

  • API REST: include in the JSON body a parameter kurentoOptions with a property allowedFilters (a string array containing the name of the filters the user will be able to apply)

    {
        "session": "6fpivlanw91qjy6n",
        "data": "user_data",
        "role": "PUBLISHER",
        "kurentoOptions": {
            "allowedFilters": ["GStreamerFilter", "FaceOverlayFilter"]
        }
    }
    
  • openvidu-java-client: call TokenOptions.Builder#kurentoOptions(KurentoOptions) to set allowedFilters value with method KurentoOptions.Builder#allowedFilters(String[])

    TokenOptions tokenOptions = new TokenOptions.Builder()
        .role(OpenViduRole.PUBLISHER)
        .data("user_data")
        .kurentoOptions(
            new KurentoOptions.Builder()
                .allowedFilters(new String[]{"GStreamerFilter", "FaceOverlayFilter"})
                .build())
        .build();
    String token = session.generateToken(tokenOptions);
    
  • openvidu-node-client: include in TokenOptions parameter a kurentoOptions property with allowedFiters array

    var tokenOptions = {
        role: "PUBLISHER",
        data: "user_data",
        kurentoOptions: {
            allowedFilters: ["GStreamerFilter", "FaceOverlayFilter"]
        }
    };
    session.generateToken(tokenOptions).then(token => { ... });
    


2.A) Initialize a Publisher object configured for using a filter from the beginning of the publishing ...

Use PublisherProperties, specifically property filter:

var OV = new OpenVidu();
var publisher = OV.initPublisher(
    targetElement,
    {filter: {type: "GStreamerFilter", options: "videoflip method=vertical-flip"}}
);

// ... user already connected to "session" with the appropriate token
session.publish(publisher);


2.B) ... or apply the filter dynamically after publishing the stream, whenever you want
// ... user already connected to the session with the appropriate token
// and successfully publishing the Publisher object

publisher.stream.applyFilter("GStreamerFilter", "videoflip method=vertical-flip")
    .then(() => {
        console.log("Video rotated!");
    })
    .catch(error => {
        console.error(error);
    });


3) You can execute any method offered by the filter
// ... user already connected to the session with the appropriate token,
// successfully publishing the Publisher object and a filter being applied to its stream

publisher.stream.filter.execMethod("setElementProperty", {"propertyName":"method","propertyValue":"horizontal-flip"})
    .then(() => {
        console.log("Video rotation direction modified!");
    })
    .catch(error => {
        console.error(error);
    });


4) You can also subscribe to any filter event (if it dispatches any), and later unsubscribe from it
// ... user already connected to the session with the appropriate token,
// successfully publishing the Publisher object and a filter being applied to its stream

publisher.stream.filter.addEventListener("FooFilterEvent", filterEvent => {
        console.log('Filter event received!. Data: ' + filterEvent.data);
    });

...

publisher.stream.filter.removeEventListener("FooFilterEvent");


4) To remove the filter
// ... user already connected to the session with the appropriate token,
// successfully publishing the Publisher object and a filter being applied to its stream

publisher.stream.removeFilter()
    .then(() => {
        console.log("Filter removed");
    })
    .catch(error => {
        console.error(error);
    });


Moderators are not only able to call all of these methods over their Publisher.stream object, but also over any Subscriber.stream object. Also, they don't need any special token permission to apply filters and can bypass any token restriction set to other user tokens



Filter samples

Kurento filters


  • FaceOverlayFilter (overlay an image over detected faces)
    publisher.stream.applyFilter("FaceOverlayFilter")
        .then(filter => {
            filter.execMethod(
                "setOverlayedImage",
                {
                    "uri":"https://cdn.pixabay.com/photo/2013/07/12/14/14/derby-148046_960_720.png",
                    "offsetXPercent":"-0.2F",
                    "offsetYPercent":"-0.8F",
                    "widthPercent":"1.3F",
                    "heightPercent":"1.0F"
                });
        });
    


  • ChromaFilter (set a chroma background)
    publisher.stream.applyFilter(
        "ChromaFilter",
        {
            "window": {
                "topRightCornerX": 0,
                "topRightCornerY": 0,
                "width": 50,
                "height": 50
            },
            "backgroundImage": "https://www.maxpixel.net/static/photo/1x/Cool-Blue-Liquid-Lake-Abstract-Background-Clear-316144.jpg"
        });
    

You must install Kurento ChromaFilter library in the same host as Kurento Media Server: sudo apt-get install kms-chroma


  • ZBarFilter (detect and read bar codes information)
    publisher.stream.applyFilter("ZBarFilter")
        .then(filter => {
            filter.addEventListener("CodeFound", filterEvent => {
                console.log('Bar code found!. Data: ' + filterEvent.data);
            }
        });
    

GStreamer filters

These filters are set with type GStreamerFilter and an options parameter like this:

publisher.stream.applyFilter("GStreamerFilter", {"command": "GSTREAMER_COMMAND"})

A list of interesting values for GSTREAMER_COMMAND parameter is stated below. Replace GSTREAMER_COMMAND in the upper code snippet for any of the examples provided in the following list items:

  • coloreffects: apply different color filters to the video stream
    Example: coloreffects preset=heat
  • videobalance: change different properties of the video stream such as brightness, contrast, hue or saturation
    Example: videobalance saturation=0.0
  • videoflip: rotate the video stream
    Example: videoflip method=vertical-flip
  • videobox: crop the video stream
    Example: videobox fill=black top=20 bottom=20 left=-10 right=-10
  • gamma: adjust gamma level of the video stream
    Example: gamma gamma=5.0
  • videomedian: add a median filter to the video stream
    Example: videomedian filtersize=9 lum-only=false
  • textoverlay: add an embedded text to the video
    Example: textoverlay text="Embedded text" valignment=top halignment=right font-desc="Cantarell 25"
  • timeoverlay: embed the time the video stream has been playing
    Example: timeoverlay valignment=bottom halignment=right font-desc="Sans, 20"
  • clockoverlay: embed a clock with the local time (in OpenVidu Server)
    Example: clockoverlay valignment=bottom halignment=right shaded-background=true font-desc="Sans, 20"
  • audioecho: add reverb to the audio stream
    Example: audioecho delay=50000000 intensity=0.6 feedback=0.4
  • audioamplify: amplifies an audio stream by a given factor
    Example: audioamplify amplification=1.5 clipping-method=wrap-positive
  • Other audio filters: check them out in GStreamer site
  • Many effects of effectv project : funny filters for the video stream like agingtv, dicetv, optv, quarktv, radioactv, revtv, rippletv, shagadelictv, streaktv, vertigotv, warptv
    Example: radioactv

All available GStreamer plugins can be found in GStreamer site.