CASE STUDY |
Development of complete multimedia sub-
system for a SIP Video Phone
A leading electronic device manufacturer was looking to develop a SIP video phone used for video conferencing
that supports 3-way calling. The objective was to achieve the calling with improved audio & video quality and
synchronization. SIP video phone was capable of handling 3-way calling with multi-resolution support globally
interoperable with IP phones. PathPartner delivered complete multimedia pipeline for the video phone thus
accelerating the time to market for our customer.
2020 PathPartner Technology Pvt. Ltd. www.pathpartnertech.com
Case study | Development of SIP Video Phone Used for Video Conferencing
Introduction
Our customer – one of the leading providers of electronics devices – was looking to develop a video conferencing device
that was capable of 3-way calling, supported multi-resolution encoding based on SIP negotiation and was compatible
with other IP phones. The main objective of the project was to develop complete video subsystem of the SIP phone on a
Texas Instruments (TI) platform. PathPartner provided solution for complete multimedia pipeline, board support package
and product testing.
Key Challenges
Technology innovations, especially for electronics devices and consumer electronics industry, are riddled with challenges
pertaining to A/V sync, resolutions, boot-time, and power consumption. Some of the key challenges that we addressed
during the implementation were:
● Lip Sync/Audio Video Sync
The solution was built on dual-chip architecture, one of which was handling audio and the other was handling
video. Each of them were running their own operating system. We utilized NTP timestamp which was accurate
in achieving clock synchronization, with an accuracy level of a millisecond. It left us with a great mechanism for
receiving the timestamp and getting the timestamp through the pipeline and using it to display the video stream.
● Performance optimization and memory management challenges
The multi-channel conferencing device included multi-pipeline creation and deletion, based on the number of
entities involved in a call. Even a byte of memory leak would leave us with no memory after running the system
for a few days. There was no memory debugging tool support to detect leaks/static analysis and generate reports
for this architecture. We enabled a Glib-based memory report generator and tuned the same to detect leaks in
Gstreamer.
With limited memory at hand, we had to recalculate each phase of driver development and the portion of memory
which is handled with direct memory access (DMA) enabled memory.
● Interoperability testing
SIP is a generic protocol involving video, voice, message and other communications, so the
packetization/depacketization of the frame needs to be accurate considering optional and mandatory fields.
Most of the interoperability issues were resolved with the Wireshark packet analyzer.
● Fallback mechanism for dynamic adjustments for varying features
In case a 3-way call gets established with levels (Video resolution/Frame rates/Encoded profile) beyond the
capabilities of the device, this case was handled with a safe fallback mechanism identifying the bottleneck of the
system. The fallback mechanism was used in order to bottleneck with a limited impact of calls, i.e. fallback to
ghost image in place of self-image (resize being the bottleneck), fallback to lower encoding stream rates and early
drop frames in boundaries of reference frames when buffering portion increases.
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd. www.pathpartnertech.com
Case study | Development of SIP Video Phone Used for Video Conferencing
PathPartner Solution Architecture Focused on Video Subsystem
PathPartner, with over a decade of experience in developing embedded multimedia solutions, provided a complete SIP
Phone multi-chip solution which included the following components:
• Host IC
• Video Subsystem IC
• Ethernet Interface
• CMOS Imaging Sensor
• LCD Panel
The Host IC’s role was to handle audio capturing, rendering, generating menu images based on touch events and taking
care of SIP network stacks. The Ethernet interface was used for command/video data exchange between two ICs. On the
other hand, the video subsystem IC handled the video pipeline, which includes capturing, encoding, streaming, receiving,
decoding and displaying. Two major hardware modules of the Video IP phone CMOS camera and the LCD Panel were
connected to the video IC.
At a broad level, the Video Subsystem Software package includes:
● Board support package
● Media pipeline
● PC based test bench
Board Support Package
As part of the complete video subsystem development, PathPartner provided the board support package which included:
● Bootloader support
● Bootloader boot-up logo support
● Camera driver support for CMOS sensor
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd. www.pathpartnertech.com
Case study | Development of SIP Video Phone Used for Video Conferencing
● Camera driver support for LCD Panel
● Video Processing Front End (VPFE) subsystem OSD (On Screen Display) driver support
● VPFE driver support for camera configuration modification
● Resizer for single-shot mode operations
Media Pipeline
SIP Video IP phone, video pipeline is showcased in the above picture with data flow. Multi-encoded streams are received
from entities in calls. The video IP phone also needs to generate its own video stream and encode it and send it across the
network. Apart from the usual roles of SIP video phones, the Video Subsystem also had the role of using video
acceleration to blend the user interface low frame rate, raw video with other video streams.
Apart from handling data flow and the above-showcased rules, following aspects/features were also part of video
subsystem development:
● Multi-resolution support QCIF, CIF, QVGA, VGA and 720P
● Gstreamer decode/encode pipeline integrated with custom pipeline
● Low latency streaming enabled with slice encoding
● Dynamic encoder parameter configuration support in Gstreamer plugins
● RTP H264 payload structures support for IP Phone interoperability
● Custom Audio/Video sync strategy implementation
● Multi-call layouts and letterboxing the video frames to maintain the aspect ratio of displayed video
● Frame rate control across streams for achieving high video quality
Customer Benefits Delivered
PathPartner team developed a complete SIP phone video solution for a video conferencing device that is compatible with
IP phones. Some of the key benefits include:
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd. www.pathpartnertech.com
Case study | Development of SIP Video Phone Used for Video Conferencing
● Lag-free and smooth video display even in higher resolution scenarios
● Enhanced Audio/Video synchronization
● Accelerated solution delivery aided by extensive prior experience in developing similar solutions
Learn more about PathPartner’s multimedia solutions and product engineering services
More resources available at
https://www.pathpartnertech.com/services/multimedia/.
For an in-depth discussion, contact us at
[email protected]© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd. www.pathpartnertech.com
Case study | Development of SIP Video Phone Used for Video Conferencing
About PathPartner
PathPartner Technology is a leader in providing technology solutions and product engineering services for developing
innovative and intelligent systems. PathPartner offers a portfolio of enabling products and comprehensive services,
spanning from embedded systems, hardware design, and system integration. We work with device manufacturers, system
integrators, and solution providers across diverse industry domains such as automotive, consumer electronics, drones,
video surveillance, medical devices, etc. Our deep technology expertise in multimedia systems, deep learning solutions,
vision-based systems, connectivity technologies, and imaging pipelines coupled with our experience in underlying
hardware platforms enables us to provide complete solutions to our customers while expediting their time to marketing
and future-proofing investments.
Headquartered in Bangalore, India, PathPartner also has presence in USA, Europe, and Japan. To learn more about us,
please visit www.pathpartnertech.com.
Contact Us
PathPartner Technology Pvt. Ltd.
#16, PSS Plaza, 1st and 2nd Floor,
New Thippasandra Main Road,
HAL III Stage, Bangalore-560075.
Other presence: Kochi, India | Bay Area, USA | Tokyo, Japan | Frankfurt, Germany
Phone: +91 80 6772 2000 | +1 408 242 7411 | +81 905546681
Email: [email protected]
Website: https://www.pathpartnertech.com/
Follow us on social media: LinkedIn | Twitter
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd.
© 2010-2020 PathPartner Technology Pvt. Ltd. www.pathpartnertech.com
