Plex Hardware Transcoding with an Intel CPU inside an Ubuntu VM on ESXi

Plex Hardware Transcoding with an Intel CPU inside an Ubuntu VM on ESXi

Required Components:
1. A reasonably current Intel CPU with Intel Quick Sync Video capabilities.
Check your processor capabilities here: https://ark.intel.com/
2. Plex Pass Subscription
3. VMware environment.  This guide is using ESXi 6.5, but 6.0 will work also.  I have not tried other versions. -A setup with the free version of ESXi installed on a standalone host would work here.

Background:
I have been running a Plex server for years and never really paid much attention to transcoding in hardware mainly because it was a feature reserved for Plex Pass Subscribers, (which I was not) and I didn't have many friends and family accessing my media.  I recently became a Plex Pass user with a lifetime subscription deal that they sent me via email.
I always run my Plex servers inside of a VM because I run a home lab and have VM resources always available and ready to do work.
My homelab consists of two I3-6100U processor 6th generation Intel NUC VMware ESXi hosts.  These NUCs are perfect for me since they were affordable, low power, quiet, extremely tiny, and easy to add to by stacking on another NUC.  Also, if you are in a situation where you are regularly moving apartments, home, dorms, etc.  You cannot beat a space optimized home lab.  Plus ESXi installs on the 5th gen NUC with no need for additional work.  My home lab, like most others is not a big consumer of CPU most of the time.

 

Home lab shown.  The cardboard is there to keep the drive trays from vibrating and making noise. 6x 8TB drives for media storage, 2x 1TB SSD drives for VM datastore, unmanaged 8 port gigabit switch, 2x Intel NUC6i3SYK NUCs configured with 32 gigs RAM each.

I did some reading and experimentation with transcoding because I was getting more friends and family hitting my Plex server recently.  I felt like I needed to monitor performance and make sure that I had enough resources for this additional load.  Most outside users were transcoding due to bandwidth limitations, whereas I was always doing direct play when watching at home.
I was surprised with the results of my testing.
Plex has a ton of information about transcoding.  I took a look at this page and learned how to enable hardware transcoding, the requirements and how to experiment with forcing transcoding:
https://support.plex.tv/articles/115002178853-using-hardware-accelerated-streaming/

To experiment, I began streaming a 1080p movie and watched my Plex VM by logging in and running the HTOP command. (It is an Ubuntu Server VM)

Direct Play didn't hit the CPU much at all.  I then changed to a transcoding resolution and my Plex VM saturated all of the CPUs granted to it (2) for quite some time.  The video played just fine, but I needed to avoid this saturation if possible.  After the initial CPU spike for upwards of 1 minute, it calmed down a bit to about 60% usage during the video playback.
I tried giving the VM 4 virtual CPUs since my I3 has two physical CPU and two hyperthreaded virtual cores.  As you can imagine, this didn't do much.
So now that I knew this was happening, I needed to enable hardware transcoding at all costs, but I didn't want to incur a cost. 
Plex says in the link above that hardware transcoding is not possible inside of a VM.  Damn!  But let's not take their word for it, let's try it anyways.
I went to Direct Access settings in my host, and I saw the video card available and configured it for access.

This change required a restart of the ESXi host, so I did that.
Then I shut down the Plex VM and added the direct access device.  Looking good so far.

Upon booting of my Plex VM, I got back into the Plex interface and tried watching the same movie.  I made no changes to the Plex VM.  I didn't look for drivers nor try to install anything on my Plex VM.  I am running Ubuntu Server 18.04 LTS.
Direct Play worked as expected, but then I switched to a transcoding resolution and was immediately impressed.
It switched over very fast and I saw the indicator that I was transcoding in hardware: (hw) was showing on the Plex web interface server status.  Wow.  That's it?  That was easy.

What does this mean for performance?  Huge for me, as I am running on underpowered hardware in the first place.  After the change, transcoding took nearly zero CPU resources since it was all being done by the onboard video portion of the Intel CPU.  There was just a slight hit well under 100% CPU consumption while the video started streaming, and nearly zero hit after the initial spike.

What is the drawback for the VM and general VMware functionality?  Well, this is the trade-off.
1. You can't do a live vMotion of the VM because it has direct attached hardware.
2. It requires you to reserve RAM for the VM.  Not a ton though.  I have 4 gigs RAM allocated to my Plex VM, and it is not using that much.  I could probably allocate 2 gigs and be fine.  I set a RAM reservation to 4 gigs.
3. You can't do a snapshot of the VM.
 
For me, these are acceptable trade-offs since I was considering setting up a dedicated NUC for Plex serving to take advantage of the hardware transcoding.  This avoids needing to do that and allows me to run additional VMs on this ESXi host like I already was, I just need to shut down the Plex VM if I need to do any ESXi host maintenance.


I don't know if this would also work with an AMD based CPU, but it would certainly work if you had an ESXi host with some graphics card solution installed.  You'd just need to research which graphics cards ESXi can recognize and allow direct access to.

***Followup on this.  I was curious if this will allow me to transcode 4k video.  I don't have any 4k content, so I got a sample 1 minute of Elysium from http://4ksamples.com/ 

Even with transcoding confirmed to be done in hardware now, I could not watch the 1 minute video without the video buffering twice and then crashing.  It started strong, but couldn't really make it past 30 seconds.  Oh well.....  4k aint no joke.