Are all BSDs created equally — 一次关于BSD内核漏洞的调查

Are all BSDs created equally — 一次关于BSD内核漏洞的调查

在今年(2017)的defcon大会上,Ilja van Sprundel做了一个名为 “所有的BSD生来是平等的吗?”(Are all BSDs created equally? ),副标题: 一次关于BSD系统内核漏洞的调查(A survey of BSD kernel vulnerabilities)。我在youtube上搜了一下,目前还没有视频。   浏览了一下pdf文件。主要内容如下: 作者花了三个月时间检查了OBSD/FBSD/NBSD的内核代码,主要涉及系统调用、驱动程序、TCP/IP栈、trap handler、文件系统以及用于兼容其他系统(如Linux)的代码,截止报告期,共发现bug或漏洞约115个,其中: FBSD: ~30 OBSD: 25 NBSD: ~60   作者列出的OpenBSD相关的bug有:   Unbound malloc and out of bound read (could panic or info leak) OpenBSD 6.1 Been there since OpenBSD 4.0 Fri Apr 28 08:34:31 2006 getcwd syscall when taking data from fuse / userland   mbuf mishandling, leading to crash Doesn’t guarantee it pulls up enough mbuf data OpenBSD…

Read More

Protecting Netflix Viewing Privacy at Scale

On the Open Connect team at Netflix, we are always working to enhance the hardware and software in the purpose-built Open Connect Appliances (OCAs) that store and serve Netflix video content. As we mentioned in a recent company blog post, since the beginning of the Open Connect program we have significantly increased the efficiency of our OCAs – from delivering 8 Gbps of throughput from a single server in 2012 to over 90 Gbps from a single server in 2016. We contribute to this effort on the software side by optimizing every aspect of the software for our unique use case – in particular, focusing on the open source FreeBSD operating system and the NGINX web server that run on the OCAs.

Members of the team will be presenting a technical session on this topic at the Intel Developer Forum (IDF16) in San Francisco this month. This blog introduces some of the work we’ve done.

Adding TLS to Video Streams

In the modern internet world, we have to focus not only on efficiency, but also security. There are many state-of-the-art security mechanisms in place at Netflix, including Transport Level Security (TLS) encryption of customer information, search queries, and other confidential data. We have always relied on pre-encoded Digital Rights Management (DRM) to secure our video streams. Over the past year, we’ve begun to use Secure HTTP (HTTP over TLS or HTTPS) to encrypt the transport of the video content as well. This helps protect member privacy, particularly when the network is insecure – ensuring that our members are safe from eavesdropping by anyone who might want to record their viewing habits.

Netflix Open Connect serves over 125 million hours of content per day, all around the world. Given our scale, adding the overhead of TLS encryption calculations to our video stream transport had the potential to greatly reduce the efficiency of our global infrastructure. We take this efficiency seriously, so we had to find creative ways to enhance the software on our OCAs to accomplish this objective.

We will describe our work in these three main areas:
  • Determining the ideal cipher for bulk encryption
  • Finding the best implementation of the chosen cipher
  • Exploring ways to improve the data path to and from the cipher implementation

Cipher Evaluation

We evaluated available and applicable ciphers and decided to primarily use the Advanced Encryption Standard (AES) cipher in Galois/Counter Mode (GCM), available starting in TLS 1.2. We chose AES-GCM over the Cipher Block Chaining (CBC) method, which comes at a higher computational cost. The AES-GCM cipher algorithm encrypts and authenticates the message simultaneously – as opposed to AES-CBC, which requires an additional pass over the data to generate keyed-hash message authentication code (HMAC). CBC can still be used as a fallback for clients that cannot support the preferred method.

All revisions of Open Connect Appliances also have Intel CPUs that support AES-NI, the extension to the x86 instruction set designed to improve encryption and decryption performance.
We needed to determine the best implementation of AES-GCM with the AES-NI instruction set, so we investigated alternatives to OpenSSL, including BoringSSL and the Intel Intelligent Storage Acceleration Library (ISA-L).

Additional Optimizations

Netflix and NGINX had previously worked together to improve our HTTP client request and response time via the use of sendfile calls to perform a zero-copy data flow from storage (HDD or SSD) to network socket, keeping the data in the kernel memory address space and relieving some of the CPU burden. The Netflix team specifically added the ability to make the sendfile calls asynchronous – further reducing the data path and enabling more simultaneous connections.

However, TLS functionality, which requires the data to be passed to the application layer, was incompatible with the sendfile approach.

To retain the benefits of the sendfile model while adding TLS functionality, we designed a hybrid TLS scheme whereby session management stays in the application space, but the bulk encryption is inserted into the sendfile data pipeline in the kernel. This extends sendfile to support encrypting data for TLS/SSL connections.

We also made some important fixes to our earlier data path implementation, including eliminating the need to repeatedly traverse mbuf linked lists to gain addresses for encryption.

Testing and Results

We tested the BoringSSL and ISA-L AES-GCM implementations with our sendfile improvements against a baseline of OpenSSL (with no sendfile changes), under typical Netflix traffic conditions on three different OCA hardware types. Our changes in both the BoringSSL and ISA-L test situations significantly increased both CPU utilization and bandwidth over baseline – increasing performance by up to 30%, depending on the OCA hardware version. We chose the ISA-L cipher implementation, which had slightly better results. With these improvements in place, we can continue the process of adding TLS to our video streams for clients that support it, without suffering prohibitive performance hits.

Read more details in this paper and the follow up paper. We continue to investigate new and novel approaches to making both security and performance a reality. If this kind of ground-breaking work is up your alley, check out our latest job openings!

By Randall Stewart, Scott Long, Drew Gallatin, Alex Gutarin, and Ellen Livengood

Read More