|By Marten Terpstra||
|January 23, 2014 09:30 AM EST||
Last week I walked through the packet formats for VXLAN and NVGRE specifically focused on ways by which the overlay packets provide information to the physical network that help the physical network. Some of the initial extreme thoughts that the overlay and physical network can and should be completely ignorant of each other have softened more recently and more pragmatic thoughts of collaborating layers are being articulated. At Plexxi we have often mentioned that we believe the physical network and the overlay need to be closely orchestrated to get the most benefit out of the total network solution. And orchestration != ECMP.
In addition to VXLAN and NVGRE, Stateless Transport Tunneling (STT) is an encapsulation mechanism used by VMware, mostly for communication between server based vSwitches. It is a bit more involved and complicated than VXLAN and NVGRE, mostly because it was designed to carry large data packets, up to 64 Kbytes. Physical networks have limitations on the size of a packet that can be transferred. Ethernet standard maximum transmission unit (MTU) used to be 1500 bytes, but most ethernet devices these days can support jumbo packets allowing packets of 4, 9 or even 16 Kbytes in size. Even at those sizes, large data transfers are somewhat hampered by the work involved in taking a large chunk of data and then chopping them up into smaller portions to be transmitted. In a response to this, hardware vendors have taken some of this functionality and added it to the Network Interface Cards (NICs) on servers and have them do most of this segmentation and re-assembly work based on how TCP takes large portions of data and chops them into smaller segments. Doing his in hardware means it can be done faster, but more importantly, it removes this burden from the server CPUs, allowing them to do other (more useful) work.
STT was designed to make use of these TCP capabilities in NICs. STT can take ethernet packets up to 64 Kbytes from a VM on a server, and tunnel it to its destination as a 64 Kbyte entity. This STT frame has to be chopped into smaller pieces to match the MTU of the physical network, but an STT packet looks just like a TCP segment to the receiving NICs, allowing them to reconstruct the original 64 Kbyte packet without needing the CPU.
When the sending tunnel endpoint receives a large chunk of data to be transmitted at another VM at the other side of a tunnel, the vSwitch takes several steps to encapsulate this packet. First, it adds an STT Frame Header to the packet.
The STT Header is 18 bytes in length and has a variety of administrative fields, but the key field is the Context ID. This is a 64 bit field and its intended use is similar to the VXLAN Network Identifier (VNI) or the NVGRE Virtual Subnet ID (VSID). While the semantics of this field are somewhat defined, its value and how to use it is left open in the latest specifications. Its main purpose is to provide the receiving tunnel endpoint the information it needs to determine where this packet needs to be sent after decapsulation.
After the STT Frame Header has been added, this new packet (original packet + new STT header) is chopped into smaller pieces so that each piece is at least 62 bytes smaller than the MTU of the physical network. Each of these new segments receives 24 byte TCP like header, a normal 20 byte IP header, and of course the final 18 byte Ethernet header before transmission. The magic (or ugliness for those less enamored by STT) is in the TCP like header. These 24 bytes are formatted just like a normal TCP header to ensure the hardware in the NICs can re-assemble segments that belong together. The traditional Acknowledgement field in TCP is used as a fragment ID, essentially telling the NIC that all packets/segments that come in with the same fragment ID belong together and should be reassembled into the larger original ethernet frame. The traditional Sequence number is used as an offset indicator, to tell the NIC in what order the fragments need to be put together.
Similar to VXLAN and NVGRE described last week, STT has a mechanism to create entropy for the physical network to distinguish flows from each other and allow them to be balanced using ECMP (or link aggregation – LAG) based deployments. In STT, the TCP source port is used to create entropy. The originating tunnel end point will use some hash calculation on the original packets header information and use the result to populate the TCP source port. Switches in the physical network can now use the TCP port information from the tunneled packet in their hash calculation for ECMP or LAG packet distribution.
While STT is likely to be more efficient than either VXLAN or NVGRE for the transfer of large amount of information because it offloads the segmentation and re-assembly, it carries significantly more overhead than either VXLAN or NVGRE in additional header information for smaller packets. STT adds 80 bytes of new header to a VM originated ethernet packet for the first segment of this packet, 62 for each following segment. Compare that to a consistent 46 bytes for each NVGRE encapsulated packet, and 54 bytes for VXLAN. For traffic between VMs on the same server this may not matter, but it certainly does for traffic carried across the physical network. For the plentiful mice flows, we have likely doubled the size and bandwidth required for each.
A probably more significant drawback of STT comes from its strength. Designed for large packet transfers, once an original packet is encapsulated with STT header, chopped into parts, then encapsulated into individual ethernet, IP and TCP (like) headers, only the first packet provides any clue or context of the original source, destination, protocol, application and other content. The relevant pieces of that will only be found in the first segment, any follow up segments only provide enough information about the tunnel endpoints and no other original context without the first segment. And that makes debugging really hard. It also makes it hard to differentiate traffic on the physical network, even at a very high level Virtual Network identifier. And every existing network based service (realizing that one of the goals of overlay networks is to push this to the vSwitches themselves) will also have a hard time deciding what to do with these packets.
At a high level the concepts of larger packets, hardware offload, reduced CPU load and interrupts all make sense. But most data center ethernet networks can easily support 9k or even 16k packets, so perhaps the gap between 16k packet based transfer and 64k semi-stream based communication is really not that much considering that the bulk of packets are small to begin with (remember those mice and elephants?). Perhaps aligning the MTU of the virtual port with that of the network may be worthwhile to have the STT and original header in each and every packet on the wire. Regardless of whether that is a real wire, or a virtual one.
[Today's fun fact: One of the primary reasons the Mayflower pilgrims ended their voyage at Plymouth Rock was pretty much the same reason people today suspend their journeys: they ran out of beer. No need for a funny punch line on that one]
The post Stateless Transport Tunneling (STT) meets the Network appeared first on Plexxi.
"ReadyTalk is an audio and web video conferencing provider. We've really come to embrace WebRTC as the platform for our future of technology," explained Dan Cunningham, CTO of ReadyTalk, in this SYS-CON.tv interview at WebRTC Summit at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 6, 2016 10:15 AM EST Reads: 520
Financial Technology has become a topic of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming 20th Cloud Expo at the Javits Center in New York, June 6-8, 2017, will find fresh new content in a new track called FinTech.
Dec. 6, 2016 09:15 AM EST Reads: 2,141
Whether your IoT service is connecting cars, homes, appliances, wearable, cameras or other devices, one question hangs in the balance – how do you actually make money from this service? The ability to turn your IoT service into profit requires the ability to create a monetization strategy that is flexible, scalable and working for you in real-time. It must be a transparent, smoothly implemented strategy that all stakeholders – from customers to the board – will be able to understand and comprehe...
Dec. 6, 2016 09:00 AM EST Reads: 3,963
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life sett...
Dec. 6, 2016 08:45 AM EST Reads: 7,124
Successful digital transformation requires new organizational competencies and capabilities. Research tells us that the biggest impediment to successful transformation is human; consequently, the biggest enabler is a properly skilled and empowered workforce. In the digital age, new individual and collective competencies are required. In his session at 19th Cloud Expo, Bob Newhouse, CEO and founder of Agilitiv, drew together recent research and lessons learned from emerging and established compa...
Dec. 6, 2016 08:30 AM EST Reads: 879
The WebRTC Summit New York, to be held June 6-8, 2017, at the Javits Center in New York City, NY, announces that its Call for Papers is now open. Topics include all aspects of improving IT delivery by eliminating waste through automated business models leveraging cloud technologies. WebRTC Summit is co-located with 20th International Cloud Expo and @ThingsExpo. WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web co...
Dec. 6, 2016 08:30 AM EST Reads: 1,335
WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web communications world. The 6th WebRTC Summit continues our tradition of delivering the latest and greatest presentations within the world of WebRTC. Topics include voice calling, video chat, P2P file sharing, and use cases that have already leveraged the power and convenience of WebRTC.
Dec. 6, 2016 08:15 AM EST Reads: 1,671
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
Dec. 6, 2016 07:15 AM EST Reads: 1,837
Extracting business value from Internet of Things (IoT) data doesn’t happen overnight. There are several requirements that must be satisfied, including IoT device enablement, data analysis, real-time detection of complex events and automated orchestration of actions. Unfortunately, too many companies fall short in achieving their business goals by implementing incomplete solutions or not focusing on tangible use cases. In his general session at @ThingsExpo, Dave McCarthy, Director of Products...
Dec. 6, 2016 07:15 AM EST Reads: 740
Businesses and business units of all sizes can benefit from cloud computing, but many don't want the cost, performance and security concerns of public cloud nor the complexity of building their own private clouds. Today, some cloud vendors are using artificial intelligence (AI) to simplify cloud deployment and management. In his session at 20th Cloud Expo, Ajay Gulati, Co-founder and CEO of ZeroStack, will discuss how AI can simplify cloud operations. He will cover the following topics: why clou...
Dec. 6, 2016 07:00 AM EST Reads: 836
The Internet of Things (IoT) promises to simplify and streamline our lives by automating routine tasks that distract us from our goals. This promise is based on the ubiquitous deployment of smart, connected devices that link everything from industrial control systems to automobiles to refrigerators. Unfortunately, comparatively few of the devices currently deployed have been developed with an eye toward security, and as the DDoS attacks of late October 2016 have demonstrated, this oversight can ...
Dec. 6, 2016 06:30 AM EST Reads: 1,131
DevOps is being widely accepted (if not fully adopted) as essential in enterprise IT. But as Enterprise DevOps gains maturity, expands scope, and increases velocity, the need for data-driven decisions across teams becomes more acute. DevOps teams in any modern business must wrangle the ‘digital exhaust’ from the delivery toolchain, "pervasive" and "cognitive" computing, APIs and services, mobile devices and applications, the Internet of Things, and now even blockchain. In this power panel at @...
Dec. 6, 2016 06:00 AM EST Reads: 524
Internet-of-Things discussions can end up either going down the consumer gadget rabbit hole or focused on the sort of data logging that industrial manufacturers have been doing forever. However, in fact, companies today are already using IoT data both to optimize their operational technology and to improve the experience of customer interactions in novel ways. In his session at @ThingsExpo, Gordon Haff, Red Hat Technology Evangelist, will share examples from a wide range of industries – includin...
Dec. 6, 2016 05:00 AM EST Reads: 1,656
"We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 6, 2016 04:45 AM EST Reads: 1,002
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
Dec. 6, 2016 04:30 AM EST Reads: 1,935
"We're a cybersecurity firm that specializes in engineering security solutions both at the software and hardware level. Security cannot be an after-the-fact afterthought, which is what it's become," stated Richard Blech, Chief Executive Officer at Secure Channels, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 6, 2016 12:45 AM EST Reads: 870
According to Forrester Research, every business will become either a digital predator or digital prey by 2020. To avoid demise, organizations must rapidly create new sources of value in their end-to-end customer experiences. True digital predators also must break down information and process silos and extend digital transformation initiatives to empower employees with the digital resources needed to win, serve, and retain customers.
Dec. 6, 2016 12:30 AM EST Reads: 1,201
The IoT is changing the way enterprises conduct business. In his session at @ThingsExpo, Eric Hoffman, Vice President at EastBanc Technologies, discussed how businesses can gain an edge over competitors by empowering consumers to take control through IoT. He cited examples such as a Washington, D.C.-based sports club that leveraged IoT and the cloud to develop a comprehensive booking system. He also highlighted how IoT can revitalize and restore outdated business models, making them profitable ...
Dec. 6, 2016 12:30 AM EST Reads: 4,593
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Dec. 6, 2016 12:00 AM EST Reads: 919
"Once customers get a year into their IoT deployments, they start to realize that they may have been shortsighted in the ways they built out their deployment and the key thing I see a lot of people looking at is - how can I take equipment data, pull it back in an IoT solution and show it in a dashboard," stated Dave McCarthy, Director of Products at Bsquare Corporation, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 5, 2016 10:30 PM EST Reads: 1,072