Parallel NFS – A Modern Protocol for High-Performance Workloads

(Re)introducing NFS

Whenever technology becomes widespread and works without drama, it gets taken for granted. NFS, the file sharing protocol that turned 40 last year, is one of these technologies. Unless you’ve been in IT a very long time, NFS has always been around, and like any utility it only draws attention when it doesn’t work, which is rare.

Unfortunately, since NFS is so common and familiar, many in IT — even storage specialists — have outdated notions about its capabilities and potential. The conventional wisdom has long been that NFS is fine for general-purpose file sharing, but it is “Not For Speed.” True high-performance file sharing requires abandoning standards to deploy a temperamental, complicated, and expensive third-party parallel file system, right?

That’s what the purveyors of bespoke file systems and scale-out storage appliances would like the world to think, but it isn’t true. A dedicated cadre of professionals has been driving NFS forward, and its capabilities have advanced far beyond its humble beginnings. NFS is now capable of competing head-to-head with HPCclass parallel file systems, both in performance and scalability. That’s right — standard NFS, included with every Linux distribution, is suitable for today’s most demanding applications, including AI and other GPUcomputing workloads.

This paper is a technical overview of modern NFS, illustrating in detail how the architecture and features of the protocol have been optimized for scalable, parallel high performance. It describes the state of the art today and provides a glimpse of enhancements now working their way through the standards process. It also discusses how Hammerspace builds on this foundation with additional data services and capabilities like data orchestration and global namespace.

A Recent History of NFS and pNFS

Before diving into the details that make NFSv4.2 and pNFS so remarkable, it’s necessary to look back to truly understand how far NFS has come. Figure 1 highlights many of the important NFS milestones.

Figure 1 – Significant NFS Milestones

NFS originated at Sun Microsystems in 1984, but v3, still the most popular version today, was introduced in 1995. NFSv3 is stateless like its predecessors and relies on ancillary protocols for mounting and file locking. This makes it “chatty” on the wire and difficult to get across firewalls, as well as introducing complicated failure modes.

NFSv4 takes a different approach, using a stateful model and collapsing all functions into a single protocol operating on a single port. It also introduces the important concepts of compound operations (or simply compounds), and delegations, which will be explored in detail shortly.

Building on v4, NFSv4.1 added important security enhancements and introduced parallel NFS, the key to achieving high performance with NFS. While an important step, pNFSv4.1 still suffered from excessive ‘getattr’ chattiness and other issues, making it unable to fully realize the performance needed to compete with HPC-class parallel file systems.

With the goal of enhancing scalability by pushing more functionality into the client, NFSv4.2 added a host of optional features. Key among them is a real-time communication channel to enable clients to report back to metadata servers their experience with storage servers, including information about performance and error conditions.

In 2018, pNFS was enhanced with a new layout type called the Flexible File layout (FlexFiles), which enables the use of any NFSv3 storage system as the storage behind pNFS. Because NFSv3 support is ubiquitous, this means that any new or existing file storage can be used with pNFS. FlexFiles turned pNFS into a truly open solution, enabling existing storage environments to upgrade to parallel file system performance.

With these and other developments, pNFSv4.2 was now on a par with other parallel file systems used in the HPC industry.

Standards are good and necessary, but until they are implemented, they are just words and hand-waving. In the case of NFS, the development of these high-performance capabilities has been tightly aligned with the Linux community and submitted through the IETF (Internet Engineering Task Force) process to ensure vendor neutrality. The result is that these improvements are now part of the standard Linux kernel and have been included in most distributions since 2019. With pNFSv4.2, every common Linux distribution today contains a high-performance parallel file system that meets or exceeds those used in the largest HPC environments.

This table summarizes the key differences between NFSv3 — what most people think of when you say “NFS” — and NFSv4.2 with pNFS and FlexFiles.

Characteristic	NFSv3	NFSv4.2 (pNFS)
Performance	Single Server	HPC-class performance with RDMA support and parallel data transfers to multiple storage servers
Scalability	Single Server	Single namespace across scale-out storage servers
Efficiency	Stateless	Stateful – most operations can reliably cache state; fewer network round trips
Security	Basic	Access Control Lists (ACLs), advanced in‑flight encryption, and rigorous client authentication
Management	Simple	Simple, even at scale, with HPC performance, and across datacenters. Folded the MOUNT operation into the NFSv4.2 protocol
Client	Standard	Standard, available in every Linux distribution since ~2017

Storage, Storage System: The devices and hardware that store data

File System: The code and structures that keep data organized, presenting it to users and applications as files and directories

The Proof is in Production

Seasoned IT professionals tend to be skeptical and conservative. That’s reasonable given the usual gaps between technology hype and reality. Parallel NFSv4.1 fell short of realizing HPC-class performance, and this left a lasting negative impression on many in the industry. But the continued flow of improvements since then — including release of pNFSv4.2 with FlexFiles — has changed all that, with the protocol now stable and performant enough to be deployed in some of the world’s largest GPU-computing environments for hyperscale AI workflows. Before diving into the details of how NFSv4.2 with pNFS works, here’s a realworld example showing it in production, at scale:

Figure 2 – Meta pNFSv4.2 Implementation

Meta uses pNFSv4.2 to feed a GPU cluster training large AI models. The system runs on Meta’s standard infrastructure consisting of commodity hardware connected with standard (non-RDMA) Ethernet. No special client-side software is required on the GPU server clients or the NVMe storage servers — just standard Linux NFS. Before choosing Hammerspace and pNFS, Meta tested many other options, including proprietary systems from all the usual suspects. No other solution was able to meet their requirements for performance and vendor-neutrality, or even came close. This real-world example proves that a standardsbased approach using pNFS can deliver extreme performance at any scale, even hyperscale. For more details on this implementation, see the customer story.

NFSv4.2 Features & I/O

The enhanced capabilities of NFSv4.2 vs. NFSv3 are the result of both major architectural choices and smaller optimizations. This section describes the key factors in detail, with one exception. pNFS deserves its own section (which follows this one) not only due to its significance, but also because it is less well known. It’s discussed in more detail because while most IT professionals have experience with NFS, very few have deployed pNFS.

The features and functions discussed below were introduced at different points in NFSv4’s development, but all are present in v4.2. The rest of this paper simply refers to “NFS” unless the version matters, in which case the version (NFSv3, for example) is specified.

Compound Operations

A major enhancement introduced in NFSv4 is the COMPOUND RPC. In NFSv3 and prior, every action was a separate RPC procedure. In contrast, NFSv4 and later contain only two RPCs, NULL and COMPOUND. Using COMPOUND, clients group multiple actions into a single request, drastically reducing the required number of roundtrips to the server. This is a fundamental and substantial improvement in efficiency compared with NFSv3, especially for latency-sensitive workloads.

A compound operation is like a small program sent by the client to be executed on the server. The server executes the operations in a COMPOUND sequentially, even passing the results of one step to the next. It returns to the client only when an error is encountered, or all the steps have been executed.

Figure 3 – NFSv4.2 Compound Operations

Locking and State

NFSv3 is technically a stateless protocol, but by its nature, locking requires the maintenance of state. To implement locking with NFSv3, a separate, stateful lock manager protocol is used. Implementation of the lock manager protocol is optional, however, and locks are advisory only. But optional and advisory locking isn’t very useful. To prevent conflicts and maintain data integrity, applications need to know whether they have exclusive use of a file or not.

Because NFSv4.2 is stateful, it can implement locking within the core protocol, vs. a separate Network Lock Manager protocol. Locking in NFSv4.2 is standard and when used, locks are mandatory, not advisory, which is much more useful for applications. All state information is owned by the server. Even when some responsibility is temporarily delegated to a client, the server retains ultimate ownership.

Categories of state in NFSv4.2 include the following:

• Byte-range Locks: Enable locking a portion of a file.
• Share Reservations: Grant a client access to open a file and the ability to deny other clients access to the same file.
• Delegations: Empower clients to act independently following certain rules, which allows caching file (and in the future directory) data and metadata.
• Layouts: Used to provide clients access to data when using pNFS.

Each of these is separately granted by the server, and either returned by the client when they are finished or recalled or revoked by the server. There is also a single global timer for each client, known as a lease. Any operation by a client extends their lease, but if a client disappears the server will invalidate its state when the lease is up, preventing the hung client from denying other clients access to data.

If a server crashes, on the other hand, there is a short grace period after it recovers during which clients may reclaim their state.

So that servers can notify clients of necessary changes, such as the recall of a delegation or lock, NFS provides a callback mechanism.

Delegations

The ability of clients to cache file data and metadata is key to performance. Client caching is not new in NFSv4.2, but it is significantly advanced compared to previous versions.

In most use cases, actual file conflicts, such as two clients wishing to write to the same portion of the same file at the same time, are very rare compared to the quantity of non-conflicting operations. Instead of optimizing for the rare conflict, NFSv4.2 attempts to minimize unnecessary client/server communication for the most common situations, which involve single writers and one or more readers of a file. This is done via delegations.

The concept of delegations enables servers to temporarily hand off responsibility for certain actions to clients. When a client finishes its work, it returns the delegation to the server. If the server receives a conflicting request from another client before the delegation is returned, it uses a callback to recall the delegation from the first client, who flushes their changes and acknowledges the recall by returning the delegation.

Today, file data and attributes are cached using delegations. Directory delegations will be added in the future to expand the ability of clients to cache state and minimize traffic over the wire.

Figure 4 – Delegations and Client Caching

Delegations enable clients to cache much more aggressively than before, since they are in bidirectional communication with the server regarding conflicting activity on the file system. Rather than continually checking with the server, they simply listen for (usually infrequent) recall requests, otherwise going about their business independently.

Since servers are responsible for all state, they must be able to recover when a client, for whatever reason, fails to return a delegation. Once a client lease expires, the server may unilaterally revoke all client state, including outstanding delegations. It is the client’s responsibility to deal with the fallout from a revocation event, verifying or reclaiming state as appropriate.

Authentication and Security

In NFSv3, security is quite basic. Regarding authentication, the server considers the “client” to be the entire machine or source IP address. The client machine is responsible for authenticating individual users, and the server trusts that this authentication is taking place. NFSv4 closes this security gap, supporting end-to-end authentication from the end-user on the client to the NFSv4.x server.

Because an NFSv4.x server may support multiple security methods, even using different security for different portions of the file system, a security negotiation must take place before a client may mount a file system. NFS enables a client to query a server about the security mechanisms required for access to that server’s file system resources. Based on the answer to this query, the client can choose a security mechanism that is acceptable to both parties.

NFSv3 only supports POSIX mode bits for controlling file access, though most NFSv3 server implementations use a separate RPC program to support the simple ACLs defined in never-ratified POSIX 1003.1e/1003.2c Draft Standard 17. Support for these “POSIX Draft ACLs” was added to the Linux kernel in 2002.

NFSv4.x includes integrated support for much richer ACLs that resemble those found in Microsoft Windows. This provides finer-grained control over permissions and simplifies interoperability in mixed Windows/Linux environments.

Figure 5 below shows an example security negotiation. In this case the client makes a security proposal, which is accepted by the server. This is the simplest case.

Figure 5 – Simple Security Negotiation

Figure 6 illustrates a more complex security negotiation. The client makes a proposal, which the server rejects. The client makes a second proposal, which the server accepts, with conditions. The client agrees, completing the negotiation.

Figure 6 – A More Complex Security Negotiation

A comprehensive description of NFS security would take many pages and is outside the scope of this document. Fortunately, this activity takes place in the background and may be taken for granted when considering the operations that follow.

Mounting

In NFSv3, client mount of an NFS export is accomplished using a separate mount protocol. In NFSv4 and later, this functionality is included in the base protocol. It’s also very simple. A constant called ROOTFH is defined by the protocol that represents the top of the logical file system tree provided by each server. From this root filehandle, any other filehandle on the server may be located with a LOOKUP operation. If there are multiple file systems exported by the server, they are connected into a single virtual tree known as a pseudo-file system. This enables clients to traverse all exports even if they are discontinuous on the local file system.

Figure 7 – The Pseudo-File System

Once authenticated, a client must obtain the actual filehandle of the root of the file system as shown in Figure 8 below. PUTROOTFH sets the current filehandle to the root filehandle, and the subsequent GETFH returns the current filehandle. This sequence implicitly mounts the server’s entire export tree.

Figure 8 – Get Root Filehandle to Mount

Parallel NFS

There’s a reason that UPS has more than one truck. The only way they can hope to deliver all the packages is by doing it in parallel. It’s no different with data. Every server and network connection has a limit, and to exceed those limits requires utilizing multiple servers and paths in parallel. This is what pNFS does – it enables the efficient use of multiple servers and paths to move data faster than a single server or path ever could.

There are techniques that can be useful by themselves or in combination with pNFS, but they aren’t enough to maximize performance. These include using the NFS nconnect mount option to multiplex TCP connections (which is easy), and aggregating multiple physical interfaces using LACP (which is not easy, involving switch reconfiguration).

pNFS Architecture

What some don’t realize is that pNFS does more than just enable moving data in parallel. It also separates the metadata communication path from the data path. This empowers clients to engage directly with storage servers to read and write data without having to proxy through an intermediate server such as a “NAS head.”

If you are familiar with high-performance parallel file systems such as Lustre, this should sound familiar, because they share these same attributes. Separating the metadata and data paths and enabling clients to talk directly to storage servers, in parallel, is how those file systems achieve their incredibly high performance. It’s a proven architecture, but until pNFS it required dealing with notoriously complex and temperamental software and installing custom clients on every system that required access. That’s no longer the case. pNFS offers the same high-performance architecture in a standards-based implementation that’s included in standard Linux.

Figure 9 – pNFS Architecture

Layouts

Before a client can communicate directly with storage servers, it needs two things: 1) to understand how to reach the storage servers, and 2) because this is a shared file system, the client needs to be granted the authority to perform operations on the storage. Both things are provided by metadata servers using data structures called layouts.

From the beginning, pNFS was intended to support any type of storage back end, including block, file, and object storage. There is a different layout defined for each of these storage types, providing the information pNFS clients need to interact with them. The specification also allows new layout types to be defined in the future.

Layouts are requested and granted for specific byte ranges, not always an entire file. This enables multiple clients to work in different areas of the same file, but it also opens the door to striping, where a client writes a file in segments across multiple storage targets.

As shown in Figure 9 above, there are three types of components in a pNFS system: clients, which are the consumers of the storage service, metadata servers, which handle all metadata and coordination of client activity, and storage servers (the specification calls them data servers), which house the storage devices. Each one of these components needs to communicate with the other two. This communication happens over various protocols.

The metadata protocol used between clients and metadata servers is pNFS. Since multiple types of storage servers are supported, there are a variety of storage access protocols that may be used between clients and storage servers, including NFS.

What about the control protocol between the metadata server and the storage servers? In pNFSv4.1, this is left undefined.

Leaving the control protocol undefined seems odd until you realize it arose from an assumption: that metadata servers and storage servers would always be components within a packaged storage appliance such as a NAS system. In this scenario, it makes no sense to standardize the control protocol, since it’s never exposed outside the storage appliance.

This assumption was self-reinforcing. Since storage vendors had to invent their own control protocol, each vendor’s metadata servers can only talk to their own storage servers. This forced customers to buy metadata servers and storage servers together, because units from different vendors couldn’t interoperate. Sellers of storage appliances like this arrangement, and it’s how customers are used to buying storage, anyhow.

Figure 10 – Older pNFSv4.1 Communication Protocols

The Flexible File Layout

Let’s question that assumption: Why couldn’t the metadata servers and storage servers be from different vendors? All that would be required is a standard control protocol. And if that control protocol used only established tools and code, any common storage servers could be used. All the uniqueness and innovation would be inside the metadata server, while the storage servers became commodities.

The Flexible File layout (aka the FlexFile layout or simply FlexFiles) does exactly this. It defines the control protocol using standard NFSv3 operations. Anything that supports NFSv3 can be a pNFS storage server, from that dusty NetApp in the corner Figure 11 – Newer pNFSv4.2 FlexFile Layout Communication Protocols to a bare Linux server housing some NVMe drives.

Figure 11 – Newer pNFSv4.2 FlexFile Layout Communication Protocols

FlexFiles Control Protocol

The control protocol must provide the ability to control access to files on the storage servers. This involves both granting clients access, and more importantly, preventing clients from accessing files when it’s not appropriate, known as “fencing.”

With the FlexFiles layout, only the pNFS metadata server has root access to the backing NFSv3 storage servers, and it owns all directories that contain data files on those servers. This secures the files from all clients, fencing them out. The metadata server then selectively provides access to clients using a clever system of synthetic UIDs (user identifiers) and GIDs (group identifiers).

The UID owner of a file is allowed read/write access, and the GID owner is allowed read-only access. The layout handed to a client by the metadata server includes the appropriate UID and GID to be used as RPC credentials to access the file. The metadata server fences off clients by changing the synthetic UID and/ or GID of the file, implicitly revoking the outstanding credentials. When a client finds its access has been revoked, it will return its layout and request a new one from the metadata server, which will hand out the new UID and GID credentials if access is still allowed.

Note that the UID and GID manipulation described above doesn’t affect the access rights that end users experience when mounting the file system – that’s separate and handled over the pNFS metadata path. These processes are just a way to use existing NFSv3 data path mechanics under the covers to enable the creation of a shared parallel file system with pNFS, using commodity components and existing storage from any vendor.

pNFS Read and Write I/O Examples

Let’s step through a simple example of opening a file for reading using pNFS and FlexFile layouts. Remember that each table represents a single compound operation and thus a single round trip from the client to the server and back. Tables with blue headers represent NFSv4.2 conversations between the client and metadata server (a Hammerspace Anvil node), and tables with green headers represent NFSv3 conversations between the client and storage server.

In the first compound shown in Figure 12 below, the client opens a session, verifies access, requests to open the file and asks for the layout type for the file. Once the layout type is received, the client can ask for the layout itself, which happens in the second compound. Among other things, the layout contains a list of device IDs.

Once the client has the device IDs, it requests the device addresses for any device IDs it hasn’t seen before (which are cached). With these addresses, the client now has what it needs to directly talk to the storage servers, which happens in the fourth and final compound.

Figure 12 – Opening a File for Reading with pNFSv4.2 with FlexFiles

Write operations follow a similar flow, shown in Figure 13 below. In the first compound, the client submits an open for the file, further specifying that it only wants the file to be created if it doesn’t already exist. It provides a “layout hint” to the server, indicating the type of layout it would like to receive – FlexFile, in this instance. Finally, it requests the filehandle and layout type actually assigned by the server, which the server returns.

Using the returned filehandle and the layout type, the client next requests the layout itself. The client requests device addresses for any device IDs in the layout that it hasn’t previously interacted with, at which point – in the final compound – it writes data directly to the storage server.

Figure 13 – Creating and Writing a File with pNFSv4.2 with Flex Files

pNFS with Client-Side Mirroring, Striping, and Erasure Coding

RFC8435, which defines the FlexFiles layout, also contains a bonus feature: a definition for client-side mirroring. With traditional mirroring, the server is responsible for creating, validating, and maintaining the data copies.

With client-side mirroring, the metadata server defines the number and location of mirrors and presents the available mirrors to clients in the layout. For reads, clients choose one of the mirrors from the list to read from. Today, the Linux NFS client always uses the first mirror on the list. With Hammerspace, the Anvil metadata server load-balances requests by changing the order the mirrors are listed with every request.

Soon, Hammerspace will put local devices (those residing on the same node as the client) at the top of the list to take advantage of LOCALIO. LOCALIO, also known as NFS protocol bypass, is a recent Linux enhancement to optimize the data path when the NFS client is running on the same host as the NFS server. When this is the case, applications can directly access local storage, bypassing the network and NFS protocol stack. This lowers latency by removing unnecessary ‘hops.’

Figure 14 – NFS Protocol Bypass, aka LOCALIO

Figure 15 – Client-side mirroring READ from a triple-mirrored file

For writes, clients are responsible for sending all writes to all mirrors. A write transaction is only considered complete when all mirrors have been successfully written.

Figure 16 – Client-side mirroring WRITE to a triple-mirrored file

When clients encounter errors writing to one or more mirrors, they use the LAYOUTERROR operation (discussed in detail in the next section) to inform the metadata server of the errors. The metadata server is then responsible for repairing the mirrored copies of the file, a process known as “resilvering.”

Because it scales with the number of clients and prevents the metadata server from becoming a bottleneck, client-side mirroring enables much higher performance in large and busy storage environments vs. traditional server-side approaches.

Figure 17 – Client-Side Mirroring and Striping Layout

In addition to client-side mirroring, the protocol standards also contain provisions for client-side striping. Striping is exciting for several reasons. It provides an opportunity for higher performance for large files by striping them in parallel across multiple storage servers. It also opens the door to more sophisticated and efficient data protection using erasure coding, where clients write erasure-coded stripes across storage servers instead of simple stripes.

Figure 18 – Client Striped Read (or Write) across Five Storage Servers

Client-side striping and erasure coding are not implemented in the Linux NFS client today, but these features are in active development at Hammerspace and will be contributed into the community in future releases.

Error Recovery and Special Operations

In any system dealing with data, the top priority is preventing data loss. The technologies and protocols at every level must be sophisticated enough to handle both the normal state when everything is working properly, as well as a full range of failure scenarios. In the case of NFS, this means being able to gracefully recover from loss of communication between clients and servers, including system crash / restart events.

NFS Client and Server Crash Recovery

As described earlier, layouts include provisions for dealing with client problems. The metadata server may recall or revoke layouts as needed, for example when a client goes dark and their lease expires. This prevents situations where locks are stranded due to clients hanging or disappearing. Each time a client starts, it receives a new, unique client ID from the server. If a client crashes and restarts, losing its state, it will be granted a new client ID and the metadata server will release all locks associated with the old client ID.

NFS also includes the ability for clients to reclaim state after a metadata server crash/restart event during a special grace period. During the grace period, the server will reject certain non-reclaim requests (such as RENAMEs or REMOVEs on open files) with an NFS4ERR_GRACE error, though it will allow requests (such as READs) that can be done safely. Normal operations begin once all clients have reclaimed their locks, which normally only takes a few seconds, or the full 45-second grace period has expired.

Figure 19 – Reclaiming State After a Metadata Server Crash / Restart

Recovery of pNFS Mirrored or Striped Writes

Mirroring and striping present special problems with recovery. With pNFS, clients are responsible for writing all mirrors, and all stripes. But if a client fails, leaving some mirrors or stripes unwritten, recovery must take place to ensure consistency of all mirrors and stripes. This recovery is completed by the metadata server.

Figure 20 – Mirrored Write Recovery From a Client Crash

Mirror recovery is straightforward. Because the client has failed in the middle of the write, the application will not receive an acknowledgement that the write was successful. The metadata server will detect that the client crashed either when it restarts and re-mounts the share, or when its lease expires. It is then responsible for ensuring all mirrors match, either by completing the update to the remaining mirrors, or rolling back the changes made by the client. This process is referred to as “resilvering” the mirror.

For striped writes the situation is more complicated, and the details are still being worked out. When erasure coding is used, the metadata server may have the ability to complete the write if enough data was successfully written by the client. This example (hypothetical, as client-side erasure coding isn’t yet implemented) is shown in the figure below.

Figure 21 – Striped Write Recovery From a Client Crash

In the event a storage server fails during a mirrored write, the client is normally able to recover transparently and complete the write without the application becoming aware. If retries fail, the client will report the error to the metadata server using LAYOUTERROR (described in the next section), request a new layout, and complete the write per the new layout.

Figure 22 – Client Recovery from Storage Server Failure During a Mirrored Write

Recovery from the failure of a storage server during a pNFS mirrored read is similar, but simpler. Remember that the layout the client receives will contain a list of all the mirrors, and only one is needed to read the data.

The client will initially attempt to read from the first mirror on the list. If that storage server is unavailable, the client will read from the next storage server in the layout.

Figure 23 – Client Recovery from Storage Server Failure During a Mirrored Read

pNFSv4.2 Storage Status Reporting

With pNFS, clients communicate directly to storage servers. This potentially opens a communication gap. Metadata servers and storage servers communicate over the control protocol, so the metadata server can detect some problems with storage servers, but only from its own point of view. The metadata server has no visibility into the experience a client may be having with a storage server during I/O operations, and the original definition of pNFS in v4.1 provides no avenue for clients to report problems communicating with storage servers.

For example, if the network path between a client and a storage server goes down, the metadata server won’t be able to detect the problem if it uses a different network path to communicate to that storage server and still has connectivity. With pNFSv4.1 there was no way for a client to communicate these sorts of problems back to the metadata server.

pNFSv4.2 introduces a new _LAYOUTERROR operation to enable clients to inform metadata servers of problems experienced with storage servers. The intent of _LAYOUTERROR is for clients to report persistent errors only — they are still expected to handle transient errors on their own. The metadata server can adjust its behavior based on the information contained in _LAYOUTERROR messages, for example not handing out layouts containing storage targets the client can’t reach.

This is fine as far as it goes, but wouldn’t it be great if there was a way for clients to report not just on errors, but on the performance characteristics of their interactions with storage servers? If the metadata server understood the storage performance experienced by each client, (including network performance), it could do intelligent things when handing out layouts, such as preferring storage targets with higher performance, or load-balancing across multiple storage servers.

pNFSv4.2 introduces another new operation called _LAYOUTSTATS to enable just this sort of functionality. LAYOUTSTATS enables clients to report layout-specific performance information back to the metadata server, which may then aggregate it and act upon it.

Taken together, _LAYOUTSTATS and _LAYOUTERROR provide vital telemetry that a pNFSv4.2 metadata server can use to intelligently re-route around failures and optimize performance. This was simply not possible in earlier versions of NFS.

Figure 24 – LAYOUTERROR and LAYOUTSTATS Operations

Hammerspace and pNFS

By design, this document has focused on the NFS protocol, to bring readers up to date regarding the state of the art. But it probably has you wondering, “Where does Hammerspace fit in?” Good question!

In a pNFSv4.2 architecture, Hammerspace provides the software for the metadata servers, and optionally some or all the storage servers, running on customer-provided x86 server hardware. Hammerspace has two node types, known as Anvils and DSX (Data Services eXtension) nodes respectively.

With pNFSv4.2, Anvils act as the metadata servers. DSX nodes present front-end data access protocols and are responsible for data movement tasks. DSX nodes can also serve as storage servers (including those in a pNFSv4.2 system) using attached block storage devices such as HDDs or NVMe flash.

Figure 25 – Generic pNFSv4.2 vs. Hammerspace Implementation

Anvil Clustering

All metadata in Hammerspace (including pNFS metadata) is handled by a cluster of Anvil nodes. Clients communicate with the Anvils using a floating IP that always points to the primary Anvil.

The metadata database is replicated synchronously between all Anvils, and a heartbeat is used to monitor cluster health.

The Anvil cluster may consist of up to three Anvil nodes and an optional quorum server. The quorum server provides additional ‘votes’ during an Anvil failover to ensure cluster consistency. A single quorum server may serve multiple Hammerspace clusters.

Figure 26 – Hammerspace Anvil Cluster

Hammerspace Builds on pNFS

Hammerspace has many other capabilities that go beyond just providing bare bones pNFS capability, many of which utilize the architectural primitives of pNFSv4.2. These include:

• Simultaneously (or separately) exporting shares via pNFSv4.2, NFSv3, v4, SMB, S3, and a CSI driver for containers
• Providing a unified control and management plane for any storage, from any vendor
• Data protection via snapshots, clones, versioning, WORM, and replication, including mirroring and erasure coding
• Back-end support for block, file, and object storage
• Data-in-place assimilation of existing storage – no painful migration necessary
• Transparent and non-disruptive live data mobility
• Rich tagging and metadata support for driving workflows and reporting
• Objectives-driven actions for automated data orchestration, data protection, tiering, and other data placement tasks to support workflow automation
• Creating a true multi-protocol global namespace that can span multiple sites and clouds, including support for cloud-bursting and hybrid cloud environments

This rich feature set makes Hammerspace a data platform that provides standards-based parallel file system performance in an enterprise-friendly way. To learn more about the full capabilities of Hammerspace, see the Technology White Paper on Hammerspace.com.

Figure 27 – Hammerspace Architectural Overview

Conclusion

Most IT professionals — even those well-versed in storage — have an outdated perspective on NFS. NFSv3 is still the most commonly deployed version, despite more recent advancements that transform and modernize the protocol. pNFSv4.2 turns NFS into a parallel file system that performs and scales linearly to levels that used to require complex HPC file systems with custom clients and often vendor-locked proprietary hardware. pNFS offers the same high-performance architecture as those systems, separating metadata and data, but does so with commodity hardware and the standard client and server software already available in every modern Linux distribution.

Innovations such as compound operations and delegations improve performance while eliminating the “chattiness” NFS used to be known for. The Flexible Files layout type for pNFS enables any storage that supports NFSv3 to be part of a pNFS cluster, from existing NAS devices to generic Linux storage servers running on commodity hardware. Client-side mirroring and (soon) striping and erasure-coding further enhance scalability by enabling clients to do more of the heavy lifting. The telemetry feedback loop between clients and metadata servers adds real-time intelligence to the system, making it more resilient at extreme scales in large and complex environments.

Hammerspace builds on the standards, offering a full suite of data services, including data orchestration and multi-protocol global namespace capability across sites and clouds. Deploying pNFSv4.2 using Hammerspace delivers a system that exhibits HPC-level performance and scalability for demanding use cases, while retaining the ease-of-use and rich functionality that enterprises require.

References and Resources

The best way to learn about the deep details of NFS is to read the standards documents that define the protocol. These are known as RFCs (Request For Comment), and are created and maintained by the Internet Engineering Task Force (IETF) using a formalized process. The main RFCs for NFS are listed in the table below. Note that Hammerspace is heavily involved in the standards process, helping to define new functionality and drive NFS forward to become even more capable and scalable.

In addition to authoring RFCs, multiple Hammerspace team members are Linux kernel maintainers, helping to ensure that as new functionality is defined, support for it is built into the standard Linux NFS client and server code.

Topic	RFC Link	Date	Notes
NFSv3 Definition	RFC1813	6/1995	Co-authored by Brian Pawlowski, now at Hammerspace
NFSv4 Definition	RFC7530	3/2015	Introduction of Compounds & Delegations, enhanced Locking Co-authored by Tom Haynes, now at Hammerspace
NFSv4.1 Definition	RFC8881	8/2020	Update of NFSv4.1, including pNFS (which was introduced in the original NFSv4.1 definition, RFC5661, in 1/2010)
NFSv4.2 Definition	RFC7862	11/2016	Authored by Tom Haynes, now at Hammerspace
Flexible Files Layout Definition	RFC8435	8/2018	Co-authored by Tom Haynes, Hammerspace

More information regarding pNFS is available in the Hammerspace pNFS Resource Center including a whitepaper discussing powering AI/DL workloads with pNFS. Additional whitepapers, tech briefs, customer case studies, and other resources are located on the Hammerspace Resources page.