Infrastructure

Announcing Narwhal & Tusk Open Source

Mysten Labs is open-sourcing Narwhal & Tusk, our high-throughput mempool and consensus engine.

Announcing Narwhal & Tusk Open Source

Mysten Labs is open-sourcing Narwhal & Tusk, our high-throughput mempool and consensus engine.

Narwhal & Tusk (N/T) powers critical aspects of our smart contracts platform Sui:

  • ordering all transactions involving shared objects using a consensus protocol
  • composing the protocol dealing with owned objects with the protocol
  • sequencing the transaction to guarantee Sui’s security properties

The dual name highlights that the system splits the responsibilities of:

  • Ensuring the availability of data submitted to consensus (Narwhal)
  • Agreeing on a specific ordering of this data (Tusk)

Narwhal & Tusk represents the latest variant of decades of work on multi-proposer, high-throughput consensus algorithms that reaches throughputs more than 130,000 transactions per second on a WAN, with production cryptography, permanent storage, and a scaled-out primary-worker architecture.

Why are we open sourcing?

It was clear that we were discussing ideas, problems and solutions that resonate with developers in regards to scalability within web3. We hope that by sharing Narwhal & Tusk, we are able to bring our learnings forward and help accelerate likeminded teams.

Our experience talking with peers across the industry indicates that teams are looking for some of the same technologies within our high-throughput mempool and consensus. By sharing the code as open source, we hope others can help the overall web3 community reuse those technologies; in fact, we’re looking forward to the concepts and features in Narwhal & Tusk to land in other scalability solutions. We will be using Narwhal & Tusk as part of Sui.

Lastly, open sourcing feeds our desire to give back and share with the community outside Mysten Labs. We believe this will lead to active engagements across teams who are working on similar engineering challenges.

Preparing for open sourcing

The codebase has been designed to be small, efficient, and easy to benchmark and modify. We’re still working on making the project ready for production, yet it already uses real cryptography (ed25519 & other signing schemes), networking (tokio), and storage (rocksdb).

The core protocols are written in Rust, but all benchmarking scripts are written in Python and run with Fabric. To deploy and benchmark a testbed of four nodes on your local machine, clone the repo and install the Python dependencies:

$ git clone <https://github.com/mystenlabs/narwhal.git>
$ cd narwhal/benchmark
$ pip install -r requirements.txt

You also need to install Clang (required by RocksDB) and tmux (which runs all nodes and clients in the background). Finally, run a local benchmark using Fabric:

$ fab local

This command may take a long time the first time you run it (compiling rust code in release mode may be slow), and you can customize a number of benchmark parameters in fabfile.py. When the benchmark terminates, it displays a summary of the execution similarly to the one below.

-----------------------------------------
 SUMMARY:
-----------------------------------------
 + CONFIG:
 Faults: 0 node(s)
 Committee size: 4 node(s)
 Worker(s) per node: 1 worker(s)
 Collocate primary and workers: True
 Input rate: 50,000 tx/s
 Transaction size: 512 B
 Execution time: 19 s Header size: 1,000 B
 Max header delay: 100 ms
 GC depth: 50 round(s)
 Sync retry delay: 10,000 ms
 Sync retry nodes: 3 node(s)
 batch size: 500,000 B
 Max batch delay: 100 ms + RESULTS:
 Consensus TPS: 46,478 tx/s
 Consensus BPS: 23,796,531 B/s
 Consensus latency: 464 ms End-to-end TPS: 46,149 tx/s
 End-to-end BPS: 23,628,541 B/s
 End-to-end latency: 557 ms
-----------------------------------------

Next steps

  1. Read the Narwhal and Tusk README.
  2. Read the paper Narwhal and Tusk: A DAG-based Mempool and Efficient BFT Consensus.
  3. Recommended to have a look at the README files of the worker and primary crates.
  4. An additional resource to better understand the Tusk consensus protocol is the paper All You Need is DAG as it describes a similar protocol.
  5. The README file of the benchmark folder explains how to benchmark the codebase and read benchmarks’ results. It also provides a step-by-step tutorial to run benchmarks on Amazon Web Services (AWS) across multiple data centers (WAN).

Further reading

Narwhal and Tusk (Danezis et al. 2021) is a consensus system leveraging directed acyclic graphs (DAG). DAG-based consensus has been developed over the last 30 years, and some of the history is summarized in (Wang & al. 2020). The closest theoretical ancestor of Narwhal & Tusk is (Keidar & al. 2021).

Narwhal & Tusk are developed in the asynchronous model. A partially synchronous variant of Narwhal and Tusk, called Bullshark, is presented in (Giridharan 2022).

Narwhal and Tusk started as a research prototype at Facebook Novi.

Appendix

  • Danezis, G., Kogias, E. K., Sonnino, A., & Spiegelman, A. (2021). Narwhal and Tusk: A DAG-based Mempool and Efficient BFT Consensus. ArXiv:2105.11827 [Cs]. http://arxiv.org/abs/2105.11827
  • Giridharan, N., Kokoris-Kogias, L., Sonnino, A., & Spiegelman, A. (2022). Bullshark: DAG BFT Protocols Made Practical. ArXiv:2201.05677 [Cs]. http://arxiv.org/abs/2201.05677
  • Keidar, I., Kokoris-Kogias, E., Naor, O., & Spiegelman, A. (2021). All You Need is DAG. ArXiv:2102.08325 [Cs]. http://arxiv.org/abs/2102.08325
  • Wang, Q., Yu, J., Chen, S., & Xiang, Y. (2020). SoK: Diving into DAG-based Blockchain Systems. ArXiv:2012.06128 [Cs]. http://arxiv.org/abs/2012.06128

Read next