How Can We Improve Decentralization for the Permaweb?

Read in Chinese

In a recent article, we covered a patch of unexpected gateway downtime caused by extreme spike in load, and gave details on how the Arweave team swiftly resolved and future-proofed the issue. In this piece, we’ll look at what application developers, miners, and Arweave enthusiasts can do to improve the decentralization of gateways – and remove the last point of failure between the permaweb and the end user.

What is a gateway?

An Arweave gateway is a combination of tools which make it easier for users to access data on the permaweb. An example of a gateway is arweave.net – the original Arweave gateway. Arweave.net fetches content from a network of miners – trusted peers which keep sync with the state of blockweave – and returns it when asked.

A key difference between a gateway and a node is that a gateway is able to accept GraphQL queries and return whole blocks of data that match. That could be anything from one transaction that matches a specific ID to every upload on ArDrive. In short, gateways choose which nodes they trust and act as http servers with the ability to search their own index.

Why run a gateway?

The recent Arweave.net crash demonstrated that even a network as radically decentralized as Arweave has single points of failure. The issue stemmed from an overload of one particular gateway, and was solved by distributing load across multiple peers via Meson – a decentralized CDN.

Here’s a vision of a better, more decentralized future for Arweave:

What if every application ran its own gateway? What about if each even ran their own nodes? The more gateways, and the more evenly traffic is distributed across each, the better the experience and reliability will be for end users.

Contrast this with the recent Fastly outage which sent Twitch, Reddit, Spotify, Stack Overflow and countless other web apps down in a single stroke. If these sites were responsible for their own content delivery, such a catastrophic failure wouldn’t have been possible. And when the majority of applications are interfacing with Arweave.net, it’s no different for the permaweb.

How to eliminate Arweave’s points of centralization

Ever since the Arweave.net gateway was knocked out with 1,000,000 requests in a single minute, the Arweave team has set about patching future issues and can already offer a number of solutions for developers.

Run Vartex

Vartex builds upon Amplify, which is a fork of the original Arweave.net gateway. It focuses on simplicity and is possible to run inside Docker. Developers can clone the source code from GitHub and follow the instructions in the readme. Either add your own node’s IP in the .env file or use an existing peer. Running Vartex on a dedicated gateway server would make it possible to own and operate a way for your application to search and submit data to the Arweave network.

Deploy an Amplify gateway

Use Amplify’s guide to deploy your own gateway on Ubuntu 20.04. The guide recommends using a minimum of 8 CPU cores and 32GB of RAM on a VPS service (Digital Ocean, AWS), but it’s also possible to run on a local machine. The process outlined in the guide is more involved than what’s required for a Vartex gateway; Vartex uses Docker to automate dependencies and database.

Amplify is currently running a gateway at amp-gw.online, and a testnet for developers at testnet.amplify.host. Its stated goals include working with key Arweave ecosystem apps like Verto, ArDrive and ArGo to migrate their current gateways over to Amplify hosts.

Run a dedicated node

It’s recommended that each gateway communicates with its own trusted node, so when setting up a gateway either choose an existing node from the peer list or run your own node for a fully decentralized approach. With the right config, nodes don’t necessarily have to be miners – it’s possible to configure a node to just sync with the network. Follow the steps in the mining guide, but remove the mining options from the start command and add sync_jobs 200 to speed up syncing data from other nodes (thanks to Ros McMahon for the info here!).

Caveat: to run a full node, a machine needs the full available data (currently ~7.5TB) plus room for growth.

Consider running nodes and gateways on your own hardware

The permaweb is only as strong as its weakest link. While centralized VPS services like Amazon and Digital Ocean have decent uptime, running the permaweb by hosting gateways across these services does not significantly improve on the web2 paradigm. It comes down to what you think is safer and can better be guaranteed.

Serve application assets with Meson

Meson is a decentralized CDN with over 36,000 nodes which are incentivized to serve content from the Arweave network. When Arweave.net was hit with unprecedented levels of traffic from the Degen Apes Academy launch, the Arweave team worked to integrate Meson with the site and distribute load across many decentralized nodes. Meson can be used to pull files from Arweave in cases where the Arweave transaction hash is know to the application. Read more here.

At a minimum, target multiple gateways

In the days when Arweave was much smaller, it was fine to direct all requests to Arweave.net, but now it’s important to add redundancy. There are plenty of alternative gateways, and targeting multiple makes use of Arweave’s decentralized nature and makes the app using it more robust.

Redstone Finance recently released arweave-multihost, an easy way for developers to declare multiple gateways for their app to communicate with, not just one. The SDK comes with default hosts, and the ability to define custom gateways too.

Watch out for Pocket Network’s Arweave integration

Here’s a bit of exclusive news: Arweave is working to integrate the Pocket Network API protocol which will allow developers to target any request that would normally hit a gateway at a Pocket endpoint. Pocket Network’s community runs thousands of incentivized nodes and already provides fast, robust and decentralized access to data from major chains including Ethereum, Binance Smart Chain, Bitcoin, Polygon and Solana.

Watch out for a full announcement on this soon. We expect this will be the missing piece that will greatly improve Arweave’s redundancy.

The Arweave team has a number of exciting projects in the works to follow Vartex, focusing on partnerships with other services in the space and ways to make it easier for developers to keep their permaweb apps running reliably and address the challenge of reliance on centralized gateways. All information will be announced on arweave.news as soon as we know about it!

Join our
Telegram / Discord / Twitter