Arweave is a permanent storage public chain that provides users with decentralized and reliable data storage services, and it is an important infrastructure for Web3. As a rigid requirement of Web3, Web3 applications are inseparable from the support of decentralized storage. For example, on the NFT project in the last round of the bull market, although the NFT contract is released on public chains such as Ethereum, most of the NFT images are stored on decentralized storage such as Arweave or Filecoin. And this is just the beginning, there will be more Web3 applications in the future, such as videos, Twitter, and other forms of projects with higher demand for decentralized storage.

As upper level applications, projects generally do not deal directly with Arweave, but more about saving and reading data with Arweave through the gateway. Let us explore the status and development of the most important infrastructure in the Arweave ecosystem – the gateway layer.

1. Brief Description of Arweave Storage Structure

Compared to Filecoin’s separating storage and chain design, Arweave implements storage and chain integration, a design that makes it easier for users to build upper-level applications. We are introducing a few storage-related structures in Arweave as below.

1.1 Storage Structure and Chunk

Arweave Storage Schematic

Arweave does not store the data directly in the chunk, the data is placed in the storage area after it is Base64 encoded, and the transaction records the Offset and Size of the data to index the data.

When we read the data, we get the data by getting a chunk. The size of each chunk is 256Kb, and when our data is larger than 256Kb, we need to cycle through the chunks until we get all the data.

For instance, if we want to get transaction data in Block 1001, the parameter is {Offset=9800, Size=800}, where Offset means the position at the end of the data, we use Start = Offset – Size to find the position at the beginning of the data, that is, 9000Kb, so the data of the transaction is in the interval {9000Kb, 9800Kb}, we call getChunk(9000kb) to get the first chunk, and then call getChunk(9256kb ) to get the second chunk, and execute getChunk in a loop until all the data is obtained, and then stitch all the data together to get the complete data.

We usually need to do three interactions to get a complete transaction from the chain:

1. Get the basic information of the transaction: Get the basic information of the transaction through the transaction ID, including From, To, Value, Tags, etc.
2. Get the Chunk’s information: Get the Offset, Size of the Chunk using the transaction ID
3. Get Chunk’s Data: Get Chunk through Offset, and Size, after several iterations, and then put them together

1.2 Tag

The data information of each transaction in Arweave is identified by Tags, which the sending traders can upload by themselves, similar to dapp name, version, usage, input and other fields. We usually use this information to determine which application a transaction belongs to.

A transaction in ViewBlock

For example, the transaction in the image above is a data upload from KYVE. and it includes some information that KYVE needs, such as Uploader, Value, etc.

Search with Tag

Data searching in Arweave is currently implemented through tags. When we upload data, we upload the key information to the tag field, and look up through the tagged tag information when fetching information. The reason behind this is that the data stored on Arweave is stored in Base64 encoding into Chunk, which also brings complexity to indexing data directly.

2. Why Do We Need a Gateway

In order to ensure the availability of the network, Arweave is only responsible for the core storage services on the chain, while the ease of use is left to the ecological infrastructure to solve.

Combined with the above structure, we can list some of the problems associated with the direct usage of Arweave:

1. High complexity in data reading
   We know that Arweave is a probabilistic storage chain, and the nodes do not necessarily store all the data, which brings great complexity to data reading. Imagine a scenario: a user wants to pull a data, he starts by visiting the nearest node, and realizes that the node does not store the data he needs, he then starts to rotate the list of nodes in turn to request data, until he finds a node with data to download.
2. Low efficiency in data reading
   It is inefficient for users in different geographic regions to pull data. For example, a user in Asia wants to download a video of several GB, but only the node in North America has this data, so the user needs to download it from the node in North America. Such efficiency is unbearable, and it is not feasible to directly access Arweave for streaming media and other services that require high-speed bandwidth, a high-speed read channel needs to be established.
3. Difficulty in data searching
   At this stage, the data content search is based on the tags of the transaction, which are scattered in each transaction of each block. When a user conducts the search, he/she needs to traverse all the block transactions to find the transaction data that matches the target tag. In addition, Arweave-based smart contract solutions, SmartWeave and SCP mode applications are all dependent on the search function of the tags, which would not be possible without the search function provided by the gateway.
4. Data uploading
   Like downloading data, when it comes to uploading data, it is required to upload transactions and Chunks separately in multiple steps.The block generation time of Arweave takes 2-10 minutes, and each block has a maximum of 1000 transactions. In order to improve the upload efficiency, we need to use the ANS-104 protocol to package the data when uploading many transactions.
   Data loss problem: transaction id is packed, but data is not uploaded for some reason and will be lost.
5. High barriers to development
   To sum up, if the project party needs to understand a large number of Arweave features based on the original Arweave chain development, the learning cost is too high for traditional web2 development. Since the development model on Arweave is also different from public chains such as Ethereum, it also takes a long time for Web3 developers to learn. This new and complex interaction process and learning cost will slow down most developers.
   As we can see, building applications directly based on Arweave is a nightmare for developers, so we need a gateway layer to deal with the above issues. Ideally, as an application developer, it does not require much expertise to develop a dapp using a gateway, and it is even as easy as developing a Web2 application, which can lower the barriers to entry for developers and attract more ecological builders.
   In addition, the implementation architecture of Permaweb is also based on top of the gateway layer. Permaweb utilizes the GraphQL interface provided by the gateway layer to implement page loading and data lookup. SmartWeave and SCP applications also rely on the services of the gateway layer.
   Secondly, due to problems such as network congestion and data loss, we need the gateway layer to optimize functions such as data upload, and data seeding, data search to improve the availability of the entire network.
   To summarize, the gateway layer provides the following functions,
   1. Data searching
      Data search is an important function of the gateway layer, and most gateways now provide lookups in the form of GraphQL interfaces. This model is important in Arweave, implementations such as Permaweb and SmartWeave also rely on GraphQL to index the tag.
   2. Read acceleration & data cache
      It takes Arweave 2-10 minutes to generate a block, and after uploading the transactions, we have to wait for the blocks to be packaged before downloading. The gateway layer provides a cache function. After the data is uploaded to the gateway, a transaction ID is generated, and the uploaded data can be accessed immediately at the gateway through the transaction ID. The gateway will bundle the transactions and ensure that they are uploaded to Arweave within a certain timeframe.
      The gateway can also provide a function similar to CDN, when we need to access resources quickly, we can read them directly from the gateway layer. Similar to the scenario mentioned above, when users in Asia need to access data stored in North American nodes, they can read directly from the cache in the gateway layer.
   3. Data Upload
      By uploading data through the gateway, in addition to reading cache acceleration, the gateway also provides a transaction packing function. Arweave limits the number of transactions per block to 1000, but there is no limit to the transaction size, hence, with the use of this feature, multiple transactions can be packaged into one transaction for uploading purposes, which can greatly increase the network throughput of Arweave.
      In addition, some gateways also support folder upload, and the entire folder content can be uploaded to Arweave, and the gateway will automatically parse the directory structure and generate a manifest file.
   4. Path resolution
      The gateway that supports folder upload will automatically generate a manifest based on the directory structure, which allows the gateway to upload a static web page at a time and generate a reference relationship between the page and the script, and the browser can render the page directly when it accesses the page address. This format of this file is as follows:
      
      You can see that the page initially accesses the index.html page, you can find the transaction id of the resource in the “paths” list, and you can use the id to access the resources in the gateway, for example, in the arweave.net gateway, it is * “https:// arweave.net/ + Transaction ID “*. The js script of “assets/index.dd9fca7d.js” will be loaded in index.html, which is also accessed by using the transaction id. For example, the path to access the index.html page in arweave.net isL
      
   5. Data seeding
      The gateway also provides the ability to broadcast data, enabling more nodes to replicate data and increasing network availability.

Arweave Ecological Service Architecture

Generally speaking, the architecture diagram of the Arweave ecosystem can be divided into four layers:

• Arweave: the lowest layer of the Arweave network provides the perpetual storage of data
• Gateway Layer: The layer above is the gateway layer, which handles core issues such as data reading, data uploading, data searching, data acceleration, and it provides an easy-to-use interface to the upper layers.
• Basic Services Layer: This layer provides the basic services required by the Arweave ecosystem, such as the well-known Bundlr, WeaveDB, Warp, everpay, etc., which are also built on the gateway layer and play the role of linking the top and bottom.
• Application layer: This layer is the ecological application that we are familiar with, such as Mirror, ReadOn, etc. are all applications, and they use the basic service layer or directly use the gateway layer to access Arweave.

The gateway layer plays the role of linking the top and bottom, providing a convenient and reliable way for upper-layer applications to access Arweave. It can be said that how easy it is for the gateway layer to be used will affect the prosperity of the upper-layer applications, and the emergence of large-scale Web3 applications requires a powerful and secure gateway layer.

3. Introduction of Existing Gateways

There have been many excellent gateway projects in the Arweave ecosystem, each of which has its own characteristics and solves different problems, the following is an introduction to several important gateways in the Arweave ecosystem.

3.1arweave.net

arweave.net is the earliest officially provided gateway and it is currently the most widely used gateway. Many projects, such as Bundlr, SmartWeave, Permaweb, etc. are built on the arweave.net gateway. The arweave.net has made great contributions to early ecological construction.

arweave.net provides rich functions, such as instant access to the data cache, data upload, GraphQL interface, etc., it also supports web page uploading and rendering. Permaweb is implemented using arweave.net, where the front-end static content is stored in Arweave, and the dynamic data is obtained through GraphQL to assemble dynamic web pages. now.arweave implements a real-time content website using this approach. (https://now.arweave.dev/)

As an early gateway, arweave.net also has its problems, such as:

• net is a centralized gateway, and although it has the official endorsement, there is still a risk of lack of trust.
• Due to its early launch and official endorsement, arweave.net has become the choice of most projects, and hence the ecological projects rely heavily on arweave.net which can be prone to single point of failure.
• There are known bugs in the arweave node transaction broadcast service. The txs received by the net gateway cannot be broadcast to other block producers in time. As a result, the arweave.net gateway pending pool accumulates a large number of txs, but other block producers have no transactions to package. The consequence is that a large number of transactions in the pending pool will expire.

Although arweave.net has accumulated a large number of users, it has shortcomings in terms of functionality and centralization. As an early gateway, we seem to see that arweave.net is being replaced by a new generation of gateways with more powerful functions, and it is also officially stated that arweave.net will be migrated to ar.io in the future.

3.2 Arseeding

Arseeding is an Arweave light node with a gateway function. Other than supporting data upload and download, it also integrates everpay for fee payment. Users can pay with ETH, USDC and other tokens. At the same time, Arseeding is also able to provide a content reading acceleration function which is similar to CDN, this is very important for application layer projects. Arseeding is a very comprehensive function and is a powerful gateway infrastructure in the Arweave ecosystem.

How Arseeding is Used

There are 2 ways for users to use Arseeding service: (refer to the development document: https://web3infra.dev)

1. Build the Arseeding light node by yourself
2. Use the official web3infra gateway directly

When it comes to building an Arseeding light node by yourself, the project party only needs to synchronize its own business data without paying attention to other data. This is very similar to the Ethereum’s wallet light node function. At the same time, Arseeding also supports data caching, project parties can read and find data quickly through the light node, which is very important for websites like YouTube and Twitter that need to provide high-speed and timely services.

If the project party does not need to build a light node by itself, the official web3infra gateway can be used instead. Both Arseeding and Web3infra support the ANS-104 packaging protocol, which allows transactions to be uploaded in bulk.

The Importance of Light Nodes

Why does Arseeding want to make light nodes? The developers of Arweave ecosystem face several serious problems:

• Ecological projects rely heavily on the net gateway and suffer from a single point of failure (Bundlr also accesses the arweave.net gateway to upload transactions)
• There are known bugs in the arweave node transaction broadcast service. The txs received by the net gateway cannot be broadcast to other block producers in time. As a result, the arweave.net gateway pending pool accumulates a large number of txs, but other block producers have no transactions to package. The consequence is that a large number of transactions in the pending pool will expire.
• Arweave nodes do not store the full amount of data, and users may have to search a number of nodes to obtain the desired data, especially for data with a low replication. At the same time, the nodes that have data may be far from what the users demand, resulting in low data transmission efficiency.

In order to solve the above problems, Arseeding implements the following functions:

Architecture diagram of the web3infra.dev

• Transaction Broadcast

Arseeding will automatically broadcast transactions to all Arweave nodes in the network and is fully compatible with the Arweave native nodes. This ensures that the transaction is received by all Arweave nodes in the pending pool in time, increasing the rate at which transactions are packaged.

• Data Seeding Service

When a developer uses Arseeding, all data will be stored in the Arseeding node, and the developer can submit a valid Arweave transaction ID. If the transaction data exists, Arseeding will initiate a scheduled task to broadcast the transaction to the Arweave nodes on the entire network. If the transaction data does not exist, Arseeding will first initiate a data fetching task from the entire network, and then broadcast to the entire network after obtaining the data, so that a copy of the data can be stored by the nodes of the entire network.

• Data Acquisition Service

Through the Arseeding node, **developers can call special interfaces to automatically request specified data from the entire network. The user only needs to submit the arId, and the light node first fetches the transaction data from the arweave.net gateway, and if the acquisition fails, it fetches the transaction data from the nodes in the entire network until the acquisition succeed. **At the same time, Arseeding will also cache the synchronized data and provide high-speed download service, similar to CDN acceleration service.

It can be seen that in addition to basic gateway services such as data upload and download, Arseeding also has functions such as data broadcasting, data seeding and CDN gateway. Arseeding strengthens the function of arweave.net gateway, and the extension of such core services is very important for ecological developers.

everpay payments

Arseeding the use of everPay for storage fee payment, which means that holders of ETH, USDC and other tokens can also use Arweave’s file storage service.

Node Open Features

Arseeding’s nodes deployed by the project owner can also be opened to all external users, and third-party applications like AskMirror (https://askmirror.xyz/) can be built using the project owner’s open nodes, making the application more open and scalable. The public Arseeding nodes will use everPay and users can use any token supported by everPay to make payments.

Arseeding’s light node model provides better flexibility and scalability to the application party, also, data synchronization and data seeding features provide protection for the data integrity, hence, everPay brings convenience to payment. The various features of Arseeding make it one of the best gateways in the Arweave infrastructure today.

3.3 ar.io

The purpose of ar.io is to build a decentralized and incentivized public gateway, under the responsibility of the ArDrive team.

At the same time, ar.io integrates the team from Vartex. Vartex is an open source gateway tool built to decentralize Arweave from Arweave.net, and their main team will join ar.io to develop the next generation gateway. Arweave.net will also transition to this new gateway client, and transfer operations from the Arweave core team to ar.io DAO.

In February 2022, ar.io raised a $17.2 million in funding, led by Arweave Team, Blockchain Capital. From the scale of financing, to the strength of the team, there are good configurations, along with the official endorsement, it can be said that at the right time and place, it strives to become the core gateway in the Arweave ecosystem.

http://ar.io/start/

ar.io is still in the early development stage. According to the information disclosed so far, the summarized features are as follows:

• Decentralization

The ar.io gateway will adopt a decentralized network architecture where any person and organization can run a node to become a gateway operator and earn incentives.

• Incentive Layer

As a decentralized gateway, the incentive layer is key. ar.io will launch an incentive token, $IO, to motivate people to join the ar.io network and provide a more secure gateway service.

The tokens will be issued using SmartWeave while providing Staking functions, and used for DAO governance. The tokens also lease the ArNS domain name.

• DAO

ar.io will operate as a community, returning control to all users and builders, with governance tokens using $IO tokens.

• Basic Function

The basic functions disclosed by ar.io include data seeding, data caching, data indexing, etc.

• Domain Name Service

ArNS (Arweave Name System) is also a part of ar.io. ArNS provides a decentralized domain name service based on Arweave, and users can use ArNS to access Permaweb applications. The existing access method is areave.net plus the transaction ID (*https://arweave.net/{transactionId})*. With ArNS, user-friendly link addresses can be created.

Each ar.io gateway acts as a cache and name resolver. Users utilize the ArNS token (ANT), which identifies the Arweave transaction ID they wish to qualify for name registration. they then register their ANT to map to an available name of their choice through the ArNS registration system, which is also supported by a separate Smartweave contract. ANT is transferable like any other Smartweave token, and ANT owners can update their appointed Arweave Transaction ID at any time.

• Payments

ar.io will support token payments in multiple chains and connect to centralized payment platforms like Stripe.

• Customized Gateways

ar.io mentioned that a customized gateway service can be built where applications only synchronize its relevant data or provide specific types of queries and acceleration services. This format is very similar to Arseeding’s light nodes.

It can be seen that ar.io provides very comprehensive functions, from basic data seeding and data indexing, to customized gateway services, to ArNS domain services, to various token payments, which can be said to include the core functions of other existing gateways. At the same time, the network and governance also adopt a decentralized form, which is more in line with the Web3 form. All in all, ar.io is a highly anticipated project, and it is expected to become the core basic gateway of the Arweave ecosystem.

3.4 Arg8

Arg8 is a lightweight decentralized gateway, its goal is to achieve an incentivized layer 2 network where anyone can participate in, users can access Arweave through the Arg8 network.

Arg8 realizes that the gateways in the Arweave ecosystem rely on a few providers, which is prone to single points of failure, as verified by the previous congestion of arweave.net. Arg8 is committed to enhance the decentralization and availability of the Arweave ecosystem.

Arg8 is the first implementation version of the P3 protocol, it uses an open layer 2 network in which anyone can participate and provide services as nodes to earn incentives. This open network format necessarily requires financial incentives to maintain the stability and availability of the network.

Permaweb Payment Protocol (P3) protocol

P3 Network Architecture

P3 is a payable service protocol that allows users to pay for the services they use on Permaweb using $AR. Permaweb is a permanent network running on Arweave, it is similar to L2, and all ecological applications run on Permaweb. Using the P3 protocol allows applications and users on Permaweb to access each other’s data, send transactions, make queries, and pay fees without having to run nodes. This is somewhat similar to the role of a gateway, P3 becomes a payment gateway after adding its payment function.

Other than this, in addition to the data access capability of the gateway, service providers can offer more complex services, it can be said that any service that conforms to the P3 protocol can run on this, such as gaming services, search services, Defi services, etc. The P3 protocol adds composability standards to the Arweave ecosystem.

• Optional service marketplace

Any node can provide the same service, and users are free to choose the nodes with low cost and good services.

• Composability

P3 allows any developer to insert their own services into the network and provide services, and as long as they use the same payment protocol “P3”, the services can call and pay each other.

P3 protocol content

The P3 protocol (P3 protocol content) defines six types of service interfaces, including service listings, price acquisition, deposits and withdrawals, service requests, service ranking, etc.

Users can make deposits to the service node in advance, and the service node will deduct according to the configured service price and provide the user with a statement, so that the service node only needs to settle the payment regularly.

Arg8 allows us to see an open and incentivized service gateway, while the P3 protocol gives more room for expansion in terms of service composability, which will be a fundamental form of future decentralized gateways. Furthermore, the P3 protocol has the potential to connect various services built on the basis of SCP on Permaweb, such as everPay, Mirror, RedStone, etc., just like each contract in EVM can be called and combined, which will build a more open network and bring more imagination to Permaweb.

Other gateways:

• Warp
  
  Warp gateway is a dedicated gateway, primarily for Warp contracts, to provide fast and reliable data access.
  

• Meson.Network
• Amplify

4. Future Direction of Exploration

1. Decentralization
   According to Permaweb’s vision, Web3 applications built on Arweave should be completely decentralized, and the only problem now is that the gateway we use is the centralized arweave.net. Hence, the future of the gateway must be decentralized, just like the operation mode of ar.io and arg8.
2. More Optimized CDN Service
   Nowadays, traditional networks have very powerful CDN services that store web content closer to the user, thus speeding up the delivery of web content, with data centers all over the world using caching. CDNs cache content such as web pages, images and videos in proxy servers close to the user’s physical location, providing high-speed access.
   The gateway layer of Arweave should also have this function, which is very important for large-scale use of web applications. The current gateway service has basic CDN functions, but in the future a more powerful and professional CDN service should emerge to cope with global user access.
   Unlike the traditional Web2 networks, traditional CDNs only store the static content of web pages, and the dynamic content is loaded from application servers. In the Arweave ecosystem, both static and dynamic content of web pages are stored on Arweave, which makes the gateway as a CDN to be able to handle such dynamic content accurately. Existing development models such as SmartWeave, SCP are also based on GraphQL provided by the gateway to obtain data, and how to provide fast and reliable data access for such applications might also be a direction.
3. Streaming Service
   Streaming services place extremely high requirements for data access speed. In addition, the data of live applications may be time-sensitive and do not need to be permanently stored. How to provide high-speed and removable services for streaming media data is also a research direction.
4. Customized gateway
   General gateways such as arweave.net provide the basic functions of a gateway, and these basic functions can be used by us on a daily basis. However, some large applications, such as uniswap, and chat applications, may have higher requirements for gateway services, and enterprise-level custom gateways will also be a direction in the gateway vertical field.
5. Data search
   Data search will be a very important direction, Arweave-based data search is very different from traditional data search engines. At this stage Arweave mainly relies on the transaction tags for search, and there is no content-based search yet. In addition, because the data ownership of Web3 is also fundamentally different from traditional Web2, in the future data validation and data transactions based on the chain may become a new type of application, which brings more unknowns and imagination to data search methods.

References：

• ANS-104
• https://github.com/joshbenaron/arweave-standards/blob/ans104/ans/ANS-104.md
• 什么是Arweave
• /what-is-arweave-ar-chinese/
• Arseeding
• https://web3infra.dev/zh-cn/docs/arseeding/introduction/lightNode
• Arg8
• https://arweave.live/#about
• P3协议
• https://kkame6mgezvgxz3tjgvrevjrv6nmykkgcfvor2hwpgoiw3j2li2a.arweave.net/UoDCeYYmamvnc0mrElUxr5rMKUYRaujo9nmci206WjQ
• Gateways
• https://arwiki.wiki/#/en/gateways
• manifest
• https://ar-io.zendesk.com/hc/en-us/articles/5300353421467-Arweave-Manifests
• ar.io
• https://ar.io/
• /ario-gateways/
• https://github.com/ar-io/arweave-gateway
• ArNS
• https://ar.io/arns/

加入PermaDAO，建设 Web3！一起做 Buidler

Translator：John Khor @ Contributer of PermaDAO

Editor ：Xiaosong HU @ Contributer of PermaDAO

Join our

Telegram / Discord / Twitter