Permaweb, a Decentralized Microservices Architecture
Web3 is booming, and Arweave is becoming a popular infrastructure choice for developers. PermaDAO is a community where everyone can contribute to the Arweave ecosystem. It's a place to propose and tackle tasks related to Arweave, with the support and feedback of the entire community. Join PermaDAO and help shape Web3!
Translator：Tom Rakis @ Builder of Permaweb
Reviewer：Viya @ Contributor of PermaDAO
Permaweb is the next-generation internet architecture proposed by the Arweave ecosystem. The key emphasis of Permaweb lies in granting applications and websites the attribute of perpetual accessibility, allowing the internet to possess memory and eliminate forgetting. Additionally, Permaweb features a serverless nature, relieving developers from deploying front-end and back-end components themselves; all developed services will be provided by the foundational service layer of Permaweb.
This article will provide a technical interpretation of Permaweb while incorporating the SCP (Storage-based Consensus Paradigm) theory.
Framework
Permaweb follows a three-layer architecture. The top layer is the Application Layer, which is the user-facing interface. The middle layer is the Service Layer, providing backend services to the application. The bottom layer is the Storage Layer, utilizing Arweave to offer data storage services.
The architecture of Permaweb is not significantly different from Web2 architecture. The Application Layer corresponds to the frontend in traditional Web2, the Service Layer corresponds to the backend, and the Storage Layer corresponds to physical servers or databases.
The most significant distinction between Permaweb and Web2 is the use of decentralized storage. Once Permaweb applications integrate permanent storage like Arweave, tampering and erasing application content becomes very difficult, granting the application decentralized properties. Although Permaweb's architecture resembles Web2, its underlying technology and design philosophy fundamentally differ.
Permaweb's architecture is shown in the figure below (from Permaweb Cookbook: https://cookbook.arweave.asia/). The following sections will provide a detailed overview of Permaweb's three-layer architecture.
Storage Layer
The storage layer is the core of Permaweb. Without blockchain technology in the storage layer, there would be no difference between Permaweb and Web2. The storage layer of Permaweb is not limited to Arweave. Theoretically, Bitcoin or Ethereum could also be used as a storage layer to create a Permaweb application, but developers and users may struggle with high storage costs. Arweave, with its permanent storage blockchain capabilities and costing only $5 for 1 GB, is the optimal choice for being the storage layer of Permaweb.
Can the storage layer use IPFS? If IPFS is used, Permaweb will lose the data traceability feature. IPFS's data CID ensures immutability, but IPFS does not use blockchain technology. IPFS data lacks block timestamps, making it impossible to discern the time of data creation. Additionally, data will be lost if IPFS service nodes are shut down, making data permanence and traceability unassured.
Service Layer
The service layer serves as a unified middleware providing APIs to applications, similar to a Web2 microservices cluster, being stateless and capable of horizontal scaling. Typically, Permaweb's service layer offers standardized interfaces and protocols (similar to Web2's gRPC or Thrift, these are entirely open-source). These interfaces and protocols are fully open-source, and any service provider that deploys services can provide the same API based on the protocol standard. As long as an application is developed using the standard protocol, it can operate and be used on different service layers.
Currently, Arweave's service layer includes four core components:
Gateway services: Caches for frequently used Arweave data. Examples include arweave.asia, arweave.dev, g8way.io, ar-io.dev, and more.
Bundling services: Using the Arweave ANS-104 standard, large amounts of data are batched and packaged onto Arweave. Examples include Bundlr, arseeding, etc. Data conforming to ANS-104 can also be queried using GraphQL.
Sequence services: Smart contract services, payment services, etc. Examples include Warp, everPay, etc.
Indexing services: Arweave Tags retrieval service, full-text indexing service. Examples include Adot, KNN3, Goldsky, etc.
Currently, the standard protocol includes only these four modules, but in the future, Permaweb's service layer could be further expanded with new standards.
Application Layer
Permaweb applications possess Serverless characteristics; developers don't need to deploy servers. As the service layer provides standard interfaces to the application layer, applications developed according to specifications can operate and be used on any service layer.
Interaction
Users only need to interact with the service layer without interacting with the underlying blockchain.
Resource loading: A standardized gateway provides all necessary file resources that the Permaweb frontend needs to load. These file resources are organized in a directory format based on the Manifest standard, making it easy for web protocols to load and assemble them.
Data writing: Writing to Permaweb usually employs the ANS-104 standard on Arweave, supporting large-scale data writing. The bundle service implemented using ANS-104 makes the writing experience on Permaweb consistent with Web2.
Data querying and indexing: A standardized indexing service enables Permaweb to load content dynamically. Indexing doesn't require waiting for data to be packed onto Arweave, once data is uploaded to the bundle service, efficient application indexes can be generated in real-time, providing users with real-time data querying capabilities.
In summary, interaction-wise, Permaweb can provide an entirely consistent experience with Web2.
Resistance to Censorship
The standard protocol and interfaces provide applications with anti-censorship features. Below are URLs of a Permaweb application named "cookbook":
You can see that any of these URLs can access the "cookbook" application. This website is served by different gateways and servers globally. If arweave.asia becomes inaccessible, users can still use other URLs to continue using the application. Even if all gateways are closed, the cookbook's data is stored using Arweave, and this data won't be lost, allowing service providers to restore the "cookbook" application at any time.
Using Arweave's permanent storage as the storage layer ensures data decentralization for every Permaweb application. Adopting standardized, open-source protocols for the service layer, can prevent censorship risks caused by the closure of individual servers due to uncontrollable circumstances. Permaweb applications possess decentralized and censorship-resistant features.
Microservices Version of SCP
The architecture of Permaweb is similar to traditional Web2 architecture. Essentially, Permaweb is a set of microservices-based application architecture built on decentralized storage.
Microservices is a software architecture pattern used in Web2 development that breaks down a large and complex application into a series of smaller, more independent service units. Each microservice is an isolated functional module that can be developed, deployed, and operated independently. These microservices communicate with each other through well-defined APIs and can use lightweight communication protocols such as HTTP or message queues to achieve interoperability.
The overall architecture of Permaweb is very similar to microservices. Each Permaweb application is composed of standardized and independent service units. This same design philosophy equips Permaweb with the capability to build complex applications on a large scale.
What sets Permaweb apart from traditional microservices is its alignment with SCP, making it an SCP-based microservices architecture.
What is SCP (Storage-based Consensus Paradigm)? It's a consensus paradigm based on storage, where the core idea is that as long as the storage is immutable and transactions on it are traceable, the result of application computation will be unique anywhere, leading to consensus. A distinctive feature of SCP is that underlying data can be infinitely combinable. As long as there are standardized ways of assembling data, applications can aggregate unique states from any storage layer, even multiple storage layers together.
When developing applications using SCP, traditional application architectures require minimal adjustments. Only the database (storage layer) must be replaced with immutable and traceable blockchain storage.
Advantages
Applications developed using this decentralized microservices architecture can acquire equivalent features of blockchain technology. These features include decentralization, immutability, traceability, resistance to censorship, and more.
Similar to traditional microservices architecture, this framework offers the following development advantages:
Modularity and Maintainability: Independent services responsible for specific functions. This standardized and modular design allows each service to be developed, tested, deployed, and maintained independently, enhancing application maintainability and flexibility.
Independence: Each service can be developed and maintained by different teams, aligning with the characteristics of DAOs. This allows various organizations and individuals to contribute their best performance and development speed to Permaweb. It also enables developers to choose the most suitable technology stack and tools. In the Arweave ecosystem, we can see different teams providing various services, such as gateways offered by ar.io, data bundling services provided by Bundlr and the open-source arseeding, and dedicated financial services offered by everPay and Permaswap, among others.
Scalability: Independent services can be horizontally scaled as needed. Each service could even be likened to the shards mentioned in ETH 2.0, but Permaweb doesn't have a limitation on the number of shards and can scale infinitely horizontally. Additionally, the same protocol standard can offer different data set services. For instance, certain gateways can optimize the access speed for specific applications by caching and processing data for those applications only.
High Availability: With multiple available gateways (microservices), the application avoids single points of failure and achieves better availability.
This architecture inherits all the advantages of microservices while overcoming the blockchain's impossible trinity. Traditional blockchain applications had to sacrifice one feature among scalability, security, and decentralization to ensure the integrity of the other two. Under the SCP theory, due to the layered architectural design, consensus will be guaranteed by the storage layer and a universal protocol. Decentralization is no longer strongly related to the number of nodes but to the openness (open-source nature) of the protocol. When discussing Ethereum's decentralization, apart from considering the number of nodes, it's crucial to recognize that the Ethereum software itself is a standard protocol. Users and miners reach consensus using the same protocol, forming a high degree of decentralization. In the case of Ethereum, the majority of users and developers currently use services provided by infura.io. Although infura.io is a centralized service, users can deploy the Ethereum protocol at any time and independently access the network. Similarly, decentralized microservices can fulfill the mentioned characteristics. However, decentralized microservices aren't tied to a specific VM architecture (like EVM). Instead, they are more flexible and closer to the theoretical model of traditional architecture. They represent the next-generation decentralized internet architecture born from the fusion of microservices and SCP theory.
Practice: From Arweave, Beyond Arweave
Currently, Permaweb has only four services: gateway, data binding, sequence, and indexing. These services have formed a certain standard. So, what exactly is a Permaweb instance, and how is it built using these four standard services?
The "Now" Application
https://now.arweave.asia is a Permaweb application (referred to as "Now" in the following text) that serves as a platform for communication among developers within the Arweave ecosystem. The primary function of Now is to allow users to initiate blogs, and other users can like the content of these blogs.
All the data of Now is stored on Arweave, ensuring decentralization through immutable storage. When a user opens Now, the browser loads a frontend framework from Arweave's gateway service, as shown in the figure below. This page framework mainly includes the Header component of Now and doesn't contain the display data of the Now application.
After the browser loads the application framework, we can see that the content display section of Now will show Loading stamps . At this point, the Now application is initiating GraphQL requests to retrieve data for the application. After a few seconds of waiting, the retrieval process is completed, and the Now application will display the latest user comments and likes (Stampers), as shown in the image below:
On this page, the initialization of the framework, all user data, and all images presented on the page are obtained through the gateway. These elements are computed and assembled into the complete Now page within the browser. The gateway provides the capability to load static resources, while the indexing service provides the ability to load dynamic data.
When a user wants to add a new blog post, they can use their wallet to upload the content to Arweave. This content will be marked with specific tags to facilitate GraphQL query functionality provided by the indexing service. After the successful upload of the blog, upon refreshing the Now application, the new content will be displayed within the content presentation of the application.
The diagram above illustrates the fundamental process of writing to and reading from Permaweb. In practical applications, both writing to and reading from Permaweb will be carried out through the service layer.
Scaling
The greatest advantage of microservices is scalability, and this feature will be fully inherited by the Now application. We can access Now using any of the following gateways:
Similar gateways can be horizontally scaled without limit, giving the Now application nearly unlimited performance capability.
The Best Practices of SCP
In the architecture of Permaweb, Arweave is only used as the storage layer. However, decentralized microservices should not be limited to this. We should deeply draw from the advantages of Permaweb and microservices architecture to explore the best engineering practices of SCP.
The following image is from the article Evolution of the Blockchain Application Model along with Changing Consensus, describing the future architecture of blockchain applications - users no longer interact with the blockchain system itself, but with the service layer. The evolution of blockchain applications is shown from Figure 1 to Figure 3, and more content can be read at:
From Figure 3, it can be seen that the blockchain relied upon by the application is not limited to Arweave, Bitcoin and Ethereum or other blockchain objects. These blockchains serve as the consensus source for the application, providing decentralization and trusted assurance. Similarly, Permaweb and decentralized microservices can also be expanded, with the storage layer not limited to Arweave. Bitcoin and Ethereum can also be used as storage layers, where the storage layer can also be other types of blockchains. One possible best practice architecture is shown in the image below:
Describing from bottom to top:
Consensus Layer: After expanding Permaweb, renaming the original storage layer to the consensus layer is more appropriate. The consensus layer can be any blockchain system, possessing the properties of immutability and traceability.
Service Layer: The key distinction between the service layer and Permaweb lies in the fact that the underlying standard protocol is not limited to Arweave storage, allowing microservices to be established for any blockchain system. Taking the indexing service as an example, Nansen provides data query capabilities for existing EVM public chains, and many DApps and frontend data can directly use Nansen as a data source. KNN3 also provides similar indexing capabilities, creating a standardized relational query layer for blockchain data and offering storage indexing compatibility with Arweave GraphQL. In this architecture, microservices have more standards and better compatibility with the consensus layer.
Application Layer: Not limited to Permaweb applications, this architecture can support decentralized applications (Web3) and Web2 applications, fully compatible with the original architectural system.
Conclusion
When looking back at the evolution of computer software engineering over the past 30 years and considering the evolution of blockchain engineering over the past 10 years, it is necessary to reflect on whether a global virtual machine (such as EVM) combined with Layer2 scaling solutions is the ultimate solution for decentralized systems.
This article did not deeply explore the feasibility of Layer2 or sharding technologies, but instead provided another possibility for the way blockchain applications are approached. Using Arweave's Permaweb as an example, this article combined the Storage-based Consensus Paradigm (SCP) and the microservices architecture to reevaluate a feasible engineering solution. This solution not only has strong scalability but also empowers applications with decentralization features. More importantly, this solution is not a mere illusion but a practical architectural system that can be implemented.
Reference
🔗 More about PermaDAO：Website | Twitter | Telegram | Discord | Medium | Youtube
