"Web3 Technology Architecture" is the second course of the Matters Web3 Online Learning Conference, taught by @刘国.

In this course, we studied and discussed the differences in the technical architecture of Web2 and Web3, the development and changes of Web3 technology from a historical perspective, the current connection between blockchain applications and decentralized networks, and the importance of decentralized networks. Meaning and efficiency problems encountered, etc.

The following is the specific content of the course. Xie Guo once again sorted out the content of the course for us in words.

Matters Web3 eLearning sessions are held every Friday at 11am on the Matters discord. Click here to enter the discord to participate.

The definition of Web3 has many facets, it can be about the way people generate and use digital assets, it can also be about decentralized networks and power structures. The topic of this discussion is the architecture of Web3 applications, so it is limited to the decentralized network structure. The following discussion assumes that participants understand the fundamentals of blockchain and are familiar with common terms such as "smart contracts".

Architecture of Web2 and Web3

Web3's technology stack is rapidly evolving while also relying on existing technologies. Users still use browsers to access web pages, and then interact with web applications through browsers. The browser itself is a complex and mature software ecosystem with slow evolution, so many Web3 technological advancements depend on the symbiosis with the browser ecosystem.

In the Web2 scenario, the user enters a web address in the browser address, such as google.com, the browser obtains the IP address of the Google server from the Domain Name System (DNS) server, then obtains the web application from the Google server, and finally renders the web application. to users. When the user interacts with the web application, the web application makes a request to Google's servers again through the browser.

In Web3, the process remains similar. At this time, the browser obtains the data fingerprint (content hash) of the web application from a blockchain-based domain name system, and then obtains the web application itself from a decentralized storage system. When a user interacts with a web application, the web application no longer initiates a request to a specific server, but reads and writes to a decentralized network such as a blockchain, a decentralized storage system, or an indexer.

Speaking of "decentralization," many discussions refer to the decentralization of API servers. But the decentralization of the domain name system is sometimes more important, especially when it comes to resistance to censorship and blocking, because the domain name system is the first step for users to use applications, and it is also the target of common attacks such as DNS pollution.

Blockchain is naturally suitable as a decentralized domain name system. The Domain Name System is a table that records the correspondence between a domain name (such as google.com) and a set of data (such as Google's server IP), and each correspondence is controlled by the owner of the domain name. This is equivalent to a global ledger, which can be implemented through the blockchain.

A common example is to use ENS on Ethereum. The above domain name can be used to correspond to both the Ethereum address and the data fingerprint of the web application. Therefore, users can use it as the abbreviation of their wallet address, and developers can also use it as the entry point of the application.

Decentralized applications (DApps) often combine blockchain and decentralized storage in this way to provide easy access to addresses. IPFS is suitable for storing static data, while ENS can dynamically point to the latest version of the application on IPFS. Users obtain data kisses through ENS, and then obtain data corresponding to fingerprints through IPFS.

There are many constraints on the cost and shape of storage on the blockchain, so applications often require additional indexes responsible for merging and processing on-chain data. A popular example is The Graph, which lets developers consolidate on-chain data into desired data structures, and then query the consolidated data using a language called GraphQL. Another example is RSS3, which integrates on-chain data into a stream of content information that is easy for clients to call.

Smart contracts often need to use real-world information off-chain. For example, a contract that buys and sells tokens based on the stock market needs to know the price of the stock market. In this case, the user needs to trust a server or a group of servers to input information into the smart contract. These servers are often referred to as "oracles," and Chainlink is a popular example.

In practice, the application will depend on more other servers. For example, most applications cannot directly connect to Ethereum or IPFS by themselves, and need to rely on service providers. These dependencies are important to the application architecture and determine the degree of decentralization and censorship resistance of the entire application.

Decentralized network evolution direction

Host Center and Content Center

The Web3 architecture we see now is the result of past technological evolution. The mental model at the beginning of the Internet design can be called "host-centric". When users use it, they first indicate the host they want to access. The browser parses the host address, renders the returned data, and trusts the host throughout the process.

For example, a user connects to Facebook's host to get her feed, and because she trusts Facebook's host at all times, Facebook can manipulate her feed to increase attention retention or insert ads. A host-centric network must be host-centric.

The TCP/IP protocol suite that makes up the Internet was originally designed to be host-centric. In the pre-Internet telephony network, a user dialed a specific machine address to initiate a conversation with another user, a mental model that was carried over as the Internet evolved. But with the further evolution of the Internet, most users are no longer using one-to-one conversations, but posting and acquiring specific data, whether it's an email from a contact, a friend's activity on a social network, or is a video file of a movie.

The host-centric model is very inefficient in this scenario, especially when specific data is very popular. For example, when two people are sitting in the same room watching the same Youtube video, the two people need to initiate a request to the remote Youtube server, establish a connection, and obtain data, but cannot share their data and bandwidth.

A more natural mental model is a content-centric network, also known as a "named data network." The idea behind it is that the user does not specify the host they want to access, but the data they want to get, making requests to many different computers. In this way, computers can easily share bandwidth. For applications, individual computers are no longer important and can replace each other. Therefore, the content-centric network is very efficient in distributing popular content, and at the same time, it is naturally easy to decentralize.

Van Jacobsen is one of the most important contributors to the TCP/IP protocol suite. He described this shift in thinking in his 2006 Google Tech Talks talk " A New Perspective on Understanding the Web ." Since then, many research groups in many countries have advanced similar lines of thought. For the progress of the update, please refer to Professor Lixia Zhang's speech " Named Networks: A New Way of Information Exchange ".

At the same time, many ideas of content-centric networks have been implemented and implemented in different projects. BitTorrent and some commercial CDNs (Content Delivery Networks) use a content-centric network structure, and Git, a common code versioning tool, is also content-centric in a sense. The recently emerging blockchain and decentralized storage are also generally structured as content-centric networks.

Node Discovery and Data Verification

In a decentralized network, a single computer is abstracted into one of the nodes. Because the user's client is no longer connected to a specific computer, it needs to constantly discover new nodes in the network. When establishing a connection with the network, the client needs a list of node addresses for startup, and then uses the node to discover more nodes in the network.

Because the client is connecting to randomly discovered nodes and cannot directly trust them, there needs to be a way to verify the authenticity of the acquired data. Therefore, when the client requests data, the way of naming the data needs to be able to verify the authenticity of the data at the same time.

One way of naming it is by the hash of the content of the request. The hash value is a digest of a piece of data. When the data is different, the digest will also be different. After receiving the data, the client can calculate the digest to verify whether the received data is requested by itself. This method is suitable for static data, and the efficiency of distribution and caching is also very high. Addresses on IPFS, transaction records in Ethereum, and commit codes in Git all use the hash value of the content.

For dynamic data, because the data can be updated, there is no way for the client to know the hash value of the data in advance. At this time, the client can verify the identity of the publisher, first request data through the hash value of the publisher's public key, and then verify the publisher's digital signature after receiving the data. Dat and IPNS verify data in this way, and virtual currencies such as Bitcoin also verify wallet balances by verifying the data publisher.

To sum up, if a decentralized and trustless network is to be formed, a client consisting of a browser and a web application needs to be able to perform node discovery and data verification locally. But browsers are very mature and complex pieces of software that evolve very slowly. Although many browser projects are experimenting with new communication protocols, such as Brave Browser, Unstoppable Browser or Agregore Browser, most web applications that want to be decentralized still rely on centralized servers.

Connecting to the Decentralized Network

In a network such as Ethereum or IPFS, data is distributed peer-to-peer between nodes. But most application clients do not run a node locally, but connect to a decentralized network through a service provider, and these connections often become centralized places. These connections also become economic exposures to the application, as application developers need to constantly pay service providers for their services, resulting in applications requiring a steady stream of cash to maintain service.

In Ethereum, the cost of storing data is paid to nodes in the network in the form of gas fees, but the bandwidth cost of reading data is not included in the design of the system. The same goes for most other blockchains. In IPFS, the network does not contain an economic model, and the cost of data storage is also handled by each node, similar to BitTorrent, Dat and other systems. So the application layer needs to use or design an incentive system to allow nodes to provide storage and bandwidth to end users.

The connection between applications and decentralized networks is changing rapidly, reflecting the evolution of Web3 infrastructure, the degree of Web3 decentralization, and the overall technological maturity.

Next we use Ethereum and IPFS for further discussion. Ethereum is representative of the blockchain, and IPFS is representative of the peer-to-peer storage network.

1. Connect to Ethereum

• Centralized Service Provider

The most common way to connect is through a centralized service provider, which can be a dedicated provider such as Alchemy or Infura, or a service provided by the application developer himself. The provider role is relatively standard and easy to replace, but still creates a single point of failure and can also be blocked or attacked. At the same time, users and clients need to unconditionally trust the data returned by the service provider, and there is no way to verify the authenticity of the data.

• Decentralized Service Provider

Service providers themselves can also be decentralized so that clients are not fixedly connected to a specific Ethereum node, but dynamically connected to one of many nodes. Decentralized service providers can use tokens to incentivize nodes to provide bandwidth to clients.

Pocket Network is an example that has recently become popular among Dapp developers. But because clients cannot start addressing locally, users still need to rely on a centralized server to connect to nodes. At the same time, because the client cannot verify the authenticity of the data, the network needs some kind of reputation system and staking mechanism to punish malicious nodes.

• index network

The index network is another way to connect to Ethereum. Nodes in the Ethereum network provide clients with data on the blockchain through standard APIs, while nodes in the index network provide clients with indexed and integrated data. Because the data structures that can be stored on the blockchain are very limited, developers sometimes need to index the network to consolidate the raw data on the blockchain into a more suitable data structure.

A common example is The Graph, which lets developers query on-chain data through a query language called GraphQL. Developers can submit code describing data structures and parsing methods to The Graph's indexing network, and then query the indexing network for custom data structures. Because clients cannot verify the authenticity of the data after integration, The Graph also needs a reputation system and a staking mechanism to punish malicious nodes.

• light node

As mentioned earlier, to form a truly decentralized network, users’ clients need to be able to verify data authenticity locally and constantly discover other nodes in the network. Nodes in Bitcoin or Ethereum operate like this, but their storage requirements are too large to run directly on the client.

Therefore, the light node in Ethereum is very important for decentralization, it is not common at present, and it is still under rapid development. Light nodes can verify the authenticity of data without storing all the data on the blockchain, so they can connect to Ethereum in a peer-to-peer manner to form a truly decentralized network.

However, because light nodes need to download data from other full nodes, they form an effect similar to the "blood-sucking donkey" in the eDonkey network, downloading more data than uploading data, resulting in increased bandwidth pressure on other full nodes. It is not yet known how severe this problem will be, as light nodes are not yet widely used. One possible solution is to increase incentives for bandwidth sharing, similar to the Pocket Network, where light nodes pay for the bandwidth used while maintaining the decentralization of the network.

2. Connect to IPFS

The most common way to read data from the IPFS network is through a public gateway, often voluntarily provided by various teams. The most common way to store data, as with Ethereum, is through a centralized service provider. These service providers can be Infura, Pinata, or Matters, and it is the cost of continuous expenditure by developers. Both of these situations also introduce a single point of failure, which is vulnerable to blockade and attack, and depends on the existence and operation of a company. They also rely on project parties or others to provide resources for free, or to find other ways to recoup their costs.

Another way to access IPFS is through a token-incentivized network, where users pay tokens to let nodes in the network store data for themselves, such as Crust Network, Arweave (via IPFS bridges), or Filecoin. Filecoin also makes users pay to read data. Meson Network, on the other hand, provides IPFS data reading, redirects clients to nearby IPFS nodes, and incentivizes these nodes to share bandwidth through tokens.

However, as long as the client cannot complete node discovery and data verification peer-to-peer, users need to unconditionally trust and rely on a centralized server, whether for data transactions or redirection. This dependency in turn will inevitably introduce a single point of failure. So the only way to form a decentralized storage network is for the client to also run an IPFS node. Compared to Ethereum nodes, IPFS requires a lot less resources, so it has been used in many desktop versions of clients, such as Brave Browser, Audius and OpenBazaar. In this case, users share data directly with each other, as BitTorrent and eDonkey do.

Decentralized category

The above discussion directly assumes that “decentralization” itself is the goal, and often it is necessary to distinguish what type of decentralization is being discussed and which ones are necessary and realistic. Vitalik Buterin has proposed to divide centralization into logical, architectural and political dimensions: whether a system can still function normally when split into two halves reflects the degree of logical centralization; how many computers and how the whole system is composed , which determines the degree of decentralization in architecture; how decisions are made in the evolution of the system determines the degree of decentralization in politics.

Logical decentralization is the most effective form of protection against censorship and attacks, as parts of the network still function as normal as the entire network. Holochain and Secure Scuttlebutt are examples of this, which are very robust and reliable in theory. However, most applications require globally shared tokens, points, rankings or other states, which requires a global ledger to become a logically centralized system.

Architectural decentralization also provides reliability, as the network does not depend on a single computer. We can also see from the previous discussion that having the client perform node discovery and data validation is important for a decentralized architecture, because it is truly independent of a single server. However, because both of these are often computationally expensive, web applications need to perform these operations efficiently through the browser. Therefore, for application developers, the degree of standardization of these operations in the browser can be regarded as an indicator of the degree of decentralization of Web3 and even the degree of technological maturity. Of course, if there is a strong enough demand for decentralization, developers still use technologies such as WebAssembly to implement these functions before browser standardization.

Political decentralization is perhaps the more important form of decentralization, because all applications evolve, and the direction of evolution should be determined by the people who use and develop the product. However, political decentralization often reduces decision-making speed and increases decision-making costs, so applications often gradually decentralize in the process of gradual maturity.

Political decentralization is outside the scope of this discussion, and probably beyond the scope of any discussion, because countless communities and societies have tried it from ancient times to the present. It is difficult to have a clear answer and solution to this topic, but it is something that each of us can and need to participate in, regardless of our understanding of technological progress.