Privacy-Respecting Analysis of Veilid Performance and Trends
A transparency report on the infrastructure run by Veilid Foundation.
Background
Veilid is a privacy-first project.
We have intentionally not baked into Veilid any sort of analytics, diagnostics, or other “phone home” mechanisms to send telemetry, statistics, or performance data back to us for measuring node or network performance, network size, or other bits of info as one might commonly find in many 21st century tech products.
In our discussions with Veilid node operators, we’ve seen a recurring theme of needing to figure out realistic system requirements for running Veilid nodes. In order to determine realistic system requirements, we need to understand how Veilid nodes and the network perform under load. We face two significant challenges in measuring node and network performance. First, there’s currently very little application traffic on the network. As can be expected at our current stage of development, there just aren’t many Veilid-based apps using the network. Second, we have the aforementioned deliberate lack of phone-home analytic data.
Our only source of performance data is the small fleet of Veilid nodes we operate. Our four headless nodes are no different than anyone else’s. They are installations of the veilid-server Debian package from our packages.veilid.net repository, just like everyone else has access to. Along with our standard headless nodes, we also maintain the two bootstrap nodes, which run the same veilid-server binary with most of the Veilid protocol capabilities deactivated in the config.
At DEF CON 32 we launched the beta test of our flagship Veilid-based app: VeilidChat. Leading up to that event, we decided to watch our own nodes more closely before, during, and after VeilidChat’s release so that we could start piecing together an idea of how the protocol and network are performing and how nodes are handling the load. We stood up a Graylog instance as an analytics tool to help us make sense of the data from our nodes. From an operational standpoint, this allows us to keep an eye open for errors after new version releases, monitor bootstrap availability, and other normal operational stuff. It also allows us a method by which we can monitor system resource usage on our nodes.
Some time after DEF CON, we were lamenting the fact that our commitment to put privacy first left us with little visibility into the makeup of the broader network. Not that we regret being privacy-first, it just sucks having no way to determine what versions of veilid-server are running in the wild, whether operators are updating, and so on. We took stock of what data we had in Graylog and came up with a solution which gives us a rough idea of how many nodes are on the network and what versions are in use in a way that we believe preserves privacy and is detailed below.
What data are we collecting?
We are currently collecting some anonymous performance metrics from 8 hosts operated by the Veilid Foundation:
- 2 Veilid bootstrap nodes
- 4 Veilid headless nodes
- Veilid build orchestration host
- Veilid package repo host
We are collecting resource utilization metrics (CPU, memory, network) from all 8 hosts using Metricbeat.
We are collecting nginx logs from the package repo host to track downloads of the Veilid Server and CLI packages.
We are collecting Veilid debug logs from the 2 bootstrap and 4 headless nodes.
What are we doing with the data?
All of the log data is sent to a Graylog instance that lives alongside the other 8 hosts in our cloud environment. The log data is parsed into well-defined fields to enable easy dashboarding and analysis. The data is retained for no less than 30 days and no more than 40 days.
How are we avoiding collection of any sensitive or identifying data?
Both the Veilid debug logs and package repo logs contain the IP addresses of the machines they’re talking to. We do not want to store anyone’s IP address, however we do want to be able to correlate what we’re seeing from those two sources. For this reason, we calculate and save the SHA256 hash of each IP and then drop the IPs from the data before it gets stored.
How are we drawing inferences about the network?
In order to get a rough idea of the number of active nodes in the network, we’re counting the number of unique IP hashes we see on a daily and weekly basis. In a recent week, we saw anywhere from 315 to 340 unique IP hashes per day and 542 unique IP hashes total for the week.
To get an idea of what versions of veilid-server are active on the network, we start with the package repo logs. When $IP_HASH downloads a veilid-server package from our repo, we add an entry to a lookup table that says “$IP_HASH is running version X.” When $IP_HASH appears in the debug logs from our headless nodes, we use the lookup table to associate the server version with the log data. If $IP_HASH isn’t found in the lookup table, we mark the logs as “unknown version.”
Currently (as of 12/15/2024), \~45% of the logs we see from our headless nodes have an unknown veilid server version, \~39% are on version 0.4.1 (the latest version), and \~12% are apparently running version 0.3.4. We believe the “unknown version” traffic falls into a few buckets:
- Headless nodes that people spun up and never updated 👎
- Headless nodes compiled from source rather than downloaded from our package repo 👍or 👎depending on if they’re updating
- VeilidChat nodes ⭐️⭐️⭐️
- Other projects or apps that people are building on the Veilid network ⭐️⭐️⭐️(As long as they’re updating)
Have we seen anything interesting?
We were able to see steady growth in the number of node operators updating to 0.3.4 when it was released in August of 2024 (the gold line):
Similarly, we were able to see that a number of node operators appear to have set up automatic updates to the latest nightly build:
And when Veilid 0.4.0 was quickly followed by 0.4.1, we were able to track how quickly folks were switching to the latest version:
In Closing
Aside from the information we get from the package repo logs, all of the above is built from data that is available to anyone who runs a headless Veilid node. If you’re a node operator who wants to build your own monitoring setup, check our GitLab to see how we are doing it.
We understand that it can be a bit of a head-scratcher to hear that a project is privacy first, but also that one can install a headless node, open veilid-cli, and see node IDs right next to their IP addresses. For nodes that move between IP addresses, we will be introducing a change to generate new node ids for each IP address used, minimizing the ability to track a node by its public node ID.
By design, however, in Veilid apps, ‘Node ID’ is not tied to your ‘identity’. We recently released an animated intro video which gives an overview of what the Veilid project is. Coming soon is a deep dive in the same format on how Veilid functions to preserve user privacy. Until that video is made and released, here is a brief placeholder:
In the context of a Veilid based application, the app user’s IP address is guarded from being associated with their identity, because node IDs are not directly used in app-level communication, rather private/safety routes are used. In short, when having a conversation on VeilidChat, the participating nodes do not know, nor do they need to know, each other’s node ID or IP address – but more to come later on that topic.
In the meantime, this rough analysis of the data we have on hand has made it clear that out-of-date headless nodes exist in significant numbers on the network and are likely strong contributors to some of the VeilidChat beta issues. In a near future release, we will introduce a “capability version” advertisement function into veilid-core. Nodes already advertise their enabled set of Veilid capabilities, so this new bit of data will likely join that packet. This will be a breaking change in that nodes from that version forward will not communicate with nodes that either do not properly advertise a version or are advertising a deprecated version. This concept is still in the early phases of design. If you want to be a part of development, please join us at veilid.com/discord!