GNS — the GNU Name System

The GNU Name System (GNS) is a decentralized database that enables users to securely resolve names to values. Names can be used to identify other users (for example, in social networking), or network services (for example, VPN services running at a peer in GNUnet, or purely IP-based services on the Internet). Users interact with GNS by typing in a hostname that ends in a top-level domain that is configured in the “GNS” section, matches an identity of the user or ends in a Base32-encoded public key.

Videos giving an overview of most of the GNS and the motivations behind it is available here and here. The remainder of this chapter targets developers that are familiar with high level concepts of GNS as presented in these talks.

Todo

Link to videos and GNS talks?

GNS-aware applications should use the GNS resolver to obtain the respective records that are stored under that name in GNS. Each record consists of a type, value, expiration time and flags.

The type specifies the format of the value. Types below 65536 correspond to DNS record types, larger values are used for GNS-specific records. Applications can define new GNS record types by reserving a number and implementing a plugin (which mostly needs to convert the binary value representation to a human-readable text format and vice-versa). The expiration time specifies how long the record is to be valid. The GNS API ensures that applications are only given non-expired values. The flags are typically irrelevant for applications, as GNS uses them internally to control visibility and validity of records.

Records are stored along with a signature. The signature is generated using the private key of the authoritative zone. This allows any GNS resolver to verify the correctness of a name-value mapping.

Internally, GNS uses the NAMECACHE to cache information obtained from other users, the NAMESTORE to store information specific to the local users, and the DHT to exchange data between users. A plugin API is used to enable applications to define new GNS record types.

libgnunetgns libgnunetgns ————

The GNS API itself is extremely simple. Clients first connect to the GNS service using GNUNET_GNS_connect. They can then perform lookups using GNUNET_GNS_lookup or cancel pending lookups using GNUNET_GNS_lookup_cancel. Once finished, clients disconnect using GNUNET_GNS_disconnect.

Looking up records

GNUNET_GNS_lookup takes a number of arguments:

handle This is simply the GNS connection handle from

GNUNET_GNS_connect.

name The client needs to specify the name to

be resolved. This can be any valid DNS or GNS hostname.

zone The client

needs to specify the public key of the GNS zone against which the resolution should be done. Note that a key must be provided, the client should look up plausible values using its configuration, the identity service and by attempting to interpret the TLD as a base32-encoded public key.

type This is the desired GNS or DNS record type

to look for. While all records for the given name will be returned, this can be important if the client wants to resolve record types that themselves delegate resolution, such as CNAME, PKEY or GNS2DNS. Resolving a record of any of these types will only work if the respective record type is specified in the request, as the GNS resolver will otherwise follow the delegation and return the records from the respective destination, instead of the delegating record.

only_cached This argument should typically be set to

GNUNET_NO. Setting it to GNUNET_YES disables resolution via the overlay network.

shorten_zone_key If GNS encounters new names during resolution,

their respective zones can automatically be learned and added to the "shorten zone". If this is desired, clients must pass the private key of the shorten zone. If NULL is passed, shortening is disabled.

proc This argument identifies

the function to call with the result. It is given proc_cls, the number of records found (possibly zero) and the array of the records as arguments. proc will only be called once. After proc,> has been called, the lookup must no longer be canceled.

proc_cls The closure for proc.

Accessing the records

The libgnunetgnsrecord library provides an API to manipulate the GNS record array that is given to proc. In particular, it offers functions such as converting record values to human-readable strings (and back). However, most libgnunetgnsrecord functions are not interesting to GNS client applications.

For DNS records, the libgnunetdnsparser library provides functions for parsing (and serializing) common types of DNS records.

Creating records

Creating GNS records is typically done by building the respective record information (possibly with the help of libgnunetgnsrecord and libgnunetdnsparser) and then using the libgnunetnamestore to publish the information. The GNS API is not involved in this operation.

Future work

In the future, we want to expand libgnunetgns to allow applications to observe shortening operations performed during GNS resolution, for example so that users can receive visual feedback when this happens.

libgnunetgnsrecord

The libgnunetgnsrecord library is used to manipulate GNS records (in plaintext or in their encrypted format). Applications mostly interact with libgnunetgnsrecord by using the functions to convert GNS record values to strings or vice-versa, or to lookup a GNS record type number by name (or vice-versa). The library also provides various other functions that are mostly used internally within GNS, such as converting keys to names, checking for expiration, encrypting GNS records to GNS blocks, verifying GNS block signatures and decrypting GNS records from GNS blocks.

We will now discuss the four commonly used functions of the API. libgnunetgnsrecord does not perform these operations itself, but instead uses plugins to perform the operation. GNUnet includes plugins to support common DNS record types as well as standard GNS record types.

Value handling

GNUNET_GNSRECORD_value_to_string can be used to convert the (binary) representation of a GNS record value to a human readable, 0-terminated UTF-8 string. NULL is returned if the specified record type is not supported by any available plugin.

GNUNET_GNSRECORD_string_to_value can be used to try to convert a human readable string to the respective (binary) representation of a GNS record value.

Type handling

GNUNET_GNSRECORD_typename_to_number can be used to obtain the numeric value associated with a given typename. For example, given the typename "A" (for DNS A reocrds), the function will return the number 1. A list of common DNS record types is here. Note that not all DNS record types are supported by GNUnet GNSRECORD plugins at this time.

GNUNET_GNSRECORD_number_to_typename can be used to obtain the typename associated with a given numeric value. For example, given the type number 1, the function will return the typename "A".

GNS plugins

Adding a new GNS record type typically involves writing (or extending) a GNSRECORD plugin. The plugin needs to implement the gnunet_gnsrecord_plugin.h API which provides basic functions that are needed by GNSRECORD to convert typenames and values of the respective record type to strings (and back). These gnsrecord plugins are typically implemented within their respective subsystems. Examples for such plugins can be found in the GNSRECORD, GNS and CONVERSATION subsystems.

The libgnunetgnsrecord library is then used to locate, load and query the appropriate gnsrecord plugin. Which plugin is appropriate is determined by the record type (which is just a 32-bit integer). The libgnunetgnsrecord library loads all block plugins that are installed at the local peer and forwards the application request to the plugins. If the record type is not supported by the plugin, it should simply return an error code.

The central functions of the block APIs (plugin and main library) are the same four functions for converting between values and strings, and typenames and numbers documented in the previous subsection.

The GNS Client-Service Protocol

The GNS client-service protocol consists of two simple messages, the LOOKUP message and the LOOKUP_RESULT. Each LOOKUP message contains a unique 32-bit identifier, which will be included in the corresponding response. Thus, clients can send many lookup requests in parallel and receive responses out-of-order. A LOOKUP request also includes the public key of the GNS zone, the desired record type and fields specifying whether shortening is enabled or networking is disabled. Finally, the LOOKUP message includes the name to be resolved.

The response includes the number of records and the records themselves in the format created by GNUNET_GNSRECORD_records_serialize. They can thus be deserialized using GNUNET_GNSRECORD_records_deserialize.

Hijacking the DNS-Traffic using gnunet-service-dns

This section documents how the gnunet-service-dns (and the gnunet-helper-dns) intercepts DNS queries from the local system. This is merely one method for how we can obtain GNS queries. It is also possible to change resolv.conf to point to a machine running gnunet-dns2gns or to modify libc’s name system switch (NSS) configuration to include a GNS resolution plugin. The method described in this chapter is more of a last-ditch catch-all approach.

gnunet-service-dns enables intercepting DNS traffic using policy based routing. We MARK every outgoing DNS-packet if it was not sent by our application. Using a second routing table in the Linux kernel these marked packets are then routed through our virtual network interface and can thus be captured unchanged.

Our application then reads the query and decides how to handle it. If the query can be addressed via GNS, it is passed to gnunet-service-gns and resolved internally using GNS. In the future, a reverse query for an address of the configured virtual network could be answered with records kept about previous forward queries. Queries that are not hijacked by some application using the DNS service will be sent to the original recipient. The answer to the query will always be sent back through the virtual interface with the original nameserver as source address.

Network Setup Details

The DNS interceptor adds the following rules to the Linux kernel:

iptables -t mangle -I OUTPUT 1 -p udp --sport $LOCALPORT --dport 53 \
-j ACCEPT iptables -t mangle -I OUTPUT 2 -p udp --dport 53 -j MARK \
--set-mark 3 ip rule add fwmark 3 table2 ip route add default via \
$VIRTUALDNS table2

Todo

FIXME: Rewrite to reflect display which is no longer content by line due to the < 74 characters limit.

Line 1 makes sure that all packets coming from a port our application opened beforehand ($LOCALPORT) will be routed normally. Line 2 marks every other packet to a DNS-Server with mark 3 (chosen arbitrarily). The third line adds a routing policy based on this mark 3 via the routing table.

Importing DNS Zones into GNS

This section discusses the challenges and problems faced when writing the Ascension tool. It also takes a look at possible improvements in the future.

Consider the following diagram that shows the workflow of Ascension:

ascension

Further the interaction between components of GNUnet are shown in the diagram below:

DNS Conversion .. _Conversions-between-DNS-and-GNS:

Conversions between DNS and GNS

The differences between the two name systems lies in the details and is not always transparent. For instance an SRV record is converted to a BOX record which is unique to GNS.

This is done by converting to a BOX record from an existing SRV record:

# SRV
# _service._proto.name. TTL class SRV priority weight port target
_sip._tcp.example.com. 14000 IN SRV     0 0 5060 www.example.com.
# BOX
# TTL BOX flags port protocol recordtype priority weight port target
14000 BOX n 5060 6 33 0 0 5060 www.example.com

Other records that need to undergo such transformation is the MX record type, as well as the SOA record type.

Transformation of a SOA record into GNS works as described in the following example. Very important to note are the rname and mname keys.

# BIND syntax for a clean SOA record
   IN SOA master.example.com. hostmaster.example.com. (
    2017030300 ; serial
    3600       ; refresh
    1800       ; retry
    604800     ; expire
    600 )      ; ttl
# Recordline for adding the record
$ gnunet-namestore -z example.com -a -n  -t SOA -V \
    rname=master.example.com mname=hostmaster.example.com  \
    2017030300,3600,1800,604800,600 -e 7200s

The transformation of MX records is done in a simple way.

# mail.example.com. 3600 IN MX 10 mail.example.com.
$ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail

Finally, one of the biggest struggling points were the NS records that are found in top level domain zones. The intended behaviour for those is to add GNS2DNS records for those so that gnunet-gns can resolve records for those domains on its own. Those require the values from DNS GLUE records, provided they are within the same zone.

The following two examples show one record with a GLUE record and the other one does not have a GLUE record. This takes place in the ‘com’ TLD.

# ns1.example.com 86400 IN A 127.0.0.1
# example.com 86400 IN NS ns1.example.com.
$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
    example.com@127.0.0.1

# example.com 86400 IN NS ns1.example.org.
$ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
    example.com@ns1.example.org

As you can see, one of the GNS2DNS records has an IP address listed and the other one a DNS name. For the first one there is a GLUE record to do the translation directly and the second one will issue another DNS query to figure out the IP of ns1.example.org.

A solution was found by creating a hierarchical zone structure in GNS and linking the zones using PKEY records to one another. This allows the resolution of the name servers to work within GNS while not taking control over unwanted zones.

Currently the following record types are supported:

  • A

  • AAAA

  • CNAME

  • MX

  • NS

  • SRV

  • TXT

This is not due to technical limitations but rather a practical ones. The problem occurs with DNSSEC enabled DNS zones. As records within those zones are signed periodically, and every new signature is an update to the zone, there are many revisions of zones. This results in a problem with bigger zones as there are lots of records that have been signed again but no major changes. Also trying to add records that are unknown that require a different format take time as they cause a CLI call of the namestore. Furthermore certain record types need transformation into a GNS compatible format which, depending on the record type, takes more time.

Further a blacklist was added to drop for instance DNSSEC related records. Also if a record type is neither in the white list nor the blacklist it is considered as a loss of data and a message is shown to the user. This helps with transparency and also with contributing, as the not supported record types can then be added accordingly.

DNS Zone Size

Another very big problem exists with very large zones. When migrating a small zone the delay between adding of records and their expiry is negligible. However when working with big zones that easily have more than a few million records this delay becomes a problem.

Records will start to expire well before the zone has finished migrating. This is usually not a problem but can cause a high CPU load when a peer is restarted and the records have expired.

A good solution has not been found yet. One of the idea that floated around was that the records should be added with the s (shadow) flag to keep the records resolvable even if they expired. However this would introduce the problem of how to detect if a record has been removed from the zone and would require deletion of said record(s).

Another problem that still persists is how to refresh records. Expired records are still displayed when calling gnunet-namestore but do not resolve with gnunet-gns. Zonemaster will sign the expired records again and make sure that the records are still valid. With a recent change this was fixed as gnunet-gns to improve the suffix lookup which allows for a fast lookup even with thousands of local egos.

Currently the pace of adding records in general is around 10 records per second. Crypto is the upper limit for adding of records. The performance of your machine can be tested with the perf_crypto_* tools. There is still a big discrepancy between the pace of Ascension and the theoretical limit.

A performance metric for measuring improvements has not yet been implemented in Ascension.

Performance

The performance when migrating a zone using the Ascension tool is limited by a handful of factors. First of all ascension is written in Python3 and calls the CLI tools of GNUnet. This is comparable to a fork and exec call which costs a few CPU cycles. Furthermore all the records that are added to the same label are signed using the zones private key. This signing operation is very resource heavy and was optimized during development by adding the ‘-R’ (Recordline) option to gnunet-namestore which allows to specify multiple records using the CLI tool. Assuming that in a TLD zone every domain has at least two name servers this halves the amount of signatures needed.

Another improvement that could be made is with the addition of multiple threads or using asynchronous subprocesses when opening the GNUnet CLI tools. This could be implemented by simply creating more workers in the program but performance improvements were not tested.

Ascension was tested using different hardware and database backends. Performance differences between SQLite and postgresql are marginal and almost non existent. What did make a huge impact on record adding performance was the storage medium. On a traditional mechanical hard drive adding of records were slow compared to a solid state disk.

In conclusion there are many bottlenecks still around in the program, namely the single threaded implementation and inefficient, sequential calls of gnunet-namestore. In the future a solution that uses the C API would be cleaner and better.

Registering names using the FCFS daemon

This section describes FCFSD, a daemon used to associate names with PKEY records following a “First Come, First Served” policy. This policy means that a certain name can not be registered again if someone registered it already.

The daemon can be started by using gnunet-namestore-fcfsd, which will start a simple HTTP server on localhost, using a port specified by the HTTPORT value in its configuration.

Communication is performed by sending GET or POST requests to specific paths (“endpoints”), as described in the following sections.

The daemon will always respond with data structured using the JSON format. The fields to be expected will be listed for each endpoint.

The only exceptions are for the “root” endpoint (i.e. /) which will return a HTML document, and two other HTML documents which will be served when certain errors are encountered, like when requesting an unknown endpoint.

FCFSD GET requests .. _Obtaining-information-from-the-daemon:

Obtaining information from the daemon

To query the daemon, a GET request must be sent to these endpoints, placing parameters in the address as per the HTTP specification, like so:

GET /endpoint?param1=value&param2=value

Each endpoint will be described using its name (/endpoint in the example above), followed by the name of each parameter (like param1 and param2.)

Endpoint: /search :bolditalic:`name`

This endpoint is used to query about the state of <name>, that is, whether it is available for registration or not.

The response JSON will contain two fields:

  • error

  • free

error can be either the string "true" or the string "false": when "true", it means there was an error within the daemon and the name could not be searched at all.

free can be either the string "true" or the string "false": when "true", the requested name can be registered.

FCFSD POST requests .. _Submitting-data-to-the-daemon:

Submitting data to the daemon

To send data to the daemon, a POST request must be sent to these endpoints, placing the data to submit in the body of the request, structured using the JSON format, like so:

POST /endpoint
Content-Type: application/json
...

{"param1": value1, "param2": value2, ...}

Each endpoint will be described using its name (/endpoint in the example above), followed by the name of each JSON field (like param1 and param2.)

Endpoint: /register :bolditalic:`name key`

This endpoint is used to register a new association between <name> and <key>.

For this operation to succeed, both <NAME> and <KEY> must not be registered already.

The response JSON will contain two fields:

  • error

  • message

error can be either the string "true" or the string "false": when "true", it means the name could not be registered. Clients can get the reason of the failure from the HTTP response code or from the message field.

message is a string which can be used by clients to let users know the result of the operation. It might be localized to the daemon operator’s locale.

Customizing the HTML output

In some situations, the daemon will serve HTML documents instead of JSON values. It is possible to configure the daemon to serve custom documents instead of the ones provided with GNUnet, by setting the HTMLDIR value in its configuration to a directory path.

Within the provided path, the daemon will search for these three files:

  • fcfsd-index.html

  • fcfsd-notfound.html

  • fcfsd-forbidden.html

The fcfsd-index.html file is the daemon’s “homepage”: operators might want to provide information about the service here, or provide a form with which it is possible to register a name.

The fcfsd-notfound.html file is used primarily to let users know they tried to access an unknown endpoint.

The fcfsd-forbidden.html file is served to users when they try to access an endpoint they should not access. For example, sending an invalid request might result in this page being served.