.. index::
   double: GNU Name System; subsystem
   see: GNS; GNU Name System

.. _GNU-Name-System-_0028GNS_0029:

GNS — the GNU Name System
=========================

The GNU Name System (GNS) is a decentralized database that enables users
to securely resolve names to values. Names can be used to identify other
users (for example, in social networking), or network services (for
example, VPN services running at a peer in GNUnet, or purely IP-based
services on the Internet). Users interact with GNS by typing in a
hostname that ends in a top-level domain that is configured in the "GNS"
section, matches an identity of the user or ends in a Base32-encoded
public key.

Videos giving an overview of most of the GNS and the motivations behind
it is available here and here. The remainder of this chapter targets
developers that are familiar with high level concepts of GNS as
presented in these talks.

.. todo:: Link to videos and GNS talks?

GNS-aware applications should use the GNS resolver to obtain the
respective records that are stored under that name in GNS. Each record
consists of a type, value, expiration time and flags.

The type specifies the format of the value. Types below 65536 correspond
to DNS record types, larger values are used for GNS-specific records.
Applications can define new GNS record types by reserving a number and
implementing a plugin (which mostly needs to convert the binary value
representation to a human-readable text format and vice-versa). The
expiration time specifies how long the record is to be valid. The GNS
API ensures that applications are only given non-expired values. The
flags are typically irrelevant for applications, as GNS uses them
internally to control visibility and validity of records.

Records are stored along with a signature. The signature is generated
using the private key of the authoritative zone. This allows any GNS
resolver to verify the correctness of a name-value mapping.

Internally, GNS uses the NAMECACHE to cache information obtained from
other users, the NAMESTORE to store information specific to the local
users, and the DHT to exchange data between users. A plugin API is used
to enable applications to define new GNS record types.


:index:`libgnunetgns <single: libgnunet; gns>`
----------------------------------------------

The GNS API itself is extremely simple. Clients first connect to the GNS
service using ``GNUNET_GNS_connect``. They can then perform lookups
using ``GNUNET_GNS_lookup`` or cancel pending lookups using
``GNUNET_GNS_lookup_cancel``. Once finished, clients disconnect using
``GNUNET_GNS_disconnect``.

.. _Looking-up-records:

Looking up records
^^^^^^^^^^^^^^^^^^

``GNUNET_GNS_lookup`` takes a number of arguments:

handle This is simply the GNS connection handle from
   ``GNUNET_GNS_connect``.

name The client needs to specify the name to
   be resolved. This can be any valid DNS or GNS hostname.

zone The client
   needs to specify the public key of the GNS zone against which the
   resolution should be done. Note that a key must be provided, the
   client should look up plausible values using its configuration, the
   identity service and by attempting to interpret the TLD as a
   base32-encoded public key.

type This is the desired GNS or DNS record type
   to look for. While all records for the given name will be returned,
   this can be important if the client wants to resolve record types
   that themselves delegate resolution, such as CNAME, PKEY or GNS2DNS.
   Resolving a record of any of these types will only work if the
   respective record type is specified in the request, as the GNS
   resolver will otherwise follow the delegation and return the records
   from the respective destination, instead of the delegating record.

only_cached This argument should typically be set to
   ``GNUNET_NO``. Setting it to ``GNUNET_YES`` disables resolution via
   the overlay network.

shorten_zone_key If GNS encounters new names during resolution,
   their respective zones can automatically be learned and added to the
   \"shorten zone\". If this is desired, clients must pass the private
   key of the shorten zone. If NULL is passed, shortening is disabled.

proc This argument identifies
   the function to call with the result. It is given proc_cls, the
   number of records found (possibly zero) and the array of the records
   as arguments. proc will only be called once. After proc,> has been
   called, the lookup must no longer be canceled.

proc_cls The closure for proc.

.. _Accessing-the-records:

Accessing the records
^^^^^^^^^^^^^^^^^^^^^

The ``libgnunetgnsrecord`` library provides an API to manipulate the GNS
record array that is given to proc. In particular, it offers functions
such as converting record values to human-readable strings (and back).
However, most ``libgnunetgnsrecord`` functions are not interesting to
GNS client applications.

For DNS records, the ``libgnunetdnsparser`` library provides functions
for parsing (and serializing) common types of DNS records.

.. _Creating-records:

Creating records
^^^^^^^^^^^^^^^^

Creating GNS records is typically done by building the respective record
information (possibly with the help of ``libgnunetgnsrecord`` and
``libgnunetdnsparser``) and then using the ``libgnunetnamestore`` to
publish the information. The GNS API is not involved in this operation.

.. _Future-work:

Future work
^^^^^^^^^^^

In the future, we want to expand ``libgnunetgns`` to allow applications
to observe shortening operations performed during GNS resolution, for
example so that users can receive visual feedback when this happens.

libgnunetgnsrecord
~~~~~~~~~~~~~~~~~~

The ``libgnunetgnsrecord`` library is used to manipulate GNS records (in
plaintext or in their encrypted format). Applications mostly interact
with ``libgnunetgnsrecord`` by using the functions to convert GNS record
values to strings or vice-versa, or to lookup a GNS record type number
by name (or vice-versa). The library also provides various other
functions that are mostly used internally within GNS, such as converting
keys to names, checking for expiration, encrypting GNS records to GNS
blocks, verifying GNS block signatures and decrypting GNS records from
GNS blocks.

We will now discuss the four commonly used functions of the
API. ``libgnunetgnsrecord`` does not perform these operations itself,
but instead uses plugins to perform the operation. GNUnet includes
plugins to support common DNS record types as well as standard GNS
record types.

.. _Value-handling:

Value handling
^^^^^^^^^^^^^^

``GNUNET_GNSRECORD_value_to_string`` can be used to convert the (binary)
representation of a GNS record value to a human readable, 0-terminated
UTF-8 string. NULL is returned if the specified record type is not
supported by any available plugin.

``GNUNET_GNSRECORD_string_to_value`` can be used to try to convert a
human readable string to the respective (binary) representation of a GNS
record value.

.. _Type-handling:

Type handling
^^^^^^^^^^^^^

``GNUNET_GNSRECORD_typename_to_number`` can be used to obtain the
numeric value associated with a given typename. For example, given the
typename \"A\" (for DNS A reocrds), the function will return the number
1. A list of common DNS record types is
`here <http://en.wikipedia.org/wiki/List_of_DNS_record_types>`__. Note
that not all DNS record types are supported by GNUnet GNSRECORD plugins
at this time.

``GNUNET_GNSRECORD_number_to_typename`` can be used to obtain the
typename associated with a given numeric value. For example, given the
type number 1, the function will return the typename \"A\".

.. _GNS-plugins:

GNS plugins
~~~~~~~~~~~

Adding a new GNS record type typically involves writing (or extending) a
GNSRECORD plugin. The plugin needs to implement the
``gnunet_gnsrecord_plugin.h`` API which provides basic functions that
are needed by GNSRECORD to convert typenames and values of the
respective record type to strings (and back). These gnsrecord plugins
are typically implemented within their respective subsystems. Examples
for such plugins can be found in the GNSRECORD, GNS and CONVERSATION
subsystems.

The ``libgnunetgnsrecord`` library is then used to locate, load and
query the appropriate gnsrecord plugin. Which plugin is appropriate is
determined by the record type (which is just a 32-bit integer). The
``libgnunetgnsrecord`` library loads all block plugins that are
installed at the local peer and forwards the application request to the
plugins. If the record type is not supported by the plugin, it should
simply return an error code.

The central functions of the block APIs (plugin and main library) are
the same four functions for converting between values and strings, and
typenames and numbers documented in the previous subsection.

.. _The-GNS-Client_002dService-Protocol:

The GNS Client-Service Protocol
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The GNS client-service protocol consists of two simple messages, the
``LOOKUP`` message and the ``LOOKUP_RESULT``. Each ``LOOKUP`` message
contains a unique 32-bit identifier, which will be included in the
corresponding response. Thus, clients can send many lookup requests in
parallel and receive responses out-of-order. A ``LOOKUP`` request also
includes the public key of the GNS zone, the desired record type and
fields specifying whether shortening is enabled or networking is
disabled. Finally, the ``LOOKUP`` message includes the name to be
resolved.

The response includes the number of records and the records themselves
in the format created by ``GNUNET_GNSRECORD_records_serialize``. They
can thus be deserialized using ``GNUNET_GNSRECORD_records_deserialize``.

.. _Hijacking-the-DNS_002dTraffic-using-gnunet_002dservice_002ddns:

Hijacking the DNS-Traffic using gnunet-service-dns
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This section documents how the gnunet-service-dns (and the
gnunet-helper-dns) intercepts DNS queries from the local system. This is
merely one method for how we can obtain GNS queries. It is also possible
to change ``resolv.conf`` to point to a machine running
``gnunet-dns2gns`` or to modify libc's name system switch (NSS)
configuration to include a GNS resolution plugin. The method described
in this chapter is more of a last-ditch catch-all approach.

``gnunet-service-dns`` enables intercepting DNS traffic using policy
based routing. We MARK every outgoing DNS-packet if it was not sent by
our application. Using a second routing table in the Linux kernel these
marked packets are then routed through our virtual network interface and
can thus be captured unchanged.

Our application then reads the query and decides how to handle it. If
the query can be addressed via GNS, it is passed to
``gnunet-service-gns`` and resolved internally using GNS. In the future,
a reverse query for an address of the configured virtual network could
be answered with records kept about previous forward queries. Queries
that are not hijacked by some application using the DNS service will be
sent to the original recipient. The answer to the query will always be
sent back through the virtual interface with the original nameserver as
source address.

.. _Network-Setup-Details:

Network Setup Details
^^^^^^^^^^^^^^^^^^^^^

The DNS interceptor adds the following rules to the Linux kernel:

::

   iptables -t mangle -I OUTPUT 1 -p udp --sport $LOCALPORT --dport 53 \
   -j ACCEPT iptables -t mangle -I OUTPUT 2 -p udp --dport 53 -j MARK \
   --set-mark 3 ip rule add fwmark 3 table2 ip route add default via \
   $VIRTUALDNS table2

.. todo:: 
   FIXME: Rewrite to reflect display which is no longer content 
   by line due to the < 74 characters limit.

Line 1 makes sure that all packets coming from a port our application
opened beforehand (``$LOCALPORT``) will be routed normally. Line 2 marks
every other packet to a DNS-Server with mark 3 (chosen arbitrarily). The
third line adds a routing policy based on this mark 3 via the routing
table.

.. _Importing-DNS-Zones-into-GNS:

Importing DNS Zones into GNS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This section discusses the challenges and problems faced when writing
the Ascension tool. It also takes a look at possible improvements in the
future.

Consider the following diagram that shows the workflow of Ascension:

|ascension|

Further the interaction between components of GNUnet are shown in the
diagram below:

DNS Conversion
.. _Conversions-between-DNS-and-GNS:

Conversions between DNS and GNS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The differences between the two name systems lies in the details and is
not always transparent. For instance an SRV record is converted to a BOX
record which is unique to GNS.

This is done by converting to a BOX record from an existing SRV record:

::

   # SRV
   # _service._proto.name. TTL class SRV priority weight port target
   _sip._tcp.example.com. 14000 IN SRV     0 0 5060 www.example.com.
   # BOX
   # TTL BOX flags port protocol recordtype priority weight port target
   14000 BOX n 5060 6 33 0 0 5060 www.example.com

Other records that need to undergo such transformation is the MX record
type, as well as the SOA record type.

Transformation of a SOA record into GNS works as described in the
following example. Very important to note are the rname and mname keys.

::

   # BIND syntax for a clean SOA record
      IN SOA master.example.com. hostmaster.example.com. (
       2017030300 ; serial
       3600       ; refresh
       1800       ; retry
       604800     ; expire
       600 )      ; ttl
   # Recordline for adding the record
   $ gnunet-namestore -z example.com -a -n  -t SOA -V \
       rname=master.example.com mname=hostmaster.example.com  \
       2017030300,3600,1800,604800,600 -e 7200s

The transformation of MX records is done in a simple way.

::

   # mail.example.com. 3600 IN MX 10 mail.example.com.
   $ gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail

Finally, one of the biggest struggling points were the NS records that
are found in top level domain zones. The intended behaviour for those is
to add GNS2DNS records for those so that gnunet-gns can resolve records
for those domains on its own. Those require the values from DNS GLUE
records, provided they are within the same zone.

The following two examples show one record with a GLUE record and the
other one does not have a GLUE record. This takes place in the 'com'
TLD.

.. code-block:: shell

   # ns1.example.com 86400 IN A 127.0.0.1
   # example.com 86400 IN NS ns1.example.com.
   $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
       example.com@127.0.0.1

   # example.com 86400 IN NS ns1.example.org.
   $ gnunet-namestore -z com -n example -R 86400 GNS2DNS n \
       example.com@ns1.example.org

As you can see, one of the GNS2DNS records has an IP address listed and
the other one a DNS name. For the first one there is a GLUE record to do
the translation directly and the second one will issue another DNS query
to figure out the IP of ns1.example.org.

A solution was found by creating a hierarchical zone structure in GNS
and linking the zones using PKEY records to one another. This allows the
resolution of the name servers to work within GNS while not taking
control over unwanted zones.

Currently the following record types are supported:

-  A

-  AAAA

-  CNAME

-  MX

-  NS

-  SRV

-  TXT

This is not due to technical limitations but rather a practical ones.
The problem occurs with DNSSEC enabled DNS zones. As records within
those zones are signed periodically, and every new signature is an
update to the zone, there are many revisions of zones. This results in a
problem with bigger zones as there are lots of records that have been
signed again but no major changes. Also trying to add records that are
unknown that require a different format take time as they cause a CLI
call of the namestore. Furthermore certain record types need
transformation into a GNS compatible format which, depending on the
record type, takes more time.

Further a blacklist was added to drop for instance DNSSEC related
records. Also if a record type is neither in the white list nor the
blacklist it is considered as a loss of data and a message is shown to
the user. This helps with transparency and also with contributing, as
the not supported record types can then be added accordingly.

.. _DNS-Zone-Size:

DNS Zone Size
^^^^^^^^^^^^^

Another very big problem exists with very large zones. When migrating a
small zone the delay between adding of records and their expiry is
negligible. However when working with big zones that easily have more
than a few million records this delay becomes a problem.

Records will start to expire well before the zone has finished
migrating. This is usually not a problem but can cause a high CPU load
when a peer is restarted and the records have expired.

A good solution has not been found yet. One of the idea that floated
around was that the records should be added with the s (shadow) flag to
keep the records resolvable even if they expired. However this would
introduce the problem of how to detect if a record has been removed from
the zone and would require deletion of said record(s).

Another problem that still persists is how to refresh records. Expired
records are still displayed when calling gnunet-namestore but do not
resolve with gnunet-gns. Zonemaster will sign the expired records again
and make sure that the records are still valid. With a recent change
this was fixed as gnunet-gns to improve the suffix lookup which allows
for a fast lookup even with thousands of local egos.

Currently the pace of adding records in general is around 10 records per
second. Crypto is the upper limit for adding of records. The performance
of your machine can be tested with the perf_crypto\_\* tools. There is
still a big discrepancy between the pace of Ascension and the
theoretical limit.

A performance metric for measuring improvements has not yet been
implemented in Ascension.

.. _Performance:

Performance
^^^^^^^^^^^

The performance when migrating a zone using the Ascension tool is
limited by a handful of factors. First of all ascension is written in
Python3 and calls the CLI tools of GNUnet. This is comparable to a fork
and exec call which costs a few CPU cycles. Furthermore all the records
that are added to the same label are signed using the zones private key.
This signing operation is very resource heavy and was optimized during
development by adding the '-R' (Recordline) option to gnunet-namestore
which allows to specify multiple records using the CLI tool. Assuming
that in a TLD zone every domain has at least two name servers this
halves the amount of signatures needed.

Another improvement that could be made is with the addition of multiple
threads or using asynchronous subprocesses when opening the GNUnet CLI
tools. This could be implemented by simply creating more workers in the
program but performance improvements were not tested.

Ascension was tested using different hardware and database backends.
Performance differences between SQLite and postgresql are marginal and
almost non existent. What did make a huge impact on record adding
performance was the storage medium. On a traditional mechanical hard
drive adding of records were slow compared to a solid state disk.

In conclusion there are many bottlenecks still around in the program,
namely the single threaded implementation and inefficient, sequential
calls of gnunet-namestore. In the future a solution that uses the C API
would be cleaner and better.

.. _Registering-names-using-the-FCFS-daemon:

Registering names using the FCFS daemon
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This section describes FCFSD, a daemon used to associate names with PKEY
records following a "First Come, First Served" policy. This policy means
that a certain name can not be registered again if someone registered it
already.

The daemon can be started by using ``gnunet-namestore-fcfsd``, which
will start a simple HTTP server on localhost, using a port specified by
the ``HTTPORT`` value in its configuration.

Communication is performed by sending GET or POST requests to specific
paths ("endpoints"), as described in the following sections.

The daemon will always respond with data structured using the JSON
format. The fields to be expected will be listed for each endpoint.

The only exceptions are for the "root" endpoint (i.e. ``/``) which will
return a HTML document, and two other HTML documents which will be
served when certain errors are encountered, like when requesting an
unknown endpoint.

FCFSD GET requests
.. _Obtaining-information-from-the-daemon:

Obtaining information from the daemon
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To query the daemon, a GET request must be sent to these endpoints,
placing parameters in the address as per the HTTP specification, like
so:

::

   GET /endpoint?param1=value&param2=value

Each endpoint will be described using its name (``/endpoint`` in the
example above), followed by the name of each parameter (like ``param1``
and ``param2``.)


**Endpoint: /search** :bolditalic:`name` 
   This endpoint is used to query about the state of <name>, that is,
   whether it is available for registration or not.

   The response JSON will contain two fields:

   -  error

   -  free

   ``error`` can be either the string ``"true"`` or the string
   ``"false"``: when ``"true"``, it means there was an error within the
   daemon and the name could not be searched at all.

   ``free`` can be either the string ``"true"`` or the string
   ``"false"``: when ``"true"``, the requested name can be registered.

FCFSD POST requests
.. _Submitting-data-to-the-daemon:

Submitting data to the daemon
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To send data to the daemon, a POST request must be sent to these
endpoints, placing the data to submit in the body of the request,
structured using the JSON format, like so:

::

   POST /endpoint
   Content-Type: application/json
   ...

   {"param1": value1, "param2": value2, ...}

Each endpoint will be described using its name (``/endpoint`` in the
example above), followed by the name of each JSON field (like ``param1``
and ``param2``.)

**Endpoint: /register** :bolditalic:`name key`
   This endpoint is used to register a new association between <name>
   and <key>.

   For this operation to succeed, both <NAME> and <KEY> **must not** be
   registered already.

   The response JSON will contain two fields:

   -  error

   -  message

   ``error`` can be either the string ``"true"`` or the string
   ``"false"``: when ``"true"``, it means the name could not be
   registered. Clients can get the reason of the failure from the HTTP
   response code or from the ``message`` field.

   ``message`` is a string which can be used by clients to let users
   know the result of the operation. It might be localized to the daemon
   operator's locale.

.. _Customizing-the-HTML-output:

Customizing the HTML output
^^^^^^^^^^^^^^^^^^^^^^^^^^^

In some situations, the daemon will serve HTML documents instead of JSON
values. It is possible to configure the daemon to serve custom documents
instead of the ones provided with GNUnet, by setting the ``HTMLDIR``
value in its configuration to a directory path.

Within the provided path, the daemon will search for these three files:

-  fcfsd-index.html

-  fcfsd-notfound.html

-  fcfsd-forbidden.html

The ``fcfsd-index.html`` file is the daemon's "homepage": operators
might want to provide information about the service here, or provide a
form with which it is possible to register a name.

The ``fcfsd-notfound.html`` file is used primarily to let users know
they tried to access an unknown endpoint.

The ``fcfsd-forbidden.html`` file is served to users when they try to
access an endpoint they should not access. For example, sending an
invalid request might result in this page being served.

.. |ascension| image:: /_static/images/ascension_ssd.png
