gnunet-service-zonemaster.c File Reference

publish records from namestore to GNUnet name system More...

#include "platform.h"
#include "gnunet_util_lib.h"
#include "gnunet_dnsparser_lib.h"
#include "gnunet_dht_service.h"
#include "gnunet_namestore_service.h"
#include "gnunet_statistics_service.h"

Include dependency graph for gnunet-service-zonemaster.c:

Go to the source code of this file.

Data Structures
struct	DhtPutActivity
	Handle for DHT PUT activity triggered from the namestore monitor. More...

Macros
#define	LOG_STRERROR_FILE(kind, syscall, filename)
#define	PUBLISH_OPS_PER_EXPIRATION   4
	How often should we (re)publish each record before it expires? More...
#define	DELTA_INTERVAL   100
	How often do we measure the delta between desired zone iteration speed and actual speed, and tell statistics service about it? More...
#define	NS_BLOCK_SIZE   1000
	How many records do we fetch in one shot from the namestore? More...
#define	DHT_QUEUE_LIMIT   2000
	How many pending DHT operations do we allow at most? More...
#define	NAMESTORE_QUEUE_LIMIT   50
	How many events may the namestore give us before it has to wait for us to keep up? More...
#define	INITIAL_ZONE_ITERATION_INTERVAL   GNUNET_TIME_UNIT_MILLISECONDS
	The initial interval in milliseconds btween puts in a zone iteration. More...
#define	MAXIMUM_ZONE_ITERATION_INTERVAL
	The upper bound for the zone iteration interval (per record). More...
#define	LATE_ITERATION_SPEEDUP_FACTOR   2
	The factor the current zone iteration interval is divided by for each additional new record. More...
#define	DHT_GNS_REPLICATION_LEVEL   5
	What replication level do we use for DHT PUT operations? More...

Functions
static void	shutdown_task (void *cls)
	Task run during shutdown. More...
static void	publish_zone_namestore_next (void *cls)
	Method called periodically that triggers iteration over authoritative records. More...
static void	publish_zone_dht_start (void *cls)
	Periodically iterate over our zone and store everything in dht. More...
static void	calculate_put_interval ()
	Calculate target_iteration_velocity_per_record. More...
static void	update_velocity (unsigned int cnt)
	Re-calculate our velocity and the desired velocity. More...
static void	check_zone_namestore_next ()
	Check if the current zone iteration needs to be continued by calling publish_zone_namestore_next(), and if so with what delay. More...
static void	dht_put_continuation (void *cls)
	Continuation called from DHT once the PUT operation is done. More...
static unsigned int	convert_records_for_export (const struct GNUNET_GNSRECORD_Data *rd, unsigned int rd_count, struct GNUNET_GNSRECORD_Data *rd_public)
	Convert namestore records from the internal format to that suitable for publication (removes private records, converts to absolute expiration time). More...
static struct GNUNET_DHT_PutHandle *	perform_dht_put (const struct GNUNET_IDENTITY_PrivateKey *key, const char *label, const struct GNUNET_GNSRECORD_Data *rd_public, unsigned int rd_public_count, struct DhtPutActivity *ma)
	Store GNS records in the DHT. More...
static void	zone_iteration_error (void *cls)
	We encountered an error in our zone iteration. More...
static void	zone_iteration_finished (void *cls)
	Zone iteration is completed. More...
static void	put_gns_record (void *cls, const struct GNUNET_IDENTITY_PrivateKey *key, const char *label, unsigned int rd_count, const struct GNUNET_GNSRECORD_Data *rd)
	Function used to put all records successively into the DHT. More...
static void	run (void *cls, const struct GNUNET_CONFIGURATION_Handle *c, struct GNUNET_SERVICE_Handle *service)
	Performe zonemaster duties: watch namestore, publish records. More...
	GNUNET_SERVICE_MAIN ("zonemaster", GNUNET_SERVICE_OPTION_NONE, &run, NULL, NULL, NULL, GNUNET_MQ_handler_end())
	Define "main" method using service macro. More...

Variables
static struct GNUNET_STATISTICS_Handle *	statistics
	Handle to the statistics service. More...
static struct GNUNET_DHT_Handle *	dht_handle
	Our handle to the DHT. More...
static struct GNUNET_NAMESTORE_Handle *	namestore_handle
	Our handle to the namestore service. More...
static struct GNUNET_NAMESTORE_ZoneIterator *	namestore_iter
	Handle to iterate over our authoritative zone in namestore. More...
static struct DhtPutActivity *	it_head
	Head of iteration put activities; kept in a DLL. More...
static struct DhtPutActivity *	it_tail
	Tail of iteration put activities; kept in a DLL. More...
static unsigned int	dht_queue_length
	Number of entries in the DHT queue it_head. More...
static unsigned long long	num_public_records
	Useful for zone update for DHT put. More...
static unsigned long long	last_num_public_records
	Last seen record count. More...
static unsigned long long	put_cnt
	Number of successful put operations performed in the current measurement cycle (as measured in check_zone_namestore_next()). More...
static struct GNUNET_TIME_Relative	target_iteration_velocity_per_record
	What is the frequency at which we currently would like to perform DHT puts (per record)? Calculated in update_velocity() from the zone_publish_time_window() and the total number of record sets we have (so far) observed in the zone. More...
static struct GNUNET_TIME_Relative	min_relative_record_time
	Minimum relative expiration time of records seem during the current zone iteration. More...
static struct GNUNET_TIME_Relative	last_min_relative_record_time
	Minimum relative expiration time of records seem during the last zone iteration. More...
static struct GNUNET_TIME_Relative	zone_publish_time_window_default
	Default time window for zone iteration. More...
static struct GNUNET_TIME_Relative	zone_publish_time_window
	Time window for zone iteration, adjusted based on relative record expiration times in our zone. More...
static struct GNUNET_TIME_Absolute	last_put_100
	When did we last start measuring the DELTA_INTERVAL successful DHT puts? Used for velocity calculations. More...
static struct GNUNET_TIME_Relative	sub_delta
	By how much should we try to increase our per-record iteration speed (over the desired speed calculated directly from the #put_interval)? Basically this value corresponds to the per-record CPU time overhead we have. More...
static struct GNUNET_SCHEDULER_Task *	zone_publish_task
	zone publish task More...
static unsigned int	ns_iteration_left
	How many more values are left for the current query before we need to explicitly ask the namestore for more? More...
static int	first_zone_iteration
	GNUNET_YES if zone has never been published before More...
static int	cache_keys
	Optimize block insertion by caching map of private keys to public keys in memory? More...

Detailed Description

publish records from namestore to GNUnet name system

Author

Christian Grothoff

Definition in file gnunet-service-zonemaster.c.

Macro Definition Documentation

LOG_STRERROR_FILE

#define LOG_STRERROR_FILE	(		kind,
			syscall,
			filename
	)

Value:

GNUNET_log_from_strerror_file (kind, "util", \
                                                                   syscall, \
                                                                   filename)

Definition at line 34 of file gnunet-service-zonemaster.c.

PUBLISH_OPS_PER_EXPIRATION

#define PUBLISH_OPS_PER_EXPIRATION   4

How often should we (re)publish each record before it expires?

Definition at line 44 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval().

DELTA_INTERVAL

#define DELTA_INTERVAL   100

How often do we measure the delta between desired zone iteration speed and actual speed, and tell statistics service about it?

Definition at line 51 of file gnunet-service-zonemaster.c.

Referenced by put_gns_record().

NS_BLOCK_SIZE

#define NS_BLOCK_SIZE   1000

How many records do we fetch in one shot from the namestore?

Definition at line 56 of file gnunet-service-zonemaster.c.

Referenced by check_zone_namestore_next(), and publish_zone_namestore_next().

DHT_QUEUE_LIMIT

#define DHT_QUEUE_LIMIT   2000

How many pending DHT operations do we allow at most?

Definition at line 61 of file gnunet-service-zonemaster.c.

Referenced by put_gns_record().

NAMESTORE_QUEUE_LIMIT

#define NAMESTORE_QUEUE_LIMIT   50

How many events may the namestore give us before it has to wait for us to keep up?

Definition at line 67 of file gnunet-service-zonemaster.c.

INITIAL_ZONE_ITERATION_INTERVAL

#define INITIAL_ZONE_ITERATION_INTERVAL   GNUNET_TIME_UNIT_MILLISECONDS

The initial interval in milliseconds btween puts in a zone iteration.

Definition at line 73 of file gnunet-service-zonemaster.c.

Referenced by run().

MAXIMUM_ZONE_ITERATION_INTERVAL

#define MAXIMUM_ZONE_ITERATION_INTERVAL

Value:

GNUNET_TIME_relative_multiply ( \
    GNUNET_TIME_UNIT_MINUTES, 15)

The upper bound for the zone iteration interval (per record).

Definition at line 79 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval(), and check_zone_namestore_next().

LATE_ITERATION_SPEEDUP_FACTOR

#define LATE_ITERATION_SPEEDUP_FACTOR   2

The factor the current zone iteration interval is divided by for each additional new record.

Definition at line 86 of file gnunet-service-zonemaster.c.

Referenced by update_velocity().

DHT_GNS_REPLICATION_LEVEL

#define DHT_GNS_REPLICATION_LEVEL   5

What replication level do we use for DHT PUT operations?

Definition at line 91 of file gnunet-service-zonemaster.c.

Referenced by perform_dht_put().

Function Documentation

shutdown_task()

static void shutdown_task ( void *  cls )

static

Task run during shutdown.

Parameters

cls	unused
tc	unused

Definition at line 248 of file gnunet-service-zonemaster.c.

References dht_queue_length, GNUNET_CONTAINER_DLL_remove, GNUNET_DHT_disconnect(), GNUNET_DHT_put_cancel(), GNUNET_ERROR_TYPE_DEBUG, GNUNET_free, GNUNET_log, GNUNET_NAMESTORE_disconnect(), GNUNET_NAMESTORE_zone_iteration_stop(), GNUNET_NO, GNUNET_SCHEDULER_cancel(), GNUNET_STATISTICS_destroy(), and DhtPutActivity::ph.

Referenced by run().

{
  struct DhtPutActivity *ma;

  (void) cls;
  GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
              "Shutting down!\n");
  while (NULL != (ma = it_head))
  {
    GNUNET_DHT_put_cancel (ma->ph);
    dht_queue_length--;
    GNUNET_CONTAINER_DLL_remove (it_head,
                                 it_tail,
                                 ma);
    dht_queue_length--;
    GNUNET_free (ma);
  }
  if (NULL != statistics)
  {
    GNUNET_STATISTICS_destroy (statistics,
                               GNUNET_NO);
    statistics = NULL;
  }
  if (NULL != zone_publish_task)
  {
    GNUNET_SCHEDULER_cancel (zone_publish_task);
    zone_publish_task = NULL;
  }
  if (NULL != namestore_iter)
  {
    GNUNET_NAMESTORE_zone_iteration_stop (namestore_iter);
    namestore_iter = NULL;
  }
  if (NULL != namestore_handle)
  {
    GNUNET_NAMESTORE_disconnect (namestore_handle);
    namestore_handle = NULL;
  }
  if (NULL != dht_handle)
  {
    GNUNET_DHT_disconnect (dht_handle);
    dht_handle = NULL;
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

publish_zone_namestore_next()

static void publish_zone_namestore_next ( void *  cls )

static

Method called periodically that triggers iteration over authoritative records.

Parameters

cls

NULL

Definition at line 300 of file gnunet-service-zonemaster.c.

References GNUNET_assert, GNUNET_NAMESTORE_zone_iterator_next(), NS_BLOCK_SIZE, ns_iteration_left, and publish_zone_dht_start().

Referenced by check_zone_namestore_next().

{
  (void) cls;
  zone_publish_task = NULL;
  GNUNET_assert (NULL != namestore_iter);
  GNUNET_assert (0 == ns_iteration_left);
  ns_iteration_left = NS_BLOCK_SIZE;
  GNUNET_NAMESTORE_zone_iterator_next (namestore_iter,
                                       NS_BLOCK_SIZE);
}

Here is the call graph for this function:

Here is the caller graph for this function:

publish_zone_dht_start()

static void publish_zone_dht_start ( void *  cls )

static

Periodically iterate over our zone and store everything in dht.

Periodically iterate over all zones and store everything in DHT.

Parameters

cls

NULL

Definition at line 796 of file gnunet-service-zonemaster.c.

References GNUNET_assert, GNUNET_ERROR_TYPE_DEBUG, GNUNET_log, GNUNET_NAMESTORE_zone_iteration_start(), GNUNET_NO, GNUNET_STATISTICS_update(), ns_iteration_left, num_public_records, put_gns_record(), zone_iteration_error(), and zone_iteration_finished().

Referenced by publish_zone_namestore_next(), run(), zone_iteration_error(), and zone_iteration_finished().

{
  (void) cls;
  zone_publish_task = NULL;
  GNUNET_STATISTICS_update (statistics,
                            "Full zone iterations launched",
                            1,
                            GNUNET_NO);
  GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
              "Starting DHT zone update!\n");
  /* start counting again */
  num_public_records = 0;
  GNUNET_assert (NULL == namestore_iter);
  ns_iteration_left = 1;
  namestore_iter
    = GNUNET_NAMESTORE_zone_iteration_start (namestore_handle,
                                             NULL, /* All zones */
                                             &zone_iteration_error,
                                             NULL,
                                             &put_gns_record,
                                             NULL,
                                             &zone_iteration_finished,
                                             NULL);
  GNUNET_assert (NULL != namestore_iter);
}

Here is the call graph for this function:

Here is the caller graph for this function:

calculate_put_interval()

static void calculate_put_interval ( )

static

Calculate target_iteration_velocity_per_record.

If no records are known (startup) or none present we can safely set the interval to the value for a single record

Definition at line 325 of file gnunet-service-zonemaster.c.

References GNUNET_ERROR_TYPE_BULK, GNUNET_ERROR_TYPE_DEBUG, GNUNET_log, GNUNET_NO, GNUNET_STATISTICS_set(), GNUNET_TIME_relative_divide(), GNUNET_TIME_relative_min(), last_min_relative_record_time, last_num_public_records, MAXIMUM_ZONE_ITERATION_INTERVAL, min_relative_record_time, num_public_records, PUBLISH_OPS_PER_EXPIRATION, GNUNET_TIME_Relative::rel_value_us, target_iteration_velocity_per_record, zone_publish_time_window, and zone_publish_time_window_default.

Referenced by update_velocity(), and zone_iteration_finished().

{
  if (0 == num_public_records)
  {target_iteration_velocity_per_record = zone_publish_time_window;
    GNUNET_log (GNUNET_ERROR_TYPE_DEBUG | GNUNET_ERROR_TYPE_BULK,
                "No records in namestore database.\n");
  }
  else
  {
    last_min_relative_record_time
      = GNUNET_TIME_relative_min (last_min_relative_record_time,
                                  min_relative_record_time);
    zone_publish_time_window
      = GNUNET_TIME_relative_min (GNUNET_TIME_relative_divide (
                                    last_min_relative_record_time,
                                    PUBLISH_OPS_PER_EXPIRATION),
                                  zone_publish_time_window_default);
    target_iteration_velocity_per_record
      = GNUNET_TIME_relative_divide (zone_publish_time_window,
                                     last_num_public_records);
  }
  target_iteration_velocity_per_record
    = GNUNET_TIME_relative_min (target_iteration_velocity_per_record,
                                MAXIMUM_ZONE_ITERATION_INTERVAL);
  GNUNET_STATISTICS_set (statistics,
                         "Minimum relative record expiration (in μs)",
                         last_min_relative_record_time.rel_value_us,
                         GNUNET_NO);
  GNUNET_STATISTICS_set (statistics,
                         "Zone publication time window (in μs)",
                         zone_publish_time_window.rel_value_us,
                         GNUNET_NO);
  GNUNET_STATISTICS_set (statistics,
                         "Target zone iteration velocity (μs)",
                         target_iteration_velocity_per_record.rel_value_us,
                         GNUNET_NO);
}

Here is the call graph for this function:

Here is the caller graph for this function:

update_velocity()

static void update_velocity ( unsigned int  cnt )

static

Re-calculate our velocity and the desired velocity.

We have succeeded in making DELTA_INTERVAL puts, so now calculate the new desired delay between puts.

Parameters

cnt	how many records were processed since the last call?

Definition at line 377 of file gnunet-service-zonemaster.c.

References calculate_put_interval(), dht_queue_length, first_zone_iteration, GNUNET_ERROR_TYPE_DEBUG, GNUNET_log, GNUNET_NO, GNUNET_STATISTICS_set(), GNUNET_STRINGS_relative_time_to_string(), GNUNET_TIME_absolute_get(), GNUNET_TIME_absolute_get_duration(), GNUNET_TIME_relative_add(), GNUNET_TIME_relative_subtract(), GNUNET_TIME_UNIT_ZERO, GNUNET_YES, last_num_public_records, last_put_100, LATE_ITERATION_SPEEDUP_FACTOR, num_public_records, GNUNET_TIME_Relative::rel_value_us, sub_delta, and target_iteration_velocity_per_record.

Referenced by check_zone_namestore_next(), and put_gns_record().

{
  struct GNUNET_TIME_Relative delta;
  unsigned long long pct = 0;

  if (0 == cnt)
    return;
  /* How fast were we really? */
  delta = GNUNET_TIME_absolute_get_duration (last_put_100);
  delta.rel_value_us /= cnt;
  last_put_100 = GNUNET_TIME_absolute_get ();

  /* calculate expected frequency */
  if ((num_public_records > last_num_public_records) &&
      (GNUNET_NO == first_zone_iteration))
  {
    GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
                "Last record count was lower than current record count.  Reducing interval.\n");
    last_num_public_records = num_public_records
                              * LATE_ITERATION_SPEEDUP_FACTOR;
    calculate_put_interval ();
  }
  GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
              "Desired global zone iteration interval is %s/record!\n",
              GNUNET_STRINGS_relative_time_to_string (
                target_iteration_velocity_per_record,
                GNUNET_YES));

  /* Tell statistics actual vs. desired speed */
  GNUNET_STATISTICS_set (statistics,
                         "Current zone iteration velocity (μs/record)",
                         delta.rel_value_us,
                         GNUNET_NO);
  /* update "sub_delta" based on difference, taking
     previous sub_delta into account! */
  if (target_iteration_velocity_per_record.rel_value_us > delta.rel_value_us)
  {
    /* We were too fast, reduce sub_delta! */
    struct GNUNET_TIME_Relative corr;

    corr = GNUNET_TIME_relative_subtract (target_iteration_velocity_per_record,
                                          delta);
    if (sub_delta.rel_value_us > delta.rel_value_us)
    {
      /* Reduce sub_delta by corr */
      sub_delta = GNUNET_TIME_relative_subtract (sub_delta,
                                                 corr);
    }
    else
    {
      /* We're doing fine with waiting the full time, this
         should theoretically only happen if we run at
         infinite speed. */
      sub_delta = GNUNET_TIME_UNIT_ZERO;
    }
  }
  else if (target_iteration_velocity_per_record.rel_value_us <
           delta.rel_value_us)
  {
    /* We were too slow, increase sub_delta! */
    struct GNUNET_TIME_Relative corr;

    corr = GNUNET_TIME_relative_subtract (delta,
                                          target_iteration_velocity_per_record);
    sub_delta = GNUNET_TIME_relative_add (sub_delta,
                                          corr);
    if (sub_delta.rel_value_us >
        target_iteration_velocity_per_record.rel_value_us)
    {
      /* CPU overload detected, we cannot go at desired speed,
         as this would mean using a negative delay. */
      /* compute how much faster we would want to be for
         the desired velocity */
      if (0 == target_iteration_velocity_per_record.rel_value_us)
        pct = UINT64_MAX;     /* desired speed is infinity ... */
      else
        pct = (sub_delta.rel_value_us
               - target_iteration_velocity_per_record.rel_value_us) * 100LLU
              / target_iteration_velocity_per_record.rel_value_us;
      sub_delta = target_iteration_velocity_per_record;
    }
  }
  GNUNET_STATISTICS_set (statistics,
                         "# size of the DHT queue (it)",
                         dht_queue_length,
                         GNUNET_NO);
  GNUNET_STATISTICS_set (statistics,
                         "% speed increase needed for target velocity",
                         pct,
                         GNUNET_NO);
  GNUNET_STATISTICS_set (statistics,
                         "# records processed in current iteration",
                         num_public_records,
                         GNUNET_NO);
}

Here is the call graph for this function:

Here is the caller graph for this function:

check_zone_namestore_next()

static void check_zone_namestore_next ( )

static

Check if the current zone iteration needs to be continued by calling publish_zone_namestore_next(), and if so with what delay.

Definition at line 479 of file gnunet-service-zonemaster.c.

References first_zone_iteration, GNUNET_assert, GNUNET_NO, GNUNET_SCHEDULER_add_delayed(), GNUNET_STATISTICS_set(), GNUNET_TIME_relative_min(), GNUNET_TIME_relative_multiply(), GNUNET_TIME_relative_subtract(), GNUNET_TIME_UNIT_ZERO, GNUNET_YES, MAXIMUM_ZONE_ITERATION_INTERVAL, NS_BLOCK_SIZE, ns_iteration_left, publish_zone_namestore_next(), put_cnt, GNUNET_TIME_Relative::rel_value_us, sub_delta, target_iteration_velocity_per_record, and update_velocity().

Referenced by put_gns_record().

{
  struct GNUNET_TIME_Relative delay;

  if (0 != ns_iteration_left)
    return; /* current NAMESTORE iteration not yet done */
  update_velocity (put_cnt);
  put_cnt = 0;
  delay = GNUNET_TIME_relative_subtract (target_iteration_velocity_per_record,
                                         sub_delta);
  /* We delay *once* per #NS_BLOCK_SIZE, so we need to multiply the
     per-record delay calculated so far with the #NS_BLOCK_SIZE */
  GNUNET_STATISTICS_set (statistics,
                         "Current artificial NAMESTORE delay (μs/record)",
                         delay.rel_value_us,
                         GNUNET_NO);
  delay = GNUNET_TIME_relative_multiply (delay,
                                         NS_BLOCK_SIZE);
  /* make sure we do not overshoot because of the #NS_BLOCK_SIZE factor */
  delay = GNUNET_TIME_relative_min (MAXIMUM_ZONE_ITERATION_INTERVAL,
                                    delay);
  /* no delays on first iteration */
  if (GNUNET_YES == first_zone_iteration)
    delay = GNUNET_TIME_UNIT_ZERO;
  GNUNET_assert (NULL == zone_publish_task);
  zone_publish_task = GNUNET_SCHEDULER_add_delayed (delay,
                                                    &publish_zone_namestore_next,
                                                    NULL);
}

Here is the call graph for this function:

Here is the caller graph for this function:

dht_put_continuation()

static void dht_put_continuation ( void *  cls )

static

Continuation called from DHT once the PUT operation is done.

Parameters

cls	a struct DhtPutActivity

Definition at line 516 of file gnunet-service-zonemaster.c.

References dht_queue_length, GNUNET_CONTAINER_DLL_remove, GNUNET_ERROR_TYPE_DEBUG, GNUNET_free, and GNUNET_log.

Referenced by perform_dht_put().

{
  struct DhtPutActivity *ma = cls;

  GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
              "PUT complete\n");
  dht_queue_length--;
  GNUNET_CONTAINER_DLL_remove (it_head,
                               it_tail,
                               ma);
  GNUNET_free (ma);
}

Here is the caller graph for this function:

convert_records_for_export()

static unsigned int convert_records_for_export ( const struct GNUNET_GNSRECORD_Data *  rd, unsigned int  rd_count, struct GNUNET_GNSRECORD_Data *  rd_public  )

static

Convert namestore records from the internal format to that suitable for publication (removes private records, converts to absolute expiration time).

Parameters

rd	input records
rd_count	size of the rd and rd_public arrays
rd_public	where to write the converted records

Returns

number of records written to rd_public

Definition at line 541 of file gnunet-service-zonemaster.c.

References GNUNET_TIME_Absolute::abs_value_us, GNUNET_GNSRECORD_RF_PRIVATE, GNUNET_GNSRECORD_RF_RELATIVE_EXPIRATION, GNUNET_MIN, GNUNET_TIME_absolute_get(), min_relative_record_time, and GNUNET_TIME_Relative::rel_value_us.

Referenced by put_gns_record().

{
  struct GNUNET_TIME_Absolute now;
  unsigned int rd_public_count;

  rd_public_count = 0;
  now = GNUNET_TIME_absolute_get ();
  for (unsigned int i = 0; i < rd_count; i++)
  {
    if (0 != (rd[i].flags & GNUNET_GNSRECORD_RF_PRIVATE))
      continue;
    if ((0 == (rd[i].flags & GNUNET_GNSRECORD_RF_RELATIVE_EXPIRATION)) &&
        (rd[i].expiration_time < now.abs_value_us))
      continue;   /* record already expired, skip it */
    if (0 != (rd[i].flags & GNUNET_GNSRECORD_RF_RELATIVE_EXPIRATION))
    {
      /* GNUNET_GNSRECORD_block_create will convert to absolute time;
         we just need to adjust our iteration frequency */
      min_relative_record_time.rel_value_us =
        GNUNET_MIN (rd[i].expiration_time,
                    min_relative_record_time.rel_value_us);
    }
    rd_public[rd_public_count++] = rd[i];
  }
  return rd_public_count;
}

Here is the call graph for this function:

Here is the caller graph for this function:

perform_dht_put()

static struct GNUNET_DHT_PutHandle* perform_dht_put ( const struct GNUNET_IDENTITY_PrivateKey *  key, const char *  label, const struct GNUNET_GNSRECORD_Data *  rd_public, unsigned int  rd_public_count, struct DhtPutActivity *  ma  )

static

Store GNS records in the DHT.

Parameters

key	key of the zone
label	label to store under
rd_public	public record data
rd_public_count	number of records in rd_public
ma	handle for the put operation

Returns

DHT PUT handle, NULL on error

Definition at line 582 of file gnunet-service-zonemaster.c.

References cache_keys, DHT_GNS_REPLICATION_LEVEL, dht_put_continuation(), GNUNET_BLOCK_TYPE_GNS_NAMERECORD, GNUNET_break, GNUNET_DHT_put(), GNUNET_DHT_RO_DEMULTIPLEX_EVERYWHERE, GNUNET_ERROR_TYPE_DEBUG, GNUNET_free, GNUNET_GNSRECORD_block_create(), GNUNET_GNSRECORD_block_create2(), GNUNET_GNSRECORD_block_get_size(), GNUNET_GNSRECORD_query_from_private_key(), GNUNET_GNSRECORD_record_get_expiration_time(), GNUNET_h2s(), GNUNET_log, GNUNET_NO, GNUNET_STATISTICS_update(), GNUNET_STRINGS_absolute_time_to_string(), num_public_records, and ret.

Referenced by put_gns_record().

{
  struct GNUNET_GNSRECORD_Block *block;
  struct GNUNET_HashCode query;
  struct GNUNET_TIME_Absolute expire;
  size_t block_size;
  struct GNUNET_DHT_PutHandle *ret;

  expire = GNUNET_GNSRECORD_record_get_expiration_time (rd_public_count,
                                                        rd_public);
  if (cache_keys)
    block = GNUNET_GNSRECORD_block_create2 (key,
                                            expire,
                                            label,
                                            rd_public,
                                            rd_public_count);
  else
    block = GNUNET_GNSRECORD_block_create (key,
                                           expire,
                                           label,
                                           rd_public,
                                           rd_public_count);
  if (NULL == block)
  {
    GNUNET_break (0);
    return NULL;   /* whoops */
  }
  block_size = GNUNET_GNSRECORD_block_get_size (block);
  GNUNET_GNSRECORD_query_from_private_key (key,
                                           label,
                                           &query);
  GNUNET_STATISTICS_update (statistics,
                            "DHT put operations initiated",
                            1,
                            GNUNET_NO);
  GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
              "Storing %u record(s) for label `%s' in DHT with expiration `%s' under key %s\n",
              rd_public_count,
              label,
              GNUNET_STRINGS_absolute_time_to_string (expire),
              GNUNET_h2s (&query));
  num_public_records++;
  ret = GNUNET_DHT_put (dht_handle,
                        &query,
                        DHT_GNS_REPLICATION_LEVEL,
                        GNUNET_DHT_RO_DEMULTIPLEX_EVERYWHERE,
                        GNUNET_BLOCK_TYPE_GNS_NAMERECORD,
                        block_size,
                        block,
                        expire,
                        &dht_put_continuation,
                        ma);
  GNUNET_free (block);
  return ret;
}

Here is the call graph for this function:

Here is the caller graph for this function:

zone_iteration_error()

static void zone_iteration_error ( void *  cls )

static

We encountered an error in our zone iteration.

Parameters

cls

NULL

Definition at line 649 of file gnunet-service-zonemaster.c.

References GNUNET_ERROR_TYPE_DEBUG, GNUNET_log, GNUNET_SCHEDULER_add_now(), GNUNET_SCHEDULER_cancel(), and publish_zone_dht_start().

Referenced by publish_zone_dht_start().

{
  (void) cls;
  GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
              "Got disconnected from namestore database, retrying.\n");
  namestore_iter = NULL;
  /* We end up here on error/disconnect/shutdown, so potentially
     while a zone publish task or a DHT put is still running; hence
     we need to cancel those. */
  if (NULL != zone_publish_task)
  {
    GNUNET_SCHEDULER_cancel (zone_publish_task);
    zone_publish_task = NULL;
  }
  zone_publish_task = GNUNET_SCHEDULER_add_now (&publish_zone_dht_start,
                                                NULL);
}

Here is the call graph for this function:

Here is the caller graph for this function:

zone_iteration_finished()

static void zone_iteration_finished ( void *  cls )

static

Zone iteration is completed.

Parameters

cls

NULL

Definition at line 674 of file gnunet-service-zonemaster.c.

References calculate_put_interval(), first_zone_iteration, GNUNET_assert, GNUNET_ERROR_TYPE_DEBUG, GNUNET_log, GNUNET_NO, GNUNET_SCHEDULER_add_delayed(), GNUNET_SCHEDULER_add_now(), GNUNET_STATISTICS_set(), GNUNET_STRINGS_relative_time_to_string(), GNUNET_TIME_UNIT_FOREVER_REL, GNUNET_YES, last_min_relative_record_time, last_num_public_records, min_relative_record_time, num_public_records, publish_zone_dht_start(), GNUNET_TIME_Relative::rel_value_us, and target_iteration_velocity_per_record.

Referenced by publish_zone_dht_start().

{
  (void) cls;
  /* we're done with one iteration, calculate when to do the next one */
  namestore_iter = NULL;
  last_num_public_records = num_public_records;
  first_zone_iteration = GNUNET_NO;
  last_min_relative_record_time = min_relative_record_time;
  calculate_put_interval ();
  /* reset for next iteration */
  min_relative_record_time
    = GNUNET_TIME_UNIT_FOREVER_REL;
  GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
              "Zone iteration finished. Adjusted zone iteration interval to %s\n",
              GNUNET_STRINGS_relative_time_to_string (
                target_iteration_velocity_per_record,
                GNUNET_YES));
  GNUNET_STATISTICS_set (statistics,
                         "Target zone iteration velocity (μs)",
                         target_iteration_velocity_per_record.rel_value_us,
                         GNUNET_NO);
  GNUNET_STATISTICS_set (statistics,
                         "Number of public records in DHT",
                         last_num_public_records,
                         GNUNET_NO);
  GNUNET_assert (NULL == zone_publish_task);
  if (0 == last_num_public_records)
  {
    zone_publish_task = GNUNET_SCHEDULER_add_delayed (
      target_iteration_velocity_per_record,
      &publish_zone_dht_start,
      NULL);
  }
  else
  {
    zone_publish_task = GNUNET_SCHEDULER_add_now (&publish_zone_dht_start,
                                                  NULL);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

put_gns_record()

static void put_gns_record ( void *  cls, const struct GNUNET_IDENTITY_PrivateKey *  key, const char *  label, unsigned int  rd_count, const struct GNUNET_GNSRECORD_Data *  rd  )

static

Function used to put all records successively into the DHT.

Parameters

cls	the closure (NULL)
key	the private key of the authority (ours)
label	the name of the records, NULL once the iteration is done
rd_count	the number of records in rd
rd	the record data

Definition at line 725 of file gnunet-service-zonemaster.c.

References check_zone_namestore_next(), convert_records_for_export(), DELTA_INTERVAL, dht_queue_length, DHT_QUEUE_LIMIT, GNUNET_CONTAINER_DLL_insert_tail, GNUNET_CONTAINER_DLL_remove, GNUNET_DHT_put_cancel(), GNUNET_ERROR_TYPE_DEBUG, GNUNET_ERROR_TYPE_WARNING, GNUNET_free, GNUNET_log, GNUNET_new, GNUNET_STRINGS_relative_time_to_string(), GNUNET_TIME_absolute_get(), GNUNET_TIME_absolute_get_duration(), GNUNET_YES, it_head, ns_iteration_left, perform_dht_put(), DhtPutActivity::ph, put_cnt, DhtPutActivity::start_date, and update_velocity().

Referenced by publish_zone_dht_start().

{
  struct GNUNET_GNSRECORD_Data rd_public[rd_count];
  unsigned int rd_public_count;
  struct DhtPutActivity *ma;

  (void) cls;
  ns_iteration_left--;
  rd_public_count = convert_records_for_export (rd,
                                                rd_count,
                                                rd_public);
  if (0 == rd_public_count)
  {
    GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
                "Record set empty, moving to next record set\n");
    check_zone_namestore_next ();
    return;
  }
  /* We got a set of records to publish */
  GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
              "Starting DHT PUT\n");
  ma = GNUNET_new (struct DhtPutActivity);
  ma->start_date = GNUNET_TIME_absolute_get ();
  ma->ph = perform_dht_put (key,
                            label,
                            rd_public,
                            rd_public_count,
                            ma);
  put_cnt++;
  if (0 == put_cnt % DELTA_INTERVAL)
    update_velocity (DELTA_INTERVAL);
  check_zone_namestore_next ();
  if (NULL == ma->ph)
  {
    GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
                "Could not perform DHT PUT, is the DHT running?\n");
    GNUNET_free (ma);
    return;
  }
  dht_queue_length++;
  GNUNET_CONTAINER_DLL_insert_tail (it_head,
                                    it_tail,
                                    ma);
  if (dht_queue_length > DHT_QUEUE_LIMIT)
  {
    ma = it_head;
    GNUNET_CONTAINER_DLL_remove (it_head,
                                 it_tail,
                                 ma);
    GNUNET_DHT_put_cancel (ma->ph);
    dht_queue_length--;
    GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
                "DHT PUT unconfirmed after %s, aborting PUT\n",
                GNUNET_STRINGS_relative_time_to_string (
                  GNUNET_TIME_absolute_get_duration (ma->start_date),
                  GNUNET_YES));
    GNUNET_free (ma);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

run()

static void run ( void *  cls, const struct GNUNET_CONFIGURATION_Handle *  c, struct GNUNET_SERVICE_Handle *  service  )

static

Performe zonemaster duties: watch namestore, publish records.

Parameters

cls	closure
server	the initialized server
c	configuration to use

Definition at line 831 of file gnunet-service-zonemaster.c.

References _, cache_keys, first_zone_iteration, GNUNET_CONFIGURATION_get_value_number(), GNUNET_CONFIGURATION_get_value_time(), GNUNET_CONFIGURATION_get_value_yesno(), GNUNET_DHT_connect(), GNUNET_DHT_DEFAULT_REPUBLISH_FREQUENCY, GNUNET_ERROR_TYPE_DEBUG, GNUNET_ERROR_TYPE_ERROR, GNUNET_log, GNUNET_MQ_handler_end, GNUNET_NAMESTORE_connect(), GNUNET_NO, GNUNET_OK, GNUNET_SCHEDULER_add_now(), GNUNET_SCHEDULER_add_shutdown(), GNUNET_SCHEDULER_shutdown(), GNUNET_SERVICE_MAIN(), GNUNET_SERVICE_OPTION_NONE, GNUNET_STATISTICS_create(), GNUNET_STATISTICS_set(), GNUNET_STRINGS_relative_time_to_string(), GNUNET_TIME_absolute_get(), GNUNET_TIME_UNIT_FOREVER_REL, GNUNET_YES, INITIAL_ZONE_ITERATION_INTERVAL, last_put_100, min_relative_record_time, publish_zone_dht_start(), GNUNET_TIME_Relative::rel_value_us, shutdown_task(), target_iteration_velocity_per_record, zone_publish_time_window, and zone_publish_time_window_default.

{
  unsigned long long max_parallel_bg_queries = 128;

  (void) cls;
  (void) service;
  last_put_100 = GNUNET_TIME_absolute_get ();  /* first time! */
  min_relative_record_time
    = GNUNET_TIME_UNIT_FOREVER_REL;
  target_iteration_velocity_per_record = INITIAL_ZONE_ITERATION_INTERVAL;
  namestore_handle = GNUNET_NAMESTORE_connect (c);
  if (NULL == namestore_handle)
  {
    GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
                _ ("Failed to connect to the namestore!\n"));
    GNUNET_SCHEDULER_shutdown ();
    return;
  }
  cache_keys = GNUNET_CONFIGURATION_get_value_yesno (c,
                                                     "namestore",
                                                     "CACHE_KEYS");
  zone_publish_time_window_default = GNUNET_DHT_DEFAULT_REPUBLISH_FREQUENCY;
  if (GNUNET_OK ==
      GNUNET_CONFIGURATION_get_value_time (c,
                                           "zonemaster",
                                           "ZONE_PUBLISH_TIME_WINDOW",
                                           &zone_publish_time_window_default))
  {
    GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
                "Time window for zone iteration: %s\n",
                GNUNET_STRINGS_relative_time_to_string (
                  zone_publish_time_window,
                  GNUNET_YES));
  }
  zone_publish_time_window = zone_publish_time_window_default;
  if (GNUNET_OK ==
      GNUNET_CONFIGURATION_get_value_number (c,
                                             "zonemaster",
                                             "MAX_PARALLEL_BACKGROUND_QUERIES",
                                             &max_parallel_bg_queries))
  {
    GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
                "Number of allowed parallel background queries: %llu\n",
                max_parallel_bg_queries);
  }
  if (0 == max_parallel_bg_queries)
    max_parallel_bg_queries = 1;
  dht_handle = GNUNET_DHT_connect (c,
                                   (unsigned int) max_parallel_bg_queries);
  if (NULL == dht_handle)
  {
    GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
                _ ("Could not connect to DHT!\n"));
    GNUNET_SCHEDULER_add_now (&shutdown_task,
                              NULL);
    return;
  }

  /* Schedule periodic put for our records. */
  first_zone_iteration = GNUNET_YES;
  statistics = GNUNET_STATISTICS_create ("zonemaster",
                                         c);
  GNUNET_STATISTICS_set (statistics,
                         "Target zone iteration velocity (μs)",
                         target_iteration_velocity_per_record.rel_value_us,
                         GNUNET_NO);
  zone_publish_task = GNUNET_SCHEDULER_add_now (&publish_zone_dht_start,
                                                NULL);
  GNUNET_SCHEDULER_add_shutdown (&shutdown_task,
                                 NULL);
}

Here is the call graph for this function:

GNUNET_SERVICE_MAIN()

GNUNET_SERVICE_MAIN	(	"zonemaster"	,
		GNUNET_SERVICE_OPTION_NONE	,
		&	run,
		NULL	,
		NULL	,
		NULL	,
		GNUNET_MQ_handler_end()
	)

Define "main" method using service macro.

Referenced by run().

Here is the caller graph for this function:

Variable Documentation

statistics

struct GNUNET_STATISTICS_Handle* statistics

static

Handle to the statistics service.

Definition at line 124 of file gnunet-service-zonemaster.c.

dht_handle

struct GNUNET_DHT_Handle* dht_handle

static

Our handle to the DHT.

Definition at line 129 of file gnunet-service-zonemaster.c.

namestore_handle

struct GNUNET_NAMESTORE_Handle* namestore_handle

static

Our handle to the namestore service.

Definition at line 134 of file gnunet-service-zonemaster.c.

namestore_iter

struct GNUNET_NAMESTORE_ZoneIterator* namestore_iter

static

Handle to iterate over our authoritative zone in namestore.

Definition at line 139 of file gnunet-service-zonemaster.c.

it_head

struct DhtPutActivity* it_head

static

Head of iteration put activities; kept in a DLL.

Definition at line 144 of file gnunet-service-zonemaster.c.

Referenced by put_gns_record().

it_tail

struct DhtPutActivity* it_tail

static

Tail of iteration put activities; kept in a DLL.

Definition at line 149 of file gnunet-service-zonemaster.c.

dht_queue_length

unsigned int dht_queue_length

static

Number of entries in the DHT queue it_head.

Definition at line 154 of file gnunet-service-zonemaster.c.

Referenced by dht_put_continuation(), put_gns_record(), shutdown_task(), and update_velocity().

num_public_records

unsigned long long num_public_records

static

Useful for zone update for DHT put.

Definition at line 159 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval(), perform_dht_put(), publish_zone_dht_start(), update_velocity(), and zone_iteration_finished().

last_num_public_records

unsigned long long last_num_public_records

static

Last seen record count.

Definition at line 164 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval(), update_velocity(), and zone_iteration_finished().

put_cnt

unsigned long long put_cnt

static

Number of successful put operations performed in the current measurement cycle (as measured in check_zone_namestore_next()).

Definition at line 170 of file gnunet-service-zonemaster.c.

Referenced by check_zone_namestore_next(), and put_gns_record().

target_iteration_velocity_per_record

struct GNUNET_TIME_Relative target_iteration_velocity_per_record

static

What is the frequency at which we currently would like to perform DHT puts (per record)? Calculated in update_velocity() from the zone_publish_time_window() and the total number of record sets we have (so far) observed in the zone.

Definition at line 179 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval(), check_zone_namestore_next(), run(), update_velocity(), and zone_iteration_finished().

min_relative_record_time

struct GNUNET_TIME_Relative min_relative_record_time

static

Minimum relative expiration time of records seem during the current zone iteration.

Definition at line 185 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval(), convert_records_for_export(), run(), and zone_iteration_finished().

last_min_relative_record_time

struct GNUNET_TIME_Relative last_min_relative_record_time

static

Minimum relative expiration time of records seem during the last zone iteration.

Definition at line 191 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval(), and zone_iteration_finished().

zone_publish_time_window_default

struct GNUNET_TIME_Relative zone_publish_time_window_default

static

Default time window for zone iteration.

Definition at line 196 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval(), and run().

zone_publish_time_window

struct GNUNET_TIME_Relative zone_publish_time_window

static

Time window for zone iteration, adjusted based on relative record expiration times in our zone.

Definition at line 202 of file gnunet-service-zonemaster.c.

Referenced by calculate_put_interval(), and run().

last_put_100

struct GNUNET_TIME_Absolute last_put_100

static

When did we last start measuring the DELTA_INTERVAL successful DHT puts? Used for velocity calculations.

Definition at line 208 of file gnunet-service-zonemaster.c.

Referenced by run(), and update_velocity().

sub_delta

struct GNUNET_TIME_Relative sub_delta

static

By how much should we try to increase our per-record iteration speed (over the desired speed calculated directly from the #put_interval)? Basically this value corresponds to the per-record CPU time overhead we have.

Definition at line 216 of file gnunet-service-zonemaster.c.

Referenced by check_zone_namestore_next(), and update_velocity().

zone_publish_task

struct GNUNET_SCHEDULER_Task* zone_publish_task

static

zone publish task

Definition at line 221 of file gnunet-service-zonemaster.c.

ns_iteration_left

unsigned int ns_iteration_left

static

How many more values are left for the current query before we need to explicitly ask the namestore for more?

Definition at line 227 of file gnunet-service-zonemaster.c.

Referenced by check_zone_namestore_next(), publish_zone_dht_start(), publish_zone_namestore_next(), and put_gns_record().

first_zone_iteration

int first_zone_iteration

static

GNUNET_YES if zone has never been published before

Definition at line 232 of file gnunet-service-zonemaster.c.

Referenced by check_zone_namestore_next(), run(), update_velocity(), and zone_iteration_finished().

cache_keys

int cache_keys

static

Optimize block insertion by caching map of private keys to public keys in memory?

Definition at line 238 of file gnunet-service-zonemaster.c.

Referenced by perform_dht_put(), and run().