GNUnet  0.11.x
Data Structures | Functions
fs_sharetree.c File Reference

code to manipulate the 'struct GNUNET_FS_ShareTreeItem' tree More...

#include "platform.h"
#include "gnunet_fs_service.h"
#include "gnunet_scheduler_lib.h"
#include <pthread.h>
Include dependency graph for fs_sharetree.c:

Go to the source code of this file.

Data Structures

struct  KeywordCounter
 Entry for each unique keyword to track how often it occurred. More...
 
struct  MetaCounter
 Aggregate information we keep for meta data in each directory. More...
 
struct  TrimContext
 A structure that forms a singly-linked list that serves as a stack for metadata-processing function. More...
 

Functions

static int add_to_keyword_counter (void *cls, const char *keyword, int is_mandatory)
 Add the given keyword to the keyword statistics tracker. More...
 
static int add_to_meta_counter (void *cls, const char *plugin_name, enum EXTRACTOR_MetaType type, enum EXTRACTOR_MetaFormat format, const char *data_mime_type, const char *data, size_t data_len)
 Function called on each meta data item. More...
 
static int remove_high_frequency_keywords (void *cls, const char *keyword, int is_mandatory)
 Remove keywords above the threshold. More...
 
static int migrate_and_drop_keywords (void *cls, const struct GNUNET_HashCode *key, void *value)
 Move "frequent" keywords over to the target ksk uri, free the counters. More...
 
static int migrate_and_drop_metadata (void *cls, const struct GNUNET_HashCode *key, void *value)
 Copy "frequent" metadata items over to the target metadata container, free the counters. More...
 
static void share_tree_trim (struct TrimContext *tc, struct GNUNET_FS_ShareTreeItem *tree)
 Process a share item tree, moving frequent keywords up and copying frequent metadata up. More...
 
void GNUNET_FS_share_tree_trim (struct GNUNET_FS_ShareTreeItem *toplevel)
 Process a share item tree, moving frequent keywords up and copying frequent metadata up. More...
 
void GNUNET_FS_share_tree_free (struct GNUNET_FS_ShareTreeItem *toplevel)
 Release memory of a share item tree. More...
 

Detailed Description

code to manipulate the 'struct GNUNET_FS_ShareTreeItem' tree

Author
LRN
Christian Grothoff

Definition in file fs_sharetree.c.

Function Documentation

◆ add_to_keyword_counter()

static int add_to_keyword_counter ( void *  cls,
const char *  keyword,
int  is_mandatory 
)
static

Add the given keyword to the keyword statistics tracker.

Parameters
clsthe multihashmap we store the keyword counters in
keywordthe keyword to count
is_mandatoryignored
Returns
always GNUNET_OK

Definition at line 155 of file fs_sharetree.c.

156 {
157  struct GNUNET_CONTAINER_MultiHashMap *mcm = cls;
158  struct KeywordCounter *cnt;
159  struct GNUNET_HashCode hc;
160  size_t klen;
161 
162  klen = strlen (keyword) + 1;
163  GNUNET_CRYPTO_hash (keyword, klen - 1, &hc);
164  cnt = GNUNET_CONTAINER_multihashmap_get (mcm, &hc);
165  if (cnt == NULL)
166  {
167  cnt = GNUNET_malloc (sizeof(struct KeywordCounter) + klen);
168  cnt->value = (const char *) &cnt[1];
169  GNUNET_memcpy (&cnt[1], keyword, klen);
172  &hc, cnt,
174  }
175  cnt->count++;
176  return GNUNET_OK;
177 }
#define GNUNET_memcpy(dst, src, n)
Call memcpy() but check for n being 0 first.
@ GNUNET_OK
Definition: gnunet_common.h:95
void GNUNET_CRYPTO_hash(const void *block, size_t size, struct GNUNET_HashCode *ret)
Compute hash of a given block.
Definition: crypto_hash.c:41
int GNUNET_CONTAINER_multihashmap_put(struct GNUNET_CONTAINER_MultiHashMap *map, const struct GNUNET_HashCode *key, void *value, enum GNUNET_CONTAINER_MultiHashMapOption opt)
Store a key-value pair in the map.
void * GNUNET_CONTAINER_multihashmap_get(const struct GNUNET_CONTAINER_MultiHashMap *map, const struct GNUNET_HashCode *key)
Given a key find a value in the map matching the key.
@ GNUNET_CONTAINER_MULTIHASHMAPOPTION_UNIQUE_ONLY
There must only be one value per key; storing a value should fail if a value under the same key alrea...
#define GNUNET_assert(cond)
Use this for fatal errors that cannot be handled.
#define GNUNET_malloc(size)
Wrapper around malloc.
Internal representation of the hash map.
A 512-bit hashcode.
Entry for each unique keyword to track how often it occurred.
Definition: fs_sharetree.c:38
unsigned int count
How many files have this keyword?
Definition: fs_sharetree.c:57
const char * value
Keyword that was found.
Definition: fs_sharetree.c:52

References KeywordCounter::count, GNUNET_assert, GNUNET_CONTAINER_multihashmap_get(), GNUNET_CONTAINER_multihashmap_put(), GNUNET_CONTAINER_MULTIHASHMAPOPTION_UNIQUE_ONLY, GNUNET_CRYPTO_hash(), GNUNET_malloc, GNUNET_memcpy, GNUNET_OK, and KeywordCounter::value.

Referenced by share_tree_trim().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ add_to_meta_counter()

static int add_to_meta_counter ( void *  cls,
const char *  plugin_name,
enum EXTRACTOR_MetaType  type,
enum EXTRACTOR_MetaFormat  format,
const char *  data_mime_type,
const char *  data,
size_t  data_len 
)
static

Function called on each meta data item.

Increments the respective counter.

Parameters
clsthe container multihashmap to update
plugin_namename of the plugin that produced this value; special values can be used (e.g. '<zlib>' for zlib being used in the main libextractor library and yielding meta data).
typelibextractor-type describing the meta data
formatbasic format information about data
data_mime_typemime-type of data (not of the original file); can be NULL (if mime-type is not known)
dataactual meta-data found
data_lennumber of bytes in data
Returns
0 to continue extracting / iterating

Definition at line 198 of file fs_sharetree.c.

203 {
204  struct GNUNET_CONTAINER_MultiHashMap *map = cls;
205  struct GNUNET_HashCode key;
206  struct MetaCounter *cnt;
207 
208  GNUNET_CRYPTO_hash (data, data_len, &key);
210  if (NULL == cnt)
211  {
212  cnt = GNUNET_new (struct MetaCounter);
213  cnt->data = data;
214  cnt->data_size = data_len;
215  cnt->plugin_name = plugin_name;
216  cnt->type = type;
217  cnt->format = format;
221  &key, cnt,
223  }
224  cnt->count++;
225  return 0;
226 }
static struct GNUNET_CONTAINER_MultiPeerMap * map
Handle to the map used to store old latency values for peers.
struct GNUNET_HashCode key
The key used in the DHT.
uint32_t data
The data value.
static char * plugin_name
Name of our plugin.
#define GNUNET_new(type)
Allocate a struct or union of the given type.
Aggregate information we keep for meta data in each directory.
Definition: fs_sharetree.c:65
enum EXTRACTOR_MetaFormat format
Format of the data.
Definition: fs_sharetree.c:104
enum EXTRACTOR_MetaType type
Type of the data.
Definition: fs_sharetree.c:99
const char * plugin_name
Name of the plugin that provided that piece of metadata.
Definition: fs_sharetree.c:79
unsigned int count
How many files have meta entries matching this value? (type and format do not have to match).
Definition: fs_sharetree.c:110
size_t data_size
Number of bytes in 'data'.
Definition: fs_sharetree.c:94
const char * data_mime_type
MIME-type of the metadata itself.
Definition: fs_sharetree.c:84
const char * data
The actual meta data.
Definition: fs_sharetree.c:89
enum GNUNET_TESTBED_UnderlayLinkModelType type
the type of this model

References MetaCounter::count, MetaCounter::data, data, MetaCounter::data_mime_type, MetaCounter::data_size, MetaCounter::format, GNUNET_assert, GNUNET_CONTAINER_multihashmap_get(), GNUNET_CONTAINER_multihashmap_put(), GNUNET_CONTAINER_MULTIHASHMAPOPTION_UNIQUE_ONLY, GNUNET_CRYPTO_hash(), GNUNET_new, GNUNET_OK, key, map, plugin_name, MetaCounter::plugin_name, MetaCounter::type, and type.

Referenced by share_tree_trim().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ remove_high_frequency_keywords()

static int remove_high_frequency_keywords ( void *  cls,
const char *  keyword,
int  is_mandatory 
)
static

Remove keywords above the threshold.

Parameters
clsthe 'struct TrimContext' with the pos to remove the keywords from
keywordthe keyword to check
is_mandatoryignored
Returns
always GNUNET_OK

Definition at line 238 of file fs_sharetree.c.

240 {
241  struct TrimContext *tc = cls;
242  struct KeywordCounter *counter;
243  struct GNUNET_HashCode hc;
244  size_t klen;
245 
246  klen = strlen (keyword) + 1;
247  GNUNET_CRYPTO_hash (keyword, klen - 1, &hc);
248  counter = GNUNET_CONTAINER_multihashmap_get (tc->keywordcounter, &hc);
249  GNUNET_assert (NULL != counter);
250  if (counter->count < tc->move_threshold)
251  return GNUNET_OK;
252  GNUNET_FS_uri_ksk_remove_keyword (tc->pos->ksk_uri,
253  counter->value);
254  return GNUNET_OK;
255 }
void GNUNET_FS_uri_ksk_remove_keyword(struct GNUNET_FS_Uri *uri, const char *keyword)
Remove the given keyword from the set of keywords represented by the URI.
Definition: fs_uri.c:790
static struct GNUNET_SCHEDULER_TaskContext tc
Task context of the current task.
Definition: scheduler.c:423
A structure that forms a singly-linked list that serves as a stack for metadata-processing function.
Definition: fs_sharetree.c:119

References KeywordCounter::count, GNUNET_assert, GNUNET_CONTAINER_multihashmap_get(), GNUNET_CRYPTO_hash(), GNUNET_FS_uri_ksk_remove_keyword(), GNUNET_OK, tc, and KeywordCounter::value.

Referenced by share_tree_trim().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ migrate_and_drop_keywords()

static int migrate_and_drop_keywords ( void *  cls,
const struct GNUNET_HashCode key,
void *  value 
)
static

Move "frequent" keywords over to the target ksk uri, free the counters.

Parameters
clsthe 'struct TrimContext'
keykey of the entry
valuethe 'struct KeywordCounter'
Returns
GNUNET_YES (always)

Definition at line 268 of file fs_sharetree.c.

270 {
271  struct TrimContext *tc = cls;
272  struct KeywordCounter *counter = value;
273 
274  if (counter->count >= tc->move_threshold)
275  {
276  if (NULL == tc->pos->ksk_uri)
277  tc->pos->ksk_uri = GNUNET_FS_uri_ksk_create_from_args (1,
278  &counter->value);
279  else
280  GNUNET_FS_uri_ksk_add_keyword (tc->pos->ksk_uri, counter->value,
281  GNUNET_NO);
282  }
284  GNUNET_CONTAINER_multihashmap_remove (tc->keywordcounter,
285  key,
286  counter));
287  GNUNET_free (counter);
288  return GNUNET_YES;
289 }
static char * value
Value of the record to add/remove.
@ GNUNET_YES
Definition: gnunet_common.h:97
@ GNUNET_NO
Definition: gnunet_common.h:94
struct GNUNET_FS_Uri * GNUNET_FS_uri_ksk_create_from_args(unsigned int argc, const char **argv)
Create an FS URI from a user-supplied command line of keywords.
Definition: fs_uri.c:1154
void GNUNET_FS_uri_ksk_add_keyword(struct GNUNET_FS_Uri *uri, const char *keyword, int is_mandatory)
Add the given keyword to the set of keywords represented by the URI.
Definition: fs_uri.c:762
int GNUNET_CONTAINER_multihashmap_remove(struct GNUNET_CONTAINER_MultiHashMap *map, const struct GNUNET_HashCode *key, const void *value)
Remove the given key-value pair from the map.
#define GNUNET_free(ptr)
Wrapper around free.

References KeywordCounter::count, GNUNET_assert, GNUNET_CONTAINER_multihashmap_remove(), GNUNET_free, GNUNET_FS_uri_ksk_add_keyword(), GNUNET_FS_uri_ksk_create_from_args(), GNUNET_NO, GNUNET_YES, key, tc, KeywordCounter::value, and value.

Referenced by share_tree_trim().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ migrate_and_drop_metadata()

static int migrate_and_drop_metadata ( void *  cls,
const struct GNUNET_HashCode key,
void *  value 
)
static

Copy "frequent" metadata items over to the target metadata container, free the counters.

Parameters
clsthe 'struct TrimContext'
keykey of the entry
valuethe 'struct KeywordCounter'
Returns
GNUNET_YES (always)

Definition at line 302 of file fs_sharetree.c.

304 {
305  struct TrimContext *tc = cls;
306  struct MetaCounter *counter = value;
307 
308  if (counter->count >= tc->move_threshold)
309  {
310  if (NULL == tc->pos->meta)
311  tc->pos->meta = GNUNET_CONTAINER_meta_data_create ();
313  counter->plugin_name,
314  counter->type,
315  counter->format,
316  counter->data_mime_type, counter->data,
317  counter->data_size);
318  }
321  key,
322  counter));
323  GNUNET_free (counter);
324  return GNUNET_YES;
325 }
struct GNUNET_CONTAINER_MetaData * GNUNET_CONTAINER_meta_data_create(void)
Create a fresh meta data container.
int GNUNET_CONTAINER_meta_data_insert(struct GNUNET_CONTAINER_MetaData *md, const char *plugin_name, enum EXTRACTOR_MetaType type, enum EXTRACTOR_MetaFormat format, const char *data_mime_type, const char *data, size_t data_size)
Extend metadata.

References MetaCounter::count, MetaCounter::data, MetaCounter::data_mime_type, MetaCounter::data_size, MetaCounter::format, GNUNET_assert, GNUNET_CONTAINER_meta_data_create(), GNUNET_CONTAINER_meta_data_insert(), GNUNET_CONTAINER_multihashmap_remove(), GNUNET_free, GNUNET_YES, key, MetaCounter::plugin_name, tc, MetaCounter::type, and value.

Referenced by share_tree_trim().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ share_tree_trim()

static void share_tree_trim ( struct TrimContext tc,
struct GNUNET_FS_ShareTreeItem tree 
)
static

Process a share item tree, moving frequent keywords up and copying frequent metadata up.

Parameters
tctrim context with hash maps to use
treetree to trim

Definition at line 336 of file fs_sharetree.c.

338 {
339  struct GNUNET_FS_ShareTreeItem *pos;
340  unsigned int num_children;
341 
342  /* first, trim all children */
343  num_children = 0;
344  for (pos = tree->children_head; NULL != pos; pos = pos->next)
345  {
346  share_tree_trim (tc, pos);
347  num_children++;
348  }
349 
350  /* consider adding filename to directory meta data */
351  if (tree->is_directory == GNUNET_YES)
352  {
353  const char *user = getenv ("USER");
354  if ((user == NULL) ||
355  (0 != strncasecmp (user, tree->short_filename, strlen (user))))
356  {
357  /* only use filename if it doesn't match $USER */
358  if (NULL == tree->meta)
360  GNUNET_CONTAINER_meta_data_insert (tree->meta, "<libgnunetfs>",
363  "text/plain", tree->short_filename,
364  strlen (tree->short_filename) + 1);
365  }
366  }
367 
368  if (1 >= num_children)
369  return; /* nothing to trim */
370 
371  /* now, count keywords and meta data in children */
372  for (pos = tree->children_head; NULL != pos; pos = pos->next)
373  {
374  if (NULL != pos->meta)
376  tc->metacounter);
377  if (NULL != pos->ksk_uri)
379  tc->keywordcounter);
380  }
381 
382  /* calculate threshold for moving keywords / meta data */
383  tc->move_threshold = 1 + (num_children / 2);
384 
385  /* remove high-frequency keywords from children */
386  for (pos = tree->children_head; NULL != pos; pos = pos->next)
387  {
388  tc->pos = pos;
389  if (NULL != pos->ksk_uri)
390  {
391  struct GNUNET_FS_Uri *ksk_uri_copy = GNUNET_FS_uri_dup (pos->ksk_uri);
392  GNUNET_FS_uri_ksk_get_keywords (ksk_uri_copy,
394  GNUNET_FS_uri_destroy (ksk_uri_copy);
395  }
396  }
397 
398  /* add high-frequency meta data and keywords to parent */
399  tc->pos = tree;
400  GNUNET_CONTAINER_multihashmap_iterate (tc->keywordcounter,
402  tc);
405  tc);
406 }
static int add_to_keyword_counter(void *cls, const char *keyword, int is_mandatory)
Add the given keyword to the keyword statistics tracker.
Definition: fs_sharetree.c:155
static int add_to_meta_counter(void *cls, const char *plugin_name, enum EXTRACTOR_MetaType type, enum EXTRACTOR_MetaFormat format, const char *data_mime_type, const char *data, size_t data_len)
Function called on each meta data item.
Definition: fs_sharetree.c:198
static int migrate_and_drop_keywords(void *cls, const struct GNUNET_HashCode *key, void *value)
Move "frequent" keywords over to the target ksk uri, free the counters.
Definition: fs_sharetree.c:268
static void share_tree_trim(struct TrimContext *tc, struct GNUNET_FS_ShareTreeItem *tree)
Process a share item tree, moving frequent keywords up and copying frequent metadata up.
Definition: fs_sharetree.c:336
static int migrate_and_drop_metadata(void *cls, const struct GNUNET_HashCode *key, void *value)
Copy "frequent" metadata items over to the target metadata container, free the counters.
Definition: fs_sharetree.c:302
static int remove_high_frequency_keywords(void *cls, const char *keyword, int is_mandatory)
Remove keywords above the threshold.
Definition: fs_sharetree.c:238
char * getenv()
@ EXTRACTOR_METAFORMAT_UTF8
0-terminated, UTF-8 encoded string.
@ EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME
int GNUNET_FS_uri_ksk_get_keywords(const struct GNUNET_FS_Uri *uri, GNUNET_FS_KeywordIterator iterator, void *iterator_cls)
Iterate over all keywords in this keyword URI.
Definition: fs_uri.c:730
void GNUNET_FS_uri_destroy(struct GNUNET_FS_Uri *uri)
Free URI.
Definition: fs_uri.c:678
struct GNUNET_FS_Uri * GNUNET_FS_uri_dup(const struct GNUNET_FS_Uri *uri)
Duplicate URI.
Definition: fs_uri.c:997
int GNUNET_CONTAINER_multihashmap_iterate(struct GNUNET_CONTAINER_MultiHashMap *map, GNUNET_CONTAINER_MulitHashMapIteratorCallback it, void *it_cls)
Iterate over all entries in the map.
int GNUNET_CONTAINER_meta_data_iterate(const struct GNUNET_CONTAINER_MetaData *md, EXTRACTOR_MetaDataProcessor iter, void *iter_cls)
Iterate over MD entries.
A node of a directory tree (produced by dirscanner)
int is_directory
GNUNET_YES if this is a directory
struct GNUNET_FS_Uri * ksk_uri
Keywords for this file or directory (derived from metadata).
char * short_filename
Base name of the file/directory.
struct GNUNET_FS_ShareTreeItem * next
This is a doubly-linked list.
struct GNUNET_FS_ShareTreeItem * children_head
This is a doubly-linked tree NULL for files and empty directories.
struct GNUNET_CONTAINER_MetaData * meta
Metadata for this file or directory.
A Universal Resource Identifier (URI), opaque.
Definition: fs_api.h:166

References add_to_keyword_counter(), add_to_meta_counter(), GNUNET_FS_ShareTreeItem::children_head, EXTRACTOR_METAFORMAT_UTF8, EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME, getenv(), GNUNET_CONTAINER_meta_data_create(), GNUNET_CONTAINER_meta_data_insert(), GNUNET_CONTAINER_meta_data_iterate(), GNUNET_CONTAINER_multihashmap_iterate(), GNUNET_FS_uri_destroy(), GNUNET_FS_uri_dup(), GNUNET_FS_uri_ksk_get_keywords(), GNUNET_YES, GNUNET_FS_ShareTreeItem::is_directory, GNUNET_FS_ShareTreeItem::ksk_uri, GNUNET_FS_ShareTreeItem::meta, migrate_and_drop_keywords(), migrate_and_drop_metadata(), GNUNET_FS_ShareTreeItem::next, remove_high_frequency_keywords(), GNUNET_FS_ShareTreeItem::short_filename, and tc.

Referenced by GNUNET_FS_share_tree_trim().

Here is the call graph for this function:
Here is the caller graph for this function: