3 * Simple generational GC.
6 * Paolo Molaro (lupus@ximian.com)
7 * Rodrigo Kumpera (kumpera@gmail.com)
9 * Copyright 2005-2011 Novell, Inc (http://www.novell.com)
10 * Copyright 2011 Xamarin Inc (http://www.xamarin.com)
12 * Thread start/stop adapted from Boehm's GC:
13 * Copyright (c) 1994 by Xerox Corporation. All rights reserved.
14 * Copyright (c) 1996 by Silicon Graphics. All rights reserved.
15 * Copyright (c) 1998 by Fergus Henderson. All rights reserved.
16 * Copyright (c) 2000-2004 by Hewlett-Packard Company. All rights reserved.
17 * Copyright 2001-2003 Ximian, Inc
18 * Copyright 2003-2010 Novell, Inc.
19 * Copyright 2011 Xamarin, Inc.
20 * Copyright (C) 2012 Xamarin Inc
22 * Licensed under the MIT license. See LICENSE file in the project root for full license information.
24 * Important: allocation provides always zeroed memory, having to do
25 * a memset after allocation is deadly for performance.
26 * Memory usage at startup is currently as follows:
28 * 64 KB internal space
30 * We should provide a small memory config with half the sizes
32 * We currently try to make as few mono assumptions as possible:
33 * 1) 2-word header with no GC pointers in it (first vtable, second to store the
35 * 2) gc descriptor is the second word in the vtable (first word in the class)
36 * 3) 8 byte alignment is the minimum and enough (not true for special structures (SIMD), FIXME)
37 * 4) there is a function to get an object's size and the number of
38 * elements in an array.
39 * 5) we know the special way bounds are allocated for complex arrays
40 * 6) we know about proxies and how to treat them when domains are unloaded
42 * Always try to keep stack usage to a minimum: no recursive behaviour
43 * and no large stack allocs.
45 * General description.
46 * Objects are initially allocated in a nursery using a fast bump-pointer technique.
47 * When the nursery is full we start a nursery collection: this is performed with a
49 * When the old generation is full we start a copying GC of the old generation as well:
50 * this will be changed to mark&sweep with copying when fragmentation becomes to severe
51 * in the future. Maybe we'll even do both during the same collection like IMMIX.
53 * The things that complicate this description are:
54 * *) pinned objects: we can't move them so we need to keep track of them
55 * *) no precise info of the thread stacks and registers: we need to be able to
56 * quickly find the objects that may be referenced conservatively and pin them
57 * (this makes the first issues more important)
58 * *) large objects are too expensive to be dealt with using copying GC: we handle them
59 * with mark/sweep during major collections
60 * *) some objects need to not move even if they are small (interned strings, Type handles):
61 * we use mark/sweep for them, too: they are not allocated in the nursery, but inside
62 * PinnedChunks regions
68 *) we could have a function pointer in MonoClass to implement
69 customized write barriers for value types
71 *) investigate the stuff needed to advance a thread to a GC-safe
72 point (single-stepping, read from unmapped memory etc) and implement it.
73 This would enable us to inline allocations and write barriers, for example,
74 or at least parts of them, like the write barrier checks.
75 We may need this also for handling precise info on stacks, even simple things
76 as having uninitialized data on the stack and having to wait for the prolog
77 to zero it. Not an issue for the last frame that we scan conservatively.
78 We could always not trust the value in the slots anyway.
80 *) modify the jit to save info about references in stack locations:
81 this can be done just for locals as a start, so that at least
82 part of the stack is handled precisely.
84 *) test/fix endianess issues
86 *) Implement a card table as the write barrier instead of remembered
87 sets? Card tables are not easy to implement with our current
88 memory layout. We have several different kinds of major heap
89 objects: Small objects in regular blocks, small objects in pinned
90 chunks and LOS objects. If we just have a pointer we have no way
91 to tell which kind of object it points into, therefore we cannot
92 know where its card table is. The least we have to do to make
93 this happen is to get rid of write barriers for indirect stores.
96 *) Get rid of write barriers for indirect stores. We can do this by
97 telling the GC to wbarrier-register an object once we do an ldloca
98 or ldelema on it, and to unregister it once it's not used anymore
99 (it can only travel downwards on the stack). The problem with
100 unregistering is that it needs to happen eventually no matter
101 what, even if exceptions are thrown, the thread aborts, etc.
102 Rodrigo suggested that we could do only the registering part and
103 let the collector find out (pessimistically) when it's safe to
104 unregister, namely when the stack pointer of the thread that
105 registered the object is higher than it was when the registering
106 happened. This might make for a good first implementation to get
107 some data on performance.
109 *) Some sort of blacklist support? Blacklists is a concept from the
110 Boehm GC: if during a conservative scan we find pointers to an
111 area which we might use as heap, we mark that area as unusable, so
112 pointer retention by random pinning pointers is reduced.
114 *) experiment with max small object size (very small right now - 2kb,
115 because it's tied to the max freelist size)
117 *) add an option to mmap the whole heap in one chunk: it makes for many
118 simplifications in the checks (put the nursery at the top and just use a single
119 check for inclusion/exclusion): the issue this has is that on 32 bit systems it's
120 not flexible (too much of the address space may be used by default or we can't
121 increase the heap as needed) and we'd need a race-free mechanism to return memory
122 back to the system (mprotect(PROT_NONE) will still keep the memory allocated if it
123 was written to, munmap is needed, but the following mmap may not find the same segment
126 *) memzero the major fragments after restarting the world and optionally a smaller
129 *) investigate having fragment zeroing threads
131 *) separate locks for finalization and other minor stuff to reduce
134 *) try a different copying order to improve memory locality
136 *) a thread abort after a store but before the write barrier will
137 prevent the write barrier from executing
139 *) specialized dynamically generated markers/copiers
141 *) Dynamically adjust TLAB size to the number of threads. If we have
142 too many threads that do allocation, we might need smaller TLABs,
143 and we might get better performance with larger TLABs if we only
144 have a handful of threads. We could sum up the space left in all
145 assigned TLABs and if that's more than some percentage of the
146 nursery size, reduce the TLAB size.
148 *) Explore placing unreachable objects on unused nursery memory.
149 Instead of memset'ng a region to zero, place an int[] covering it.
150 A good place to start is add_nursery_frag. The tricky thing here is
151 placing those objects atomically outside of a collection.
153 *) Allocation should use asymmetric Dekker synchronization:
154 http://blogs.oracle.com/dave/resource/Asymmetric-Dekker-Synchronization.txt
155 This should help weak consistency archs.
162 #define _XOPEN_SOURCE
163 #define _DARWIN_C_SOURCE
169 #ifdef HAVE_PTHREAD_H
172 #ifdef HAVE_PTHREAD_NP_H
173 #include <pthread_np.h>
182 #include "mono/sgen/sgen-gc.h"
183 #include "mono/sgen/sgen-cardtable.h"
184 #include "mono/sgen/sgen-protocol.h"
185 #include "mono/sgen/sgen-memory-governor.h"
186 #include "mono/sgen/sgen-hash-table.h"
187 #include "mono/sgen/sgen-pinning.h"
188 #include "mono/sgen/sgen-workers.h"
189 #include "mono/sgen/sgen-client.h"
190 #include "mono/sgen/sgen-pointer-queue.h"
191 #include "mono/sgen/gc-internal-agnostic.h"
192 #include "mono/utils/mono-proclib.h"
193 #include "mono/utils/mono-memory-model.h"
194 #include "mono/utils/hazard-pointer.h"
196 #include <mono/utils/memcheck.h>
197 #include <mono/utils/mono-mmap-internals.h>
199 #undef pthread_create
201 #undef pthread_detach
204 * ######################################################################
205 * ######## Types and constants used by the GC.
206 * ######################################################################
209 /* 0 means not initialized, 1 is initialized, -1 means in progress */
210 static int gc_initialized = 0;
211 /* If set, check if we need to do something every X allocations */
212 gboolean has_per_allocation_action;
213 /* If set, do a heap check every X allocation */
214 guint32 verify_before_allocs = 0;
215 /* If set, do a minor collection before every X allocation */
216 guint32 collect_before_allocs = 0;
217 /* If set, do a whole heap check before each collection */
218 static gboolean whole_heap_check_before_collection = FALSE;
219 /* If set, do a remset consistency check at various opportunities */
220 static gboolean remset_consistency_checks = FALSE;
221 /* If set, do a mod union consistency check before each finishing collection pause */
222 static gboolean mod_union_consistency_check = FALSE;
223 /* If set, check whether mark bits are consistent after major collections */
224 static gboolean check_mark_bits_after_major_collection = FALSE;
225 /* If set, check that all nursery objects are pinned/not pinned, depending on context */
226 static gboolean check_nursery_objects_pinned = FALSE;
227 /* If set, do a few checks when the concurrent collector is used */
228 static gboolean do_concurrent_checks = FALSE;
229 /* If set, do a plausibility check on the scan_starts before and after
231 static gboolean do_scan_starts_check = FALSE;
233 static gboolean disable_minor_collections = FALSE;
234 static gboolean disable_major_collections = FALSE;
235 static gboolean do_verify_nursery = FALSE;
236 static gboolean do_dump_nursery_content = FALSE;
237 static gboolean enable_nursery_canaries = FALSE;
239 static gboolean precleaning_enabled = TRUE;
240 static gboolean dynamic_nursery = FALSE;
241 static size_t min_nursery_size = 0;
242 static size_t max_nursery_size = 0;
244 #ifdef HEAVY_STATISTICS
245 guint64 stat_objects_alloced_degraded = 0;
246 guint64 stat_bytes_alloced_degraded = 0;
248 guint64 stat_copy_object_called_nursery = 0;
249 guint64 stat_objects_copied_nursery = 0;
250 guint64 stat_copy_object_called_major = 0;
251 guint64 stat_objects_copied_major = 0;
253 guint64 stat_scan_object_called_nursery = 0;
254 guint64 stat_scan_object_called_major = 0;
256 guint64 stat_slots_allocated_in_vain;
258 guint64 stat_nursery_copy_object_failed_from_space = 0;
259 guint64 stat_nursery_copy_object_failed_forwarded = 0;
260 guint64 stat_nursery_copy_object_failed_pinned = 0;
261 guint64 stat_nursery_copy_object_failed_to_space = 0;
263 static guint64 stat_wbarrier_add_to_global_remset = 0;
264 static guint64 stat_wbarrier_arrayref_copy = 0;
265 static guint64 stat_wbarrier_generic_store = 0;
266 static guint64 stat_wbarrier_generic_store_atomic = 0;
267 static guint64 stat_wbarrier_set_root = 0;
270 static guint64 stat_pinned_objects = 0;
272 static guint64 time_minor_pre_collection_fragment_clear = 0;
273 static guint64 time_minor_pinning = 0;
274 static guint64 time_minor_scan_remsets = 0;
275 static guint64 time_minor_scan_major_blocks = 0;
276 static guint64 time_minor_scan_los = 0;
277 static guint64 time_minor_scan_pinned = 0;
278 static guint64 time_minor_scan_roots = 0;
279 static guint64 time_minor_finish_gray_stack = 0;
280 static guint64 time_minor_fragment_creation = 0;
282 static guint64 time_major_pre_collection_fragment_clear = 0;
283 static guint64 time_major_pinning = 0;
284 static guint64 time_major_scan_pinned = 0;
285 static guint64 time_major_scan_roots = 0;
286 static guint64 time_major_scan_mod_union_blocks = 0;
287 static guint64 time_major_scan_mod_union_los = 0;
288 static guint64 time_major_finish_gray_stack = 0;
289 static guint64 time_major_free_bigobjs = 0;
290 static guint64 time_major_los_sweep = 0;
291 static guint64 time_major_sweep = 0;
292 static guint64 time_major_fragment_creation = 0;
294 static guint64 time_max = 0;
296 static int sgen_max_pause_time = SGEN_DEFAULT_MAX_PAUSE_TIME;
297 static float sgen_max_pause_margin = SGEN_DEFAULT_MAX_PAUSE_MARGIN;
299 static SGEN_TV_DECLARE (time_major_conc_collection_start);
300 static SGEN_TV_DECLARE (time_major_conc_collection_end);
302 int gc_debug_level = 0;
304 static char* gc_params_options;
305 static char* gc_debug_options;
309 mono_gc_flush_info (void)
311 fflush (gc_debug_file);
315 #define TV_DECLARE SGEN_TV_DECLARE
316 #define TV_GETTIME SGEN_TV_GETTIME
317 #define TV_ELAPSED SGEN_TV_ELAPSED
319 static SGEN_TV_DECLARE (sgen_init_timestamp);
321 NurseryClearPolicy nursery_clear_policy = CLEAR_AT_TLAB_CREATION;
323 #define object_is_forwarded SGEN_OBJECT_IS_FORWARDED
324 #define object_is_pinned SGEN_OBJECT_IS_PINNED
325 #define pin_object SGEN_PIN_OBJECT
327 #define ptr_in_nursery sgen_ptr_in_nursery
329 #define LOAD_VTABLE SGEN_LOAD_VTABLE
332 nursery_canaries_enabled (void)
334 return enable_nursery_canaries;
337 #define safe_object_get_size sgen_safe_object_get_size
342 SGEN_MAJOR_CONCURRENT,
343 SGEN_MAJOR_CONCURRENT_PARALLEL
349 SGEN_MINOR_SIMPLE_PARALLEL,
356 SGEN_MODE_THROUGHPUT,
361 * ######################################################################
362 * ######## Global data.
363 * ######################################################################
365 MonoCoopMutex gc_mutex;
367 #define SCAN_START_SIZE SGEN_SCAN_START_SIZE
369 size_t degraded_mode = 0;
371 static mword bytes_pinned_from_failed_allocation = 0;
373 GCMemSection *nursery_section = NULL;
374 static volatile mword lowest_heap_address = ~(mword)0;
375 static volatile mword highest_heap_address = 0;
377 MonoCoopMutex sgen_interruption_mutex;
379 int current_collection_generation = -1;
380 volatile gboolean concurrent_collection_in_progress = FALSE;
382 /* objects that are ready to be finalized */
383 static SgenPointerQueue fin_ready_queue = SGEN_POINTER_QUEUE_INIT (INTERNAL_MEM_FINALIZE_READY);
384 static SgenPointerQueue critical_fin_queue = SGEN_POINTER_QUEUE_INIT (INTERNAL_MEM_FINALIZE_READY);
386 /* registered roots: the key to the hash is the root start address */
388 * Different kinds of roots are kept separate to speed up pin_from_roots () for example.
390 SgenHashTable roots_hash [ROOT_TYPE_NUM] = {
391 SGEN_HASH_TABLE_INIT (INTERNAL_MEM_ROOTS_TABLE, INTERNAL_MEM_ROOT_RECORD, sizeof (RootRecord), sgen_aligned_addr_hash, NULL),
392 SGEN_HASH_TABLE_INIT (INTERNAL_MEM_ROOTS_TABLE, INTERNAL_MEM_ROOT_RECORD, sizeof (RootRecord), sgen_aligned_addr_hash, NULL),
393 SGEN_HASH_TABLE_INIT (INTERNAL_MEM_ROOTS_TABLE, INTERNAL_MEM_ROOT_RECORD, sizeof (RootRecord), sgen_aligned_addr_hash, NULL)
395 static mword roots_size = 0; /* amount of memory in the root set */
397 /* The size of a TLAB */
398 /* The bigger the value, the less often we have to go to the slow path to allocate a new
399 * one, but the more space is wasted by threads not allocating much memory.
401 * FIXME: Make this self-tuning for each thread.
403 guint32 tlab_size = (1024 * 4);
405 #define MAX_SMALL_OBJ_SIZE SGEN_MAX_SMALL_OBJ_SIZE
407 #define ALLOC_ALIGN SGEN_ALLOC_ALIGN
409 #define ALIGN_UP SGEN_ALIGN_UP
411 #ifdef SGEN_DEBUG_INTERNAL_ALLOC
412 MonoNativeThreadId main_gc_thread = NULL;
415 /*Object was pinned during the current collection*/
416 static mword objects_pinned;
419 * ######################################################################
420 * ######## Macros and function declarations.
421 * ######################################################################
424 /* forward declarations */
425 static void scan_from_registered_roots (char *addr_start, char *addr_end, int root_type, ScanCopyContext ctx);
427 static void pin_from_roots (void *start_nursery, void *end_nursery, ScanCopyContext ctx);
428 static void finish_gray_stack (int generation, ScanCopyContext ctx);
431 SgenMajorCollector major_collector;
432 SgenMinorCollector sgen_minor_collector;
434 static SgenRememberedSet remset;
437 * The gray queue a worker job must use. If we're not parallel or
438 * concurrent, we use the main gray queue.
440 static SgenGrayQueue*
441 sgen_workers_get_job_gray_queue (WorkerData *worker_data, SgenGrayQueue *default_gray_queue)
444 return &worker_data->private_gray_queue;
445 SGEN_ASSERT (0, default_gray_queue, "Why don't we have a default gray queue when we're not running in a worker thread?");
446 return default_gray_queue;
450 gray_queue_redirect (SgenGrayQueue *queue)
452 sgen_workers_take_from_queue (current_collection_generation, queue);
456 sgen_scan_area_with_callback (char *start, char *end, IterateObjectCallbackFunc callback, void *data, gboolean allow_flags, gboolean fail_on_canaries)
458 while (start < end) {
462 if (!*(void**)start) {
463 start += sizeof (void*); /* should be ALLOC_ALIGN, really */
468 if (!(obj = (char *)SGEN_OBJECT_IS_FORWARDED (start)))
474 if (!sgen_client_object_is_array_fill ((GCObject*)obj)) {
475 CHECK_CANARY_FOR_OBJECT ((GCObject*)obj, fail_on_canaries);
476 size = ALIGN_UP (safe_object_get_size ((GCObject*)obj));
477 callback ((GCObject*)obj, size, data);
478 CANARIFY_SIZE (size);
480 size = ALIGN_UP (safe_object_get_size ((GCObject*)obj));
488 * sgen_add_to_global_remset:
490 * The global remset contains locations which point into newspace after
491 * a minor collection. This can happen if the objects they point to are pinned.
493 * LOCKING: If called from a parallel collector, the global remset
494 * lock must be held. For serial collectors that is not necessary.
497 sgen_add_to_global_remset (gpointer ptr, GCObject *obj)
499 SGEN_ASSERT (5, sgen_ptr_in_nursery (obj), "Target pointer of global remset must be in the nursery");
501 HEAVY_STAT (++stat_wbarrier_add_to_global_remset);
503 if (!major_collector.is_concurrent) {
504 SGEN_ASSERT (5, current_collection_generation != -1, "Global remsets can only be added during collections");
506 if (current_collection_generation == -1)
507 SGEN_ASSERT (5, sgen_concurrent_collection_in_progress (), "Global remsets outside of collection pauses can only be added by the concurrent collector");
510 if (!object_is_pinned (obj))
511 SGEN_ASSERT (5, sgen_minor_collector.is_split || sgen_concurrent_collection_in_progress (), "Non-pinned objects can only remain in nursery if it is a split nursery");
512 else if (sgen_cement_lookup_or_register (obj))
515 remset.record_pointer (ptr);
517 sgen_pin_stats_register_global_remset (obj);
519 SGEN_LOG (8, "Adding global remset for %p", ptr);
520 binary_protocol_global_remset (ptr, obj, (gpointer)SGEN_LOAD_VTABLE (obj));
524 * sgen_drain_gray_stack:
526 * Scan objects in the gray stack until the stack is empty. This should be called
527 * frequently after each object is copied, to achieve better locality and cache
532 sgen_drain_gray_stack (ScanCopyContext ctx)
534 SGEN_ASSERT (0, ctx.ops->drain_gray_stack, "Why do we have a scan/copy context with a missing drain gray stack function?");
536 return ctx.ops->drain_gray_stack (ctx.queue);
540 * Addresses in the pin queue are already sorted. This function finds
541 * the object header for each address and pins the object. The
542 * addresses must be inside the nursery section. The (start of the)
543 * address array is overwritten with the addresses of the actually
544 * pinned objects. Return the number of pinned objects.
547 pin_objects_from_nursery_pin_queue (gboolean do_scan_objects, ScanCopyContext ctx)
549 GCMemSection *section = nursery_section;
550 void **start = sgen_pinning_get_entry (section->pin_queue_first_entry);
551 void **end = sgen_pinning_get_entry (section->pin_queue_last_entry);
552 void *start_nursery = section->data;
553 void *end_nursery = section->end_data;
558 void *pinning_front = start_nursery;
560 void **definitely_pinned = start;
561 ScanObjectFunc scan_func = ctx.ops->scan_object;
562 SgenGrayQueue *queue = ctx.queue;
564 sgen_nursery_allocator_prepare_for_pinning ();
566 while (start < end) {
567 GCObject *obj_to_pin = NULL;
568 size_t obj_to_pin_size = 0;
573 SGEN_ASSERT (0, addr >= start_nursery && addr < end_nursery, "Potential pinning address out of range");
574 SGEN_ASSERT (0, addr >= last, "Pin queue not sorted");
581 SGEN_LOG (5, "Considering pinning addr %p", addr);
582 /* We've already processed everything up to pinning_front. */
583 if (addr < pinning_front) {
589 * Find the closest scan start <= addr. We might search backward in the
590 * scan_starts array because entries might be NULL. In the worst case we
591 * start at start_nursery.
593 idx = ((char*)addr - (char*)section->data) / SCAN_START_SIZE;
594 SGEN_ASSERT (0, idx < section->num_scan_start, "Scan start index out of range");
595 search_start = (void*)section->scan_starts [idx];
596 if (!search_start || search_start > addr) {
599 search_start = section->scan_starts [idx];
600 if (search_start && search_start <= addr)
603 if (!search_start || search_start > addr)
604 search_start = start_nursery;
608 * If the pinning front is closer than the scan start we found, start
609 * searching at the front.
611 if (search_start < pinning_front)
612 search_start = pinning_front;
615 * Now addr should be in an object a short distance from search_start.
617 * search_start must point to zeroed mem or point to an object.
620 size_t obj_size, canarified_obj_size;
623 if (!*(void**)search_start) {
624 search_start = (void*)ALIGN_UP ((mword)search_start + sizeof (gpointer));
625 /* The loop condition makes sure we don't overrun addr. */
629 canarified_obj_size = obj_size = ALIGN_UP (safe_object_get_size ((GCObject*)search_start));
632 * Filler arrays are marked by an invalid sync word. We don't
633 * consider them for pinning. They are not delimited by canaries,
636 if (!sgen_client_object_is_array_fill ((GCObject*)search_start)) {
637 CHECK_CANARY_FOR_OBJECT (search_start, TRUE);
638 CANARIFY_SIZE (canarified_obj_size);
640 if (addr >= search_start && (char*)addr < (char*)search_start + obj_size) {
641 /* This is the object we're looking for. */
642 obj_to_pin = (GCObject*)search_start;
643 obj_to_pin_size = canarified_obj_size;
648 /* Skip to the next object */
649 search_start = (void*)((char*)search_start + canarified_obj_size);
650 } while (search_start <= addr);
652 /* We've searched past the address we were looking for. */
654 pinning_front = search_start;
655 goto next_pin_queue_entry;
659 * We've found an object to pin. It might still be a dummy array, but we
660 * can advance the pinning front in any case.
662 pinning_front = (char*)obj_to_pin + obj_to_pin_size;
665 * If this is a dummy array marking the beginning of a nursery
666 * fragment, we don't pin it.
668 if (sgen_client_object_is_array_fill (obj_to_pin))
669 goto next_pin_queue_entry;
672 * Finally - pin the object!
674 desc = sgen_obj_get_descriptor_safe (obj_to_pin);
675 if (do_scan_objects) {
676 scan_func (obj_to_pin, desc, queue);
678 SGEN_LOG (4, "Pinned object %p, vtable %p (%s), count %d\n",
679 obj_to_pin, *(void**)obj_to_pin, sgen_client_vtable_get_name (SGEN_LOAD_VTABLE (obj_to_pin)), count);
680 binary_protocol_pin (obj_to_pin,
681 (gpointer)LOAD_VTABLE (obj_to_pin),
682 safe_object_get_size (obj_to_pin));
684 pin_object (obj_to_pin);
685 GRAY_OBJECT_ENQUEUE_SERIAL (queue, obj_to_pin, desc);
686 sgen_pin_stats_register_object (obj_to_pin, GENERATION_NURSERY);
687 definitely_pinned [count] = obj_to_pin;
690 if (concurrent_collection_in_progress)
691 sgen_pinning_register_pinned_in_nursery (obj_to_pin);
693 next_pin_queue_entry:
697 sgen_client_nursery_objects_pinned (definitely_pinned, count);
698 stat_pinned_objects += count;
703 pin_objects_in_nursery (gboolean do_scan_objects, ScanCopyContext ctx)
707 if (nursery_section->pin_queue_first_entry == nursery_section->pin_queue_last_entry)
710 reduced_to = pin_objects_from_nursery_pin_queue (do_scan_objects, ctx);
711 nursery_section->pin_queue_last_entry = nursery_section->pin_queue_first_entry + reduced_to;
715 * This function is only ever called (via `collector_pin_object()` in `sgen-copy-object.h`)
716 * when we can't promote an object because we're out of memory.
719 sgen_pin_object (GCObject *object, SgenGrayQueue *queue)
721 SGEN_ASSERT (0, sgen_ptr_in_nursery (object), "We're only supposed to use this for pinning nursery objects when out of memory.");
724 * All pinned objects are assumed to have been staged, so we need to stage as well.
725 * Also, the count of staged objects shows that "late pinning" happened.
727 sgen_pin_stage_ptr (object);
729 SGEN_PIN_OBJECT (object);
730 binary_protocol_pin (object, (gpointer)LOAD_VTABLE (object), safe_object_get_size (object));
733 sgen_pin_stats_register_object (object, GENERATION_NURSERY);
735 GRAY_OBJECT_ENQUEUE_SERIAL (queue, object, sgen_obj_get_descriptor_safe (object));
738 /* Sort the addresses in array in increasing order.
739 * Done using a by-the book heap sort. Which has decent and stable performance, is pretty cache efficient.
742 sgen_sort_addresses (void **array, size_t size)
747 for (i = 1; i < size; ++i) {
750 size_t parent = (child - 1) / 2;
752 if (array [parent] >= array [child])
755 tmp = array [parent];
756 array [parent] = array [child];
763 for (i = size - 1; i > 0; --i) {
766 array [i] = array [0];
772 while (root * 2 + 1 <= end) {
773 size_t child = root * 2 + 1;
775 if (child < end && array [child] < array [child + 1])
777 if (array [root] >= array [child])
781 array [root] = array [child];
790 * Scan the memory between start and end and queue values which could be pointers
791 * to the area between start_nursery and end_nursery for later consideration.
792 * Typically used for thread stacks.
795 sgen_conservatively_pin_objects_from (void **start, void **end, void *start_nursery, void *end_nursery, int pin_type)
799 SGEN_ASSERT (0, ((mword)start & (SIZEOF_VOID_P - 1)) == 0, "Why are we scanning for references in unaligned memory ?");
801 #if defined(VALGRIND_MAKE_MEM_DEFINED_IF_ADDRESSABLE) && !defined(_WIN64)
802 VALGRIND_MAKE_MEM_DEFINED_IF_ADDRESSABLE (start, (char*)end - (char*)start);
805 while (start < end) {
807 * *start can point to the middle of an object
808 * note: should we handle pointing at the end of an object?
809 * pinning in C# code disallows pointing at the end of an object
810 * but there is some small chance that an optimizing C compiler
811 * may keep the only reference to an object by pointing
812 * at the end of it. We ignore this small chance for now.
813 * Pointers to the end of an object are indistinguishable
814 * from pointers to the start of the next object in memory
815 * so if we allow that we'd need to pin two objects...
816 * We queue the pointer in an array, the
817 * array will then be sorted and uniqued. This way
818 * we can coalesce several pinning pointers and it should
819 * be faster since we'd do a memory scan with increasing
820 * addresses. Note: we can align the address to the allocation
821 * alignment, so the unique process is more effective.
823 mword addr = (mword)*start;
824 addr &= ~(ALLOC_ALIGN - 1);
825 if (addr >= (mword)start_nursery && addr < (mword)end_nursery) {
826 SGEN_LOG (6, "Pinning address %p from %p", (void*)addr, start);
827 sgen_pin_stage_ptr ((void*)addr);
828 binary_protocol_pin_stage (start, (void*)addr);
829 sgen_pin_stats_register_address ((char*)addr, pin_type);
835 SGEN_LOG (7, "found %d potential pinned heap pointers", count);
839 * The first thing we do in a collection is to identify pinned objects.
840 * This function considers all the areas of memory that need to be
841 * conservatively scanned.
844 pin_from_roots (void *start_nursery, void *end_nursery, ScanCopyContext ctx)
848 SGEN_LOG (2, "Scanning pinned roots (%d bytes, %d/%d entries)", (int)roots_size, roots_hash [ROOT_TYPE_NORMAL].num_entries, roots_hash [ROOT_TYPE_PINNED].num_entries);
849 /* objects pinned from the API are inside these roots */
850 SGEN_HASH_TABLE_FOREACH (&roots_hash [ROOT_TYPE_PINNED], void **, start_root, RootRecord *, root) {
851 SGEN_LOG (6, "Pinned roots %p-%p", start_root, root->end_root);
852 sgen_conservatively_pin_objects_from (start_root, (void**)root->end_root, start_nursery, end_nursery, PIN_TYPE_OTHER);
853 } SGEN_HASH_TABLE_FOREACH_END;
854 /* now deal with the thread stacks
855 * in the future we should be able to conservatively scan only:
856 * *) the cpu registers
857 * *) the unmanaged stack frames
858 * *) the _last_ managed stack frame
859 * *) pointers slots in managed frames
861 sgen_client_scan_thread_data (start_nursery, end_nursery, FALSE, ctx);
865 single_arg_user_copy_or_mark (GCObject **obj, void *gc_data)
867 ScanCopyContext *ctx = (ScanCopyContext *)gc_data;
868 ctx->ops->copy_or_mark_object (obj, ctx->queue);
872 * The memory area from start_root to end_root contains pointers to objects.
873 * Their position is precisely described by @desc (this means that the pointer
874 * can be either NULL or the pointer to the start of an object).
875 * This functions copies them to to_space updates them.
877 * This function is not thread-safe!
880 precisely_scan_objects_from (void** start_root, void** end_root, char* n_start, char *n_end, SgenDescriptor desc, ScanCopyContext ctx)
882 CopyOrMarkObjectFunc copy_func = ctx.ops->copy_or_mark_object;
883 ScanPtrFieldFunc scan_field_func = ctx.ops->scan_ptr_field;
884 SgenGrayQueue *queue = ctx.queue;
886 switch (desc & ROOT_DESC_TYPE_MASK) {
887 case ROOT_DESC_BITMAP:
888 desc >>= ROOT_DESC_TYPE_SHIFT;
890 if ((desc & 1) && *start_root) {
891 copy_func ((GCObject**)start_root, queue);
892 SGEN_LOG (9, "Overwrote root at %p with %p", start_root, *start_root);
898 case ROOT_DESC_COMPLEX: {
899 gsize *bitmap_data = (gsize *)sgen_get_complex_descriptor_bitmap (desc);
900 gsize bwords = (*bitmap_data) - 1;
901 void **start_run = start_root;
903 while (bwords-- > 0) {
904 gsize bmap = *bitmap_data++;
905 void **objptr = start_run;
907 if ((bmap & 1) && *objptr) {
908 copy_func ((GCObject**)objptr, queue);
909 SGEN_LOG (9, "Overwrote root at %p with %p", objptr, *objptr);
914 start_run += GC_BITS_PER_WORD;
918 case ROOT_DESC_VECTOR: {
921 for (p = start_root; p < end_root; p++) {
923 scan_field_func (NULL, (GCObject**)p, queue);
927 case ROOT_DESC_USER: {
928 SgenUserRootMarkFunc marker = sgen_get_user_descriptor_func (desc);
929 marker (start_root, single_arg_user_copy_or_mark, &ctx);
932 case ROOT_DESC_RUN_LEN:
933 g_assert_not_reached ();
935 g_assert_not_reached ();
940 reset_heap_boundaries (void)
942 lowest_heap_address = ~(mword)0;
943 highest_heap_address = 0;
947 sgen_update_heap_boundaries (mword low, mword high)
952 old = lowest_heap_address;
955 } while (SGEN_CAS_PTR ((gpointer*)&lowest_heap_address, (gpointer)low, (gpointer)old) != (gpointer)old);
958 old = highest_heap_address;
961 } while (SGEN_CAS_PTR ((gpointer*)&highest_heap_address, (gpointer)high, (gpointer)old) != (gpointer)old);
965 * Allocate and setup the data structures needed to be able to allocate objects
966 * in the nursery. The nursery is stored in nursery_section.
969 alloc_nursery (gboolean dynamic, size_t min_size, size_t max_size)
976 min_size = SGEN_DEFAULT_NURSERY_MIN_SIZE;
978 max_size = SGEN_DEFAULT_NURSERY_MAX_SIZE;
980 SGEN_ASSERT (0, min_size == max_size, "We can't have nursery ranges for static configuration.");
982 min_size = max_size = SGEN_DEFAULT_NURSERY_SIZE;
985 SGEN_ASSERT (0, !nursery_section, "Why are we allocating the nursery twice?");
986 SGEN_LOG (2, "Allocating nursery size: %zu, initial %zu", max_size, min_size);
988 /* FIXME: handle OOM */
989 nursery_section = (GCMemSection *)sgen_alloc_internal (INTERNAL_MEM_SECTION);
991 /* If there isn't enough space even for the nursery we should simply abort. */
992 g_assert (sgen_memgov_try_alloc_space (max_size, SPACE_NURSERY));
995 * The nursery section range represents the memory section where objects
996 * can be found. This is used when iterating for objects in the nursery,
997 * pinning etc. sgen_nursery_max_size represents the total allocated space
998 * for the nursery. sgen_nursery_size represents the current size of the
999 * nursery and it is used for allocation limits, heuristics etc. The
1000 * nursery section is not always identical to the current nursery size
1001 * because it can contain pinned objects from when the nursery was larger.
1003 * sgen_nursery_size <= nursery_section size <= sgen_nursery_max_size
1005 data = (char *)major_collector.alloc_heap (max_size, max_size);
1006 sgen_update_heap_boundaries ((mword)data, (mword)(data + max_size));
1007 nursery_section->data = data;
1008 nursery_section->end_data = data + min_size;
1009 scan_starts = (max_size + SCAN_START_SIZE - 1) / SCAN_START_SIZE;
1010 nursery_section->scan_starts = (char **)sgen_alloc_internal_dynamic (sizeof (char*) * scan_starts, INTERNAL_MEM_SCAN_STARTS, TRUE);
1011 nursery_section->num_scan_start = scan_starts;
1013 sgen_nursery_allocator_set_nursery_bounds (data, min_size, max_size);
1017 mono_gc_get_logfile (void)
1019 return gc_debug_file;
1023 mono_gc_params_set (const char* options)
1025 if (gc_params_options)
1026 g_free (gc_params_options);
1028 gc_params_options = g_strdup (options);
1032 mono_gc_debug_set (const char* options)
1034 if (gc_debug_options)
1035 g_free (gc_debug_options);
1037 gc_debug_options = g_strdup (options);
1041 scan_finalizer_entries (SgenPointerQueue *fin_queue, ScanCopyContext ctx)
1043 CopyOrMarkObjectFunc copy_func = ctx.ops->copy_or_mark_object;
1044 SgenGrayQueue *queue = ctx.queue;
1047 for (i = 0; i < fin_queue->next_slot; ++i) {
1048 GCObject *obj = (GCObject *)fin_queue->data [i];
1051 SGEN_LOG (5, "Scan of fin ready object: %p (%s)\n", obj, sgen_client_vtable_get_name (SGEN_LOAD_VTABLE (obj)));
1052 copy_func ((GCObject**)&fin_queue->data [i], queue);
1057 generation_name (int generation)
1059 switch (generation) {
1060 case GENERATION_NURSERY: return "nursery";
1061 case GENERATION_OLD: return "old";
1062 default: g_assert_not_reached ();
1067 sgen_generation_name (int generation)
1069 return generation_name (generation);
1073 finish_gray_stack (int generation, ScanCopyContext ctx)
1077 int done_with_ephemerons, ephemeron_rounds = 0;
1078 char *start_addr = generation == GENERATION_NURSERY ? sgen_get_nursery_start () : NULL;
1079 char *end_addr = generation == GENERATION_NURSERY ? sgen_get_nursery_end () : (char*)-1;
1080 SgenGrayQueue *queue = ctx.queue;
1082 binary_protocol_finish_gray_stack_start (sgen_timestamp (), generation);
1084 * We copied all the reachable objects. Now it's the time to copy
1085 * the objects that were not referenced by the roots, but by the copied objects.
1086 * we built a stack of objects pointed to by gray_start: they are
1087 * additional roots and we may add more items as we go.
1088 * We loop until gray_start == gray_objects which means no more objects have
1089 * been added. Note this is iterative: no recursion is involved.
1090 * We need to walk the LO list as well in search of marked big objects
1091 * (use a flag since this is needed only on major collections). We need to loop
1092 * here as well, so keep a counter of marked LO (increasing it in copy_object).
1093 * To achieve better cache locality and cache usage, we drain the gray stack
1094 * frequently, after each object is copied, and just finish the work here.
1096 sgen_drain_gray_stack (ctx);
1098 SGEN_LOG (2, "%s generation done", generation_name (generation));
1101 Reset bridge data, we might have lingering data from a previous collection if this is a major
1102 collection trigged by minor overflow.
1104 We must reset the gathered bridges since their original block might be evacuated due to major
1105 fragmentation in the meanwhile and the bridge code should not have to deal with that.
1107 if (sgen_client_bridge_need_processing ())
1108 sgen_client_bridge_reset_data ();
1111 * Mark all strong toggleref objects. This must be done before we walk ephemerons or finalizers
1112 * to ensure they see the full set of live objects.
1114 sgen_client_mark_togglerefs (start_addr, end_addr, ctx);
1117 * Walk the ephemeron tables marking all values with reachable keys. This must be completely done
1118 * before processing finalizable objects and non-tracking weak links to avoid finalizing/clearing
1119 * objects that are in fact reachable.
1121 done_with_ephemerons = 0;
1123 done_with_ephemerons = sgen_client_mark_ephemerons (ctx);
1124 sgen_drain_gray_stack (ctx);
1126 } while (!done_with_ephemerons);
1128 if (sgen_client_bridge_need_processing ()) {
1129 /*Make sure the gray stack is empty before we process bridge objects so we get liveness right*/
1130 sgen_drain_gray_stack (ctx);
1131 sgen_collect_bridge_objects (generation, ctx);
1132 if (generation == GENERATION_OLD)
1133 sgen_collect_bridge_objects (GENERATION_NURSERY, ctx);
1136 Do the first bridge step here, as the collector liveness state will become useless after that.
1138 An important optimization is to only proccess the possibly dead part of the object graph and skip
1139 over all live objects as we transitively know everything they point must be alive too.
1141 The above invariant is completely wrong if we let the gray queue be drained and mark/copy everything.
1143 This has the unfortunate side effect of making overflow collections perform the first step twice, but
1144 given we now have heuristics that perform major GC in anticipation of minor overflows this should not
1147 sgen_client_bridge_processing_stw_step ();
1151 Make sure we drain the gray stack before processing disappearing links and finalizers.
1152 If we don't make sure it is empty we might wrongly see a live object as dead.
1154 sgen_drain_gray_stack (ctx);
1157 We must clear weak links that don't track resurrection before processing object ready for
1158 finalization so they can be cleared before that.
1160 sgen_null_link_in_range (generation, ctx, FALSE);
1161 if (generation == GENERATION_OLD)
1162 sgen_null_link_in_range (GENERATION_NURSERY, ctx, FALSE);
1165 /* walk the finalization queue and move also the objects that need to be
1166 * finalized: use the finalized objects as new roots so the objects they depend
1167 * on are also not reclaimed. As with the roots above, only objects in the nursery
1168 * are marked/copied.
1170 sgen_finalize_in_range (generation, ctx);
1171 if (generation == GENERATION_OLD)
1172 sgen_finalize_in_range (GENERATION_NURSERY, ctx);
1173 /* drain the new stack that might have been created */
1174 SGEN_LOG (6, "Precise scan of gray area post fin");
1175 sgen_drain_gray_stack (ctx);
1178 * This must be done again after processing finalizable objects since CWL slots are cleared only after the key is finalized.
1180 done_with_ephemerons = 0;
1182 done_with_ephemerons = sgen_client_mark_ephemerons (ctx);
1183 sgen_drain_gray_stack (ctx);
1185 } while (!done_with_ephemerons);
1187 sgen_client_clear_unreachable_ephemerons (ctx);
1190 * We clear togglerefs only after all possible chances of revival are done.
1191 * This is semantically more inline with what users expect and it allows for
1192 * user finalizers to correctly interact with TR objects.
1194 sgen_client_clear_togglerefs (start_addr, end_addr, ctx);
1197 SGEN_LOG (2, "Finalize queue handling scan for %s generation: %lld usecs %d ephemeron rounds", generation_name (generation), (long long)TV_ELAPSED (atv, btv), ephemeron_rounds);
1200 * handle disappearing links
1201 * Note we do this after checking the finalization queue because if an object
1202 * survives (at least long enough to be finalized) we don't clear the link.
1203 * This also deals with a possible issue with the monitor reclamation: with the Boehm
1204 * GC a finalized object my lose the monitor because it is cleared before the finalizer is
1207 g_assert (sgen_gray_object_queue_is_empty (queue));
1209 sgen_null_link_in_range (generation, ctx, TRUE);
1210 if (generation == GENERATION_OLD)
1211 sgen_null_link_in_range (GENERATION_NURSERY, ctx, TRUE);
1212 if (sgen_gray_object_queue_is_empty (queue))
1214 sgen_drain_gray_stack (ctx);
1217 g_assert (sgen_gray_object_queue_is_empty (queue));
1219 binary_protocol_finish_gray_stack_end (sgen_timestamp (), generation);
1223 sgen_check_section_scan_starts (GCMemSection *section)
1226 for (i = 0; i < section->num_scan_start; ++i) {
1227 if (section->scan_starts [i]) {
1228 mword size = safe_object_get_size ((GCObject*) section->scan_starts [i]);
1229 SGEN_ASSERT (0, size >= SGEN_CLIENT_MINIMUM_OBJECT_SIZE && size <= MAX_SMALL_OBJ_SIZE, "Weird object size at scan starts.");
1235 check_scan_starts (void)
1237 if (!do_scan_starts_check)
1239 sgen_check_section_scan_starts (nursery_section);
1240 major_collector.check_scan_starts ();
1244 scan_from_registered_roots (char *addr_start, char *addr_end, int root_type, ScanCopyContext ctx)
1248 SGEN_HASH_TABLE_FOREACH (&roots_hash [root_type], void **, start_root, RootRecord *, root) {
1249 SGEN_LOG (6, "Precise root scan %p-%p (desc: %p)", start_root, root->end_root, (void*)root->root_desc);
1250 precisely_scan_objects_from (start_root, (void**)root->end_root, addr_start, addr_end, root->root_desc, ctx);
1251 } SGEN_HASH_TABLE_FOREACH_END;
1257 static gboolean inited = FALSE;
1262 mono_counters_register ("Collection max time", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME | MONO_COUNTER_MONOTONIC, &time_max);
1264 mono_counters_register ("Minor fragment clear", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_minor_pre_collection_fragment_clear);
1265 mono_counters_register ("Minor pinning", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_minor_pinning);
1266 mono_counters_register ("Minor scan remembered set", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_minor_scan_remsets);
1267 mono_counters_register ("Minor scan major blocks", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_minor_scan_major_blocks);
1268 mono_counters_register ("Minor scan los", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_minor_scan_los);
1269 mono_counters_register ("Minor scan pinned", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_minor_scan_pinned);
1270 mono_counters_register ("Minor scan roots", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_minor_scan_roots);
1271 mono_counters_register ("Minor fragment creation", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_minor_fragment_creation);
1273 mono_counters_register ("Major fragment clear", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_pre_collection_fragment_clear);
1274 mono_counters_register ("Major pinning", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_pinning);
1275 mono_counters_register ("Major scan pinned", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_scan_pinned);
1276 mono_counters_register ("Major scan roots", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_scan_roots);
1277 mono_counters_register ("Major scan mod union blocks", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_scan_mod_union_blocks);
1278 mono_counters_register ("Major scan mod union los", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_scan_mod_union_los);
1279 mono_counters_register ("Major finish gray stack", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_finish_gray_stack);
1280 mono_counters_register ("Major free big objects", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_free_bigobjs);
1281 mono_counters_register ("Major LOS sweep", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_los_sweep);
1282 mono_counters_register ("Major sweep", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_sweep);
1283 mono_counters_register ("Major fragment creation", MONO_COUNTER_GC | MONO_COUNTER_ULONG | MONO_COUNTER_TIME, &time_major_fragment_creation);
1285 mono_counters_register ("Number of pinned objects", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_pinned_objects);
1287 #ifdef HEAVY_STATISTICS
1288 mono_counters_register ("WBarrier remember pointer", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_wbarrier_add_to_global_remset);
1289 mono_counters_register ("WBarrier arrayref copy", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_wbarrier_arrayref_copy);
1290 mono_counters_register ("WBarrier generic store called", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_wbarrier_generic_store);
1291 mono_counters_register ("WBarrier generic atomic store called", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_wbarrier_generic_store_atomic);
1292 mono_counters_register ("WBarrier set root", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_wbarrier_set_root);
1294 mono_counters_register ("# objects allocated degraded", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_objects_alloced_degraded);
1295 mono_counters_register ("bytes allocated degraded", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_bytes_alloced_degraded);
1297 mono_counters_register ("# copy_object() called (nursery)", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_copy_object_called_nursery);
1298 mono_counters_register ("# objects copied (nursery)", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_objects_copied_nursery);
1299 mono_counters_register ("# copy_object() called (major)", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_copy_object_called_major);
1300 mono_counters_register ("# objects copied (major)", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_objects_copied_major);
1302 mono_counters_register ("# scan_object() called (nursery)", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_scan_object_called_nursery);
1303 mono_counters_register ("# scan_object() called (major)", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_scan_object_called_major);
1305 mono_counters_register ("Slots allocated in vain", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_slots_allocated_in_vain);
1307 mono_counters_register ("# nursery copy_object() failed from space", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_nursery_copy_object_failed_from_space);
1308 mono_counters_register ("# nursery copy_object() failed forwarded", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_nursery_copy_object_failed_forwarded);
1309 mono_counters_register ("# nursery copy_object() failed pinned", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_nursery_copy_object_failed_pinned);
1310 mono_counters_register ("# nursery copy_object() failed to space", MONO_COUNTER_GC | MONO_COUNTER_ULONG, &stat_nursery_copy_object_failed_to_space);
1312 sgen_nursery_allocator_init_heavy_stats ();
1320 reset_pinned_from_failed_allocation (void)
1322 bytes_pinned_from_failed_allocation = 0;
1326 sgen_set_pinned_from_failed_allocation (mword objsize)
1328 bytes_pinned_from_failed_allocation += objsize;
1332 sgen_collection_is_concurrent (void)
1334 switch (current_collection_generation) {
1335 case GENERATION_NURSERY:
1337 case GENERATION_OLD:
1338 return concurrent_collection_in_progress;
1340 g_error ("Invalid current generation %d", current_collection_generation);
1346 sgen_concurrent_collection_in_progress (void)
1348 return concurrent_collection_in_progress;
1352 SgenThreadPoolJob job;
1353 SgenObjectOperations *ops;
1354 SgenGrayQueue *gc_thread_gray_queue;
1359 int job_index, job_split_count;
1363 static ScanCopyContext
1364 scan_copy_context_for_scan_job (void *worker_data_untyped, ScanJob *job)
1366 WorkerData *worker_data = (WorkerData *)worker_data_untyped;
1370 * For jobs enqueued on workers we set the ops at job runtime in order
1371 * to be able to profit from on the fly optimized object ops or other
1372 * object ops changes, like forced concurrent finish.
1374 SGEN_ASSERT (0, sgen_workers_is_worker_thread (mono_native_thread_id_get ()), "We need a context for the scan job");
1375 job->ops = sgen_workers_get_idle_func_object_ops (worker_data);
1378 return CONTEXT_FROM_OBJECT_OPERATIONS (job->ops, sgen_workers_get_job_gray_queue (worker_data, job->gc_thread_gray_queue));
1386 } ScanFromRegisteredRootsJob;
1389 job_scan_from_registered_roots (void *worker_data_untyped, SgenThreadPoolJob *job)
1391 ScanFromRegisteredRootsJob *job_data = (ScanFromRegisteredRootsJob*)job;
1392 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, &job_data->scan_job);
1394 scan_from_registered_roots (job_data->heap_start, job_data->heap_end, job_data->root_type, ctx);
1401 } ScanThreadDataJob;
1404 job_scan_thread_data (void *worker_data_untyped, SgenThreadPoolJob *job)
1406 ScanThreadDataJob *job_data = (ScanThreadDataJob*)job;
1407 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, &job_data->scan_job);
1409 sgen_client_scan_thread_data (job_data->heap_start, job_data->heap_end, TRUE, ctx);
1414 SgenPointerQueue *queue;
1415 } ScanFinalizerEntriesJob;
1418 job_scan_finalizer_entries (void *worker_data_untyped, SgenThreadPoolJob *job)
1420 ScanFinalizerEntriesJob *job_data = (ScanFinalizerEntriesJob*)job;
1421 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, &job_data->scan_job);
1423 scan_finalizer_entries (job_data->queue, ctx);
1427 job_scan_wbroots (void *worker_data_untyped, SgenThreadPoolJob *job)
1429 ScanJob *job_data = (ScanJob*)job;
1430 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, job_data);
1432 sgen_wbroots_scan_card_table (ctx);
1436 job_scan_major_card_table (void *worker_data_untyped, SgenThreadPoolJob *job)
1438 SGEN_TV_DECLARE (atv);
1439 SGEN_TV_DECLARE (btv);
1440 ParallelScanJob *job_data = (ParallelScanJob*)job;
1441 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, (ScanJob*)job_data);
1443 SGEN_TV_GETTIME (atv);
1444 major_collector.scan_card_table (CARDTABLE_SCAN_GLOBAL, ctx, job_data->job_index, job_data->job_split_count, job_data->data);
1445 SGEN_TV_GETTIME (btv);
1446 time_minor_scan_major_blocks += SGEN_TV_ELAPSED (atv, btv);
1448 if (worker_data_untyped)
1449 ((WorkerData*)worker_data_untyped)->major_scan_time += SGEN_TV_ELAPSED (atv, btv);
1453 job_scan_los_card_table (void *worker_data_untyped, SgenThreadPoolJob *job)
1455 SGEN_TV_DECLARE (atv);
1456 SGEN_TV_DECLARE (btv);
1457 ParallelScanJob *job_data = (ParallelScanJob*)job;
1458 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, (ScanJob*)job_data);
1460 SGEN_TV_GETTIME (atv);
1461 sgen_los_scan_card_table (CARDTABLE_SCAN_GLOBAL, ctx, job_data->job_index, job_data->job_split_count);
1462 SGEN_TV_GETTIME (btv);
1463 time_minor_scan_los += SGEN_TV_ELAPSED (atv, btv);
1465 if (worker_data_untyped)
1466 ((WorkerData*)worker_data_untyped)->los_scan_time += SGEN_TV_ELAPSED (atv, btv);
1470 job_scan_major_mod_union_card_table (void *worker_data_untyped, SgenThreadPoolJob *job)
1472 SGEN_TV_DECLARE (atv);
1473 SGEN_TV_DECLARE (btv);
1474 ParallelScanJob *job_data = (ParallelScanJob*)job;
1475 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, (ScanJob*)job_data);
1477 g_assert (concurrent_collection_in_progress);
1478 SGEN_TV_GETTIME (atv);
1479 major_collector.scan_card_table (CARDTABLE_SCAN_MOD_UNION, ctx, job_data->job_index, job_data->job_split_count, job_data->data);
1480 SGEN_TV_GETTIME (btv);
1481 time_major_scan_mod_union_blocks += SGEN_TV_ELAPSED (atv, btv);
1483 if (worker_data_untyped)
1484 ((WorkerData*)worker_data_untyped)->major_scan_time += SGEN_TV_ELAPSED (atv, btv);
1488 job_scan_los_mod_union_card_table (void *worker_data_untyped, SgenThreadPoolJob *job)
1490 SGEN_TV_DECLARE (atv);
1491 SGEN_TV_DECLARE (btv);
1492 ParallelScanJob *job_data = (ParallelScanJob*)job;
1493 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, (ScanJob*)job_data);
1495 g_assert (concurrent_collection_in_progress);
1496 SGEN_TV_GETTIME (atv);
1497 sgen_los_scan_card_table (CARDTABLE_SCAN_MOD_UNION, ctx, job_data->job_index, job_data->job_split_count);
1498 SGEN_TV_GETTIME (btv);
1499 time_major_scan_mod_union_los += SGEN_TV_ELAPSED (atv, btv);
1501 if (worker_data_untyped)
1502 ((WorkerData*)worker_data_untyped)->los_scan_time += SGEN_TV_ELAPSED (atv, btv);
1506 job_major_mod_union_preclean (void *worker_data_untyped, SgenThreadPoolJob *job)
1508 SGEN_TV_DECLARE (atv);
1509 SGEN_TV_DECLARE (btv);
1510 ParallelScanJob *job_data = (ParallelScanJob*)job;
1511 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, (ScanJob*)job_data);
1513 g_assert (concurrent_collection_in_progress);
1514 SGEN_TV_GETTIME (atv);
1515 major_collector.scan_card_table (CARDTABLE_SCAN_MOD_UNION_PRECLEAN, ctx, job_data->job_index, job_data->job_split_count, job_data->data);
1516 SGEN_TV_GETTIME (btv);
1518 g_assert (worker_data_untyped);
1519 ((WorkerData*)worker_data_untyped)->major_scan_time += SGEN_TV_ELAPSED (atv, btv);
1523 job_los_mod_union_preclean (void *worker_data_untyped, SgenThreadPoolJob *job)
1525 SGEN_TV_DECLARE (atv);
1526 SGEN_TV_DECLARE (btv);
1527 ParallelScanJob *job_data = (ParallelScanJob*)job;
1528 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, (ScanJob*)job_data);
1530 g_assert (concurrent_collection_in_progress);
1531 SGEN_TV_GETTIME (atv);
1532 sgen_los_scan_card_table (CARDTABLE_SCAN_MOD_UNION_PRECLEAN, ctx, job_data->job_index, job_data->job_split_count);
1533 SGEN_TV_GETTIME (btv);
1535 g_assert (worker_data_untyped);
1536 ((WorkerData*)worker_data_untyped)->los_scan_time += SGEN_TV_ELAPSED (atv, btv);
1540 job_scan_last_pinned (void *worker_data_untyped, SgenThreadPoolJob *job)
1542 ScanJob *job_data = (ScanJob*)job;
1543 ScanCopyContext ctx = scan_copy_context_for_scan_job (worker_data_untyped, job_data);
1545 g_assert (concurrent_collection_in_progress);
1547 sgen_scan_pin_queue_objects (ctx);
1551 workers_finish_callback (void)
1553 ParallelScanJob *psj;
1555 size_t num_major_sections = major_collector.get_num_major_sections ();
1556 int split_count = sgen_workers_get_job_split_count (GENERATION_OLD);
1558 /* Mod union preclean jobs */
1559 for (i = 0; i < split_count; i++) {
1560 psj = (ParallelScanJob*)sgen_thread_pool_job_alloc ("preclean major mod union cardtable", job_major_mod_union_preclean, sizeof (ParallelScanJob));
1561 psj->scan_job.gc_thread_gray_queue = NULL;
1563 psj->job_split_count = split_count;
1564 psj->data = num_major_sections / split_count;
1565 sgen_workers_enqueue_job (GENERATION_OLD, &psj->scan_job.job, TRUE);
1568 for (i = 0; i < split_count; i++) {
1569 psj = (ParallelScanJob*)sgen_thread_pool_job_alloc ("preclean los mod union cardtable", job_los_mod_union_preclean, sizeof (ParallelScanJob));
1570 psj->scan_job.gc_thread_gray_queue = NULL;
1572 psj->job_split_count = split_count;
1573 sgen_workers_enqueue_job (GENERATION_OLD, &psj->scan_job.job, TRUE);
1576 sj = (ScanJob*)sgen_thread_pool_job_alloc ("scan last pinned", job_scan_last_pinned, sizeof (ScanJob));
1577 sj->gc_thread_gray_queue = NULL;
1578 sgen_workers_enqueue_job (GENERATION_OLD, &sj->job, TRUE);
1582 init_gray_queue (SgenGrayQueue *gc_thread_gray_queue)
1584 sgen_gray_object_queue_init (gc_thread_gray_queue, NULL, TRUE);
1588 enqueue_scan_remembered_set_jobs (SgenGrayQueue *gc_thread_gray_queue, SgenObjectOperations *ops, gboolean enqueue)
1590 int i, split_count = sgen_workers_get_job_split_count (GENERATION_NURSERY);
1591 size_t num_major_sections = major_collector.get_num_major_sections ();
1594 sj = (ScanJob*)sgen_thread_pool_job_alloc ("scan wbroots", job_scan_wbroots, sizeof (ScanJob));
1596 sj->gc_thread_gray_queue = gc_thread_gray_queue;
1597 sgen_workers_enqueue_job (GENERATION_NURSERY, &sj->job, enqueue);
1599 for (i = 0; i < split_count; i++) {
1600 ParallelScanJob *psj;
1602 psj = (ParallelScanJob*)sgen_thread_pool_job_alloc ("scan major remsets", job_scan_major_card_table, sizeof (ParallelScanJob));
1603 psj->scan_job.ops = ops;
1604 psj->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
1606 psj->job_split_count = split_count;
1607 psj->data = num_major_sections / split_count;
1608 sgen_workers_enqueue_job (GENERATION_NURSERY, &psj->scan_job.job, enqueue);
1610 psj = (ParallelScanJob*)sgen_thread_pool_job_alloc ("scan LOS remsets", job_scan_los_card_table, sizeof (ParallelScanJob));
1611 psj->scan_job.ops = ops;
1612 psj->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
1614 psj->job_split_count = split_count;
1615 sgen_workers_enqueue_job (GENERATION_NURSERY, &psj->scan_job.job, enqueue);
1620 enqueue_scan_from_roots_jobs (SgenGrayQueue *gc_thread_gray_queue, char *heap_start, char *heap_end, SgenObjectOperations *ops, gboolean enqueue)
1622 ScanFromRegisteredRootsJob *scrrj;
1623 ScanThreadDataJob *stdj;
1624 ScanFinalizerEntriesJob *sfej;
1626 /* registered roots, this includes static fields */
1628 scrrj = (ScanFromRegisteredRootsJob*)sgen_thread_pool_job_alloc ("scan from registered roots normal", job_scan_from_registered_roots, sizeof (ScanFromRegisteredRootsJob));
1629 scrrj->scan_job.ops = ops;
1630 scrrj->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
1631 scrrj->heap_start = heap_start;
1632 scrrj->heap_end = heap_end;
1633 scrrj->root_type = ROOT_TYPE_NORMAL;
1634 sgen_workers_enqueue_job (current_collection_generation, &scrrj->scan_job.job, enqueue);
1636 if (current_collection_generation == GENERATION_OLD) {
1637 /* During minors we scan the cardtable for these roots instead */
1638 scrrj = (ScanFromRegisteredRootsJob*)sgen_thread_pool_job_alloc ("scan from registered roots wbarrier", job_scan_from_registered_roots, sizeof (ScanFromRegisteredRootsJob));
1639 scrrj->scan_job.ops = ops;
1640 scrrj->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
1641 scrrj->heap_start = heap_start;
1642 scrrj->heap_end = heap_end;
1643 scrrj->root_type = ROOT_TYPE_WBARRIER;
1644 sgen_workers_enqueue_job (current_collection_generation, &scrrj->scan_job.job, enqueue);
1649 stdj = (ScanThreadDataJob*)sgen_thread_pool_job_alloc ("scan thread data", job_scan_thread_data, sizeof (ScanThreadDataJob));
1650 stdj->scan_job.ops = ops;
1651 stdj->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
1652 stdj->heap_start = heap_start;
1653 stdj->heap_end = heap_end;
1654 sgen_workers_enqueue_job (current_collection_generation, &stdj->scan_job.job, enqueue);
1656 /* Scan the list of objects ready for finalization. */
1658 sfej = (ScanFinalizerEntriesJob*)sgen_thread_pool_job_alloc ("scan finalizer entries", job_scan_finalizer_entries, sizeof (ScanFinalizerEntriesJob));
1659 sfej->scan_job.ops = ops;
1660 sfej->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
1661 sfej->queue = &fin_ready_queue;
1662 sgen_workers_enqueue_job (current_collection_generation, &sfej->scan_job.job, enqueue);
1664 sfej = (ScanFinalizerEntriesJob*)sgen_thread_pool_job_alloc ("scan critical finalizer entries", job_scan_finalizer_entries, sizeof (ScanFinalizerEntriesJob));
1665 sfej->scan_job.ops = ops;
1666 sfej->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
1667 sfej->queue = &critical_fin_queue;
1668 sgen_workers_enqueue_job (current_collection_generation, &sfej->scan_job.job, enqueue);
1672 * Perform a nursery collection.
1674 * Return whether any objects were late-pinned due to being out of memory.
1677 collect_nursery (const char *reason, gboolean is_overflow, SgenGrayQueue *unpin_queue)
1679 gboolean needs_major, is_parallel = FALSE;
1680 mword fragment_total;
1681 SgenGrayQueue gc_thread_gray_queue;
1682 SgenObjectOperations *object_ops_nopar, *object_ops_par = NULL;
1683 ScanCopyContext ctx;
1686 SGEN_TV_DECLARE (last_minor_collection_start_tv);
1687 SGEN_TV_DECLARE (last_minor_collection_end_tv);
1688 guint64 major_scan_start = time_minor_scan_major_blocks;
1689 guint64 los_scan_start = time_minor_scan_los;
1690 guint64 finish_gray_start = time_minor_finish_gray_stack;
1692 if (disable_minor_collections)
1695 TV_GETTIME (last_minor_collection_start_tv);
1696 atv = last_minor_collection_start_tv;
1698 binary_protocol_collection_begin (InterlockedRead (&gc_stats.minor_gc_count), GENERATION_NURSERY);
1700 object_ops_nopar = sgen_concurrent_collection_in_progress ()
1701 ? &sgen_minor_collector.serial_ops_with_concurrent_major
1702 : &sgen_minor_collector.serial_ops;
1703 if (sgen_minor_collector.is_parallel && sgen_nursery_size >= SGEN_PARALLEL_MINOR_MIN_NURSERY_SIZE) {
1704 object_ops_par = sgen_concurrent_collection_in_progress ()
1705 ? &sgen_minor_collector.parallel_ops_with_concurrent_major
1706 : &sgen_minor_collector.parallel_ops;
1710 if (do_verify_nursery || do_dump_nursery_content)
1711 sgen_debug_verify_nursery (do_dump_nursery_content);
1713 current_collection_generation = GENERATION_NURSERY;
1715 SGEN_ASSERT (0, !sgen_collection_is_concurrent (), "Why is the nursery collection concurrent?");
1717 reset_pinned_from_failed_allocation ();
1719 check_scan_starts ();
1721 sgen_nursery_alloc_prepare_for_minor ();
1726 SGEN_LOG (1, "Start nursery collection %" G_GINT32_FORMAT " %p-%p, size: %d", InterlockedRead (&gc_stats.minor_gc_count), nursery_section->data, nursery_section->end_data, (int)(nursery_section->end_data - nursery_section->data));
1728 /* world must be stopped already */
1730 time_minor_pre_collection_fragment_clear += TV_ELAPSED (atv, btv);
1732 sgen_client_pre_collection_checks ();
1734 major_collector.start_nursery_collection ();
1736 sgen_memgov_minor_collection_start ();
1738 init_gray_queue (&gc_thread_gray_queue);
1739 ctx = CONTEXT_FROM_OBJECT_OPERATIONS (object_ops_nopar, &gc_thread_gray_queue);
1741 InterlockedIncrement (&gc_stats.minor_gc_count);
1743 sgen_process_fin_stage_entries ();
1745 /* pin from pinned handles */
1746 sgen_init_pinning ();
1747 if (concurrent_collection_in_progress)
1748 sgen_init_pinning_for_conc ();
1749 sgen_client_binary_protocol_mark_start (GENERATION_NURSERY);
1750 pin_from_roots (nursery_section->data, nursery_section->end_data, ctx);
1751 /* pin cemented objects */
1752 sgen_pin_cemented_objects ();
1753 /* identify pinned objects */
1754 sgen_optimize_pin_queue ();
1755 sgen_pinning_setup_section (nursery_section);
1757 pin_objects_in_nursery (FALSE, ctx);
1758 sgen_pinning_trim_queue_to_section (nursery_section);
1759 if (concurrent_collection_in_progress)
1760 sgen_finish_pinning_for_conc ();
1762 if (remset_consistency_checks)
1763 sgen_check_remset_consistency ();
1765 if (whole_heap_check_before_collection) {
1766 sgen_clear_nursery_fragments ();
1767 sgen_check_whole_heap (FALSE);
1771 time_minor_pinning += TV_ELAPSED (btv, atv);
1772 SGEN_LOG (2, "Finding pinned pointers: %zd in %lld usecs", sgen_get_pinned_count (), (long long)TV_ELAPSED (btv, atv));
1773 SGEN_LOG (4, "Start scan with %zd pinned objects", sgen_get_pinned_count ());
1775 remset.start_scan_remsets ();
1777 enqueue_scan_remembered_set_jobs (&gc_thread_gray_queue, is_parallel ? NULL : object_ops_nopar, is_parallel);
1779 /* we don't have complete write barrier yet, so we scan all the old generation sections */
1781 time_minor_scan_remsets += TV_ELAPSED (atv, btv);
1782 SGEN_LOG (2, "Old generation scan: %lld usecs", (long long)TV_ELAPSED (atv, btv));
1784 sgen_pin_stats_report ();
1786 /* FIXME: Why do we do this at this specific, seemingly random, point? */
1787 sgen_client_collecting_minor (&fin_ready_queue, &critical_fin_queue);
1790 time_minor_scan_pinned += TV_ELAPSED (btv, atv);
1792 enqueue_scan_from_roots_jobs (&gc_thread_gray_queue, nursery_section->data, nursery_section->end_data, is_parallel ? NULL : object_ops_nopar, is_parallel);
1795 gray_queue_redirect (&gc_thread_gray_queue);
1796 sgen_workers_start_all_workers (GENERATION_NURSERY, object_ops_nopar, object_ops_par, NULL);
1797 sgen_workers_join (GENERATION_NURSERY);
1801 time_minor_scan_roots += TV_ELAPSED (atv, btv);
1803 finish_gray_stack (GENERATION_NURSERY, ctx);
1806 time_minor_finish_gray_stack += TV_ELAPSED (btv, atv);
1807 sgen_client_binary_protocol_mark_end (GENERATION_NURSERY);
1809 if (objects_pinned) {
1810 sgen_optimize_pin_queue ();
1811 sgen_pinning_setup_section (nursery_section);
1815 * This is the latest point at which we can do this check, because
1816 * sgen_build_nursery_fragments() unpins nursery objects again.
1818 if (remset_consistency_checks)
1819 sgen_check_remset_consistency ();
1822 if (sgen_max_pause_time) {
1826 duration = (int)(TV_ELAPSED (last_minor_collection_start_tv, btv) / 10000);
1827 if (duration > (sgen_max_pause_time * sgen_max_pause_margin))
1828 sgen_resize_nursery (TRUE);
1830 sgen_resize_nursery (FALSE);
1832 sgen_resize_nursery (FALSE);
1835 /* walk the pin_queue, build up the fragment list of free memory, unmark
1836 * pinned objects as we go, memzero() the empty fragments so they are ready for the
1839 sgen_client_binary_protocol_reclaim_start (GENERATION_NURSERY);
1840 fragment_total = sgen_build_nursery_fragments (nursery_section, unpin_queue);
1841 if (!fragment_total)
1844 /* Clear TLABs for all threads */
1845 sgen_clear_tlabs ();
1847 sgen_client_binary_protocol_reclaim_end (GENERATION_NURSERY);
1849 time_minor_fragment_creation += TV_ELAPSED (atv, btv);
1850 SGEN_LOG (2, "Fragment creation: %lld usecs, %lu bytes available", (long long)TV_ELAPSED (atv, btv), (unsigned long)fragment_total);
1852 if (remset_consistency_checks)
1853 sgen_check_major_refs ();
1855 major_collector.finish_nursery_collection ();
1857 TV_GETTIME (last_minor_collection_end_tv);
1858 InterlockedAdd64 (&gc_stats.minor_gc_time, TV_ELAPSED (last_minor_collection_start_tv, last_minor_collection_end_tv));
1860 sgen_debug_dump_heap ("minor", InterlockedRead (&gc_stats.minor_gc_count) - 1, NULL);
1862 /* prepare the pin queue for the next collection */
1863 sgen_finish_pinning ();
1864 if (sgen_have_pending_finalizers ()) {
1865 SGEN_LOG (4, "Finalizer-thread wakeup");
1866 sgen_client_finalize_notify ();
1868 sgen_pin_stats_reset ();
1869 /* clear cemented hash */
1870 sgen_cement_clear_below_threshold ();
1872 sgen_gray_object_queue_dispose (&gc_thread_gray_queue);
1874 check_scan_starts ();
1876 binary_protocol_flush_buffers (FALSE);
1878 sgen_memgov_minor_collection_end (reason, is_overflow);
1880 /*objects are late pinned because of lack of memory, so a major is a good call*/
1881 needs_major = objects_pinned > 0;
1882 current_collection_generation = -1;
1886 binary_protocol_collection_end_stats (0, 0, time_minor_finish_gray_stack - finish_gray_start);
1888 binary_protocol_collection_end_stats (
1889 time_minor_scan_major_blocks - major_scan_start,
1890 time_minor_scan_los - los_scan_start,
1891 time_minor_finish_gray_stack - finish_gray_start);
1893 binary_protocol_collection_end (InterlockedRead (&gc_stats.minor_gc_count) - 1, GENERATION_NURSERY, 0, 0);
1895 if (check_nursery_objects_pinned && !sgen_minor_collector.is_split)
1896 sgen_check_nursery_objects_pinned (unpin_queue != NULL);
1902 COPY_OR_MARK_FROM_ROOTS_SERIAL,
1903 COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT,
1904 COPY_OR_MARK_FROM_ROOTS_FINISH_CONCURRENT
1905 } CopyOrMarkFromRootsMode;
1908 major_copy_or_mark_from_roots (SgenGrayQueue *gc_thread_gray_queue, size_t *old_next_pin_slot, CopyOrMarkFromRootsMode mode, SgenObjectOperations *object_ops_nopar, SgenObjectOperations *object_ops_par)
1913 /* FIXME: only use these values for the precise scan
1914 * note that to_space pointers should be excluded anyway...
1916 char *heap_start = NULL;
1917 char *heap_end = (char*)-1;
1918 ScanCopyContext ctx = CONTEXT_FROM_OBJECT_OPERATIONS (object_ops_nopar, gc_thread_gray_queue);
1919 gboolean concurrent = mode != COPY_OR_MARK_FROM_ROOTS_SERIAL;
1921 SGEN_ASSERT (0, !!concurrent == !!concurrent_collection_in_progress, "We've been called with the wrong mode.");
1923 if (mode == COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT) {
1924 /*This cleans up unused fragments */
1925 sgen_nursery_allocator_prepare_for_pinning ();
1927 if (do_concurrent_checks)
1928 sgen_debug_check_nursery_is_clean ();
1930 /* The concurrent collector doesn't touch the nursery. */
1931 sgen_nursery_alloc_prepare_for_major ();
1936 /* Pinning depends on this */
1937 sgen_clear_nursery_fragments ();
1939 if (whole_heap_check_before_collection)
1940 sgen_check_whole_heap (TRUE);
1943 time_major_pre_collection_fragment_clear += TV_ELAPSED (atv, btv);
1947 sgen_client_pre_collection_checks ();
1949 if (mode != COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT) {
1950 /* Remsets are not useful for a major collection */
1951 remset.clear_cards ();
1954 sgen_process_fin_stage_entries ();
1957 sgen_init_pinning ();
1958 if (mode == COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT)
1959 sgen_init_pinning_for_conc ();
1960 SGEN_LOG (6, "Collecting pinned addresses");
1961 pin_from_roots ((void*)lowest_heap_address, (void*)highest_heap_address, ctx);
1962 if (mode == COPY_OR_MARK_FROM_ROOTS_FINISH_CONCURRENT) {
1963 /* Pin cemented objects that were forced */
1964 sgen_pin_cemented_objects ();
1966 sgen_optimize_pin_queue ();
1967 if (mode == COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT) {
1969 * Cemented objects that are in the pinned list will be marked. When
1970 * marking concurrently we won't mark mod-union cards for these objects.
1971 * Instead they will remain cemented until the next major collection,
1972 * when we will recheck if they are still pinned in the roots.
1974 sgen_cement_force_pinned ();
1977 sgen_client_collecting_major_1 ();
1980 * pin_queue now contains all candidate pointers, sorted and
1981 * uniqued. We must do two passes now to figure out which
1982 * objects are pinned.
1984 * The first is to find within the pin_queue the area for each
1985 * section. This requires that the pin_queue be sorted. We
1986 * also process the LOS objects and pinned chunks here.
1988 * The second, destructive, pass is to reduce the section
1989 * areas to pointers to the actually pinned objects.
1991 SGEN_LOG (6, "Pinning from sections");
1992 /* first pass for the sections */
1993 sgen_find_section_pin_queue_start_end (nursery_section);
1994 /* identify possible pointers to the insize of large objects */
1995 SGEN_LOG (6, "Pinning from large objects");
1996 for (bigobj = los_object_list; bigobj; bigobj = bigobj->next) {
1998 if (sgen_find_optimized_pin_queue_area ((char*)bigobj->data, (char*)bigobj->data + sgen_los_object_size (bigobj), &dummy, &dummy)) {
1999 binary_protocol_pin (bigobj->data, (gpointer)LOAD_VTABLE (bigobj->data), safe_object_get_size (bigobj->data));
2001 if (sgen_los_object_is_pinned (bigobj->data)) {
2002 SGEN_ASSERT (0, mode == COPY_OR_MARK_FROM_ROOTS_FINISH_CONCURRENT, "LOS objects can only be pinned here after concurrent marking.");
2005 sgen_los_pin_object (bigobj->data);
2006 if (SGEN_OBJECT_HAS_REFERENCES (bigobj->data))
2007 GRAY_OBJECT_ENQUEUE_SERIAL (gc_thread_gray_queue, bigobj->data, sgen_obj_get_descriptor ((GCObject*)bigobj->data));
2008 sgen_pin_stats_register_object (bigobj->data, GENERATION_OLD);
2009 SGEN_LOG (6, "Marked large object %p (%s) size: %lu from roots", bigobj->data,
2010 sgen_client_vtable_get_name (SGEN_LOAD_VTABLE (bigobj->data)),
2011 (unsigned long)sgen_los_object_size (bigobj));
2013 sgen_client_pinned_los_object (bigobj->data);
2017 pin_objects_in_nursery (mode == COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT, ctx);
2018 if (check_nursery_objects_pinned && !sgen_minor_collector.is_split)
2019 sgen_check_nursery_objects_pinned (mode != COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT);
2021 major_collector.pin_objects (gc_thread_gray_queue);
2022 if (old_next_pin_slot)
2023 *old_next_pin_slot = sgen_get_pinned_count ();
2026 time_major_pinning += TV_ELAPSED (atv, btv);
2027 SGEN_LOG (2, "Finding pinned pointers: %zd in %lld usecs", sgen_get_pinned_count (), (long long)TV_ELAPSED (atv, btv));
2028 SGEN_LOG (4, "Start scan with %zd pinned objects", sgen_get_pinned_count ());
2030 if (mode == COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT)
2031 sgen_finish_pinning_for_conc ();
2033 major_collector.init_to_space ();
2035 SGEN_ASSERT (0, sgen_workers_all_done (), "Why are the workers not done when we start or finish a major collection?");
2036 if (mode == COPY_OR_MARK_FROM_ROOTS_FINISH_CONCURRENT) {
2037 if (object_ops_par != NULL)
2038 sgen_workers_set_num_active_workers (GENERATION_OLD, 0);
2039 if (object_ops_par == NULL && sgen_workers_have_idle_work (GENERATION_OLD)) {
2041 * We force the finish of the worker with the new object ops context
2042 * which can also do copying. We need to have finished pinning. On the
2043 * parallel collector, there is no need to drain the private queues
2044 * here, since we can do it as part of the finishing work, achieving
2045 * better work distribution.
2047 sgen_workers_start_all_workers (GENERATION_OLD, object_ops_nopar, object_ops_par, NULL);
2049 sgen_workers_join (GENERATION_OLD);
2053 #ifdef SGEN_DEBUG_INTERNAL_ALLOC
2054 main_gc_thread = mono_native_thread_self ();
2057 sgen_client_collecting_major_2 ();
2060 time_major_scan_pinned += TV_ELAPSED (btv, atv);
2062 sgen_client_collecting_major_3 (&fin_ready_queue, &critical_fin_queue);
2064 enqueue_scan_from_roots_jobs (gc_thread_gray_queue, heap_start, heap_end, object_ops_nopar, FALSE);
2067 time_major_scan_roots += TV_ELAPSED (atv, btv);
2070 * We start the concurrent worker after pinning and after we scanned the roots
2071 * in order to make sure that the worker does not finish before handling all
2074 if (mode == COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT) {
2075 sgen_workers_set_num_active_workers (GENERATION_OLD, 1);
2076 gray_queue_redirect (gc_thread_gray_queue);
2077 if (precleaning_enabled) {
2078 sgen_workers_start_all_workers (GENERATION_OLD, object_ops_nopar, object_ops_par, workers_finish_callback);
2080 sgen_workers_start_all_workers (GENERATION_OLD, object_ops_nopar, object_ops_par, NULL);
2084 if (mode == COPY_OR_MARK_FROM_ROOTS_FINISH_CONCURRENT) {
2085 int i, split_count = sgen_workers_get_job_split_count (GENERATION_OLD);
2086 size_t num_major_sections = major_collector.get_num_major_sections ();
2087 gboolean parallel = object_ops_par != NULL;
2089 /* If we're not parallel we finish the collection on the gc thread */
2091 gray_queue_redirect (gc_thread_gray_queue);
2093 /* Mod union card table */
2094 for (i = 0; i < split_count; i++) {
2095 ParallelScanJob *psj;
2097 psj = (ParallelScanJob*)sgen_thread_pool_job_alloc ("scan mod union cardtable", job_scan_major_mod_union_card_table, sizeof (ParallelScanJob));
2098 psj->scan_job.ops = parallel ? NULL : object_ops_nopar;
2099 psj->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
2101 psj->job_split_count = split_count;
2102 psj->data = num_major_sections / split_count;
2103 sgen_workers_enqueue_job (GENERATION_OLD, &psj->scan_job.job, parallel);
2105 psj = (ParallelScanJob*)sgen_thread_pool_job_alloc ("scan LOS mod union cardtable", job_scan_los_mod_union_card_table, sizeof (ParallelScanJob));
2106 psj->scan_job.ops = parallel ? NULL : object_ops_nopar;
2107 psj->scan_job.gc_thread_gray_queue = gc_thread_gray_queue;
2109 psj->job_split_count = split_count;
2110 sgen_workers_enqueue_job (GENERATION_OLD, &psj->scan_job.job, parallel);
2115 * If we enqueue a job while workers are running we need to sgen_workers_ensure_awake
2116 * in order to make sure that we are running the idle func and draining all worker
2117 * gray queues. The operation of starting workers implies this, so we start them after
2118 * in order to avoid doing this operation twice. The workers will drain the main gray
2119 * stack that contained roots and pinned objects and also scan the mod union card
2122 sgen_workers_start_all_workers (GENERATION_OLD, object_ops_nopar, object_ops_par, NULL);
2123 sgen_workers_join (GENERATION_OLD);
2127 sgen_pin_stats_report ();
2129 if (mode == COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT) {
2130 sgen_finish_pinning ();
2132 sgen_pin_stats_reset ();
2134 if (do_concurrent_checks)
2135 sgen_debug_check_nursery_is_clean ();
2140 major_start_collection (SgenGrayQueue *gc_thread_gray_queue, const char *reason, gboolean concurrent, size_t *old_next_pin_slot)
2142 SgenObjectOperations *object_ops_nopar, *object_ops_par = NULL;
2144 binary_protocol_collection_begin (InterlockedRead (&gc_stats.major_gc_count), GENERATION_OLD);
2146 current_collection_generation = GENERATION_OLD;
2148 sgen_workers_assert_gray_queue_is_empty (GENERATION_OLD);
2151 sgen_cement_reset ();
2154 g_assert (major_collector.is_concurrent);
2155 concurrent_collection_in_progress = TRUE;
2157 object_ops_nopar = &major_collector.major_ops_concurrent_start;
2158 if (major_collector.is_parallel)
2159 object_ops_par = &major_collector.major_ops_conc_par_start;
2162 object_ops_nopar = &major_collector.major_ops_serial;
2165 reset_pinned_from_failed_allocation ();
2167 sgen_memgov_major_collection_start (concurrent, reason);
2169 //count_ref_nonref_objs ();
2170 //consistency_check ();
2172 check_scan_starts ();
2175 SGEN_LOG (1, "Start major collection %" G_GINT32_FORMAT, InterlockedRead (&gc_stats.major_gc_count));
2176 InterlockedIncrement (&gc_stats.major_gc_count);
2178 if (major_collector.start_major_collection)
2179 major_collector.start_major_collection ();
2181 major_copy_or_mark_from_roots (gc_thread_gray_queue, old_next_pin_slot, concurrent ? COPY_OR_MARK_FROM_ROOTS_START_CONCURRENT : COPY_OR_MARK_FROM_ROOTS_SERIAL, object_ops_nopar, object_ops_par);
2185 major_finish_collection (SgenGrayQueue *gc_thread_gray_queue, const char *reason, gboolean is_overflow, size_t old_next_pin_slot, gboolean forced)
2187 ScannedObjectCounts counts;
2188 SgenObjectOperations *object_ops_nopar;
2189 mword fragment_total;
2192 guint64 major_scan_start = time_major_scan_mod_union_blocks;
2193 guint64 los_scan_start = time_major_scan_mod_union_los;
2194 guint64 finish_gray_start = time_major_finish_gray_stack;
2196 if (concurrent_collection_in_progress) {
2197 SgenObjectOperations *object_ops_par = NULL;
2199 object_ops_nopar = &major_collector.major_ops_concurrent_finish;
2200 if (major_collector.is_parallel)
2201 object_ops_par = &major_collector.major_ops_conc_par_finish;
2203 major_copy_or_mark_from_roots (gc_thread_gray_queue, NULL, COPY_OR_MARK_FROM_ROOTS_FINISH_CONCURRENT, object_ops_nopar, object_ops_par);
2205 #ifdef SGEN_DEBUG_INTERNAL_ALLOC
2206 main_gc_thread = NULL;
2209 object_ops_nopar = &major_collector.major_ops_serial;
2212 sgen_workers_assert_gray_queue_is_empty (GENERATION_OLD);
2215 finish_gray_stack (GENERATION_OLD, CONTEXT_FROM_OBJECT_OPERATIONS (object_ops_nopar, gc_thread_gray_queue));
2217 time_major_finish_gray_stack += TV_ELAPSED (btv, atv);
2219 SGEN_ASSERT (0, sgen_workers_all_done (), "Can't have workers working after joining");
2221 if (objects_pinned) {
2222 g_assert (!concurrent_collection_in_progress);
2225 * This is slow, but we just OOM'd.
2227 * See comment at `sgen_pin_queue_clear_discarded_entries` for how the pin
2228 * queue is laid out at this point.
2230 sgen_pin_queue_clear_discarded_entries (nursery_section, old_next_pin_slot);
2232 * We need to reestablish all pinned nursery objects in the pin queue
2233 * because they're needed for fragment creation. Unpinning happens by
2234 * walking the whole queue, so it's not necessary to reestablish where major
2235 * heap block pins are - all we care is that they're still in there
2238 sgen_optimize_pin_queue ();
2239 sgen_find_section_pin_queue_start_end (nursery_section);
2243 reset_heap_boundaries ();
2244 sgen_update_heap_boundaries ((mword)sgen_get_nursery_start (), (mword)sgen_get_nursery_end ());
2246 /* walk the pin_queue, build up the fragment list of free memory, unmark
2247 * pinned objects as we go, memzero() the empty fragments so they are ready for the
2250 fragment_total = sgen_build_nursery_fragments (nursery_section, NULL);
2251 if (!fragment_total)
2253 SGEN_LOG (4, "Free space in nursery after major %ld", (long)fragment_total);
2255 if (do_concurrent_checks && concurrent_collection_in_progress)
2256 sgen_debug_check_nursery_is_clean ();
2258 /* prepare the pin queue for the next collection */
2259 sgen_finish_pinning ();
2261 /* Clear TLABs for all threads */
2262 sgen_clear_tlabs ();
2264 sgen_pin_stats_reset ();
2266 sgen_cement_clear_below_threshold ();
2268 if (check_mark_bits_after_major_collection)
2269 sgen_check_heap_marked (concurrent_collection_in_progress);
2272 time_major_fragment_creation += TV_ELAPSED (atv, btv);
2274 binary_protocol_sweep_begin (GENERATION_OLD, !major_collector.sweeps_lazily);
2275 sgen_memgov_major_pre_sweep ();
2278 time_major_free_bigobjs += TV_ELAPSED (btv, atv);
2283 time_major_los_sweep += TV_ELAPSED (atv, btv);
2285 major_collector.sweep ();
2287 binary_protocol_sweep_end (GENERATION_OLD, !major_collector.sweeps_lazily);
2290 time_major_sweep += TV_ELAPSED (btv, atv);
2292 sgen_debug_dump_heap ("major", InterlockedRead (&gc_stats.major_gc_count) - 1, reason);
2294 if (sgen_have_pending_finalizers ()) {
2295 SGEN_LOG (4, "Finalizer-thread wakeup");
2296 sgen_client_finalize_notify ();
2299 sgen_memgov_major_collection_end (forced, concurrent_collection_in_progress, reason, is_overflow);
2300 current_collection_generation = -1;
2302 memset (&counts, 0, sizeof (ScannedObjectCounts));
2303 major_collector.finish_major_collection (&counts);
2305 sgen_workers_assert_gray_queue_is_empty (GENERATION_OLD);
2307 SGEN_ASSERT (0, sgen_workers_all_done (), "Can't have workers working after major collection has finished");
2308 if (concurrent_collection_in_progress)
2309 concurrent_collection_in_progress = FALSE;
2311 check_scan_starts ();
2313 binary_protocol_flush_buffers (FALSE);
2315 //consistency_check ();
2316 if (major_collector.is_parallel)
2317 binary_protocol_collection_end_stats (0, 0, time_major_finish_gray_stack - finish_gray_start);
2319 binary_protocol_collection_end_stats (
2320 time_major_scan_mod_union_blocks - major_scan_start,
2321 time_major_scan_mod_union_los - los_scan_start,
2322 time_major_finish_gray_stack - finish_gray_start);
2324 binary_protocol_collection_end (InterlockedRead (&gc_stats.major_gc_count) - 1, GENERATION_OLD, counts.num_scanned_objects, counts.num_unique_scanned_objects);
2328 major_do_collection (const char *reason, gboolean is_overflow, gboolean forced)
2330 TV_DECLARE (time_start);
2331 TV_DECLARE (time_end);
2332 size_t old_next_pin_slot;
2333 SgenGrayQueue gc_thread_gray_queue;
2335 if (disable_major_collections)
2338 if (major_collector.get_and_reset_num_major_objects_marked) {
2339 long long num_marked = major_collector.get_and_reset_num_major_objects_marked ();
2340 g_assert (!num_marked);
2343 /* world must be stopped already */
2344 TV_GETTIME (time_start);
2346 init_gray_queue (&gc_thread_gray_queue);
2347 major_start_collection (&gc_thread_gray_queue, reason, FALSE, &old_next_pin_slot);
2348 major_finish_collection (&gc_thread_gray_queue, reason, is_overflow, old_next_pin_slot, forced);
2349 sgen_gray_object_queue_dispose (&gc_thread_gray_queue);
2351 TV_GETTIME (time_end);
2352 InterlockedAdd64 (&gc_stats.major_gc_time, TV_ELAPSED (time_start, time_end));
2354 /* FIXME: also report this to the user, preferably in gc-end. */
2355 if (major_collector.get_and_reset_num_major_objects_marked)
2356 major_collector.get_and_reset_num_major_objects_marked ();
2358 return bytes_pinned_from_failed_allocation > 0;
2362 major_start_concurrent_collection (const char *reason)
2364 TV_DECLARE (time_start);
2365 TV_DECLARE (time_end);
2366 long long num_objects_marked;
2367 SgenGrayQueue gc_thread_gray_queue;
2369 if (disable_major_collections)
2372 TV_GETTIME (time_start);
2373 SGEN_TV_GETTIME (time_major_conc_collection_start);
2375 num_objects_marked = major_collector.get_and_reset_num_major_objects_marked ();
2376 g_assert (num_objects_marked == 0);
2378 binary_protocol_concurrent_start ();
2380 init_gray_queue (&gc_thread_gray_queue);
2381 // FIXME: store reason and pass it when finishing
2382 major_start_collection (&gc_thread_gray_queue, reason, TRUE, NULL);
2383 sgen_gray_object_queue_dispose (&gc_thread_gray_queue);
2385 num_objects_marked = major_collector.get_and_reset_num_major_objects_marked ();
2387 TV_GETTIME (time_end);
2388 InterlockedAdd64 (&gc_stats.major_gc_time, TV_ELAPSED (time_start, time_end));
2390 current_collection_generation = -1;
2394 * Returns whether the major collection has finished.
2397 major_should_finish_concurrent_collection (void)
2399 return sgen_workers_all_done ();
2403 major_update_concurrent_collection (void)
2405 TV_DECLARE (total_start);
2406 TV_DECLARE (total_end);
2408 TV_GETTIME (total_start);
2410 binary_protocol_concurrent_update ();
2412 major_collector.update_cardtable_mod_union ();
2413 sgen_los_update_cardtable_mod_union ();
2415 TV_GETTIME (total_end);
2416 InterlockedAdd64 (&gc_stats.major_gc_time, TV_ELAPSED (total_start, total_end));
2420 major_finish_concurrent_collection (gboolean forced)
2422 SgenGrayQueue gc_thread_gray_queue;
2423 TV_DECLARE (total_start);
2424 TV_DECLARE (total_end);
2426 TV_GETTIME (total_start);
2428 binary_protocol_concurrent_finish ();
2431 * We need to stop all workers since we're updating the cardtable below.
2432 * The workers will be resumed with a finishing pause context to avoid
2433 * additional cardtable and object scanning.
2435 sgen_workers_stop_all_workers (GENERATION_OLD);
2437 SGEN_TV_GETTIME (time_major_conc_collection_end);
2438 InterlockedAdd64 (&gc_stats.major_gc_time_concurrent, SGEN_TV_ELAPSED (time_major_conc_collection_start, time_major_conc_collection_end));
2440 major_collector.update_cardtable_mod_union ();
2441 sgen_los_update_cardtable_mod_union ();
2443 if (mod_union_consistency_check)
2444 sgen_check_mod_union_consistency ();
2446 current_collection_generation = GENERATION_OLD;
2447 sgen_cement_reset ();
2448 init_gray_queue (&gc_thread_gray_queue);
2449 major_finish_collection (&gc_thread_gray_queue, "finishing", FALSE, -1, forced);
2450 sgen_gray_object_queue_dispose (&gc_thread_gray_queue);
2452 TV_GETTIME (total_end);
2453 InterlockedAdd64 (&gc_stats.major_gc_time, TV_ELAPSED (total_start, total_end));
2455 current_collection_generation = -1;
2459 * Ensure an allocation request for @size will succeed by freeing enough memory.
2461 * LOCKING: The GC lock MUST be held.
2464 sgen_ensure_free_space (size_t size, int generation)
2466 int generation_to_collect = -1;
2467 const char *reason = NULL;
2469 if (generation == GENERATION_OLD) {
2470 if (sgen_need_major_collection (size)) {
2471 reason = "LOS overflow";
2472 generation_to_collect = GENERATION_OLD;
2475 if (degraded_mode) {
2476 if (sgen_need_major_collection (size)) {
2477 reason = "Degraded mode overflow";
2478 generation_to_collect = GENERATION_OLD;
2480 } else if (sgen_need_major_collection (size)) {
2481 reason = concurrent_collection_in_progress ? "Forced finish concurrent collection" : "Minor allowance";
2482 generation_to_collect = GENERATION_OLD;
2484 generation_to_collect = GENERATION_NURSERY;
2485 reason = "Nursery full";
2489 if (generation_to_collect == -1) {
2490 if (concurrent_collection_in_progress && sgen_workers_all_done ()) {
2491 generation_to_collect = GENERATION_OLD;
2492 reason = "Finish concurrent collection";
2496 if (generation_to_collect == -1)
2498 sgen_perform_collection (size, generation_to_collect, reason, FALSE, TRUE);
2502 * LOCKING: Assumes the GC lock is held.
2505 sgen_perform_collection (size_t requested_size, int generation_to_collect, const char *reason, gboolean wait_to_finish, gboolean stw)
2507 TV_DECLARE (gc_total_start);
2508 TV_DECLARE (gc_total_end);
2509 int overflow_generation_to_collect = -1;
2510 int oldest_generation_collected = generation_to_collect;
2511 const char *overflow_reason = NULL;
2512 gboolean finish_concurrent = concurrent_collection_in_progress && (major_should_finish_concurrent_collection () || generation_to_collect == GENERATION_OLD);
2514 binary_protocol_collection_requested (generation_to_collect, requested_size, wait_to_finish ? 1 : 0);
2516 SGEN_ASSERT (0, generation_to_collect == GENERATION_NURSERY || generation_to_collect == GENERATION_OLD, "What generation is this?");
2519 sgen_stop_world (generation_to_collect);
2521 SGEN_ASSERT (0, sgen_is_world_stopped (), "We can only collect if the world is stopped");
2524 TV_GETTIME (gc_total_start);
2526 // FIXME: extract overflow reason
2527 // FIXME: minor overflow for concurrent case
2528 if (generation_to_collect == GENERATION_NURSERY && !finish_concurrent) {
2529 if (concurrent_collection_in_progress)
2530 major_update_concurrent_collection ();
2532 if (collect_nursery (reason, FALSE, NULL) && !concurrent_collection_in_progress) {
2533 overflow_generation_to_collect = GENERATION_OLD;
2534 overflow_reason = "Minor overflow";
2536 } else if (finish_concurrent) {
2537 major_finish_concurrent_collection (wait_to_finish);
2538 oldest_generation_collected = GENERATION_OLD;
2540 SGEN_ASSERT (0, generation_to_collect == GENERATION_OLD, "We should have handled nursery collections above");
2541 if (major_collector.is_concurrent && !wait_to_finish) {
2542 collect_nursery ("Concurrent start", FALSE, NULL);
2543 major_start_concurrent_collection (reason);
2544 oldest_generation_collected = GENERATION_NURSERY;
2545 } else if (major_do_collection (reason, FALSE, wait_to_finish)) {
2546 overflow_generation_to_collect = GENERATION_NURSERY;
2547 overflow_reason = "Excessive pinning";
2551 if (overflow_generation_to_collect != -1) {
2552 SGEN_ASSERT (0, !concurrent_collection_in_progress, "We don't yet support overflow collections with the concurrent collector");
2555 * We need to do an overflow collection, either because we ran out of memory
2556 * or the nursery is fully pinned.
2559 if (overflow_generation_to_collect == GENERATION_NURSERY)
2560 collect_nursery (overflow_reason, TRUE, NULL);
2562 major_do_collection (overflow_reason, TRUE, wait_to_finish);
2564 oldest_generation_collected = MAX (oldest_generation_collected, overflow_generation_to_collect);
2567 SGEN_LOG (2, "Heap size: %lu, LOS size: %lu", (unsigned long)sgen_gc_get_total_heap_allocation (), (unsigned long)los_memory_usage);
2569 /* this also sets the proper pointers for the next allocation */
2570 if (generation_to_collect == GENERATION_NURSERY && !sgen_can_alloc_size (requested_size)) {
2571 /* TypeBuilder and MonoMethod are killing mcs with fragmentation */
2572 SGEN_LOG (1, "nursery collection didn't find enough room for %zd alloc (%zd pinned)", requested_size, sgen_get_pinned_count ());
2573 sgen_dump_pin_queue ();
2577 TV_GETTIME (gc_total_end);
2578 time_max = MAX (time_max, TV_ELAPSED (gc_total_start, gc_total_end));
2581 sgen_restart_world (oldest_generation_collected);
2585 * ######################################################################
2586 * ######## Memory allocation from the OS
2587 * ######################################################################
2588 * This section of code deals with getting memory from the OS and
2589 * allocating memory for GC-internal data structures.
2590 * Internal memory can be handled with a freelist for small objects.
2596 G_GNUC_UNUSED static void
2597 report_internal_mem_usage (void)
2599 printf ("Internal memory usage:\n");
2600 sgen_report_internal_mem_usage ();
2601 printf ("Pinned memory usage:\n");
2602 major_collector.report_pinned_memory_usage ();
2606 * ######################################################################
2607 * ######## Finalization support
2608 * ######################################################################
2612 * If the object has been forwarded it means it's still referenced from a root.
2613 * If it is pinned it's still alive as well.
2614 * A LOS object is only alive if we have pinned it.
2615 * Return TRUE if @obj is ready to be finalized.
2617 static inline gboolean
2618 sgen_is_object_alive (GCObject *object)
2620 if (ptr_in_nursery (object))
2621 return sgen_nursery_is_object_alive (object);
2623 return sgen_major_is_object_alive (object);
2627 * This function returns true if @object is either alive and belongs to the
2628 * current collection - major collections are full heap, so old gen objects
2629 * are never alive during a minor collection.
2632 sgen_is_object_alive_and_on_current_collection (GCObject *object)
2634 if (ptr_in_nursery (object))
2635 return sgen_nursery_is_object_alive (object);
2637 if (current_collection_generation == GENERATION_NURSERY)
2640 return sgen_major_is_object_alive (object);
2645 sgen_gc_is_object_ready_for_finalization (GCObject *object)
2647 return !sgen_is_object_alive (object);
2651 sgen_queue_finalization_entry (GCObject *obj)
2653 gboolean critical = sgen_client_object_has_critical_finalizer (obj);
2655 sgen_pointer_queue_add (critical ? &critical_fin_queue : &fin_ready_queue, obj);
2657 sgen_client_object_queued_for_finalization (obj);
2661 sgen_object_is_live (GCObject *obj)
2663 return sgen_is_object_alive_and_on_current_collection (obj);
2667 * `System.GC.WaitForPendingFinalizers` first checks `sgen_have_pending_finalizers()` to
2668 * determine whether it can exit quickly. The latter must therefore only return FALSE if
2669 * all finalizers have really finished running.
2671 * `sgen_gc_invoke_finalizers()` first dequeues a finalizable object, and then finalizes it.
2672 * This means that just checking whether the queues are empty leaves the possibility that an
2673 * object might have been dequeued but not yet finalized. That's why we need the additional
2674 * flag `pending_unqueued_finalizer`.
2677 static volatile gboolean pending_unqueued_finalizer = FALSE;
2678 volatile gboolean sgen_suspend_finalizers = FALSE;
2681 sgen_set_suspend_finalizers (void)
2683 sgen_suspend_finalizers = TRUE;
2687 sgen_gc_invoke_finalizers (void)
2691 g_assert (!pending_unqueued_finalizer);
2693 /* FIXME: batch to reduce lock contention */
2694 while (sgen_have_pending_finalizers ()) {
2700 * We need to set `pending_unqueued_finalizer` before dequeing the
2701 * finalizable object.
2703 if (!sgen_pointer_queue_is_empty (&fin_ready_queue)) {
2704 pending_unqueued_finalizer = TRUE;
2705 mono_memory_write_barrier ();
2706 obj = (GCObject *)sgen_pointer_queue_pop (&fin_ready_queue);
2707 } else if (!sgen_pointer_queue_is_empty (&critical_fin_queue)) {
2708 pending_unqueued_finalizer = TRUE;
2709 mono_memory_write_barrier ();
2710 obj = (GCObject *)sgen_pointer_queue_pop (&critical_fin_queue);
2716 SGEN_LOG (7, "Finalizing object %p (%s)", obj, sgen_client_vtable_get_name (SGEN_LOAD_VTABLE (obj)));
2724 /* the object is on the stack so it is pinned */
2725 /*g_print ("Calling finalizer for object: %p (%s)\n", obj, sgen_client_object_safe_name (obj));*/
2726 sgen_client_run_finalize (obj);
2729 if (pending_unqueued_finalizer) {
2730 mono_memory_write_barrier ();
2731 pending_unqueued_finalizer = FALSE;
2738 sgen_have_pending_finalizers (void)
2740 if (sgen_suspend_finalizers)
2742 return pending_unqueued_finalizer || !sgen_pointer_queue_is_empty (&fin_ready_queue) || !sgen_pointer_queue_is_empty (&critical_fin_queue);
2746 * ######################################################################
2747 * ######## registered roots support
2748 * ######################################################################
2752 * We do not coalesce roots.
2755 sgen_register_root (char *start, size_t size, SgenDescriptor descr, int root_type, int source, const char *msg)
2757 RootRecord new_root;
2760 for (i = 0; i < ROOT_TYPE_NUM; ++i) {
2761 RootRecord *root = (RootRecord *)sgen_hash_table_lookup (&roots_hash [i], start);
2762 /* we allow changing the size and the descriptor (for thread statics etc) */
2764 size_t old_size = root->end_root - start;
2765 root->end_root = start + size;
2766 SGEN_ASSERT (0, !!root->root_desc == !!descr, "Can't change whether a root is precise or conservative.");
2767 SGEN_ASSERT (0, root->source == source, "Can't change a root's source identifier.");
2768 SGEN_ASSERT (0, !!root->msg == !!msg, "Can't change a root's message.");
2769 root->root_desc = descr;
2771 roots_size -= old_size;
2777 new_root.end_root = start + size;
2778 new_root.root_desc = descr;
2779 new_root.source = source;
2782 sgen_hash_table_replace (&roots_hash [root_type], start, &new_root, NULL);
2785 SGEN_LOG (3, "Added root for range: %p-%p, descr: %llx (%d/%d bytes)", start, new_root.end_root, (long long)descr, (int)size, (int)roots_size);
2792 sgen_deregister_root (char* addr)
2798 for (root_type = 0; root_type < ROOT_TYPE_NUM; ++root_type) {
2799 if (sgen_hash_table_remove (&roots_hash [root_type], addr, &root))
2800 roots_size -= (root.end_root - addr);
2806 sgen_wbroots_iterate_live_block_ranges (sgen_cardtable_block_callback cb)
2810 SGEN_HASH_TABLE_FOREACH (&roots_hash [ROOT_TYPE_WBARRIER], void **, start_root, RootRecord *, root) {
2811 cb ((mword)start_root, (mword)root->end_root - (mword)start_root);
2812 } SGEN_HASH_TABLE_FOREACH_END;
2815 /* Root equivalent of sgen_client_cardtable_scan_object */
2817 sgen_wbroot_scan_card_table (void** start_root, mword size, ScanCopyContext ctx)
2819 ScanPtrFieldFunc scan_field_func = ctx.ops->scan_ptr_field;
2820 guint8 *card_data = sgen_card_table_get_card_scan_address ((mword)start_root);
2821 guint8 *card_base = card_data;
2822 mword card_count = sgen_card_table_number_of_cards_in_range ((mword)start_root, size);
2823 guint8 *card_data_end = card_data + card_count;
2824 mword extra_idx = 0;
2825 char *obj_start = sgen_card_table_align_pointer (start_root);
2826 char *obj_end = (char*)start_root + size;
2827 #ifdef SGEN_HAVE_OVERLAPPING_CARDS
2828 guint8 *overflow_scan_end = NULL;
2831 #ifdef SGEN_HAVE_OVERLAPPING_CARDS
2832 /*Check for overflow and if so, setup to scan in two steps*/
2833 if (card_data_end >= SGEN_SHADOW_CARDTABLE_END) {
2834 overflow_scan_end = sgen_shadow_cardtable + (card_data_end - SGEN_SHADOW_CARDTABLE_END);
2835 card_data_end = SGEN_SHADOW_CARDTABLE_END;
2841 card_data = sgen_find_next_card (card_data, card_data_end);
2843 for (; card_data < card_data_end; card_data = sgen_find_next_card (card_data + 1, card_data_end)) {
2844 size_t idx = (card_data - card_base) + extra_idx;
2845 char *start = (char*)(obj_start + idx * CARD_SIZE_IN_BYTES);
2846 char *card_end = start + CARD_SIZE_IN_BYTES;
2847 char *elem = start, *first_elem = start;
2850 * Don't clean first and last card on 32bit systems since they
2851 * may also be part from other roots.
2853 if (card_data != card_base && card_data != (card_data_end - 1))
2854 sgen_card_table_prepare_card_for_scanning (card_data);
2856 card_end = MIN (card_end, obj_end);
2858 if (elem < (char*)start_root)
2859 first_elem = elem = (char*)start_root;
2861 for (; elem < card_end; elem += SIZEOF_VOID_P) {
2862 if (*(GCObject**)elem)
2863 scan_field_func (NULL, (GCObject**)elem, ctx.queue);
2866 binary_protocol_card_scan (first_elem, elem - first_elem);
2869 #ifdef SGEN_HAVE_OVERLAPPING_CARDS
2870 if (overflow_scan_end) {
2871 extra_idx = card_data - card_base;
2872 card_base = card_data = sgen_shadow_cardtable;
2873 card_data_end = overflow_scan_end;
2874 overflow_scan_end = NULL;
2881 sgen_wbroots_scan_card_table (ScanCopyContext ctx)
2886 SGEN_HASH_TABLE_FOREACH (&roots_hash [ROOT_TYPE_WBARRIER], void **, start_root, RootRecord *, root) {
2887 SGEN_ASSERT (0, (root->root_desc & ROOT_DESC_TYPE_MASK) == ROOT_DESC_VECTOR, "Unsupported root type");
2889 sgen_wbroot_scan_card_table (start_root, (mword)root->end_root - (mword)start_root, ctx);
2890 } SGEN_HASH_TABLE_FOREACH_END;
2894 * ######################################################################
2895 * ######## Thread handling (stop/start code)
2896 * ######################################################################
2900 sgen_get_current_collection_generation (void)
2902 return current_collection_generation;
2906 sgen_thread_attach (SgenThreadInfo* info)
2908 info->tlab_start = info->tlab_next = info->tlab_temp_end = info->tlab_real_end = NULL;
2910 sgen_client_thread_attach (info);
2916 sgen_thread_detach_with_lock (SgenThreadInfo *p)
2918 sgen_client_thread_detach_with_lock (p);
2922 * ######################################################################
2923 * ######## Write barriers
2924 * ######################################################################
2928 * Note: the write barriers first do the needed GC work and then do the actual store:
2929 * this way the value is visible to the conservative GC scan after the write barrier
2930 * itself. If a GC interrupts the barrier in the middle, value will be kept alive by
2931 * the conservative scan, otherwise by the remembered set scan.
2935 * mono_gc_wbarrier_arrayref_copy:
2938 mono_gc_wbarrier_arrayref_copy (gpointer dest_ptr, gpointer src_ptr, int count)
2940 HEAVY_STAT (++stat_wbarrier_arrayref_copy);
2941 /*This check can be done without taking a lock since dest_ptr array is pinned*/
2942 if (ptr_in_nursery (dest_ptr) || count <= 0) {
2943 mono_gc_memmove_aligned (dest_ptr, src_ptr, count * sizeof (gpointer));
2947 #ifdef SGEN_HEAVY_BINARY_PROTOCOL
2948 if (binary_protocol_is_heavy_enabled ()) {
2950 for (i = 0; i < count; ++i) {
2951 gpointer dest = (gpointer*)dest_ptr + i;
2952 gpointer obj = *((gpointer*)src_ptr + i);
2954 binary_protocol_wbarrier (dest, obj, (gpointer)LOAD_VTABLE (obj));
2959 remset.wbarrier_arrayref_copy (dest_ptr, src_ptr, count);
2963 * mono_gc_wbarrier_generic_nostore:
2966 mono_gc_wbarrier_generic_nostore (gpointer ptr)
2970 HEAVY_STAT (++stat_wbarrier_generic_store);
2972 sgen_client_wbarrier_generic_nostore_check (ptr);
2974 obj = *(gpointer*)ptr;
2976 binary_protocol_wbarrier (ptr, obj, (gpointer)LOAD_VTABLE (obj));
2979 * We need to record old->old pointer locations for the
2980 * concurrent collector.
2982 if (!ptr_in_nursery (obj) && !concurrent_collection_in_progress) {
2983 SGEN_LOG (8, "Skipping remset at %p", ptr);
2987 SGEN_LOG (8, "Adding remset at %p", ptr);
2989 remset.wbarrier_generic_nostore (ptr);
2993 * mono_gc_wbarrier_generic_store:
2996 mono_gc_wbarrier_generic_store (gpointer ptr, GCObject* value)
2998 SGEN_LOG (8, "Wbarrier store at %p to %p (%s)", ptr, value, value ? sgen_client_vtable_get_name (SGEN_LOAD_VTABLE (value)) : "null");
2999 SGEN_UPDATE_REFERENCE_ALLOW_NULL (ptr, value);
3000 if (ptr_in_nursery (value) || concurrent_collection_in_progress)
3001 mono_gc_wbarrier_generic_nostore (ptr);
3002 sgen_dummy_use (value);
3006 * mono_gc_wbarrier_generic_store_atomic:
3007 * Same as \c mono_gc_wbarrier_generic_store but performs the store
3008 * as an atomic operation with release semantics.
3011 mono_gc_wbarrier_generic_store_atomic (gpointer ptr, GCObject *value)
3013 HEAVY_STAT (++stat_wbarrier_generic_store_atomic);
3015 SGEN_LOG (8, "Wbarrier atomic store at %p to %p (%s)", ptr, value, value ? sgen_client_vtable_get_name (SGEN_LOAD_VTABLE (value)) : "null");
3017 InterlockedWritePointer ((volatile gpointer *)ptr, value);
3019 if (ptr_in_nursery (value) || concurrent_collection_in_progress)
3020 mono_gc_wbarrier_generic_nostore (ptr);
3022 sgen_dummy_use (value);
3026 sgen_wbarrier_range_copy (gpointer _dest, gpointer _src, int size)
3028 remset.wbarrier_range_copy (_dest,_src, size);
3032 * ######################################################################
3033 * ######## Other mono public interface functions.
3034 * ######################################################################
3038 sgen_gc_collect (int generation)
3043 sgen_perform_collection (0, generation, "user request", TRUE, TRUE);
3048 sgen_gc_collection_count (int generation)
3050 return InterlockedRead (generation == GENERATION_NURSERY ? &gc_stats.minor_gc_count : &gc_stats.major_gc_count);
3054 sgen_gc_get_used_size (void)
3058 tot = los_memory_usage;
3059 tot += nursery_section->end_data - nursery_section->data;
3060 tot += major_collector.get_used_size ();
3061 /* FIXME: account for pinned objects */
3067 sgen_env_var_error (const char *env_var, const char *fallback, const char *description_format, ...)
3071 va_start (ap, description_format);
3073 fprintf (stderr, "Warning: In environment variable `%s': ", env_var);
3074 vfprintf (stderr, description_format, ap);
3076 fprintf (stderr, " - %s", fallback);
3077 fprintf (stderr, "\n");
3083 parse_double_in_interval (const char *env_var, const char *opt_name, const char *opt, double min, double max, double *result)
3086 double val = strtod (opt, &endptr);
3087 if (endptr == opt) {
3088 sgen_env_var_error (env_var, "Using default value.", "`%s` must be a number.", opt_name);
3091 else if (val < min || val > max) {
3092 sgen_env_var_error (env_var, "Using default value.", "`%s` must be between %.2f - %.2f.", opt_name, min, max);
3100 parse_sgen_minor (const char *opt)
3103 return SGEN_MINOR_DEFAULT;
3105 if (!strcmp (opt, "simple")) {
3106 return SGEN_MINOR_SIMPLE;
3107 } else if (!strcmp (opt, "simple-par")) {
3108 return SGEN_MINOR_SIMPLE_PARALLEL;
3109 } else if (!strcmp (opt, "split")) {
3110 return SGEN_MINOR_SPLIT;
3112 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Using default instead.", "Unknown minor collector `%s'.", opt);
3113 return SGEN_MINOR_DEFAULT;
3118 parse_sgen_major (const char *opt)
3121 return SGEN_MAJOR_DEFAULT;
3123 if (!strcmp (opt, "marksweep")) {
3124 return SGEN_MAJOR_SERIAL;
3125 } else if (!strcmp (opt, "marksweep-conc")) {
3126 return SGEN_MAJOR_CONCURRENT;
3127 } else if (!strcmp (opt, "marksweep-conc-par")) {
3128 return SGEN_MAJOR_CONCURRENT_PARALLEL;
3130 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Using default instead.", "Unknown major collector `%s'.", opt);
3131 return SGEN_MAJOR_DEFAULT;
3137 parse_sgen_mode (const char *opt)
3140 return SGEN_MODE_NONE;
3142 if (!strcmp (opt, "balanced")) {
3143 return SGEN_MODE_BALANCED;
3144 } else if (!strcmp (opt, "throughput")) {
3145 return SGEN_MODE_THROUGHPUT;
3146 } else if (!strcmp (opt, "pause") || g_str_has_prefix (opt, "pause:")) {
3147 return SGEN_MODE_PAUSE;
3149 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Using default configurations.", "Unknown mode `%s'.", opt);
3150 return SGEN_MODE_NONE;
3155 init_sgen_minor (SgenMinor minor)
3158 case SGEN_MINOR_DEFAULT:
3159 case SGEN_MINOR_SIMPLE:
3160 sgen_simple_nursery_init (&sgen_minor_collector, FALSE);
3162 case SGEN_MINOR_SIMPLE_PARALLEL:
3163 sgen_simple_nursery_init (&sgen_minor_collector, TRUE);
3165 case SGEN_MINOR_SPLIT:
3166 sgen_split_nursery_init (&sgen_minor_collector);
3169 g_assert_not_reached ();
3174 init_sgen_major (SgenMajor major)
3176 if (major == SGEN_MAJOR_DEFAULT)
3177 major = DEFAULT_MAJOR;
3180 case SGEN_MAJOR_SERIAL:
3181 sgen_marksweep_init (&major_collector);
3183 case SGEN_MAJOR_CONCURRENT:
3184 sgen_marksweep_conc_init (&major_collector);
3186 case SGEN_MAJOR_CONCURRENT_PARALLEL:
3187 sgen_marksweep_conc_par_init (&major_collector);
3190 g_assert_not_reached ();
3195 * If sgen mode is set, major/minor configuration is fixed. The other gc_params
3196 * are parsed and processed after major/minor initialization, so it can potentially
3197 * override some knobs set by the sgen mode. We can consider locking out additional
3198 * configurations when gc_modes are used.
3201 init_sgen_mode (SgenMode mode)
3203 SgenMinor minor = SGEN_MINOR_DEFAULT;
3204 SgenMajor major = SGEN_MAJOR_DEFAULT;
3207 case SGEN_MODE_BALANCED:
3209 * Use a dynamic parallel nursery with a major concurrent collector.
3210 * This uses the default values for max pause time and nursery size.
3212 minor = SGEN_MINOR_SIMPLE;
3213 major = SGEN_MAJOR_CONCURRENT;
3214 dynamic_nursery = TRUE;
3216 case SGEN_MODE_THROUGHPUT:
3218 * Use concurrent major to let the mutator do more work. Use a larger
3219 * nursery, without pause time constraints, in order to collect more
3220 * objects in parallel and avoid repetitive collection tasks (pinning,
3221 * root scanning etc)
3223 minor = SGEN_MINOR_SIMPLE_PARALLEL;
3224 major = SGEN_MAJOR_CONCURRENT;
3225 dynamic_nursery = TRUE;
3226 sgen_max_pause_time = 0;
3228 case SGEN_MODE_PAUSE:
3230 * Use concurrent major and dynamic nursery with a more
3231 * aggressive shrinking relative to pause times.
3233 minor = SGEN_MINOR_SIMPLE_PARALLEL;
3234 major = SGEN_MAJOR_CONCURRENT;
3235 dynamic_nursery = TRUE;
3236 sgen_max_pause_margin = SGEN_PAUSE_MODE_MAX_PAUSE_MARGIN;
3239 g_assert_not_reached ();
3242 init_sgen_minor (minor);
3243 init_sgen_major (major);
3251 SgenMajor sgen_major = SGEN_MAJOR_DEFAULT;
3252 SgenMinor sgen_minor = SGEN_MINOR_DEFAULT;
3253 SgenMode sgen_mode = SGEN_MODE_NONE;
3254 char *params_opts = NULL;
3255 char *debug_opts = NULL;
3256 size_t max_heap = 0;
3257 size_t soft_limit = 0;
3259 gboolean debug_print_allowance = FALSE;
3260 double allowance_ratio = 0, save_target = 0;
3261 gboolean cement_enabled = TRUE;
3264 result = InterlockedCompareExchange (&gc_initialized, -1, 0);
3267 /* already inited */
3270 /* being inited by another thread */
3271 mono_thread_info_usleep (1000);
3274 /* we will init it */
3277 g_assert_not_reached ();
3279 } while (result != 0);
3281 SGEN_TV_GETTIME (sgen_init_timestamp);
3283 #ifdef SGEN_WITHOUT_MONO
3284 mono_thread_smr_init ();
3287 mono_coop_mutex_init (&gc_mutex);
3289 gc_debug_file = stderr;
3291 mono_coop_mutex_init (&sgen_interruption_mutex);
3293 if ((env = g_getenv (MONO_GC_PARAMS_NAME)) || gc_params_options) {
3294 params_opts = g_strdup_printf ("%s,%s", gc_params_options ? gc_params_options : "", env ? env : "");
3299 opts = g_strsplit (params_opts, ",", -1);
3300 for (ptr = opts; *ptr; ++ptr) {
3302 if (g_str_has_prefix (opt, "major=")) {
3303 opt = strchr (opt, '=') + 1;
3304 sgen_major = parse_sgen_major (opt);
3305 } else if (g_str_has_prefix (opt, "minor=")) {
3306 opt = strchr (opt, '=') + 1;
3307 sgen_minor = parse_sgen_minor (opt);
3308 } else if (g_str_has_prefix (opt, "mode=")) {
3309 opt = strchr (opt, '=') + 1;
3310 sgen_mode = parse_sgen_mode (opt);
3318 sgen_init_internal_allocator ();
3319 sgen_init_nursery_allocator ();
3320 sgen_init_fin_weak_hash ();
3321 sgen_init_hash_table ();
3322 sgen_init_descriptors ();
3323 sgen_init_gray_queues ();
3324 sgen_init_allocator ();
3325 sgen_init_gchandles ();
3327 sgen_register_fixed_internal_mem_type (INTERNAL_MEM_SECTION, SGEN_SIZEOF_GC_MEM_SECTION);
3328 sgen_register_fixed_internal_mem_type (INTERNAL_MEM_GRAY_QUEUE, sizeof (GrayQueueSection));
3330 sgen_client_init ();
3332 if (sgen_mode != SGEN_MODE_NONE) {
3333 if (sgen_minor != SGEN_MINOR_DEFAULT || sgen_major != SGEN_MAJOR_DEFAULT)
3334 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Ignoring major/minor configuration", "Major/minor configurations cannot be used with sgen modes");
3335 init_sgen_mode (sgen_mode);
3337 init_sgen_minor (sgen_minor);
3338 init_sgen_major (sgen_major);
3342 gboolean usage_printed = FALSE;
3344 for (ptr = opts; *ptr; ++ptr) {
3346 if (!strcmp (opt, ""))
3348 if (g_str_has_prefix (opt, "major="))
3350 if (g_str_has_prefix (opt, "minor="))
3352 if (g_str_has_prefix (opt, "mode=")) {
3353 if (g_str_has_prefix (opt, "mode=pause:")) {
3354 char *str_pause = strchr (opt, ':') + 1;
3355 int pause = atoi (str_pause);
3357 sgen_max_pause_time = pause;
3359 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Using default", "Invalid maximum pause time for `pause` sgen mode");
3363 if (g_str_has_prefix (opt, "max-heap-size=")) {
3364 size_t page_size = mono_pagesize ();
3365 size_t max_heap_candidate = 0;
3366 opt = strchr (opt, '=') + 1;
3367 if (*opt && mono_gc_parse_environment_string_extract_number (opt, &max_heap_candidate)) {
3368 max_heap = (max_heap_candidate + page_size - 1) & ~(size_t)(page_size - 1);
3369 if (max_heap != max_heap_candidate)
3370 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Rounding up.", "`max-heap-size` size must be a multiple of %d.", page_size);
3372 sgen_env_var_error (MONO_GC_PARAMS_NAME, NULL, "`max-heap-size` must be an integer.");
3376 if (g_str_has_prefix (opt, "soft-heap-limit=")) {
3377 opt = strchr (opt, '=') + 1;
3378 if (*opt && mono_gc_parse_environment_string_extract_number (opt, &soft_limit)) {
3379 if (soft_limit <= 0) {
3380 sgen_env_var_error (MONO_GC_PARAMS_NAME, NULL, "`soft-heap-limit` must be positive.");
3384 sgen_env_var_error (MONO_GC_PARAMS_NAME, NULL, "`soft-heap-limit` must be an integer.");
3388 if (g_str_has_prefix (opt, "nursery-size=")) {
3390 opt = strchr (opt, '=') + 1;
3391 if (*opt && mono_gc_parse_environment_string_extract_number (opt, &val)) {
3392 if ((val & (val - 1))) {
3393 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Using default value.", "`nursery-size` must be a power of two.");
3397 if (val < SGEN_MAX_NURSERY_WASTE) {
3398 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Using default value.",
3399 "`nursery-size` must be at least %d bytes.", SGEN_MAX_NURSERY_WASTE);
3403 min_nursery_size = max_nursery_size = val;
3404 dynamic_nursery = FALSE;
3406 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Using default value.", "`nursery-size` must be an integer.");
3411 if (g_str_has_prefix (opt, "save-target-ratio=")) {
3413 opt = strchr (opt, '=') + 1;
3414 if (parse_double_in_interval (MONO_GC_PARAMS_NAME, "save-target-ratio", opt,
3415 SGEN_MIN_SAVE_TARGET_RATIO, SGEN_MAX_SAVE_TARGET_RATIO, &val)) {
3420 if (g_str_has_prefix (opt, "default-allowance-ratio=")) {
3422 opt = strchr (opt, '=') + 1;
3423 if (parse_double_in_interval (MONO_GC_PARAMS_NAME, "default-allowance-ratio", opt,
3424 SGEN_MIN_ALLOWANCE_NURSERY_SIZE_RATIO, SGEN_MAX_ALLOWANCE_NURSERY_SIZE_RATIO, &val)) {
3425 allowance_ratio = val;
3430 if (!strcmp (opt, "cementing")) {
3431 cement_enabled = TRUE;
3434 if (!strcmp (opt, "no-cementing")) {
3435 cement_enabled = FALSE;
3439 if (!strcmp (opt, "precleaning")) {
3440 precleaning_enabled = TRUE;
3443 if (!strcmp (opt, "no-precleaning")) {
3444 precleaning_enabled = FALSE;
3448 if (!strcmp (opt, "dynamic-nursery")) {
3449 if (sgen_minor_collector.is_split)
3450 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Using default value.",
3451 "dynamic-nursery not supported with split-nursery.");
3453 dynamic_nursery = TRUE;
3456 if (!strcmp (opt, "no-dynamic-nursery")) {
3457 dynamic_nursery = FALSE;
3461 if (major_collector.handle_gc_param && major_collector.handle_gc_param (opt))
3464 if (sgen_minor_collector.handle_gc_param && sgen_minor_collector.handle_gc_param (opt))
3467 if (sgen_client_handle_gc_param (opt))
3470 sgen_env_var_error (MONO_GC_PARAMS_NAME, "Ignoring.", "Unknown option `%s`.", opt);
3475 fprintf (stderr, "\n%s must be a comma-delimited list of one or more of the following:\n", MONO_GC_PARAMS_NAME);
3476 fprintf (stderr, " max-heap-size=N (where N is an integer, possibly with a k, m or a g suffix)\n");
3477 fprintf (stderr, " soft-heap-limit=n (where N is an integer, possibly with a k, m or a g suffix)\n");
3478 fprintf (stderr, " mode=MODE (where MODE is 'balanced', 'throughput' or 'pause[:N]' and N is maximum pause in milliseconds)\n");
3479 fprintf (stderr, " nursery-size=N (where N is an integer, possibly with a k, m or a g suffix)\n");
3480 fprintf (stderr, " major=COLLECTOR (where COLLECTOR is `marksweep', `marksweep-conc', `marksweep-par')\n");
3481 fprintf (stderr, " minor=COLLECTOR (where COLLECTOR is `simple' or `split')\n");
3482 fprintf (stderr, " wbarrier=WBARRIER (where WBARRIER is `remset' or `cardtable')\n");
3483 fprintf (stderr, " [no-]cementing\n");
3484 fprintf (stderr, " [no-]dynamic-nursery\n");
3485 if (major_collector.print_gc_param_usage)
3486 major_collector.print_gc_param_usage ();
3487 if (sgen_minor_collector.print_gc_param_usage)
3488 sgen_minor_collector.print_gc_param_usage ();
3489 sgen_client_print_gc_params_usage ();
3490 fprintf (stderr, " Experimental options:\n");
3491 fprintf (stderr, " save-target-ratio=R (where R must be between %.2f - %.2f).\n", SGEN_MIN_SAVE_TARGET_RATIO, SGEN_MAX_SAVE_TARGET_RATIO);
3492 fprintf (stderr, " default-allowance-ratio=R (where R must be between %.2f - %.2f).\n", SGEN_MIN_ALLOWANCE_NURSERY_SIZE_RATIO, SGEN_MAX_ALLOWANCE_NURSERY_SIZE_RATIO);
3493 fprintf (stderr, "\n");
3495 usage_printed = TRUE;
3501 g_free (params_opts);
3503 alloc_nursery (dynamic_nursery, min_nursery_size, max_nursery_size);
3505 sgen_pinning_init ();
3506 sgen_cement_init (cement_enabled);
3508 if ((env = g_getenv (MONO_GC_DEBUG_NAME)) || gc_debug_options) {
3509 debug_opts = g_strdup_printf ("%s,%s", gc_debug_options ? gc_debug_options : "", env ? env : "");
3514 gboolean usage_printed = FALSE;
3516 opts = g_strsplit (debug_opts, ",", -1);
3517 for (ptr = opts; ptr && *ptr; ptr ++) {
3519 if (!strcmp (opt, ""))
3521 if (opt [0] >= '0' && opt [0] <= '9') {
3522 gc_debug_level = atoi (opt);
3527 char *rf = g_strdup_printf ("%s.%d", opt, mono_process_current_pid ());
3528 gc_debug_file = fopen (rf, "wb");
3530 gc_debug_file = stderr;
3533 } else if (!strcmp (opt, "print-allowance")) {
3534 debug_print_allowance = TRUE;
3535 } else if (!strcmp (opt, "print-pinning")) {
3536 sgen_pin_stats_enable ();
3537 } else if (!strcmp (opt, "verify-before-allocs")) {
3538 verify_before_allocs = 1;
3539 has_per_allocation_action = TRUE;
3540 } else if (g_str_has_prefix (opt, "max-valloc-size=")) {
3541 size_t max_valloc_size;
3542 char *arg = strchr (opt, '=') + 1;
3543 if (*opt && mono_gc_parse_environment_string_extract_number (arg, &max_valloc_size)) {
3544 mono_valloc_set_limit (max_valloc_size);
3546 sgen_env_var_error (MONO_GC_DEBUG_NAME, NULL, "`max-valloc-size` must be an integer.");
3549 } else if (g_str_has_prefix (opt, "verify-before-allocs=")) {
3550 char *arg = strchr (opt, '=') + 1;
3551 verify_before_allocs = atoi (arg);
3552 has_per_allocation_action = TRUE;
3553 } else if (!strcmp (opt, "collect-before-allocs")) {
3554 collect_before_allocs = 1;
3555 has_per_allocation_action = TRUE;
3556 } else if (g_str_has_prefix (opt, "collect-before-allocs=")) {
3557 char *arg = strchr (opt, '=') + 1;
3558 has_per_allocation_action = TRUE;
3559 collect_before_allocs = atoi (arg);
3560 } else if (!strcmp (opt, "verify-before-collections")) {
3561 whole_heap_check_before_collection = TRUE;
3562 } else if (!strcmp (opt, "check-remset-consistency")) {
3563 remset_consistency_checks = TRUE;
3564 nursery_clear_policy = CLEAR_AT_GC;
3565 } else if (!strcmp (opt, "mod-union-consistency-check")) {
3566 if (!major_collector.is_concurrent) {
3567 sgen_env_var_error (MONO_GC_DEBUG_NAME, "Ignoring.", "`mod-union-consistency-check` only works with concurrent major collector.");
3570 mod_union_consistency_check = TRUE;
3571 } else if (!strcmp (opt, "check-mark-bits")) {
3572 check_mark_bits_after_major_collection = TRUE;
3573 } else if (!strcmp (opt, "check-nursery-pinned")) {
3574 check_nursery_objects_pinned = TRUE;
3575 } else if (!strcmp (opt, "clear-at-gc")) {
3576 nursery_clear_policy = CLEAR_AT_GC;
3577 } else if (!strcmp (opt, "clear-nursery-at-gc")) {
3578 nursery_clear_policy = CLEAR_AT_GC;
3579 } else if (!strcmp (opt, "clear-at-tlab-creation")) {
3580 nursery_clear_policy = CLEAR_AT_TLAB_CREATION;
3581 } else if (!strcmp (opt, "debug-clear-at-tlab-creation")) {
3582 nursery_clear_policy = CLEAR_AT_TLAB_CREATION_DEBUG;
3583 } else if (!strcmp (opt, "check-scan-starts")) {
3584 do_scan_starts_check = TRUE;
3585 } else if (!strcmp (opt, "verify-nursery-at-minor-gc")) {
3586 do_verify_nursery = TRUE;
3587 } else if (!strcmp (opt, "check-concurrent")) {
3588 if (!major_collector.is_concurrent) {
3589 sgen_env_var_error (MONO_GC_DEBUG_NAME, "Ignoring.", "`check-concurrent` only works with concurrent major collectors.");
3592 nursery_clear_policy = CLEAR_AT_GC;
3593 do_concurrent_checks = TRUE;
3594 } else if (!strcmp (opt, "dump-nursery-at-minor-gc")) {
3595 do_dump_nursery_content = TRUE;
3596 } else if (!strcmp (opt, "disable-minor")) {
3597 disable_minor_collections = TRUE;
3598 } else if (!strcmp (opt, "disable-major")) {
3599 disable_major_collections = TRUE;
3600 } else if (g_str_has_prefix (opt, "heap-dump=")) {
3601 char *filename = strchr (opt, '=') + 1;
3602 nursery_clear_policy = CLEAR_AT_GC;
3603 sgen_debug_enable_heap_dump (filename);
3604 } else if (g_str_has_prefix (opt, "binary-protocol=")) {
3605 char *filename = strchr (opt, '=') + 1;
3606 char *colon = strrchr (filename, ':');
3609 if (!mono_gc_parse_environment_string_extract_number (colon + 1, &limit)) {
3610 sgen_env_var_error (MONO_GC_DEBUG_NAME, "Ignoring limit.", "Binary protocol file size limit must be an integer.");
3615 binary_protocol_init (filename, (long long)limit);
3616 } else if (!strcmp (opt, "nursery-canaries")) {
3617 do_verify_nursery = TRUE;
3618 enable_nursery_canaries = TRUE;
3619 } else if (!sgen_client_handle_gc_debug (opt)) {
3620 sgen_env_var_error (MONO_GC_DEBUG_NAME, "Ignoring.", "Unknown option `%s`.", opt);
3625 fprintf (stderr, "\n%s must be of the format [<l>[:<filename>]|<option>]+ where <l> is a debug level 0-9.\n", MONO_GC_DEBUG_NAME);
3626 fprintf (stderr, "Valid <option>s are:\n");
3627 fprintf (stderr, " collect-before-allocs[=<n>]\n");
3628 fprintf (stderr, " verify-before-allocs[=<n>]\n");
3629 fprintf (stderr, " max-valloc-size=N (where N is an integer, possibly with a k, m or a g suffix)\n");
3630 fprintf (stderr, " check-remset-consistency\n");
3631 fprintf (stderr, " check-mark-bits\n");
3632 fprintf (stderr, " check-nursery-pinned\n");
3633 fprintf (stderr, " verify-before-collections\n");
3634 fprintf (stderr, " verify-nursery-at-minor-gc\n");
3635 fprintf (stderr, " dump-nursery-at-minor-gc\n");
3636 fprintf (stderr, " disable-minor\n");
3637 fprintf (stderr, " disable-major\n");
3638 fprintf (stderr, " check-concurrent\n");
3639 fprintf (stderr, " clear-[nursery-]at-gc\n");
3640 fprintf (stderr, " clear-at-tlab-creation\n");
3641 fprintf (stderr, " debug-clear-at-tlab-creation\n");
3642 fprintf (stderr, " check-scan-starts\n");
3643 fprintf (stderr, " print-allowance\n");
3644 fprintf (stderr, " print-pinning\n");
3645 fprintf (stderr, " heap-dump=<filename>\n");
3646 fprintf (stderr, " binary-protocol=<filename>[:<file-size-limit>]\n");
3647 fprintf (stderr, " nursery-canaries\n");
3648 sgen_client_print_gc_debug_usage ();
3649 fprintf (stderr, "\n");
3651 usage_printed = TRUE;
3658 g_free (debug_opts);
3660 if (check_mark_bits_after_major_collection)
3661 nursery_clear_policy = CLEAR_AT_GC;
3663 if (major_collector.post_param_init)
3664 major_collector.post_param_init (&major_collector);
3666 sgen_thread_pool_start ();
3668 sgen_memgov_init (max_heap, soft_limit, debug_print_allowance, allowance_ratio, save_target);
3670 memset (&remset, 0, sizeof (remset));
3672 sgen_card_table_init (&remset);
3674 sgen_register_root (NULL, 0, sgen_make_user_root_descriptor (sgen_mark_normal_gc_handles), ROOT_TYPE_NORMAL, MONO_ROOT_SOURCE_GC_HANDLE, "normal gc handles");
3678 sgen_init_bridge ();
3682 sgen_gc_initialized ()
3684 return gc_initialized > 0;
3688 sgen_get_nursery_clear_policy (void)
3690 return nursery_clear_policy;
3696 mono_coop_mutex_lock (&gc_mutex);
3700 sgen_gc_unlock (void)
3702 mono_coop_mutex_unlock (&gc_mutex);
3706 sgen_major_collector_iterate_live_block_ranges (sgen_cardtable_block_callback callback)
3708 major_collector.iterate_live_block_ranges (callback);
3712 sgen_major_collector_iterate_block_ranges (sgen_cardtable_block_callback callback)
3714 major_collector.iterate_block_ranges (callback);
3718 sgen_get_major_collector (void)
3720 return &major_collector;
3724 sgen_get_minor_collector (void)
3726 return &sgen_minor_collector;
3730 sgen_get_remset (void)
3736 count_cards (long long *major_total, long long *major_marked, long long *los_total, long long *los_marked)
3738 sgen_get_major_collector ()->count_cards (major_total, major_marked);
3739 sgen_los_count_cards (los_total, los_marked);
3742 static gboolean world_is_stopped = FALSE;
3744 /* LOCKING: assumes the GC lock is held */
3746 sgen_stop_world (int generation)
3748 long long major_total = -1, major_marked = -1, los_total = -1, los_marked = -1;
3750 SGEN_ASSERT (0, !world_is_stopped, "Why are we stopping a stopped world?");
3752 binary_protocol_world_stopping (generation, sgen_timestamp (), (gpointer) (gsize) mono_native_thread_id_get ());
3754 sgen_client_stop_world (generation);
3756 world_is_stopped = TRUE;
3758 if (binary_protocol_is_heavy_enabled ())
3759 count_cards (&major_total, &major_marked, &los_total, &los_marked);
3760 binary_protocol_world_stopped (generation, sgen_timestamp (), major_total, major_marked, los_total, los_marked);
3763 /* LOCKING: assumes the GC lock is held */
3765 sgen_restart_world (int generation)
3767 long long major_total = -1, major_marked = -1, los_total = -1, los_marked = -1;
3770 SGEN_ASSERT (0, world_is_stopped, "Why are we restarting a running world?");
3772 if (binary_protocol_is_heavy_enabled ())
3773 count_cards (&major_total, &major_marked, &los_total, &los_marked);
3774 binary_protocol_world_restarting (generation, sgen_timestamp (), major_total, major_marked, los_total, los_marked);
3776 world_is_stopped = FALSE;
3778 sgen_client_restart_world (generation, &stw_time);
3780 binary_protocol_world_restarted (generation, sgen_timestamp ());
3782 if (sgen_client_bridge_need_processing ())
3783 sgen_client_bridge_processing_finish (generation);
3785 sgen_memgov_collection_end (generation, stw_time);
3789 sgen_is_world_stopped (void)
3791 return world_is_stopped;
3795 sgen_check_whole_heap_stw (void)
3797 sgen_stop_world (0);
3798 sgen_clear_nursery_fragments ();
3799 sgen_check_whole_heap (TRUE);
3800 sgen_restart_world (0);
3804 sgen_timestamp (void)
3806 SGEN_TV_DECLARE (timestamp);
3807 SGEN_TV_GETTIME (timestamp);
3808 return SGEN_TV_ELAPSED (sgen_init_timestamp, timestamp);
3811 #endif /* HAVE_SGEN_GC */