Write OS kernel from scratch-implement heap and malloc

Series catalog

• preface
• preparation work
• BIOS boot to real mode
• GDT and protection mode
• A preliminary study of virtual memory
• loads and enters the kernel
• display and print
• Global Descriptor Table GDT
• Interrupt handling
• virtual memory perfect
• Implement heap and malloc
• first kernel thread
• multi-thread switching
• lock and multi-thread synchronization
• enter user mode
• process
• system call
• simple file system
• Load executable program
• keyboard driver
• run shell

Black box malloc

So far, all the memory has been statically allocated. This is obviously not an efficient approach and cannot meet the needs of subsequent kernel development. Dynamic allocation of memory, malloc is a function we must implement.

Presumably you have used malloc in C programming before, knowing that it is used to allocate a piece of memory of a specified size and return it to you, and you have to manually free when it is used up; if you still feel mysterious about it, don’t know what it does internally Yes, or I don’t even know where the allocated memory is. I don’t even know malloc is. It needs to be introspected.

Don't be nervous, the significance of this project is to take you to examine these underlying knowledge and principles. malloc indeed something we always use, but we may never really observe it. This article will explain and implement a simple malloc library function.

Heap

The memory allocated by malloc heap heap , where heap data structure, it is just a pure memory area, for example, heap in user space is located in stack and the program Between loading areas:

malloc in the kernel space, it is also necessary to operate on a large block of heap . Back to our kernel space, its current state is like this:

The first three 4MB are already occupied. Starting from 0xC0C00000 is free space. We might as heap delineate the 060fecddc8d8c7 space of the kernel from here and end somewhere, which is the pink area in the figure. From now on, this is the happy planet where we are digging memory. .

malloc on the heap

heap is a large pool. malloc does is to enclose memory in the heap. For example, if you need 32 bytes of memory, it will find an unused area on the heap and give you a 32-byte area. That’s it. That's it.

This looks very simple, but in fact it is a very large and complex subject. There are several core issues that need to be resolved:

• Establish a data structure to manage the allocation and release of memory blocks to ensure that everything is correct and orderly;
• Fast speed, less memory fragmentation;

Let's look at the first question first. Creating a data structure to manage this memory is not as simple as it seems. The data structure here is not created elsewhere, but is located on heap , which means it is built-in, because this is still a layer problem. Data structures in the usual sense, such as linked lists, trees, hash tables, etc., generally require dynamic allocation of memory, but the original intention of heap is to solve the problem of dynamic allocation of memory, which is back to the original point. Therefore, the management of the heap's own data structure is self-woven inside the heap, which is very similar to the disk file system to be discussed later.

For the second problem, it is essentially a performance problem, which is not the focus of our project. The first thing we need to ensure is correctness, otherwise all performance will be out of the question. The issue of dynamic memory allocation management is really a huge and complicated subject. There are various technical papers and implementation methods discussing these issues. The implementation of malloc We are limited to time and our own level, and will not dig deep in this area, but adopt the simplest way to achieve it.

kmalloc design ideas

A k is added here, called kmalloc , that is, kernel malloc , which means that this is a malloc function in the kernel space. The core APIs are as follows:

void* kmalloc(uint32 size);
void* kmalloc_aligned(uint32 size);
void free(void* ptr);

Among them, kmalloc_aligned represents the allocated memory block, and the starting address is page aligned, which is a common requirement in kernel development.

The implementation method I used here is to copy the method in the previously recommended tutorial JamesM's kernel development tutorials , because this should indeed be the simplest method for the mentally retarded. However, the code in the above tutorial should be problematic after I actually tested it, so I used it to implement it completely by myself, and added the test. At present, there is no major problem. Of course, if there is a bug, it is unavoidable. Welcome , the code is 160fecddc8da35 src/mem/kheap.c .

The realization idea is very simple, which is to store the positions of all the blank areas in an ordered array:

In this way, when you need to allocate a piece of memory with a specified size of t, you can find the first blank area larger than t from this array, then you can cut an area of size t from it, and then return the remaining blank area Go back and make the array still in order.

In fact, the array does not require order, the purpose of ordering is only to find suitable blank blocks more efficiently. You can traverse one by one, or you can use binary search.

When you want free , just put it back into the array. Of course, there are some special cases here. For example, the neighbors of the free area are also blank areas, so you can merge them into a large blank area. For heap, of course, large blank areas are more welcome, because large blanks have a stronger ability to divide, which means that there are fewer fragments ( fragment ), and fragments are annoying.

---

Therefore, our heap can be planned into two areas:

1. The ordered array ordered array mentioned above;
2. A large memory area used to actually allocate memory, we call it the memory pool memory pool ;

And each block (white or gray) in the memory pool is not 100% completely given to the user. It also contains some information about this color block:

We memory pool in each of a region called block its enlarged as shown above, it has a header , is defined as:

struct kheap_block_header {
  uint32 magic;
  uint8 is_hole;
  uint32 size;
} __attribute__((packed));

in:

• magic is used to identify the header;
• is_hole indicates whether this block is used (gray) or free (white);
• size represents the size of the memory area that can be actually allocated in this block, that is, the blue slash in the figure;

Correspondingly, block also has a footer :

struct kheap_block_footer {
  uint32 magic;
  kheap_block_header_t* header;
} __attribute__((packed));

in:

• magic is the same as the one in the header;
• header is a pointer to the header, because in some cases we need to start from the footer and locate the header;

Well, the design idea part is over here, and the next step is to realize it.

kmalloc implementation

Ordered_array

First, we need to implement ordered array . We can encapsulate it as an abstract class. My code is implemented in src/utils/ordered_array.c for reference.

typedef void* type_t;

typedef struct ordered_array {
  type_t array;
  uint32 size;
  uint32 max_size;
  comparator_t comparator;
} ordered_array_t;

For this class, the core concept that needs to be recognized is that this is a pointer array . The field array represents this array, so it is a generic type. This class also needs to pass in a comparator to sort the array:

// An comparator function is used to sort elements.
//
// Returns -1, 0 or 1 if the first element
// is <, == or > the second, respectively.
typedef int32 (*comparator_t)(type_t, type_t);

The specific implementation details are not much to say, it is relatively simple, it is just insertion sort;

In addition, note that the size of this array is dynamically variable. It is 0 at the beginning. As elements are inserted and deleted, size keeps changing; the maximum capacity is controlled max_size

heap structure

Next, define the structure of kheap

typedef struct kernel_heap {
  ordered_array_t index;
  uint32 start_address;
  uint32 end_address;
  uint32 size;
  uint32 max_address;
} kheap_t;

Some fields about the size of the heap are defined here: start_address and end_address indicate the start and end positions of the current heap. It should be noted here that the heap is relatively small at the beginning, and then can gradually become larger during use. When a suitable blank area is not found to allocate memory, the current heap expand

Next, it is the ordered_array , and the field is named index , which is used to store all the blank blocks in an orderly manner. Pay attention to the way the index is initialized here. The array start address of the entire heap. kheap_block_comparator compares the size of two blocks:

kheap_t create_kheap(uint32 start,
                     uint32 end,
                     uint32 max) {
  kheap_t kheap;

  // Initialize the index array.
  kheap.index = ordered_array_create(
      (type_t*)start, KHEAP_INDEX_NUM, &kheap_block_comparator);
  
  // ...

  make_block(start, end - start - BLOCK_META_SIZE, IS_HOLE);
  ordered_array_insert(&kheap.index, (type_t)start);

  // ...
}

The entire heap initialized with only 1 large blank block:

malloc implementation

With the above design and paving, malloc actually a matter of course. As long as index array, cut the required memory on it, turn it into a gray block and assign it to the user, and then allocate the rest index the white part of the file back into the 060fecddc8de37 array. Its core function is alloc .

alloc is the processing of the remaining part after cutting. In any case, the remaining part needs to be returned to kheap. At the very least, the size of the remaining part needs to be greater than header + footer . This should be well understood.

In addition, there are more worthy of discussion when asked page_align , that call kmalloc_aligned , the situation will become complicated. In general, free block is unlikely to be page aligned, so look for the first page aligned place inside it, and start cutting from there:

The dotted line marks the position of page aligned. This kind of division brings about a problem that there may be remaining areas in front and behind the cut part, so they need to be returned to kheap again. In my code implementation, the size of the reserved area in front of it is specified, and it must be able to accommodate a complete block (greater than header + footer ), otherwise it will start cutting at the next page aligned place.

free implementation

free function is relatively simple. What needs to be considered is the free block. If the neighbor is also free, then it can be merged into a large free block:

Summarize

This one is actually a very independent chapter, even if it is not included in the kernel project, it can be taken out as a separate topic. Dynamic memory allocation is a very deep subject. We just adopted one of the simplest implementation methods here, but even if it is the simplest, it is still not easy to implement it correctly. I spent a few days in the development and conducted a lot of testing before it was basically adjusted. Just like the realization of paging , kheap must also be absolutely correct, because in the subsequent development, it will be the main battlefield for us to allocate memory.