Text/Data segments

[Strogian]

I'm reading this page right now: http://www.iecc.com/linker/linker04.html

It says this:
Many Unix systems use a trick that saves file space by starting the data immediately after the text in the object file, and mapping that page in the file into virtual memory twice, once read-only for the text and once copy-on-write for the data. In that case, the data addresses logically start exactly one page beyond the end of the text, so rather than rounding up, the data addresses start exactly 4K or whatever the page size is beyond the end of the text.

So the file looks like this:

Page:[ 1 ][ 2 ][ 3 ][ 4 ][ 5 ]
...--------------------------|-----------------------...
text data

But in memory it looks like this:

Page:[ 1 ][ 2 ][ 3 ][ 3 ][ 4 ][ 5 ]
...--------------------------|---{-----}-----------------------...
text (text) data

(I actually tested this with a C program, and it's true ;))

My question is, why doesn't it just shift the memory maps a little bit, to do this instead:

Page:[ 1 ][ 2 ][ 3 ][ a ][ b ][ c ]
...--------------------------|---|----------------------------...
text ^ ^ data
\ /
same location on disk
(start of data)


Does that make sense? That way, it's not wasting that extra little bit of space that happens when you just re-map page 3 twice.

[dchidelf]

This is really interesting.
Thanks for posting the link.

The following is my best guess. It could be completely incorrect.

If the text and data segments are loaded into separate memory pages, as is generally the case...
I would assume this is to allow the text and data segments to have different access permissions.
the size of the text segment has to be rounded up to a full page and the data and BSS segment locations correspondingly adjusted.

Lets imagine we have a 5K text segment and a 7K data segment (with 4K memory pages that is 3 pages).

111122223333 - Page number
TTTTTDDDDDDD - segment T=text D=data

We round 5K up to 8K to fit a full page, shifting the data (and BSS) segment over by 3K.

1111222233334444
TTTTT DDDDDDD

This would use 16K of memory.

As I understand it, the trick used by "Many Unix Systems" is to remap the same memory into virtual memory twice. By doing that, it actually uses less physical memory.
Time to pull out the ascii art...here it goes!

Legend : page number [ KB1 KB2 KB3 KB4 ]


Shift by 3K to move Data into it's own page
-------------------------------------------

T T T T T D D D D D D D
Physical Memory Pages 1[ 0 1 2 3 ] 2[ 4 5 6 7 ] 3[ 8 9 A B ] 4[ C D E F ]
| | | | | | | | | | | | | | | |
Virtual Memory Pages 1[ 0 1 2 3 ] 2[ 4 5 6 7 ] 3[ 8 9 A B ] 4[ C D E F ]
T T T T T D D D D D D D
16K Physical memory used.
16K Virtual memory used.

Remapping physical memory twice to get Data into it's own page
--------------------------------------------------------------

T T T T T D D D D D D D
Physical Memory Pages 1[ 0 1 2 3 ] 2[ 4 5 6 7 ] 3[ 8 9 A B ]
| | | | | | | | | | | |
Virtual Memory Pages 1[ 0 1 2 3 ] 2[ 4 5 6 7 ] 4[ C D E F ]
3[ 8 9 A B ]
T T T T T D D D D D D D
12K Physical memory used.
16K Virtual memory used.

The question I have...
Would it ever be possible for the virtual memory system to think that virtual memory page 2 was in memory, but page 3 was not? If so, would a reference to page 3 cause a page fault?
If so, wouldn't we just end up using 16K of physical memory again?

I would guess the answer to my first question is no, because the other 2 questions are too ugly.

[Strogian]

Actually, I wasn't thinking much about virtual-vs-physical memory when I posted the question. It _might_ use the same physical for both mapped pages initially, but once you try and change any data on that page, it will have to make another copy of it.

What you started with is actually pretty accurate -- it's explained pretty well earlier in that book. :)

1111222233334444
TTTTT DDDDDDD

That's actually how unix initially did things, because that saves RAM. But, those places you have "blanked out" there actually have to exist on-disk. When the program is loaded, no space (or at least, a minimal amount of space) is wasted in RAM, but those blank spots had been filled with zero-bits (on disk) in order to get that to work out. (aligning the data segment correctly)

The trick that many unix systems use is to _not_ put in the zero-bits. So on disk it looks like this:
TTTTTDDDDDDD
The problem with that is (apparently) there is a logical page on disk with both text parts and data parts. The text parts must be loaded read-only, while the data parts are writable. So, that page must be loaded twice, once read-only, and once read-write. I just don't know why it can't just "misalign" the page->disk mappings a little, and still load the data at the start of a memory page.

[Strogian]

OK, I'm thinking about the second half of your post now. :) Can you really map two different virtual pages to the same physical page? I guess you can -- that's what the COW (copy-on-write) is all about. But there, the virtual memory system knows that both virtual pages point to the same physical page, and makes sure that "messy stuff" doesn't happen. Only way it could work. :)

But now that I look at it, maybe that actually is the reason why they don't re-align how the data segment is loaded. Maybe it's so that the two pages will be identical, and can share the same physical RAM. I'd still like to know if it's even possible, though.

Geez, I always get confused when I start thinking about this virtual memory stuff. :D

[Strogian]

OK, more info. :) From the mmap(2) man page:


void * mmap(void *start, size_t length, int prot , int flags, int fd, off_t offset);
...
The mmap function asks to map length bytes starting at offset offset from the file (or other object) specified by the file descriptor fd into memory, preferably at address start. This latter address is a hint only, and is usually specified as 0. The actual place where the object is mapped is returned by mmap, and is never 0.
...
offset should be a multiple of the page size as returned by getpagesize(2).
...
A file is mapped in multiples of the page size.


Wow, this makes it a lot easier to explain my question. :) So, does anyone know why offset has to be a multiple of the page size? That's all I want to know, really. :D

[dchidelf]

After reading your post I thought about it and thought about it and thought to myself, "Why does it!?"

Looking around on the web I found this discussion:

> Whem mapping a file, the mmap() systems call requires the offset to be
> page aligned. This is not necessary. Of course performance is better
> if a memory page does not cross a filesystem block. But the vnode pager
> also works if the file offset is not page aligned.

I would recommend against this change. The reason for restricting
the mapping offset is to allow the VM system to (eventually) have a
fully coherent page cache. If you allow one file block to be mapped
at more than one offset within a page, then you can not use a single
physical page to represent that block. That makes it incoherent.

Any fully coherent page cache will require the restriction:

(file_offset % pagsize) == (mapping_offset % pagesize)

OK, actually, you could add a per-file constant to one side of
the above equality, but that is typically not useful.


Without this restriction, you could fill your entire memory with mmap pages each offset by one byte. Depending on how a program mmapped shared objects, it might not be able to share the object at all, because it has a different offset in virtual memory.

[Strogian]

Excellent! That is EXACTLY what I was looking for. Thank you very much! :)

So, it looks like you were right on with the "remap the same memory into virtual memory twice" thing, then eh? ;)