How to read this
Entry format:
* <text> | <resources to read>
“man *” in the resources means “run
man <word>
for every word from the <text> which looks like a system call."
Do not read localized (i.e. in languages other than English) manuals as they usually have astronomical numbers of errors. You have been warned.
I use the following invocation to get the default version:
LANG=C man <arguments>
Each and every word in a manual is meaningful. Do not just skip everything that looks hard to parse.
Introduction to UNIX
- UNIX text editing: vim, Emacs. Using man, searching in man, apropos | man man, “/” key, man apropos
- bash scripting. Syntax. Keyboard combos. | man bash, http://tldp.org/LDP/abs/html/
- IO redirections. Expansion and substitution. | man bash
- Version control management with mercurial and git | man hg, man git
- Standart tools: more, less, ls, cp, mv, rmdir, rm, touch, find, … | man *
- User management and access control in UNIXes, file modes. /etc/passwd, /etc/shadow.
- Standart tools: chown, chmod, useradd, usermod, sudo, … | man *
- File system hierarchy: /bin /sbin /boot /dev /etc /home /root /lib /var /media /mnt /proc /sys /usr/* /usr/local/*. | man hier
- Noninteractive text editing: ed, sed, awk, diff, patch. | man *
- UNIX history | http://en.wikipedia.org/wiki/Unix#History, http://www.levenez.com/unix/unix_a4.pdf.
OS Theory and POSIX
- Operating systems’ kernels: micro-, macro-, exo-, monolithic, modular. | http://en.wikipedia.org/wiki/Microkernel, http://wiki.osdev.org/Exokernel, http://en.wikipedia.org/wiki/Exokernel, http://pdos.csail.mit.edu/exo.html, at least abstracts of http://pdos.csail.mit.edu/PDOS-papers.html#Exokernels, at least 1.1.1 from http://u.cs.biu.ac.il/~wiseman/2os/microkernels/exokernel.pdf, at least abstracts of http://www.cl.cam.ac.uk/research/srg/netos/old-projects/nemesis/documentation.html, http://en.wikipedia.org/wiki/Monolithic_kernel, http://tldp.org/LDP/tlk/tlk.html
- Kernel-to-userspace interfaces: system calls (syscalls), memory, signals. | [1], http://lwn.net/Articles/250967/,
man 7 signal
- Access control: discretionary, mandatory, role-based. | http://en.wikipedia.org/wiki/Discretionary_access_control, http://en.wikipedia.org/wiki/Mandatory_access_control, http://en.wikipedia.org/wiki/Role-based_access_control, http://en.wikipedia.org/wiki/Trusted_Computer_System_Evaluation_Criteria, http://en.wikipedia.org/wiki/Security-Enhanced_Linux, http://www.nsa.gov/research/selinux/faqs.shtml
- POSIX access control. Real, effective and saved uid/gids, supplementary gids, setuid bit in file modes. |
man 7 credentials
- Process models: new process from scratch, cloning with inheritance and tuning, exokernel process models. | see the first question of this block, see next two questions, see section 3.2 of http://www.cl.cam.ac.uk/Research/SRG/netos/old-projects/pegasus/papers/usenix.ps.gz
- POSIX process model: fork, exec. Parent-child relationship. init process. |
man <term>, man 7 signal
- Threads in POSIX and Linux. Process groups and sessions. |
man 7 credentials, man 7 pthreads, man 2 clone
- POSIX signals: standard and realtime. Signal delivery and handling. |
man 7 signal
- POSIX file model: inode, directory entry, VFS, namespace, file descriptor. | TODO,
man 7 path_resolution
- VFS and namespace operations. VFS features in Linux and Plan 9.
- Hard disk partitioning, MBR. System pre-boot from BIOS to GRUB stage 2.
- Boot from stage 2, kernel startup, initrd, switching root, system init.
- Standart daemons: System V init, at, cron, udev, sshd, getty, … | man *
- Standart files in /etc: fstab, mtab, sysctl.conf, issue, nologin, …
- Software licensing: free software and open source. Common licenses. | [1]
- Quines. Thompson compiler hack. | [1]
ABI
- ABI. Compilation and linking: COFF, PE, ELF; o[bject]-files, exe[cutable]-files, s[hared]o[bject]-files. | [1]
- Building projects under UNIX: config.h, make, autotools, scons, cmake, … | man *
- Resources: read(2)ing files and etc, mmap(2)ing memory pages. | man *
- The geographical map of virtual memory or “where is the”: code, data, heap, stack.| [1]
- Doing syscalls with assembler. Kernel syscall dispatch. Context switching.| [1]
- CPU caches, cache associativity, virtual memory and caching, TLB.| http://lwn.net/Articles/250967/, [1]
- Compilation and linking: relocable code, GOT, PIC. Dynamic loading: simple relocation w/o GOT, w/ GOT, caching relocations, PIC. | [1]
- Dynamic loading under Windows and Linux. Lazy dynamic loading. Subtle differences between exe- and so- linking under Linux. | man ld-linux, [1], http://www.symantec.com/connect/articles/dynamic-linking-linux-and-windows-part-one, http://www.symantec.com/connect/articles/dynamic-linking-linux-and-windows-part-two, http://www.akkadia.org/drepper/dsohowto.pdf, http://www.macieira.org/blog/2012/01/sorry-state-of-dynamic-libraries-on-linux/
- libdl and
LD_PRELOAD. | man ld-linux, http://www.yolinux.com/TUTORIALS/LibraryArchives-StaticAndDynamic.html
- Debugging programs with gdb. Stepping lazy dynamically loaded programs. | http://beej.us/guide/bggdb/, run gdb and read help
Programming with POSIX
Process Control
- POSIX process control and signaling. System calls: fork, exec, kill, signal, sigaction. |
man 2 <term>
- Syscalls that are ok to call from signal handlers. Syscall interruption and restarting. |
man 7 signal
- Access control with syscalls: set(|r|e)uid, set(|r|e)gid, setre(u|g)id, setres(u|g)id. Capabilities. |
man <term>, man 7 capabilities
- High level access control with PAM. |
man 7 PAM
Basic IO
- POSIX synchronous IO API: open, read, write, close, pipe, named pipe with mknod, socketpair. | man *,
man 7 fifo
- File control. Syncing, locking, leasing, duplicating, …: fsync, fdatasync, fcntl, flock, dup, dup2. |
man 2 <term>
- File descriptor flags and options. Kernel structures behind the file descriptor table. |
man 2 fcntl
O_NONBLOCK IO and POSIX AIO. | man 2 fcntl, man 7 aio
Memory
- Memory management. Doing malloc(3) with sbrk(2) (deprecated) and mmap(2). | man *
- Memory allocators: SLAB, Hoard. Common tricks: mmaping aligned memory, per-thread storage.
Advanced IO
- IP networking. Kernel route table, IP filter. Network configuration for glibc: /etc/nsswitch.conf /etc/resolv.conf /etc/services. | man *, man iptables, http://tldp.org/HOWTO/NET3-4-HOWTO.html, [TODO][]
- BSD sockets. Ideas behind and resulting implementation idiosyncrasies. Common creation patterns and usage. | http://beej.us/guide/bgnet/
- glibc networking API. Name resolving, service queries, … | man gethostbyname and do breadth-first search through “SEE ALSO”
- IO multiplexing: select, poll. Throttling bandwidth by reading carefully. | man *
- Classical handling, synchronous interface to asynchronous IO with continuations (co-routines).
- Linux epoll and inotify, BSD kqueue, Linux NETLINK, TX-ring. | man epoll, man inotify, man kqueue (on BSD),
man 7 netlink
- Push and pull data processing models. Throughput versus latency. | http://insanecoding.blogspot.com/2007/05/sorry-state-of-sound-in-linux.html, http://insanecoding.blogspot.com/2009/06/state-of-sound-in-linux-not-so-sorry.html (read the comments!)
- Threads and synchronisation: clone(2), pthreads, shared memory, message queues, semaphores, synchronisation with file locks. |
man 2 clone, man 2 pthreads, man 2 fcntl, man 2 flock, man 2 mmap, http://beej.us/guide/bgipc/
- Debugging programs with ptrace (without details). | man ptrace
Back to the Theory
Scheduling
- Process scheduling: basic, multi-rate, cooperative, priority FIFO, pre-emptive, priority proportional, deadline. | http://en.wikipedia.org/wiki/Scheduling_(computing), http://www.personal.kent.edu/~rmuhamma/OpSystems/Myos/cpuScheduling.htm, http://homepages.uel.ac.uk/u0214248/FELIX1.htm, http://wiki.osdev.org/Scheduling_Algorithms, https://www.cs.drexel.edu/~wmm24/cs370/resources/Scheduler.pdf, http://oreilly.com/catalog/linuxkernel/chapter/ch10.html, http://cs.boisestate.edu/~amit/teaching/597/scheduling.pdf
- Linux process scheduler, process states. O(1) rescheduling (resched). | http://oreilly.com/catalog/linuxkernel/chapter/ch10.html, http://cs.boisestate.edu/~amit/teaching/597/scheduling.pdf
- Special schedulers: FIFO, OTHER, BATCH. |
man 2 sched_setscheduler
- Realtime scheduling: hi/low interrupts and preemption, process context preemption, kernel process preemption, (magic) scheduler preemption. | see questions above
- Artifacts: priority fluctuation with O(1) resched, priority inversion. | see questions above, http://en.wikipedia.org/wiki/Priority_inversion, http://www.cs.uiuc.edu/class/sp10/cs423/lectures/08-sched.pdf
- IO scheduling: noop, single queue, elevator, multiqueue. Linux: anticipatory, deadline, CFQ. | http://en.wikipedia.org/wiki/I/O_scheduling, http://www.devshed.com/c/a/BrainDump/Linux-IO-Schedulers/
- Artifacts: priority inversion.
File systems
- File systems (detailed): tmpfs, FAT, ext2. | http://www.nongnu.org/ext2-doc/
- Journaling. Journaling in ext3: journal, ordered, writeback.
- Data structures used: groups and tables, B-trees, dancing trees. Log-structured file system.
- Block and extent allocation, multiroot, snapshots, object-level RAID, transactions (and why they are not that easy in POSIX).
- File systems (main features only): ext3/4, raiser 3/4, XFS, btrfs, iso9660, squashfs.
- Device mapper, software RAIDs with mdadm, logical volume management with LVM.
- Network volumes: nbd. Network file systems: NFS, sshfs, 9P.
- Cluster file systems and volumes: Redhat GFS 1/2, Clustered LVM. Cluster management: synchronisation, fencing.
- Distributed filesystems: Google GFS, git.
- Linux FUSE interface.
Good to Know About