#linux 换页系统的抖动问题

## 参考资料
[1]http://stackoverflow.com/questions/896226/what-is-working-set
[2]https://en.wikipedia.org/wiki/Working_set
[3]https://msdn.microsoft.com/en-us/library/windows/desktop/cc441804%28v=vs.85%29.aspx
[4]Swap Thrashing, can nothing be done? 
  http://www.linuxquestions.org/questions/linux-kernel-70/swap-thrashing-can-nothing-be-done-612945/
[5] Simple-Stupid user-space program doing "kill -STOP" and "kill -CONT" to protect from thrashing
  https://github.com/tobixen/thrash-protect
[6] go 语言版 thrash-protect
  https://github.com/stone/tprotect
  
[11] OOM killer not working?
http://askubuntu.com/questions/398236/oom-killer-not-working
[12] Kernel fails to execute code for killing OOM task if oom_kill_allocating_task = 0 
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1359766
[13]http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting
[14]https://www.kernel.org/doc/Documentation/sysctl/vm.txt
[15]http://ayende.com/blog/170242/linux-debts-and-out-of-memory-killer
[16]http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
[17]http://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer
[18]http://serverfault.com/questions/362589/effects-of-configuring-vm-overcommit-memory
[19] mlock(2) - Linux man page
  https://linux.die.net/man/2/mlock
[20] mmap(2) - Linux man page
  https://linux.die.net/man/2/mmap
[21]http://stackoverflow.com/questions/24302945/tell-linux-to-not-swap-until-a-certain-amount-of-ram
[22]http://unix.stackexchange.com/questions/24625/how-to-completely-disable-swap
[23]http://superuser.com/questions/298740/how-to-configure-linux-to-call-oom-kill-on-its-own
[24] Can I run an Oracle server without swap?
  https://serverfault.com/questions/817010/can-i-run-an-oracle-server-without-swap#
[25] Linux: don't create swap partition?
  https://superuser.com/questions/639618/linux-dont-create-swap-partition
[26] Can I tell Linux not to swap out a particular processes' memory?
  https://stackoverflow.com/questions/578137/can-i-tell-linux-not-to-swap-out-a-particular-processes-memory
[27] linux memory lock浅析 
  https://yq.aliyun.com/articles/8869
[28] linux页面回收浅析 
  https://yq.aliyun.com/articles/8904?spm=5176.100239.blogcont8869.11.oZXZff
[29] linux kswapd浅析 
  https://yq.aliyun.com/articles/8865?spm=5176.100239.blogcont8904.14.Eguffn
[30] linux文件读写浅析 
  https://yq.aliyun.com/articles/8905?spm=5176.100239.blogcont8904.13.Eguffn

[31]http://unix.stackexchange.com/questions/24625/how-to-completely-disable-swap
[32]https://bugzilla.kernel.org/show_bug.cgi?id=99471
[33]system nearly loose response when 2 apps consume most RAM on box without swap
  https://bugzilla.kernel.org/show_bug.cgi?id=24692
[34]How do you empty the buffers and cache on a Linux system?
  https://unix.stackexchange.com/questions/87908/how-do-you-empty-the-buffers-and-cache-on-a-linux-system
[35]linux-ftools
  https://github.com/david415/linux-ftools
[36]pcstat - get page cache statistics for files
  https://github.com/tobert/pcstat
[37]vmtouch file system cache diagnostics and control
  https://github.com/hoytech/vmtouch
[38][Linux] Linux下谁在消耗我们的cache
  http://www.cnblogs.com/langdashu/p/5953222.html
  
[41] Android low memory killer vs. memory pressure notifications
http://lkml.iu.edu/hypermail/linux/kernel/1112.2/00811.html
[42] Android Low Memory Killer
http://www.programering.com/a/MjNzADMwATE.html
[43] Tuning Android for low RAM
http://events.linuxfoundation.org/sites/events/files/slides/android-lowmemory-abs-2014.pdf
[44] Low Memory Killer的原理
http://www.cnblogs.com/angeldevil/archive/2013/05/21/3090872.html

[45] Is it possible to make the OOM killer intervent earlier?
https://superuser.com/questions/406101/is-it-possible-to-make-the-oom-killer-intervent-earlier/708430
[46] Is it possible to trigger OOM-killer on forced swapping?
https://unix.stackexchange.com/questions/38507/is-it-possible-to-trigger-oom-killer-on-forced-swapping
[47] The Early OOM Daemon
https://github.com/rfjakob/earlyoom

## 描述
占用过多内存的程序可导致 linux 持续换页，整个系统将失去响应。

## 参考
### 工作集（Working Set）
工作集是过去一段时间内，一个进程访问过的内存空间；也就是过去一段时间内一个进程的活跃的内存空间。
这个概念对理解和判断系统的实际内存压力和抖动现象很有用。
如果系统中所有进程（包括内核）的工作集的之和（去掉重复，因为有共享的内存）的大小超过了物理内存的大小，那么抖动就不可避免，反之则不容易发生抖动。
原因很简单，系统的交换机制是要确保将不活跃的内存换出，而把活跃的内存换入，但如果活跃内存的大小超过了物理内存，那么CPU几乎在执行任何代码时都需要等待交换，这样交换就会变得过于频繁，从而发生抖动[1,2]。
这种情况下，基于一般性内存占用的判据（虚拟内存、驻留内存（位于物理内存中的部分，Resident set）、提交内存等的大小）都不是好方法，理论上说，在表观的内存占用只是稍微超过物理内存时，仍有可能发生抖动。

这可能也是linux上抖动难以根治的原因：swapness, overcommit_memory, cgroup 等机制考虑的都是一般性内存占用，而不是工作集。linux 内核似乎没有提供工作集的信息。在windows上，工作集其实是指驻留内存[3]，这可能与真正的工作集并不完全吻合，取决于换页算法。

解决“大工作集抖动”的方法除了杀掉进程以外，暂时挂起一部分进程也是有效的，因为挂起的进程的（非共享）内存空间一定是不活跃的，这等于减小了系统工作集的大小。

### thrash-protect
Simple-Stupid user-space program doing "kill -STOP" and "kill -CONT" to protect from thrashing [5,6]

### swapping-or-trashing-with-vast-amounts-of-unmapped-pagecache
http://www.windowslinuxosx.com/q/answers-swapping-or-trashing-with-vast-amounts-of-unmapped-pagecache-124366.html
To change this for a session, call sysctl again and pass it a swappiness value to use:

sudo sysctl vm.swappiness=30

Besides free, you can of course monitor the effects of doing this via the excellent htop or iotop utilities.

If you like what you see, and want to retain this value across reboots, just place "vm.swappiness=30" in the /etc/sysctl.conf file.

vm.swappiness 并不是个硬约束，即使等于 0，系统仍然会使用交换分区。

### overcommit, OOM killer

使用命令行 sysctl vm.overcommit_memory=2
或者写入 /etc/sysctl.conf

vm.overcommit_memory 的 0 表示系统自动确定 OOM 条件；2 表示 commit 的内存不能超过 overcommit_ratio|kbytes 指定的极限，否则触发OOM；1假定总是有足够的内存。

vm.overcommit_memory = 2
vm.overcommit_ratio = 100

overcommit 极限是根据 (vm.overcommit_ratio / 100) * phy_mem + swap 或者 overcommit_kbytes + swap 来计算的。
overcommit 极限只在 vm.overcommit_memory = 2 时才有意义，不过这时候应该设置较大的极限（例如物理内存的2-2.5倍），
因为有很多程序会申请很多内存但不去使用（例如JVM），偏小的极限会过早触发OOM killer。

vm.overcommit_memory = 0 的问题是 OOM killer 往往触发得太晚，系统已经由于抖动失去了响应。
vm.overcommit_memory = 2 似乎要好一些，不过由于 OOM killer 杀死的进程（或者因为内存分配失败自己崩溃的进程）往往是随机的，也可能造成一些问题。

/proc/sys/vm/oom_kill_allocating_task 文件用于控制 OOM killer 执行的方式。0 表示扫描所有的进程找出最应该被杀死的，1表示立即杀死触发OOM的进程。

查看最近被OOM killer杀死的进程：
dmesg | egrep -i 'killed process'
或者
egrep -i -r 'killed process' /var/log/

总的来说，OOM killer 是不靠谱的，它经常触发的太晚，从而导致系统失去响应直至崩溃，参考[11,12]。
因此希望能早一些（系统尚有一些剩余内存时）就触发[45,46]。但 OOM killer 很难被调整这样做，所以产生了用户态的程序——earlyoom[46]。
顾名思义它会较早的触发（默认条件是可用物理内存和交换分区都不足10%），杀掉内存消耗最多的进程，避免系统卡死。
经过测试还是挺有效的，用 android-studio 编译一个大工程时，它先后杀掉了 chrome、 firefox、vs code，编译得以顺利完成。

### 关闭交换
因为 overcommit 要根据 swap 来计算，所以打开关闭交换后需要调整overcommit_ratio。

sudo swapoff -a
sysctl vm.overcommit_ratio=150

sudo swapon -a
sysctl vm.overcommit_ratio=25

但是关闭交换分区（文件）也大概并不能解决抖动问题，因为linux的交换并不只是发生在交换分区（文件）中，也会发生在内存映射文件中。即被映射到内存中的文件在内存不足的情况下也会不断地换入换出。详见[22]。

根据[24]的使用经验，数据库、缓存服务器系统最好关闭交换，这是因为数据库、缓存服务器通常善于自己管理缓存的换入换出。
如果启用交换，在耗尽内存时，服务器软件会逐渐变慢，这是不希望的。如果没有交换，在耗尽内存时，服务器软件可能立即崩溃，然后后备服务器上线，这往往比逐渐变慢更好。

根据[25]的使用经验，桌面系统关闭交换也更好。对于有内存泄露的进程，在无交换的系统上会较快的触发 low memory killer，这可能造成数据丢失；
在有交换的系统上，会导致其它进程/数据被换出而失去响应，无法调出命令行来杀掉内存泄露的进程，最终不得不关闭电源，仍然会造成数据丢失。

### mlock, MAP_LOCKED
linux 系统调用 mlock() [19] 可以阻止一个进程的部分或全部内存空间被换页。这对实时进程是很有用的。
mlock() 涵盖直接内存分配（包括函数调用栈的内存），也可能影响内存映射文件。mlock() 可导致分配物理内存给进程，如果物理内存不足，mlock() 会失败。
内存映射文件 mmap() 有个选项 MAP_LOCKED 可以阻止映射的内存被释放，类似 mlock() [20, 27]。

### kswapd
也有人怀疑系统抖动是kernel的bug，特别是kswapd[31][33]。有的机器在开始利用交换分区时系统就会失去响应，但也有的机器只是变慢，或许这与磁盘的特性也有关系。
有的内核开发者认为这是正常的行为，建议配置 container 或者 overcommit[33]。

### android 风格的 low memory killer driver
[41-44]。
其实 Android Low Memory Killer 仍然是基于 OOM killer 的（但现在可能不是了[44]），主要改进是：
* 适当地设定oom kill优先级
* 能够发送内存不足提醒到应用程序，应用程序保存自己的状态，从而可以被安全地结束。

### 用户空间的 OOM killer/notify
Taming the OOM killer（综述OOM killer的改进方案，包括基于进程号/名的黑名单、oom cgroup、低内存提醒等，但似乎没有一个真正实现了）
  http://lwn.net/Articles/317814/

根据名称的黑名单
https://lkml.org/lkml/2009/1/12/235

oom cgroup
https://lkml.org/lkml/2009/1/29/220

### Wait IO problem
摘录自[1]：
#vmstat 2
From here you can see the pre-last column is the "wa", it is always with some value there and it means your system is constantly waiting IO operations. So, your Linux Box will never works well. 

 One of most possible reason which causes the problem is the "Journal Flushing Operation". It periodically flushes journal commits and other modifications to disk. To determine if it is the cause, using this command:

#ps auxf
So, just as mentioned above, if a process with its stat with "D"(Uninterruptible sleep (usually IO)), it means it is actually taking all CPU resource with no any possible interruption. This means your Linux Box will wait on IO and does not responding any other commands if such process is always there. 

摘录自[2]：
To identify whether I/O is causing system slowness you can use several commands such as top.
From the CPU(s) line you can see the current percentage of CPU in I/O Wait (wa)。

[1]http://www.chileoffshore.com/en/interesting-articles/126-linux-wait-io-problem
[2]http://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/
[3]http://serverfault.com/questions/12679/can-anyone-explain-precisely-what-iowait-is

### 其他因素
* hald disk polling by hald-addon-storage[1]
  deepin 似乎没有用这个东西？
* devkit polling /usr/libexec/devkit-disks-daemon[1]
  deepin 似乎没有用这个东西？
* Filesystem journaling[1]
  关闭 ext4/3 的日志可能有作用。

[1] http://forums.linuxmint.com/viewtopic.php?f=47&t=40111

### Memory
http://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6

### Understanding Virtual Memory
http://www.redhat.com/magazine/001nov04/features/vm/

## 工具
### 综述
http://unix.stackexchange.com/questions/117742/diagram-of-linux-kernel-vs-performance-tools

### 内核io审计（针对线程）
内核数据结构（task_io_accounting.h）[1]：

 11 struct task_io_accounting {
 12 #ifdef CONFIG_TASK_XACCT
 13         /* bytes read */
 14         u64 rchar;
 15         /*  bytes written */
 16         u64 wchar;
 17         /* # of read syscalls */
 18         u64 syscr;
 19         /* # of write syscalls */
 20         u64 syscw;
 21 #endif /* CONFIG_TASK_XACCT */
 22 
 23 #ifdef CONFIG_TASK_IO_ACCOUNTING
 24         /*
 25          * The number of bytes which this task has caused to be read from
 26          * storage.
 27          */
 28         u64 read_bytes;
 29 
 30         /*
 31          * The number of bytes which this task has caused, or shall cause to be
 32          * written to disk.
 33          */
 34         u64 write_bytes;
 35 
 36         /*
 37          * A task can cause "negative" IO too.  If this task truncates some
 38          * dirty pagecache, some IO which another task has been accounted for
 39          * (in its write_bytes) will not be happening.  We _could_ just
 40          * subtract that from the truncating task's write_bytes, but there is
 41          * information loss in doing that.
 42          */
 43         u64 cancelled_write_bytes;
 44 #endif /* CONFIG_TASK_IO_ACCOUNTING */
 45 };

这个统计是针对线程的，也是 iotop/atop 等工具的数据来源。

/proc/[pid]/io 也使用了上述数据[2,3]。

根据 [3]，rchar/wchar 表示通过 read()/write() 等系统调用的数据量，因此会包括已经缓存的数据，以及非磁盘文件来源的数据（网络、管道、终端等），
但是却不会包括mmap、换页等机制产生的IO数据。
read_bytes/write_bytes 是通过块设备接口的数据量，因此不会包括已经缓存的数据、网络和网络文件系统（NFS等），会包括mmap[2]；经过实验，也会包括换页（page faults）产生的数据，这意味着加载可执行文件本身产生的IO也被计算在内（如果尚未被缓存）。

[1]http://lxr.free-electrons.com/source/include/linux/task_io_accounting.h
[2]http://stackoverflow.com/questions/10637290/does-rchar-include-read-bytes-proc-pid-io
[3]http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/proc.txt


### 内核io审计（针对磁盘）
/proc/diskstats
/sys/block/hd<X>/stat
大致上有相同的数据

/sys/block/hd<X>/stat 字段解释[3]
Name            units         description
----            -----         -----------
read I/Os       requests      number of read I/Os processed
read merges     requests      number of read I/Os merged with in-queue I/O
read sectors    sectors       number of sectors read
read ticks      milliseconds  total wait time for read requests
write I/Os      requests      number of write I/Os processed
write merges    requests      number of write I/Os merged with in-queue I/O
write sectors   sectors       number of sectors written
write ticks     milliseconds  total wait time for write requests
in_flight       requests      number of I/Os currently in flight
io_ticks        milliseconds  total time this block device has been active
time_in_queue   milliseconds  total wait time for all requests

[1]http://unix.stackexchange.com/questions/87920/how-to-detect-if-a-disk-is-seeking
[2]https://www.kernel.org/doc/Documentation/iostats.txt
[3]https://www.kernel.org/doc/Documentation/block/stat.txt

### 清除磁盘缓存
/proc/sys/vm/drop_caches 或者 sysctl vm.drop_caches=n
n=1：释放 pagecache
n=2：释放 dentries 和 inodes；
n=3：释放 pagecache, dentries 和 inodes；

### 分析磁盘缓存
linux-ftool[35]，pcstat[36]，vmtouch[37] 都可以分析一个文件的哪些部分被缓存了，并且可以做其他一些事情。
linux-ftool 已经废弃了，pcstat 是 go 语言的，推荐 vmtouch。

### 清除交换分区
swapoff -a
swapon -a

### Page Faults

Minor
If the page is loaded in memory at the time the fault is generated, but is not marked in the memory management unit as being loaded in memory, then it is called a minor or soft page fault. 

Major
If the page is not loaded in memory at the time of the fault, then it is called a major or hard page fault. 

Invalid
If a page fault occurs for a reference to an address that is not part of the virtual address space, meaning there cannot be a page in memory corresponding to it, then it is called an invalid page fault. 

Example: ps command

Use the ps command to view page faults for PID #1, enter:

ps -o min_flt,maj_flt 1

Sample outputs:

 MINFL  MAJFL
  3104     36

To see every process on the system:
# ps -eo min_flt,maj_flt,cmd,pid | less

[1]http://www.cyberciti.biz/faq/linux-command-to-see-major-minor-pagefaults/
[2]https://en.wikipedia.org/wiki/Page_fault

### atop
http://www.atoptool.nl/ <- 主站

One-stop performance analysis using atop
https://lwn.net/Articles/387202/

交互模式子命令：
s - CPU
m - 内存
d - 磁盘
n - 网络

日志：
/var/log/atop/atop_YYYYMMDD
默认时间间隔是10分钟，默认保留时间是一个月。

The log file can be viewed using atop -r log_filename. The subcommand t forwards to the next sample in the atop logfile (i.e. 10 minutes by default), subcommand T rewinds one sample. Subcommand b branches to a specific time in the current logfile. All other subcommands to zoom in on specific resources also work. The logfile that the atop daemon creates can also be viewed using a sar-like interface using the command atopsar. 

### iotop
http://guichaz.free.fr/iotop/ <- 作者站点
https://www.rootusers.com/my-top-3-linux-commands-for-logging-problems/
http://linux.die.net/man/1/iotop

iotop -to -d 5 > /var/log/iotop.txt

This will basically log the contents of iotop to /var/log/iotop.txt every 5 seconds as defined by -d, -t is used to add a timestamp to each line so that when we view the log file later we can see when events happened, -o is used to only show processes or threads actually doing I/O otherwise the log will get extremely large 

-a, --accumulated 显示累计的IO数量，而不是速率
--only , -o , 或者交互模式下按 o，只显示有IO的进程/线程
-P, --processes 只显示进程

iotop 的数据来源是内核的 task_io_accounting::read_bytes|write_bytes|cancelled_write_bytes，
参考：http://lxr.free-electrons.com/source/include/linux/task_io_accounting.h，iotop/data.py:75: ('read_bytes', 248)

### iostat
用于查看整个磁盘的统计数据
iostat -dx 2

### mortadelo
https://gitorious.org/mortadelo
Mortadelo is a pseudo-clone of the Windows Sysinternals Filemon. This is a graphical viewer for all the file-related system calls that happen in your Linux system. It requires Systemtap to run.

### sysctl
sysctl -a 列出所有变量的值

sysctl name=value 设置值

### dstat
https://www.rootusers.com/my-top-3-linux-commands-for-logging-problems/

Dstat allows you to see all system resources and compare them against each other, this is very useful for seeing what is happening to the resources on a server, if a particular resource has run out for example it can cause performance problems.

apt-get install dstat

dstat -tcdrgilmns --output /var/log/dstat.csv --noupdate 5


    -t: Time – enables timestamps on the logs, very useful when logging at logs later
    -c: CPU stats (system, user, idle, wait, hardware interrupt, software interrupt)
    -d: Disk stats (read, write)
    -r: I/O request stats (read, write)
    -g: Page stats (page in, page out)
    -i: Interrupt stats
    -l: Load stats (1 minute average, 5 minute average, 15 minute average)
    -m: Memory stats (used, buffers, cache, free)
    -n: Network stats (received, sent)
    -s: Swap stats (used, free)

### /proc 文件系统
/proc/<pid>/status 文件包含了内存数据。用 man proc 或 https://www.kernel.org/doc/Documentation/filesystems/proc.txt 查看其意义。
例如：

 VmPeak                      peak virtual memory size
 VmSize                      total program size
 VmLck                       locked memory size
 VmPin                       pinned memory size
 VmHWM                       peak resident set size ("high water mark")
 VmRSS                       size of memory portions. It contains the three
                             following parts (VmRSS = RssAnon + RssFile + RssShmem)
 RssAnon                     size of resident anonymous memory
 RssFile                     size of resident file mappings
 RssShmem                    size of resident shmem memory (includes SysV shm,
                             mapping of tmpfs and shared anonymous mappings)
 VmData                      size of private data segments
 VmStk                       size of stack segments
 VmExe                       size of text segment
 VmLib                       size of shared library code
 VmPTE                       size of page table entries
 VmPMD                       size of second level page tables
 VmSwap                      amount of swap used by anonymous private data
                             (shmem swap usage is not included)

[1] https://utcc.utoronto.ca/~cks/space/blog/linux/LinuxMemoryStats

### eatmemory
https://github.com/julman99/eatmemory
消耗指定数量的内存。

## IO scheduler
### 设置
查询现在使用的和候选的：
cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
改变：
sudo sh -c "echo cfq > /sys/block/sda/queue/scheduler"

### 资料
http://en.wikipedia.org/wiki/CFQ
http://en.wikipedia.org/wiki/Deadline_scheduler
http://www.cyberciti.biz/faq/linux-change-io-scheduler-for-harddisk/
https://github.com/yksuh/thrashing_study/

## 限制资源使用
通过限制进程、用户的内存（特别是虚拟内存）使用，可以减少内存抖动现象
### ulimit
ulimit -v number 可以限制虚拟内存，单位KB。
ulimit -a 查看所有限制。

参考
  通过 ulimit 改善系统性能 http://www.ibm.com/developerworks/cn/linux/l-cn-ulimit/index.html
  ulimit and sysctl http://www.linuxhowtos.org/tips%20and%20tricks/ulimit.htm
  linux process memory limit http://stackoverflow.com/questions/4651234/linux-process-memory-limit

### cgroups
#### cgconfig.conf/cgexec

ubuntu 上，解析 /etc/cgconfig.conf 需要安装 cgroup-bin：
apt-get install cgroup-bin 

 have been dealing with this exact problem and have found the best solution if you are using 2.6.24 or higher is to use cgroup / cgroups. An example would be like this, here is the default /etc/cgconfig.conf file with an added control group at the bottom. This control group limits the amout of physical memory to 100MB and the total virtual memory allocation to 200MB.
 

mount {
    cpuset  = /cgroup/cpuset;
    cpu = /cgroup/cpu;
    cpuacct = /cgroup/cpuacct;
    memory  = /cgroup/memory;
    devices = /cgroup/devices;
    freezer = /cgroup/freezer;
    net_cls = /cgroup/net_cls;
    blkio   = /cgroup/blkio;
}

group daemons/java_limited_process {
    memory {
        memory.limit_in_bytes = "104857600";
        memory.memsw.limit_in_bytes = "209715200";
    }
}

Once I have this configured, I can add process to the group like so

cgexec -g memory:daemons/java_limited_process /usr/local/tomcat/bin/startup.sh

This will limit the memory of the main process and any children it spawns. It also has facilities to query memory usage in the controllers.

#### cgroup file system
cd /sys/fs/cgroup/blkio/
sudo mkdir ntfs-3g
cd ntfs-3g
ls -l
这个目录里自动创建了许多文件，包括
blkio.throttle.write_iops_device
blkio.throttle.read_iops_device
等。

sudo sh -c "echo '8:0 20' > blkio.throttle.write_iops_device"
sudo sh -c "echo '8:0 20' > blkio.throttle.read_iops_device"

其中格式是 "<major>:<minor>  <value>"，<major>:<minor> 可以通过 ls -l 查看：
ls -l /dev/sda

ps -e | grep ntfs
  747 ?        00:00:00 mount.ntfs-3g
  
sudo sh -c "echo 747 > tasks"

svn upgrade 在 ntfs-3g 上操作，如果不限制 io，则系统会卡死，界面不可操作，只能重启。限制到 iops 80 一下不会卡死，但直到20仍会偶尔卡顿。

参考
  http://www.aikaiyuan.com/5900.html
  http://www.cnblogs.com/jackiedai/articles/2871814.html
  http://www.linuxcertif.com/man/5/cgconfig.conf/
  http://www.oschina.net/code/explore/pf-kernel/Documentation/cgroups/blkio-controller.txt
