CentOS / RHEL CacheFS: Speed Up Network File System (NFS) File Access

CentOS / RHEL CacheFS: Speed Up Network File System (NFS) File Access


We are using NFS v4 server under RHEL 6.x+. How do I configure CacheFS for NFS under Red Hat Enterprise Linux or CentOS to speed up file access and reduce load on our NFS server?

Linux comes with CacheFS which is developed by David Howells. The Linux CacheFS currently is designed to operate on Andrew File System and Network File System. You need to install a package called cachefilesd which is a CacheFiles userspace management daemon. The cachefilesd daemon manages the caching files and directory that are used by network filesystems such a AFS and NFS to do persistent caching to the local disk.

Step #1: Install cachefilesd

Use the yum command to install cachefilesd – cacheFiles userspace management daemon:
# yum -y install cachefilesd

Sample outputs:

Loaded plugins: product-id, protectbase, rhnplugin, subscription-manager
Updating certificate-based repositories.
0 packages excluded due to repository protections
Setting up Install Process
Resolving Dependencies
–> Running transaction check
—> Package cachefilesd.x86_64 0:0.10.2-1.el6 will be installed
–> Finished Dependency Resolution
Dependencies Resolved
 Package               Arch             Version                Repository                      Size
 cachefilesd           x86_64           0.10.2-1.el6           rhel-x86_64-server-6            35 k
Transaction Summary
Install       1 Package(s)
Total download size: 35 k
Installed size: 0
Downloading Packages:
cachefilesd-0.10.2-1.el6.x86_64.rpm                                          |  35 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : cachefilesd-0.10.2-1.el6.x86_64                                                  1/1
Installed products updated.
  cachefilesd.x86_64 0:0.10.2-1.el6
Step #2: Configure cachefilesd

You need to edit /etc/cachefilesd.conf, enter:
# vi /etc/cachefilesd.conf

Sample configuration file:

dir /ssd/fscache
tag mycache
brun 10%
bcull 7%
bstop 3%
frun 10%
fcull 7%
fstop 3%
# Assuming you’re using SELinux with the default security policy included in
# this package
secctx system_u:system_r:cachefiles_kernel_t:s0

dir /ssd/fscache – The default directory is set to /var/cache/fscache. This directory act as the root of the cache. In this example, my nfs client is mounted on RAID-1 and cache is on single ssd disk at mounted at /ssd/. The cache is good while reading files but I have way too many small files and it causes the cache opposite. Particularly on slow harddrives. So I put cache to another harddrive. If you can afford used ssd disks.
tag mycache This command specifies a tag to FS-Cache to use in distinguishing multiple caches. This is only required if more than one cache is going to be used. The default is “CacheFiles”.
secctx system_u:system_r:cachefiles_kernel_t:s0 – Specify an LSM security context as which the kernel will perform operations to access the cache. The default is to use cachefilesd’s security context. If you change the directory you need to use the these instructions to set new LSM security context.
brun 10%, bcull 7%, bstop 3%, frun 10%, fcull 7%, fstop 3% – The cache may need culling occasionally to make space. This involves discarding objects from the cache that have been used less recently than anything else. Culling is based on the access time of data objects. Empty directories are culled if not in use. See cachefilesd.conf man page or this url for more information.
Step #3: How do I start / stop / restart cachefilesd?

Simply type the following commands:

## start it ##
/sbin/service cachefilesd  start
# or #
/etc/init.d/cachefilesd start
## stop it ##
/sbin/service cachefilesd  stop
## restart it ##
/sbin/service cachefilesd  restart
# or #
/etc/init.d/cachefilesd restart
## get status ##
/sbin/service cachefilesd  status
Step #4: How do I mount nfs client with CacheFS support?

You need to pass the fsc option to mount command as follows:

mount -t nfs -o fsc,optio2 nas01:/export/dir1/ /destination/mnt/point
In this example mount /var/www/html from nas042 on local /var/www/html using fsc (and other options) as follows:
# mount -t nfs4 -o rsize=32768,wsize=32768,intr,hard,proto=tcp,sync,fsc nas042:/var/www/html /var/www/html

Step #5: How do I verify that nfs client is working with CacheFS?

Cd to /ssd/fscache (or default /var/cache/fscache) directory and type the following command:
# cd /ssd/fscache
# ls -Z

Sample outputs:

drwx——. root root system_u:object_r:cachefiles_var_t:s0 cache
drwx——. root root system_u:object_r:cachefiles_var_t:s0 graveyard
When the cache is set up correctly, you will see two directories as above. You can type the following command to list files and cache size:
# find
# du -sh

Sample outputs:

How do I see FS-Cache statistics?

Simply type the following command:
# cat /proc/fs/fscache/stats

Sample outputs:

FS-Cache statistics
Cookies: idx=30 dat=7895 spc=0
Objects: alc=7164 nal=0 avl=7164 ded=4261
ChkAux : non=0 ok=3727 upd=0 obs=3
Pages  : mrk=59000 unc=37195
Acquire: n=7925 nul=0 noc=0 ok=7925 nbf=0 oom=0
Lookups: n=7164 neg=3429 pos=3735 crt=3429 tmo=0
Updates: n=0 nul=0 run=0
Relinqs: n=5022 nul=0 wcr=0 rtr=22
AttrChg: n=0 ok=0 nbf=0 oom=0 run=0
Allocs : n=0 ok=0 wt=0 nbf=0 int=0
Allocs : ops=0 owt=0 abt=0
Retrvls: n=7307 ok=3324 wt=1540 nod=3243 nbf=740 int=0 oom=0
Retrvls: ops=6567 owt=1672 abt=0
Stores : n=32683 ok=32683 agn=0 nbf=0 oom=0
Stores : ops=5842 run=38525 pgs=32683 rxd=32683 olm=0
VmScan : nos=181 gon=0 bsy=0 can=0
Ops    : pend=1695 run=12409 enq=38525 can=0 rej=0
Ops    : dfr=21 rel=12409 gc=21
CacheOp: alo=0 luo=0 luc=0 gro=0
CacheOp: upo=0 dro=0 pto=0 atc=0 syn=0
CacheOp: rap=0 ras=0 alp=0 als=0 wrp=0 ucp=0 dsp=0
You can see the caching state of individual mountpoints can be examined through other /proc files.
# cat /proc/fs/nfsfs/servers

Sample outputs:

v4 0a0a1d44  801   1 nas042
# cat /proc/fs/nfsfs/volumes

Sample outputs:

NV SERVER   PORT DEV     FSID              FSC
v4 0a0a1d44  801 0:20    147f55ffe88547d2  yes
Note down the output of the “FSC” column. It always says “yes” when the system has been asked to cache a particular NFS, and “no” when it hasn’t.

How do I test CacheFS?

Copy files from NFS server to local hard drive:

time cp /path/to/nfs/mnt/point/bigfile.gz /tmp
#### this should speed up as bigfile.gz is in cache now #####
time cp /path/to/nfs/mnt/point/bigfile.gz /dev/null
In this example copy data.tar.bz2 (78MB) from nfs to /tmp (init cache):
$ time cp data.tar.bz2 /tmp

Sample outputs:

real 0m7.023s
user 0m0.000s
sys 0m0.185s
Copy data.tar.bz2 to /dev/null from NFS again (i.e. get it from cache):
$ time cp data.tar.bz2 /dev/null

Sample outputs:

real 0m0.027s
user 0m0.000s
sys 0m0.026s


슈퍼컴퓨팅 전문 기업 클루닉스/ 상무(기술이사)/ 정보시스템감리사/ 시스존 블로그 운영자

You may also like...

1 Response

  1. ethereum pow sohbet 말해보세요:

    Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

페이스북/트위트/구글 계정으로 댓글 가능합니다.