How to install and use GlusterFS Cinder

How to install and use GlusterFS Cinder

OpenStack Cinder와 GlusterFS를 연동하는 방법에 대한 정리

1. Introduction

GlusterFS and Cinder integration provides a system for data storage that enables users to access the same data, both as an object and as a file, thus simplifying management and controlling storage costs.

GlusterFS - GlusterFS is an open source, distributed file system capable of scaling to several petabytes and handling thousands of clients. GlusterFS clusters together storage building blocks over Infiniband RDMA or TCP/IP interconnect, aggregating disk and memory resources and managing data in a single global namespace. GlusterFS is based on a stackable user space design and can deliver exceptional performance for diverse workloads.

Cinder - Cinder is the OpenStack service which is responsible for handling persistent storage for virtual machines. This is persistent block storage for the instances running in Nova. Snapshots can be taken for backing up and data, either for restoring data, or to be used to create new block storage volumes.

With Enterprise Linux 6, configuring OpenStack Grizzly to use GlusterFS for its Cinder (block) storage is fairly simple.

These instructions have been tested with both GlusterFS 3.3 and GlusterFS 3.4. Other releases may also work, but have not been tested.

2. Prerequisites

2.1. GlusterFS

For information on prerequsities and instructions for installing GlusterFS, see http://www.gluster.org/community/documentation/index.php

2.2. Cinder

For information on prerequsities and instructions for installing Cinder, see http://docs.openstack.org/

Before beginning, you must ensure there are no existing volumes in Cinder. Use "cinder delete" to remove any, and "cinder list" to verify that they are deleted. If you do not delete the existing cinder volumes, it will cause errors later in the process, breaking your Cinder installation.

NOTE - Unlike other software, the "openstack-config" and "cinder" commands generally require you to run them as a root user. Without prior configuration, running them through sudo generally does not work. (This can be changed, but is beyond the scope of this HOW-TO.)

3. Install GlusterFS Cinder

3.1. Installing GlusterFS Client on Cinder hosts

> On each Cinder host, install the GlusterFS client packages

• 필수 패키지 install

# apt-get update # apt-get upgrade # apt-get install gcc flex bison python openssl gcc-multilib libssl-dev make

• 다운로드 받은 소스 파일을 압축 해제

# tar xvzf glusterfs-3.4.2.tar.gz # cd glusterfs-3.4.2

• 소스 파일 설치

# ./configure # make # make install

• lib link 파일 설정

# ln -s /usr/local/lib/libglusterfs.so.0 /usr/lib/libglusterfs.so.0 # ln -s /usr/local/lib/libgfxdr.so.0 /usr/lib/libgfxdr.so.0 # ln -s /usr/local/lib/libgfrpc.so.0 /usr/lib/libgfrpc.so.0

• Log 위치 재설정
• make 로 컴파일시 엉뚱한 위치에 log 디렉토리 생성됨
• 해당 Log Directory 이동 후 심볼 링크 생성

# mv /usr/local/var/log/glusterfs /var/log/ # ln -s /var/log/glusterfs /usr/local/var/log/

3.2. Configuring Cinder to Add GlusterFS

• Stop cinder service

# service cinder-api stop ; \

service cinder-scheduler stop ; \

service cinder-volume stop

• On each Cinder host, run the following commands to add GlusterFS to the Cinder configuration:

# openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver \ cinder.volume.drivers.glusterfs.GlusterfsDriver # openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_shares_config \ /etc/cinder/shares.conf # openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_mount_point_base \ /var/lib/cinder/volumes

> Edit /etc/cinder/cinder.conf

[DEFAULT] # volume_group = cinder-volumes volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver glusterfs_shares_config = /etc/cinder/shares.conf glusterfs_mount_point_base = /var/lib/cinder/volumes

3.3. Creating GlusterFS Volume List

• On each of the Cinder nodes, create a simple text file /etc/cinder/shares.conf.
• This file is a simple list of the GlusterFS volumes to be used, one per line, using the following format:
• HOSTNAME:VOLUME_NAME 형태로 입력
• 입력에 앞서 /etc/hosts 파일에 해당 호스트명 등록되었는지 확인

# vi /etc/cinder/shares.conf GLUSTERHOST:VOLUME GLUSTERHOST:NEXTVOLUME GLUSTERHOST2:SOMEOTHERVOLUME

For example:

  myglusterbox.example.org:myglustervol

3.4. Updating Firewall for GlusterFS

> You must update the firewall rules on each Cinder node to communicate with the GlusterFS nodes.

The ports to open are explained in Step 3:

  http://gluster.org/community/documentation/index.php/Gluster_3.2:_Installing_GlusterFS_on_Red_Hat_Package_Manager_(RPM)_Distributions

If you are using iptables as your firewall, these lines can be added under :OUTPUT ACCEPT in the "*filter" section. You should probably adjust them to suit your environment (eg. only accept connections from your GlusterFS servers).

  -A INPUT -m state --state NEW -m tcp -p tcp --dport 111 -j ACCEPT

  -A INPUT -m state --state NEW -m tcp -p tcp --dport 24007 -j ACCEPT

  -A INPUT -m state --state NEW -m tcp -p tcp --dport 24008 -j ACCEPT

  -A INPUT -m state --state NEW -m tcp -p tcp --dport 24009 -j ACCEPT

  -A INPUT -m state --state NEW -m tcp -p tcp --dport 24010 -j ACCEPT

  -A INPUT -m state --state NEW -m tcp -p tcp --dport 24011 -j ACCEPT

  -A INPUT -m state --state NEW -m tcp -p tcp --dport 38465:38469 -j ACCEPT

Restart the firewall service:

  $ sudo service iptables restart

> 방화벽 port는 위와 같이 적용, Gluster Version 별로 port 정보가 다를수 있으니 확인 필요

3.5. Restarting Cinder Services

3.5.1. Restart the Cinder services

> Configuration is complete and now you must restart the Cinder services to make it active.

$ for i in api scheduler volume; do sudo service openstack-cinder-${i} start; done

OR

$ for a in `ls /etc/init.d/ | grep cinder`;do service $a restart;done

OR

# service cinder-api restart ; \

service cinder-scheduler restart ; \

service cinder-volume restart

3.5.2. Check Error

> Check the Cinder volume log to make sure that there are no errors:

$ sudo tail -50 /var/log/cinder/cinder-volume.log

3.6. Verify GlusterFS Integration with Cinder

> To verify if the installation and configuration is successful, create a Cinder volume then check using GlusterFS.

3.6.1. Create a Cinder volume

# cinder create --display_name myvol 10

+---------------------+--------------------------------------+ |    Property   |             Value              | +---------------------+--------------------------------------+ | attachments |               []               | |  availability_zone  |              nova              | |    bootable   |             false              | |   created_at |   2014-06-27T04:10:08.588923   | | display_description |              None              | | display_name |             myvol              | |       id      | 3f7801e8-b118-4b91-869b-aea2a6f5477c | |    metadata   |               {}               | |      size     |               1                | | snapshot_id |              None              | | source_volid |              None              | |     status    |            creating            | | volume_type |              None              | +---------------------+--------------------------------------+

3.6.2. Check a Cinder list

> Volume creation takes a few seconds. Once created, run the following command:

# cinder list

+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ |               ID               |   Status  | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 3f7801e8-b118-4b91-869b-aea2a6f5477c | available | myvol |  1   | None |  false   |          | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

> The volume should be in "available" status. Now, look for a new file in the GlusterFS volume directory:

$ sudo ls -lah /var/lib/cinder/volumes/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/

(the XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX will be a number specific to your installation)

A newly created file should be inside that directory which is the new volume you just created. A new file will appear each time you create a volume.

For example:

$ ls -alh /var/lib/cinder/volumes/8ab634a36a4925e53a186fd7b3abae51

합계 4.0K

drwxrwxr-x 3 root   cinder   73  6월 27 13:17 .

drwxr-x--- 4 cinder cinder 4.0K  6월 27 12:04 ..

-rw-rw-rw- 1 root   root   1.0G  6월 27 13:10 volume-3f7801e8-b118-4b91-869b-aea2a6f5477c

3.7. Gluster volume

3.7.1. Cinder volume을 위한 glusterfs 설정

• cinder volume에 사용할 gluster volume에 대한 option 정보 수정
• 수정하지 않을 시 instance 생성 후 부팅 시 아래와 같이 에러 발생

[[32m  OK  [0m] Found device /dev/disk/by-uuid/c3dbe26e-e200-496e-af8e-d3071afe1a29.      Starting File System Check on /dev/...e-e200-496e-af8e-d3071afe1a29... [ 0.925628] systemd-fsck[174]: /dev/vda1: clean, 17061/192000 files, 115414/768000 blocks [[32m  OK  [0m] Started File System Check on /dev/d...26e-e200-496e-af8e-d3071afe1a29. [ 1.259661] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 [[32m  OK  [0m] Started dracut initqueue hook.      Mounting /sysroot... [[32m  OK  [0m] Reached target Remote File Systems (Pre). [[32m  OK  [0m] Reached target Remote File Systems. [ 1.478128] end_request: I/O error, dev vda, sector 2050 [ 1.481067] EXT4-fs (vda1): unable to read superblock [[1;31mFAILED[0m] Failed to mount /sysroot. See "systemctl status sysroot.mount" for details. [[1;33mDEPEND[0m] Dependency failed for Initrd Root File System. [[1;33mDEPEND[0m] Dependency failed for Reload Configuration from the Real Root. [[32m  OK  [0m] Stopped dracut pre-pivot and cleanup hook. [[32m  OK  [0m] Stopped target Initrd Default Target. [[32m  OK  [0m] Stopped dracut mount hook. [[32m  OK  [0m] Reached target Initrd File Systems. [[32m  OK  [0m] Stopped target Basic System. [[32m  OK  [0m] Stopped target System Initialization.      Starting Setup Virtual Console... [[32m  OK  [0m] Started Setup Virtual Console.      Starting Emergency Shell... Generating "/run/initramfs/rdsosreport.txt"

• Volume 정보 수정

gluster volume set N02_cinder01 storage.owner-gid 165 ;\ gluster volume set N02_cinder01 storage.owner-uid 165 ;\ gluster volume set N02_cinder01 network.remote-dio enable ;\ gluster volume set N02_cinder01 cluster.eager-lock enable ;\ gluster volume set N02_cinder01 performance.stat-prefetch off ;\ gluster volume set N02_cinder01 performance.io-cache off ;\ gluster volume set N02_cinder01 performance.read-ahead off ;\ gluster volume set N02_cinder01 performance.quick-read off

• URL
• http://blog.gluster.org/category/glusterfs-openstack-cinder/

4. Error

4.1. mount fail with glusterfs

> glusterfs 로 마운트 하려고 하자 실패하여 안되는 현상 발생

>> mount 명령어 실패 확인

# tail /var/log/cinder/cinder-volume.log 2014-06-27 11:59:57.330 2534 INFO cinder.volume.manager [-] Updating volume status 2014-06-27 12:02:04.708 2534 WARNING cinder.volume.drivers.glusterfs [-] Exception during mounting Unexpected error while running command. Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs node2:/cinder_vol /var/lib/cinder/volumes/58e3cafd448b9d31e8f3090cafb561f3 Exit code: 1 Stdout: 'Mount failed. Please check the log file for more details.\n' Stderr: ''

> mount test error check

>> glusterfs 로 mount시 실패하는 현상 확인

# less /var/log/glusterfs/mnt-cinder_vol.log [2014-06-27 02:22:20.122615] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2014-06-27 02:22:20.122640] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2014-06-27 02:22:20.126427] W [socket.c:514:__socket_rwv] 0-glusterfs: readv failed (No data available) [2014-06-27 02:22:20.126443] W [socket.c:1962:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (No data available), peer (115.144.181.36:24007) [2014-06-27 02:22:20.126729] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xf0) [0x7f3bdaae8de0] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7f3bdaae7333] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f3bdaae724e]))) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2014-06-27 02:22:20.124794 (xid=0x1x) [2014-06-27 02:22:20.126744] E [glusterfsd-mgmt.c:1674:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:/cinder_vol) [2014-06-27 02:22:20.126766] W [glusterfsd.c:1002:cleanup_and_exit] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f3bdaae724e] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1e9) [0x7f3bdaae7159] (-->/usr/local/sbin/glusterfs(mgmt_getspec_cbk+0x3ef) [0x40bc5f]))) 0-: received signum (0), shutting down [2014-06-27 02:22:20.126776] I [fuse-bridge.c:5260:fini] 0-fuse: Unmounting '/mnt/cinder_vol'.

> 해결 방안

>> gluster 관련 port 정보를 모두 open

>> 방화벽 상황 점검

4.2. Mount point warning

• glusterN01:cinder_vol 형식이 맞지 않다고 경고
• 형식 바꾸기
• /etc/cinder/shares.conf 변경
• glusterN01:/cinder_vol

2014-07-25 10:09:06.067 25078 INFO cinder.volume.manager [req-7e74fc04-fb3e-441e-9a49-f78b83b3f7b8 - - - - -] Updating volume status 2014-07-25 10:09:06.068 25078 WARNING cinder.volume.drivers.nfs [req-7e74fc04-fb3e-441e-9a49-f78b83b3f7b8 - - - - -] Share glusterN01:cinder_vol ignored due to invalid format.  Must be of form address:/export.

• Openstack icehouse on Ubuntu 14.04 에서 위와 같은 현상이 발생했으며 glusterfs 가 정확히 support 되지 않는것 같다

4.3. Cinder Volume Error

• GlusterFS node 시작 후 OpenStack Cinder Volume 구성 동작 시 구동 순서가 잘못된 경우 에러가 발생
• GlusterFS node 기동 후 서비스 확인
• # service glusterd status
• OpenStack Controller Node Start

5. 참고 사항

5.1. URL

> 기본 참조 문서

http://www.gluster.org/community/documentation/index.php/GlusterFS_Cinder


OpenStack Havana on Ubuntu

https://docs.google.com/document/d/1RmGkTaB2XawO0Cvv9wRL0MqVJDM6p28v3hF9qSBWkdU/edit?usp=sharing