Search Results

Search found 242 results on 10 pages for 'mdadm'.

Page 9/10 | < Previous Page | 5 6 7 8 9 10  | Next Page >

• How to stop a Linux LVM volume group?

  - by thkala

  I am currently dealing with a multiple disk failure on a Linux LVM Volume Group that is backed up by a RAID-5 md device. One disk has been taken out completely and another one is showing a limited number of corrupt sectors, due to what seems to have been a misbehaving power supply. The problem is that once an I/O error hits, md takes the array down, since it does not have enough devices for it to be operational. Where md the only one involved, I could mdadm --stop the array and then recreate it to get all devices active again. Unfortunately, the array is a PV in an LVM volume group and I cannot seem to get the kernel to release it. vgchange -an does not seem to do anything, bar spew out a couple of I/O errors. I am obviously missing something, but how in the name of -insert-favorite-deity- do I get LVM to release the underlying PV without rebooting the server?

  Read the article

• Linux software RAID 10 implementation

  - by fabrik

  Hello there! I don't want to force anybody to make it on behalf of me but trust me: i've looked hundreds of sites and i can't find a good starting point for this. I have 4x500Gb HDD's which i want to set up in RAID 10. The most promising description is here, but it's a little old and unclear for me, above all i prefer Debian over Ubuntu (i know there are slight or no differences). Is it possible to build RAID 10 with Debian's installer or i need to build RAID 1 first in the installer then use mdadm later? What is the best practice for building software RAID 10 under Linux (Debian)? Thanks for your time, fabrik

  Read the article

• Commercial NAS RAID1 disks moved to Software Raid system?

  - by Rolnik

  I've got a couple of commercial NAS boxes and I'm wondering if they (ReadyNas duo, DLink DNS-323) or any other NAS is suitable for having their RAIDed disks moved to a software-based NAS. To be specific, I'm a big fan of the (largely) Debian-based Ubuntu. Can the aforementioned NAS drives be migrated to Ubuntu (e.g. using the mdadm Linux command)? Secondly, is there any commercial NAS that can be migrated over? Incidentally, here is a link to somebody who succeeded in a migration: http://www.linuxquestions.org/questions/slackware-14/moving-raid1-drives-into-computer-with-same-md-numbers-862312/ My specific scenario I'd like to prepare for, is the eventual (sudden) death of one of the NAS motherboards.

  Read the article

• Build Advise for Home Web/NAS Server with Ubuntu Server 12.04 [closed]

  - by razor7

  I need to have a personal Webserver with NAS capabilities. The Webserver to test some LAMP projects I develop for clients, and also NAS to be able to stream media to local network. I want to have full control of the box, so I'm planning to build it with some spare parts and Ubuntu Server. The services/software that will run are (remember, is for personal and testing use only): SAMBA/CIFS SSH Server Apache 2 MySQL 5 Mercurial Repo PHP 5.3 Ruby on Rails OwnCloud Dovecot Webmin Postfix PureFTPd ClamAV The Hardware: Intel Dual Core E2180 2.0 GHz MSI P35 Neo Kinkston 1GB DDR2, 667 MSI Nvidia 7300le PCIe x16 256mb RAM HDD SATA WD Green 2TB x2 (RAID-1 with MDADM RAID Controller) 16 GB USB Pendrive (For server system installation) My idea is to build this system, using the pendrive for the Ubuntu Server software, and packages, and the RAID-1 for gross data storage. What do you think? Thanks a lot!

  Read the article

• Backup Mongodb on EC2 through EBS snapshots - timing issue

  - by DmitrySemenov

  I'm following this guidance http://docs.mongodb.org/ecosystem/tutorial/backup-and-restore-mongodb-on-amazon-ec2/ I have 4 EBS 1000 IOPS volumes assigned to instance These 4 volumes through MDADM assembled into software RAID10 array. I want to do backups through EBS Snapshots as explained in the article above Question: Mongodb says - that I need to mongo shelldb.runCommand({fsync:1,lock:1}); -- this will lock the db for writing ....run snapshot creation... mongo shell db.$cmd.sys.unlock.findOne(); -- this will unlock the db for writing so do I need to unlock the DB for writing after I issued the comand ec2-create-snapshot or after it's finished and the actual snapshot is created thanks, Dmitry

  Read the article

• Software RAID underneath ESXi datastore

  - by carlpett

  I'm building an virtual environment for a small business. It is based around a single ESXi 5.1 host, which will host half a dozen or so VMs. I'm having some doubts regarding how to implement the storage though. I naturally want the datastore to be fault tolerant, but I can't get the funds for a separate storage machine, nor expensive hardware RAID solutions, so I would like to use some software RAID (lvm/mdadm, most likely). How can this be implemented? My only idea so far would be to create a VM which has the storage adapter as passthrough, puts some software RAID on top of the disks and then presents the resulting volumes "back" to the ESXi host which then creates a datastore from which other VMs get their storage presented. This does seem kind of round-about, do I have any better options? From my research, passthrough seems to come with quite a few drawbacks, such as no suspend/resume etc.

  Read the article

• How can I create a 4TB partition on my software RAID5 device?

  - by Kris Harper

  I have set up a RAID5 device with three 2TB hard drives using mdadm. The device was successfully created, but I cannot seem to create a partition on the device. When I try to make an ext3 or ext4 partition via Disk Utility, I get the following error Error creating partition: helper exited with exit code 1: In part_add_partition: device_file=/dev/md0, start=0, size=4000526106624, type= Entering MS-DOS parser (offset=0, size=4000526106624) MSDOS_MAGIC found found partition type 0xee => protective MBR for GPT Exiting MS-DOS parser Entering EFI GPT parser GPT magic found partition_entry_lba=2 num_entries=128 size_of_entry=128 Leaving EFI GPT parser EFI GPT partition table detected containing partition table scheme = 3 got it got disk new partition guid '' is not valid type '' for GPT appear to be malformed I have seen this question, but that seems to suggest using gparted to do the partitioning. I'm fine with doing that, but my RAID device doesn't show up in the list of gparted devices. I suspect because this is a RAID and not a regular disk. I have already created a GPT partition table on the device. How can I add a partition to my device?

  Read the article

• RAID5 over LVM on Ubuntu Server 12.04.3

  - by April Ethereal

  I'm trying to create a RAID5 software array using LVM. I use VirtualBox as I'm only learning how LVM works. So I've created 4 virtual SCSI drives and then did the following: pvcreate /dev/sd[b-e] vgcreate /dev/sd[b-e] raid5_vg lvcreate --type raid5 -i 3 -L 1G -n raid_lv raid5_vg However, I get an error after the last command: WARNING: Unrecognised segment type raid5 Using default stripesize 64.00 KiB Rounding size (256 extents) up to stripe boundary size (258 extents) Cannot update volume group raid5_vg with unknown segments in it! So it looks like raid5 is not a valid segment type. "lvm segtypes" also doesn't contain 'raid5' entry: root@ubuntu-lvm:~# lvm segtypes striped zero error free snapshot mirror So my question is - how could I create RAID5 logical volume using LVM only? It seems that it is possible, I saw a few references (not for Ubuntu, unfortunately) for RedHat and Gentoo systems. I don't want to use mdadm for now, until I find out that it is mandatory. Some info about my system is below: root@ubuntu-lvm:~# uname -a Linux ubuntu-lvm 3.8.0I use Ubuntu Server 12.04.3 (i686)-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 15:31:16 UTC 2013 i686 i686 i386 GNU/Linux root@ubuntu-lvm:~# dpkg -l | grep lvm ii lvm2 2.02.66-4ubuntu7.3 The Linux Logical Volume Manager Thanks.

  Read the article

• How to mount an external HDD?

  - by Slash

  I have Ubuntu Linux 12.04 version the latest right now.I want to mount an external HDD NTFS 1TB.I have followed many guides but still no success.The error I'm getting is this: Failed to read last sector (1953523119): Invalid argument HINTS: Either the volume is a RAID/LDM but it wasn't setup yet, or it was not setup correctly (e.g. by not using mdadm --build ...), or a wrong device is tried to be mounted, or the partition table is corrupt (partition is smaller than NTFS), or the NTFS boot sector is corrupt (NTFS size is not valid). Failed to mount '/dev/sdb1': Invalid argument The device '/dev/sdb1' doesn't seem to have a valid NTFS. Maybe the wrong device is used? Or the whole disk instead of a partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around? Using Storage Device MAnager i get this error:Error mounting: mount exited with exit code 1: helper failed with: mount: only root can mount /dev/sdb1 on /media/Skliros_Diskos {external disk name} When I use sudo fdisk -l, this is the output: Disk /dev/sda: 320.1 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders, total 625142448 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000e0bc6 Device Boot Start End Blocks Id System /dev/sda1 * 2048 618854399 309426176 83 Linux /dev/sda2 618856446 625141759 3142657 5 Extended /dev/sda5 618856448 625141759 3142656 82 Linux swap / Solaris Disk /dev/sdb: 1000.2 GB, 1000202043392 bytes 255 heads, 63 sectors/track, 121600 cylinders, total 1953519616 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0002093a Device Boot Start End Blocks Id System /dev/sdb1 2048 1953525167 976761560 7 HPFS/NTFS/exFAT

  Read the article

• Is a disk/ata timeout exception dangerous?

  - by j-g-faustus

  I have a few hard drives in mdadm RAID 5 configured to go to standby after a few minutes of inactivity. (Using hdparm.conf spindown_time.) At irregular intervals I get messages like these in dmesg: [ 1840.251661] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 1840.251722] ata4.00: failed command: SMART [ 1840.251758] ata4.00: cmd b0/d5:01:06:4f:c2/00:00:00:00:00/00 tag 0 pio 512 in [ 1840.251759] res 40/00:14:50:2e:04/00:00:02:00:00/40 Emask 0x4 (timeout) [ 1840.251858] ata4.00: status: { DRDY } [ 1840.251888] ata4: hard resetting link [ 1840.600742] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1840.601521] ata4.00: configured for UDMA/133 [ 1840.601547] ata4: EH complete [337877.713988] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [337877.714019] ata4.00: failed command: SMART [337877.714038] ata4.00: cmd b0/d5:01:06:4f:c2/00:00:00:00:00/00 tag 0 pio 512 in [337877.714039] res 40/00:04:90:10:81/00:00:00:00:00/40 Emask 0x4 (timeout) [337877.714089] ata4.00: status: { DRDY } [337877.714107] ata4: hard resetting link [337878.063085] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [337878.063743] ata4.00: configured for UDMA/133 [337878.063764] ata4: EH complete I think the exception is caused by smartd when a drive does not wake up quickly enough. There are no issues (that I can tell) in accessing the drives normally through the file system - it takes a few seconds longer than normal when they are asleep, but there are no exceptions. Is this something I should worry about, as a potential symptom on something that could corrupt a drive over time? Or can I safely ignore it as part of normal operation? Edit: By request: smartctl -a for sdaand sde, both disks are members of the array. If ata4is the same as scsi-4 then sde is the one that gave the error above, according to /dev/disk/by-path.

  Read the article

• Unable to mount NTFS Partition after resizing

  - by sam

  I was having only 15 GB space allocated to LINUX. I wanted to have more space available to linux. So I just re sized one of my ntfs partition using GParted. But after resizing I am not able to open the partition neither in Ubuntu nor in windows. OS: Dual Boot Win7/Ubuntu 10.10 The error message i get is the following: Error mounting: mount exited with exit code 12: Failed to read last sector (395458824): Invalid argument HINTS: Either the volume is a RAID/LDM but it wasn't setup yet, or it was not setup correctly (e.g. by not using mdadm --build ...), or a wrong device is tried to be mounted, or the partition table is corrupt (partition is smaller than NTFS), or the NTFS boot sector is corrupt (NTFS size is not valid). Failed to mount '/dev/sda5': Invalid argument The device '/dev/sda5' doesn't seem to have a valid NTFS. Maybe the wrong device is used? Or the whole disk instead of a partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around?

  Read the article

• How to switch to a generic kernel in a headless Ubuntu Server 12.04?

  - by chmike

  I just got a dedicated server with Ubuntu 12.04 installed with a custom compiled kernel. Since I would like to install VirtualBox and this custom kernel doesn't support dynamic module loading (for security) I need to change the kernel. I'm running some Ubuntu servers for years but never palyed with grub and a headless computer. When the command update-grub is run it shows the different kernel it finds. Here is what I see Generating grub.cfg ... Found linux image: /boot/bzImage-3.2.13-xxxx-grs-ipv6-64 Found linux image: /boot/vmlinuz-3.2.0-34-generic Found initrd image: /boot/initrd.img-3.2.0-34-generic No volume groups found done The first one is the active one as seen with uname -r. To me it looks like the second kernel is the one I should use. But I don't know how to configure grub2 to use it. The computer is also configured with a software RAID using mdadm I guess. Never used that before. I don't know if playing with the grub of changing kernel could brake this. What must I do to set the generic kernel as the default one so that I can get VirtualBox running.

  Read the article

• preseeded installation keeps asking for confirmation while creating RAID-Partitions on certain hardware-platform

  - by Marc Shennon

  I am aware of the partman-md/confirm_nooverwrite thing, that was the solution to most of this problems in the past. The thing is, that the preseed-file works for almost all hardware-platforms I tested, but only for one (Primergy MX130) it keeps asking for confirmation, before writing the partition-layout to the disks. All machines I tested are running with two SATA Disks, nothing special. I'm not really sure, what information could be needed in order to investigate the cause of this behaviour, but I would of course be willing to provide more information, if someone has an idea. Relevant part of the preseed file is the following: d-i partman-auto/disk string /dev/sda /dev/sdb d-i partman-auto/method string raid d-i partman-md/confirm boolean true d-i partman-partitioning/confirm_write_new_label boolean true d-i partman-md/device_remove_md boolean true d-i partman/choose_partition select finish d-i partman-md/confirm_nooverwrite boolean true # Write the changes to disks? d-i partman/confirm boolean true d-i mdadm/boot_degraded boolean true # RECIPE # Next you need to specify the physical partitions that will be used. d-i partman-auto/expert_recipe string \ multiraid :: \ 500 10000 1000000000 raid $lvmignore{ }\ $primary{ } \ method{ raid } \ . \ 512 1000 786 raid $lvmignore{ }\ $primary{ } \ method{ raid } \ . \ 8192 10240 10240 raid $lvmignore{ }\ method{ raid } \ . # Parameters are: # <raidtype> <devcount> <sparecount> <fstype> <mountpoint> <devices> <sparedevices> d-i partman-auto-raid/recipe string \ 1 2 0 ext4 / /dev/sda1#/dev/sdb1 . \ 1 2 0 ext2 /boot /dev/sda2#/dev/sdb2 . \ 1 2 0 swap - /dev/sda5#/dev/sdb5 .

  Read the article

• HighPoint RocketRAID 62x Controller

  - by TeXnewbie

  I have the subject card recently installed in Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-31-generic x86_64). See partial lspci -vv listing below (complete listing played havoc with pre tags): 03:00.0 RAID bus controller: HighPoint Technologies, Inc. Device 0622 (rev 01) Subsystem: HighPoint Technologies, Inc. Device 0001 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 11 Region 0: I/O ports at 9c00 [size=8] Region 1: I/O ports at 9800 [size=4] Region 2: I/O ports at 9400 [size=8] Region 3: I/O ports at 9000 [size=4] Region 4: I/O ports at 8c00 [size=16] Region 5: Memory at fdbff000 (32-bit, non-prefetchable) [size=2K] Expansion ROM at fdbe0000 [disabled] [size=64K] Capabilities: I followed instructions I found at https://help.ubuntu.com/community/RocketRaid to compile the drivers for it, and although performing the process described there seemed to work fine with no noticeable errors, when I rebooted after performing that procedure I could not boot. During dkms steps, I noticed messages indicating that (If next boot fails, revert to initrd.img-3.2.0-31-generic.old-dkms image) update-initramfs................ so I booted using a Ubuntu 12.10 LiveDVD and reverted to the old-dkms initrd.img as suggested above, but this failed to repair the boot problem. Ultimately, I used https://help.ubuntu.com/community/Boot-Repair in Ubuntu-Secure-Remix to fix the boot problem and was able to boot normally again, but now with the newly generated initrd.img in place again (which now boots normally), when I modprobe the rr62x kernel module, I immediately get a hard crash with messages to console about a kernel paging request that seems to have caused the problem. I've tried on multiple occasions now to use the newly built kernel module so as to allow me to use an eSATA port multiplier plugged into the card, but to no avail. Any suggestions on fixes or workarounds (I've read that some of the HighPoint cards (2720SGL) seem to work as a host bus adapter and thus may not need a custom driver, but that seems not to be the case for mine) would be most appreciated. My goal is to use the card as described here and with software RAID mdadm utilities. If necessary, I can hand-copy the console messages after the hard crash into a follow-up message, but I obviously can't do a cut/paste. I'll gladly provide any other details that are needed, but not sure what those would be at this point, so I'll refrain from adding other details for now. Thanks in advance for any help.

  Read the article

• NFS server generating "invalid extent" on EXT4 system disk?

  - by Stephen Winnall

  I have a server running Xen 4.1 with Oneiric in the dom0 and each of the 4 domUs. The system disks of the domUs are LVM2 volumes built on top of an mdadm RAID1. All the domU system disks are EXT4 and are created using snapshots of the same original template. 3 of them run perfectly, but one (called s-ub-02) keeps on being remounted read-only. A subsequent e2fsck results in a single "invalid extent" diagnosis: e2fsck 1.41.14 (22-Dec-2010) /dev/domu/s-ub-02-root contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Inode 525418 has an invalid extent (logical block 8959, invalid physical block 0, len 0) Clear<y>? yes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/domu/s-ub-02-root: 77757/655360 files (0.3% non-contiguous), 360592/2621440 blocks The console shows typically the following errors for the system disk (xvda2): [101980.903416] EXT4-fs error (device xvda2): ext4_ext_find_extent:732: inode #525418: comm apt-get: bad header/extent: invalid extent entries - magic f30a, entries 12, max 340(340), depth 0(0) [101980.903473] EXT4-fs (xvda2): Remounting filesystem read-only I have created new versions of the system disk. The same thing always happens. This, and the fact that the disk is ultimately on a RAID1, leads me to preclude a hardware disk error. The only obvious distinguishing feature of this domU is the presence of nfs-kernel-server, so I suspect that. Its exports file looks like this: /exports/users 192.168.0.0/255.255.248.0(rw,sync,no_subtree_check) /exports/media/music 192.168.0.0/255.255.248.0(rw,sync,no_subtree_check) /exports/media/pictures 192.168.0.0/255.255.248.0(rw,sync,no_subtree_check) /exports/opt 192.168.0.0/255.255.248.0(rw,sync,no_subtree_check) /exports/users and /exports/opt are LVM2 volumes from the same volume group as the system disk. /exports/media is an EXT2 volume. (There is an issue where clients see /exports/media/pictures as being a read-only volume, which I mention for completeness.) With the exception of the read-only problem, the NFS server appears to work correctly under light load for several hours before the "invalid extent" problem occurs. There are no helpful entries in /var/log. All of a sudden, no more files are written, so you can see when the disk was remounted read-only, but there is no indication of what the cause might be. Can anyone help me with this problem? Steve

  Read the article

• SATA drives or chipset throwing DRDY ERR and ICRC ABRT

  - by Matt

  I have an SD-VIA-1A2S PCI card with 2 sata ports (and one ATA-133 that isn't used). Two new Western Digital Caviar Green drives (WD10EARS 1TB) throw repeated errors in kern.log (removed date/time/host info for brevity): [ 7.376475] ata2.00: exception Emask 0x12 SAct 0x0 SErr 0x1000500 action 0x6 [ 7.376480] ata2.00: BMDMA stat 0x5 [ 7.376483] ata2: SError: { UnrecovData Proto TrStaTrns } [ 7.376489] ata2.00: cmd c8/00:40:20:00:00/00:00:00:00:00/e0 tag 0 dma 32768 in [ 7.376490] res 51/84:2f:20:00:00/00:00:00:00:00/e0 Emask 0x12 (ATA bus error) [ 7.376493] ata2.00: status: { DRDY ERR } [ 7.376495] ata2.00: error: { ICRC ABRT } [ 7.376504] ata2: hard resetting link I'm using Ubuntu 9.04 - 2.6.28-18-generic, though I have tried live cds of Ubuntu 9.10, Fedora 12 and OpenSUSE 11.2 - all running various 2.6.31 kernels - and all received the same error. Based on testing these drives and this card in two other machines and combos of connecting the drives directly to the motherboard or the add-in card, I'm relatively convinced that it's the VIA chipset that is the problem. Another computer that also has an onboard VIA SATA chipset (like the add-in card) produces the same errors when the drives are directly on that motherboard. I have been able to verify that the drives are perfectly good, and I tried everything I can think of in terms of swapping cables, psu isn't overloaded, etc. The error happens on boot once or twice, after using fdisk on the drive once or twice, and constantly when attempting to sync a new mdadm raid 1 array created on the two drives. Any thoughts on where to go from here - driver/kernel wise? I'm completely open to buying a new PCI add-in card if someone can recommend one with 2 internal sata ports that works well in Debian/Ubuntu. Thanks!

  Read the article

• high load average, high wait, dmesg raid error messages (debian nfs server)

  - by John Stumbles

  Debian 6 on HP proliant (2 CPU) with raid (2*1.5T RAID1 + 2*2T RAID1 joined RAID0 to make 3.5T) running mainly nfs & imapd (plus samba for windows share & local www for previewing web pages); with local ubuntu desktop client mounting $HOME, laptops accessing imap & odd files (e.g. videos) via nfs/smb; boxes connected 100baseT or wifi via home router/switch uname -a Linux prole 2.6.32-5-686 #1 SMP Wed Jan 11 12:29:30 UTC 2012 i686 GNU/Linux Setup has been working for months but prone to intermittently going very slow (user experience on desktop mounting $HOME from server, or laptop playing videos) and now consistently so bad I've had to delve into it to try to find what's wrong(!) Server seems OK at low load e.g. (laptop) client (with $HOME on local disk) connecting to server's imapd and nfs mounting RAID to access 1 file: top shows load ~ 0.1 or less, 0 wait but when (desktop) client mounts $HOME and starts user KDE session (all accessing server) then top shows e.g. top - 13:41:17 up 3:43, 3 users, load average: 9.29, 9.55, 8.27 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 0.4%us, 0.4%sy, 0.0%ni, 49.0%id, 49.7%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 903856k total, 851784k used, 52072k free, 171152k buffers Swap: 0k total, 0k used, 0k free, 476896k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3935 root 20 0 2456 1088 784 R 2 0.1 0:00.02 top 1 root 20 0 2028 680 584 S 0 0.1 0:01.14 init 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0 4 root 20 0 0 0 0 S 0 0.0 0:00.12 ksoftirqd/0 5 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1 7 root 20 0 0 0 0 S 0 0.0 0:00.16 ksoftirqd/1 8 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1 9 root 20 0 0 0 0 S 0 0.0 0:00.42 events/0 10 root 20 0 0 0 0 S 0 0.0 0:02.26 events/1 11 root 20 0 0 0 0 S 0 0.0 0:00.00 cpuset 12 root 20 0 0 0 0 S 0 0.0 0:00.00 khelper 13 root 20 0 0 0 0 S 0 0.0 0:00.00 netns 14 root 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr 15 root 20 0 0 0 0 S 0 0.0 0:00.00 pm 16 root 20 0 0 0 0 S 0 0.0 0:00.02 sync_supers 17 root 20 0 0 0 0 S 0 0.0 0:00.02 bdi-default 18 root 20 0 0 0 0 S 0 0.0 0:00.00 kintegrityd/0 19 root 20 0 0 0 0 S 0 0.0 0:00.00 kintegrityd/1 20 root 20 0 0 0 0 S 0 0.0 0:00.02 kblockd/0 21 root 20 0 0 0 0 S 0 0.0 0:00.08 kblockd/1 22 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpid 23 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpi_notify 24 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpi_hotplug 25 root 20 0 0 0 0 S 0 0.0 0:00.00 kseriod 28 root 20 0 0 0 0 S 0 0.0 0:04.19 kondemand/0 29 root 20 0 0 0 0 S 0 0.0 0:02.93 kondemand/1 30 root 20 0 0 0 0 S 0 0.0 0:00.00 khungtaskd 31 root 20 0 0 0 0 S 0 0.0 0:00.18 kswapd0 32 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd 33 root 20 0 0 0 0 S 0 0.0 0:00.00 aio/0 34 root 20 0 0 0 0 S 0 0.0 0:00.00 aio/1 35 root 20 0 0 0 0 S 0 0.0 0:00.00 crypto/0 36 root 20 0 0 0 0 S 0 0.0 0:00.00 crypto/1 203 root 20 0 0 0 0 S 0 0.0 0:00.00 ksuspend_usbd 204 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd 205 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/0 206 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/1 207 root 20 0 0 0 0 S 0 0.0 0:00.14 ata_aux 208 root 20 0 0 0 0 S 0 0.0 0:00.01 scsi_eh_0 dmesg suggests there's a disk problem: .............. (previous episode) [13276.966004] raid1:md0: read error corrected (8 sectors at 489900360 on sdc7) [13276.966043] raid1: sdb7: redirecting sector 489898312 to another mirror [13279.569186] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13279.569211] ata4.00: irq_stat 0x40000008 [13279.569230] ata4.00: failed command: READ FPDMA QUEUED [13279.569257] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13279.569262] res 41/40:00:05:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13279.569306] ata4.00: status: { DRDY ERR } [13279.569321] ata4.00: error: { UNC } [13279.575362] ata4.00: configured for UDMA/133 [13279.575388] ata4: EH complete [13283.169224] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13283.169246] ata4.00: irq_stat 0x40000008 [13283.169263] ata4.00: failed command: READ FPDMA QUEUED [13283.169289] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13283.169294] res 41/40:00:07:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13283.169331] ata4.00: status: { DRDY ERR } [13283.169345] ata4.00: error: { UNC } [13283.176071] ata4.00: configured for UDMA/133 [13283.176104] ata4: EH complete [13286.224814] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13286.224837] ata4.00: irq_stat 0x40000008 [13286.224853] ata4.00: failed command: READ FPDMA QUEUED [13286.224879] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13286.224884] res 41/40:00:06:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13286.224922] ata4.00: status: { DRDY ERR } [13286.224935] ata4.00: error: { UNC } [13286.231277] ata4.00: configured for UDMA/133 [13286.231303] ata4: EH complete [13288.802623] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13288.802646] ata4.00: irq_stat 0x40000008 [13288.802662] ata4.00: failed command: READ FPDMA QUEUED [13288.802688] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13288.802693] res 41/40:00:05:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13288.802731] ata4.00: status: { DRDY ERR } [13288.802745] ata4.00: error: { UNC } [13288.808901] ata4.00: configured for UDMA/133 [13288.808927] ata4: EH complete [13291.380430] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13291.380453] ata4.00: irq_stat 0x40000008 [13291.380470] ata4.00: failed command: READ FPDMA QUEUED [13291.380496] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13291.380501] res 41/40:00:05:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13291.380577] ata4.00: status: { DRDY ERR } [13291.380594] ata4.00: error: { UNC } [13291.386517] ata4.00: configured for UDMA/133 [13291.386543] ata4: EH complete [13294.347147] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13294.347169] ata4.00: irq_stat 0x40000008 [13294.347186] ata4.00: failed command: READ FPDMA QUEUED [13294.347211] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13294.347217] res 41/40:00:06:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13294.347254] ata4.00: status: { DRDY ERR } [13294.347268] ata4.00: error: { UNC } [13294.353556] ata4.00: configured for UDMA/133 [13294.353583] sd 3:0:0:0: [sdc] Unhandled sense code [13294.353590] sd 3:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [13294.353599] sd 3:0:0:0: [sdc] Sense Key : Medium Error [current] [descriptor] [13294.353610] Descriptor sense data with sense descriptors (in hex): [13294.353616] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 [13294.353635] 23 05 6a 06 [13294.353644] sd 3:0:0:0: [sdc] Add. Sense: Unrecovered read error - auto reallocate failed [13294.353657] sd 3:0:0:0: [sdc] CDB: Read(10): 28 00 23 05 6a 00 00 00 08 00 [13294.353675] end_request: I/O error, dev sdc, sector 587557382 [13294.353726] ata4: EH complete [13294.366953] raid1:md0: read error corrected (8 sectors at 489900544 on sdc7) [13294.366992] raid1: sdc7: redirecting sector 489898496 to another mirror and they're happening quite frequently, which I guess is liable to account for the performance problem(?) # dmesg | grep mirror [12433.561822] raid1: sdc7: redirecting sector 489900464 to another mirror [12449.428933] raid1: sdb7: redirecting sector 489900504 to another mirror [12464.807016] raid1: sdb7: redirecting sector 489900512 to another mirror [12480.196222] raid1: sdb7: redirecting sector 489900520 to another mirror [12495.585413] raid1: sdb7: redirecting sector 489900528 to another mirror [12510.974424] raid1: sdb7: redirecting sector 489900536 to another mirror [12526.374933] raid1: sdb7: redirecting sector 489900544 to another mirror [12542.619938] raid1: sdc7: redirecting sector 489900608 to another mirror [12559.431328] raid1: sdc7: redirecting sector 489900616 to another mirror [12576.553866] raid1: sdc7: redirecting sector 489900624 to another mirror [12592.065265] raid1: sdc7: redirecting sector 489900632 to another mirror [12607.621121] raid1: sdc7: redirecting sector 489900640 to another mirror [12623.165856] raid1: sdc7: redirecting sector 489900648 to another mirror [12638.699474] raid1: sdc7: redirecting sector 489900656 to another mirror [12655.610881] raid1: sdc7: redirecting sector 489900664 to another mirror [12672.255617] raid1: sdc7: redirecting sector 489900672 to another mirror [12672.288746] raid1: sdc7: redirecting sector 489900680 to another mirror [12672.332376] raid1: sdc7: redirecting sector 489900688 to another mirror [12672.362935] raid1: sdc7: redirecting sector 489900696 to another mirror [12674.201177] raid1: sdc7: redirecting sector 489900704 to another mirror [12698.045050] raid1: sdc7: redirecting sector 489900712 to another mirror [12698.089309] raid1: sdc7: redirecting sector 489900720 to another mirror [12698.111999] raid1: sdc7: redirecting sector 489900728 to another mirror [12698.134006] raid1: sdc7: redirecting sector 489900736 to another mirror [12719.034376] raid1: sdc7: redirecting sector 489900744 to another mirror [12734.545775] raid1: sdc7: redirecting sector 489900752 to another mirror [12734.590014] raid1: sdc7: redirecting sector 489900760 to another mirror [12734.624050] raid1: sdc7: redirecting sector 489900768 to another mirror [12734.647308] raid1: sdc7: redirecting sector 489900776 to another mirror [12734.664657] raid1: sdc7: redirecting sector 489900784 to another mirror [12734.710642] raid1: sdc7: redirecting sector 489900792 to another mirror [12734.721919] raid1: sdc7: redirecting sector 489900800 to another mirror [12734.744732] raid1: sdc7: redirecting sector 489900808 to another mirror [12734.779330] raid1: sdc7: redirecting sector 489900816 to another mirror [12782.604564] raid1: sdb7: redirecting sector 1242934216 to another mirror [12798.264153] raid1: sdc7: redirecting sector 1242935080 to another mirror [13245.832193] raid1: sdb7: redirecting sector 489898296 to another mirror [13261.376929] raid1: sdb7: redirecting sector 489898304 to another mirror [13276.966043] raid1: sdb7: redirecting sector 489898312 to another mirror [13294.366992] raid1: sdc7: redirecting sector 489898496 to another mirror although the arrays are still running on all disks - they haven't given up on any yet: # cat /proc/mdstat Personalities : [raid1] [raid0] md10 : active raid0 md0[0] md1[1] 3368770048 blocks super 1.2 512k chunks md1 : active raid1 sde2[2] sdd2[1] 1464087824 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdb7[0] sdc7[2] 1904684920 blocks super 1.2 [2/2] [UU] unused devices: <none> So I think I have some idea what the problem is but I am not a linux sysadmin expert by the remotest stretch of the imagination and would really appreciate some clue checking here with my diagnosis and what do I need to do: obviously I need to source another drive for sdc. (I'm guessing I could buy a larger drive if the price is right: I'm thinking that one day I'll need to grow the size of the array and that would be one less drive to replace with a larger one) then use mdadm to fail out the existing sdc, remove it and fit the new drive fdisk the new drive with the same size partition for the array as the old one had use mdadm to add the new drive into the array that sound OK?

  Read the article

• Preseeding Ubuntu partman recipe using LVM and RAID

  - by Swav

  I'm trying to preseed Ubuntu 12.04 server installation and created a recipe that would create RAID 1 on 2 drives and then partition that using LVM. Unfortunately partman complains when creating LVM volumes saying there no partitions in recipe that could be used with LVM (in console it complains about unusable recipe). The layout I'm after is RAID 1 on sdb and sdc (installing from USB stick so it takes sda) and then use LVM to create boot, root and swap. The odd thing is that if I change the mount point of boot_lv to home the recipe works fine (apart from mounting in wrong place), but when mounting at /boot it fails I know I could use separate /boot primary partition, but can anybody tell me why it fails. Recipe and relevant options below. ## Partitioning using RAID d-i partman-auto/disk string /dev/sdb /dev/sdc d-i partman-auto/method string raid d-i partman-lvm/device_remove_lvm boolean true d-i partman-md/device_remove_md boolean true #d-i partman-lvm/confirm boolean true d-i partman-auto-lvm/new_vg_name string main_vg d-i partman-auto/expert_recipe string \ multiraid :: \ 100 512 -1 raid \ $lvmignore{ } \ $primary{ } \ method{ raid } \ . \ 256 512 256 ext3 \ $defaultignore{ } \ $lvmok{ } \ method{ format } \ format{ } \ use_filesystem{ } \ filesystem{ ext3 } \ mountpoint{ /boot } \ lv_name{ boot_lv } \ . \ 2000 5000 -1 ext4 \ $defaultignore{ } \ $lvmok{ } \ method{ format } \ format{ } \ use_filesystem{ } \ filesystem{ ext4 } \ mountpoint{ / } \ lv_name{ root_lv } \ . \ 64 512 300% linux-swap \ $defaultignore{ } \ $lvmok{ } \ method{ swap } \ format{ } \ lv_name{ swap_lv } \ . d-i partman-auto-raid/recipe string \ 1 2 0 lvm - \ /dev/sdb1#/dev/sdc1 \ . d-i mdadm/boot_degraded boolean true #d-i partman-md/confirm boolean true #d-i partman-partitioning/confirm_write_new_label boolean true #d-i partman/choose_partition select Finish partitioning and write changes to disk #d-i partman/confirm boolean true #d-i partman-md/confirm_nooverwrite boolean true #d-i partman/confirm_nooverwrite boolean true EDIT: After a bit of googling I found below snippet of code from partman-auto-lvm, but I still don't understand why would they prevent that setup if it's possible to do manually and booting from boot partition on LVM is possible. # Make sure a boot partition isn't marked as lvmok if echo "$scheme" | grep lvmok | grep -q "[[:space:]]/boot[[:space:]]"; then bail_out unusable_recipe fi

  Read the article

• Ubuntu raid 1 write errors

  - by Micah

  I have an Ubuntu server set up with two SATA drives in a RAID 1 configuration with MDADM. The machine is used to record raw video, which involves a lot of writing to the disk. Sometimes during video recording the computer will crash, will the following errors in kern.log: Mar 15 10:39:41 video kernel: [414501.629864] ata2.00: exception Emask 0x10 SAct 0x0 SErr 0x400100 action 0x6 Mar 15 10:39:41 video kernel: [414501.629870] ata2.00: BMDMA stat 0x26 Mar 15 10:39:41 video kernel: [414501.629875] ata2.00: SError: { UnrecovData Handshk } Mar 15 10:39:41 video kernel: [414501.629880] ata2.00: failed command: WRITE DMA EXT Mar 15 10:39:41 video kernel: [414501.629889] ata2.00: cmd 35/00:00:28:6d:f6/00:04:06:00:00/e0 tag 0 dma 524288 out Mar 15 10:39:41 video kernel: [414501.629891] res 51/84:b1:77:6e:f6/84:02:06:00:00/e0 Emask 0x30 (host bus error) Mar 15 10:39:41 video kernel: [414501.629896] ata2.00: status: { DRDY ERR } Mar 15 10:39:41 video kernel: [414501.629899] ata2.00: error: { ICRC ABRT } Mar 15 10:39:41 video kernel: [414501.629910] ata2.00: hard resetting link Mar 15 10:39:41 video kernel: [414501.973009] ata2.01: hard resetting link Mar 15 10:39:41 video kernel: [414502.482642] ata2.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Mar 15 10:39:41 video kernel: [414502.482658] ata2.01: SATA link down (SStatus 0 SControl 300) Mar 15 10:39:41 video kernel: [414502.546160] ata2.00: configured for UDMA/133 Mar 15 10:39:41 video kernel: [414502.546203] ata2: EH complete Is this the result of faulty drives? Is software RAID just not performant enough for data rates ~15 MB/s, even with a quad-core i7? Thanks for your help. Edit: cat /proc/mdstat returns this: Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid1 sdb1[1] sda1[0] 976760768 blocks [2/2] [UU] unused devices: <none>

  Read the article

• Ubuntu 10.04 - Add RAID 1 Array?

  - by N Rahl

  I have an existing Ubuntu 10.04 desktop system setup and running on a hard drive (Drive A). I'd like to add 2 more hard drives (Drives B & C, same size) to the system and mount them as a RAID 1 array. How do I do that? I know how to create RAID arrays during the installation, but I don't want to reinstall my system, and I shouldn't have to since my system files will stay on their own drive separate from the RAID array. I've physically added both drives to the system, and formatted them as EXT3 with gparted. Ubuntu's disk utility has a "create raid" option, but it won't let me select any of my drives (it thinks they're all full). I don't mind using mdadm, but I've found several guides that are old, and give conflicting advice. Some say I have to edit an /etc/raidtab file, some say this is done automatically. So what's the current (Ubuntu 10.04) preferred way of adding a RAID 1 to an existing system? It should turn into a raid at boot, and mount itself in /home/myname/files/. Thanks!

  Read the article

• How do I diagnose a bottleneck in an Intel Atom based Ubuntu server?

  - by Jon Cage

  I have a small media server at home which has software raid and a gigabit link to the rest of my network. For some reason though, I only get ~10MB/s transfers when copying to/from the server. I use software RAID5 (mdadm) over 4 1TB disks. On top of that I then use LVM to give me a huge pool of disk space which is then split up into multiple partitions which can be resized as and when they need it. I'm guessing this it most likely the cause, but I'd like to know for sure where the root cause is. So, how can I benchmark network throughput (Windows 7 desktop <- Ubuntu server) and hard disk performance to try and identify where my bottleneck might be? [Edit] If anyone's interested, the motherboard is an Intel Desktop Board D945GCLF2. So that's a 300 series Atom processor with the Intel® 945GC Express Chipset [Edit2] I feel like such a fool! I just checked my desktop and I had the slower of the two onboard NICs plugged in so the server is probably not at fault here. Transferring a copy of ubuntu off the server I get ~35-40MB/s according to Windows 7. I'll do those HD tests when I get a chance though (just for completeness).

  Read the article

• My client's solution of a Windows SBS 2011 VM on an Ubuntu host and VirtualBox is pinning the host CPU

  - by Scott Stamp

  Here's my situation, I've got a client hosting two servers (one VM), with the host providing VMware Zimbra, the other Windows Small Business Server 2011. Unfortunately, the person before me had configured this setup as follows. Host: Ubuntu Desktop Edition 10.04 (I know, again, not my choice) running VMware Zimbra 8GB of RAM On-board RAID1 of two 320GB Seagate Barracuda drives for the OS Software RAID5 of four 500GB WD Caviar Black drives on MDADM for bulk storage (sorry, I don't know the model #) A relatively competent quad-core Intel Core i7 CPU from the Nehalem architecture (not suspicious of this as the bottleneck) Guest: Windows Small Business Server 2011 4GB of RAM Host-equivalent CPU allocation VDI file for OS hosted on the on-board RAID, VDI file for storage hosted on the on-board RAID For some reason when running, the VM locks up when sitting nearly idle, and the VirtualBox process reports values of 240%+ in top (how is that even possible?!). Anyone have any ideas or suggestions? I'm totally stumped on this one. Happy to provide whatever logs you'd like to take a look at. Ideally I'd drop VirtualBox and provision this with VMware Workstation, but the client has objected to the (very nominal) costs involved. If hardware needs to be purchased to help, it will be, but we're considering upgrades a last-resort at this time. Thanks in advance! *fingers crossed*

  Read the article

• Converting software RAID1 to RAID10 for /boot

  - by luckytaxi

  Array info: /dev/md0 - /dev/sda1 and /dev/sdb1 /dev/md2 - /dev/sda2 and /dev/sdb2 Partition info: /boot - /dev/md0 / - /dev/md1 I have two drives that are setup as RAID1 using software RAID on Redhat. I added two additional drives (same size) and I would like to conver the RAID1 to a RAID10. The problem I'm having is adding the last drive to the array. I've gotten as far as creating a RAID10 with two missing devices but as soon as I add the last drive, all hell breaks loose. It seems /dev/sda1 is the culprit. What I'm not too sure about is how to create the RAID10. I've tried the following mdadm --create /dev/md2 --level=raid10 --raid-device=4 /dev/sdc1 missing /dev/sdd1 missing I then proceeded to fail /dev/sdb1 from /dev/md0 and added that partition to /dev/md2. I proceeded to install the MBR on EACH partition since boot resides on /dev/sdx1 on each drive. As a test, all is well, I'm able to boot back into the system once I do a quick reboot. Now, when I go add the last drive /dev/sda1, it breaks. I attempted to install grub on /dev/sda1 and I get the following ... grub> root (hd0,0) /dev/sda root (hd0,0) /dev/sda Filesystem type is ext2fs, partition type 0xfd grub> setup (hd0) setup (hd0) Checking if "/boot/grub/stage1" exists... no Checking if "/grub/stage1" exists... no Error 2: Bad file or directory type At this point, the array is hosed I believe. I rebooted the server and it refuses to boot.

  Read the article

• mkfs Operation Takes Very Long on Linux Software Raid 5

  - by Elmar Weber

  I've set-up a Linux software raid level 5 consisting of 4 * 2 TB disks. The disk array was created with a 64k stripe size and no other configuration parameters. After the initial rebuild I tried to create a filesystem and this step takes very long (about half an hour or more). I tried to create an xfs and ext3 filesystem, both took a long time, with mkfs.ext3 I observed the following behaviour, which might be helpful: writing inode tables runs fast until it reaches 1053 (~ 1 second), then it writes about 50, waits for two seconds, then the next 50 are written (according to the console display) when I try to cancel the operation with Control+C it hangs for half a minute before it is really canceled The performance of the disks individually is very good, I've run bonnie++ on each one separately with write / read values of around 95 / 110MB/s. Even when I run bonnie++ on every drive in parallel the values are only reduced by about 10 MB. So I'm excluding hardware / I/O scheduling in general as a problem source. I tried different configuration parameters for stripe_cache_size and readahead size without success, but I don't think they are that relevant for the file system creation operation. The server details: Linux server 2.6.35-27-generic #48-Ubuntu SMP x86_64 GNU/Linux mdadm - v2.6.7.1 Does anyone has a suggestion on how to further debug this?

  Read the article

• mkfs Operation Takes Very Long on Linux Software Raid 5

  - by Elmar Weber

  I've set-up a Linux software raid level 5 consisting of 4 * 2 TB disks. The disk array was created with a 64k stripe size and no other configuration parameters. After the initial rebuild I tried to create a filesystem and this step takes very long (about half an hour or more). I tried to create an xfs and ext3 filesystem, both took a long time, with mkfs.ext3 I observed the following behaviour, which might be helpful: writing inode tables runs fast until it reaches 1053 (~ 1 second), then it writes about 50, waits for two seconds, then the next 50 are written (according to the console display) when I try to cancel the operation with Control+C it hangs for half a minute before it is really canceled The performance of the disks individually is very good, I've run bonnie++ on each one separately with write / read values of around 95 / 110MB/s. Even when I run bonnie++ on every drive in parallel the values are only reduced by about 10 MB. So I'm excluding hardware / I/O scheduling in general as a problem source. I tried different configuration parameters for stripe_cache_size and readahead size without success, but I don't think they are that relevant for the file system creation operation. The server details: Linux server 2.6.35-27-generic #48-Ubuntu SMP x86_64 GNU/Linux mdadm - v2.6.7.1 Does anyone has a suggestion on how to further debug this?

  Read the article

< Previous Page | 5 6 7 8 9 10  | Next Page >