I recently had a problem with a server running a RAID-6 array (via mdadm). When I looked at /proc/mdstat, the array in question had only one member.
I found the elements by running an fdisk and determining which drives were supposed to be used by the md set. When I ran an examine on them individually, they all checked out just fine. I also ran smartctl on the drives and, again, they were just fine (it was a controller issue).
The error message was:
mdadm: /dev/md3 assembled from 1 drive - not enough to start the array.
I searched the web and found some answers, which were not as helpful as I would have hoped since it came down to running mdadm --assemble --force
and I was unclear what would happen if I did that and it failed..
The answer is, it just worked. When I forced the assemble (I had nothing to lose at this point), if found one drive that had "something" messed up on it and it could not join the array. However, it got the array up and running, then I added the drive that could not be used back in. After 8 hours of syncing, the array was back up.
What basically happened was, for some reason, all the drives except one were marked Failed (though this did not show up on the mdadm --examine output). When I used the --force parameter, it attempted to clear the Fail flag on the drives and then mount them.
I am including a transcript of the session. The drives which were part of md3 were sda, sdb, sde, sdf and sdg, the shorthand for which is sd[abefg] when using bash. I've included a full transcript, only changing the server name to "myserver".
Bottom line is, it appears --force is ok to use if there is nothing horribly wrong. It appears to clear some flags, etc. Note, however, how the mdadm --examine differs for the devices. With /dev/sda, it shows all the other drives faulty, yet when I --examine the other drives, they show them all good. There was nothing I could see that would indicate that /dev/sdb was the one with the problem (and, nothing to prove that was the case).
Transcript. Some redundant lines removed.
root@myserver:/root# cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md1 : active (auto-read-only) raid1 sdc3[0] sdd3[1] 1460253560 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdc1[0] sdd1[1] 3904500 blocks super 1.2 [2/2] [UU] unused devices: root@myserver:/root# mdadm --examine /dev/sd[abefg] /dev/sda: Magic : a92b4efc Version : 0.90.00 UUID : 46309546:32ea358a:811bc394:cc8bee49 Creation Time : Wed Jul 21 14:01:53 2010 Raid Level : raid6 Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB) Array Size : 5860543488 (5589.05 GiB 6001.20 GB) Raid Devices : 5 Total Devices : 5 Preferred Minor : 3 Update Time : Sat Jan 18 03:42:41 2014 State : clean Active Devices : 1 Working Devices : 1 Failed Devices : 4 Spare Devices : 0 Checksum : 83ad5f04 - correct Events : 1465922 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 0 8 0 0 active sync /dev/sda 0 0 8 0 0 active sync /dev/sda 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 0 0 3 faulty removed 4 4 0 0 4 faulty removed /dev/sdb: Magic : a92b4efc Version : 0.90.00 UUID : 46309546:32ea358a:811bc394:cc8bee49 Creation Time : Wed Jul 21 14:01:53 2010 Raid Level : raid6 Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB) Array Size : 5860543488 (5589.05 GiB 6001.20 GB) Raid Devices : 5 Total Devices : 5 Preferred Minor : 3 Update Time : Sat Jan 18 03:39:13 2014 State : clean Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Checksum : 83ad5e02 - correct Events : 1465918 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 3 8 16 3 active sync /dev/sdb 0 0 8 0 0 active sync /dev/sda 1 1 8 96 1 active sync /dev/sdg 2 2 8 80 2 active sync /dev/sdf 3 3 8 16 3 active sync /dev/sdb 4 4 8 64 4 active sync /dev/sde /dev/sde: Magic : a92b4efc Version : 0.90.00 UUID : 46309546:32ea358a:811bc394:cc8bee49 Creation Time : Wed Jul 21 14:01:53 2010 Raid Level : raid6 Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB) Array Size : 5860543488 (5589.05 GiB 6001.20 GB) Raid Devices : 5 Total Devices : 5 Preferred Minor : 3 Update Time : Sat Jan 18 03:39:20 2014 State : active Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Checksum : 8396fffd - correct Events : 1465919 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 4 8 64 4 active sync /dev/sde 0 0 8 0 0 active sync /dev/sda 1 1 8 96 1 active sync /dev/sdg 2 2 8 80 2 active sync /dev/sdf 3 3 8 16 3 active sync /dev/sdb 4 4 8 64 4 active sync /dev/sde /dev/sdf: Magic : a92b4efc Version : 0.90.00 UUID : 46309546:32ea358a:811bc394:cc8bee49 Creation Time : Wed Jul 21 14:01:53 2010 Raid Level : raid6 Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB) Array Size : 5860543488 (5589.05 GiB 6001.20 GB) Raid Devices : 5 Total Devices : 5 Preferred Minor : 3 Update Time : Sat Jan 18 03:39:20 2014 State : active Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Checksum : 83970009 - correct Events : 1465919 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 2 8 80 2 active sync /dev/sdf 0 0 8 0 0 active sync /dev/sda 1 1 8 96 1 active sync /dev/sdg 2 2 8 80 2 active sync /dev/sdf 3 3 8 16 3 active sync /dev/sdb 4 4 8 64 4 active sync /dev/sde /dev/sdg: Magic : a92b4efc Version : 0.90.00 UUID : 46309546:32ea358a:811bc394:cc8bee49 Creation Time : Wed Jul 21 14:01:53 2010 Raid Level : raid6 Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB) Array Size : 5860543488 (5589.05 GiB 6001.20 GB) Raid Devices : 5 Total Devices : 5 Preferred Minor : 3 Update Time : Sat Jan 18 03:39:20 2014 State : active Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Checksum : 83970017 - correct Events : 1465919 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 1 8 96 1 active sync /dev/sdg 0 0 8 0 0 active sync /dev/sda 1 1 8 96 1 active sync /dev/sdg 2 2 8 80 2 active sync /dev/sdf 3 3 8 16 3 active sync /dev/sdb 4 4 8 64 4 active sync /dev/sde root@myserver:/root# mdadm --assemble /dev/md3 /dev/sd[abefg] mdadm: /dev/md3 assembled from 1 drive - not enough to start the array. root@myserver:/root# cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md3 : inactive sda[0](S) sde[4](S) sdb[3](S) sdf[2](S) sdg[1](S) 9767572480 blocks md1 : active (auto-read-only) raid1 sdc3[0] sdd3[1] 1460253560 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdc1[0] sdd1[1] 3904500 blocks super 1.2 [2/2] [UU] unused devices: root@myserver:/root# mdadm --stop /dev/md3 mdadm: stopped /dev/md3 root@myserver:/root# mdadm --assemble --force /dev/md3 /dev/sd[abefg] mdadm: forcing event count in /dev/sdg(1) from 1465919 upto 1465922 mdadm: forcing event count in /dev/sdf(2) from 1465919 upto 1465922 mdadm: forcing event count in /dev/sde(4) from 1465919 upto 1465922 mdadm: clearing FAULTY flag for device 4 in /dev/md3 for /dev/sdg mdadm: clearing FAULTY flag for device 3 in /dev/md3 for /dev/sdf mdadm: clearing FAULTY flag for device 2 in /dev/md3 for /dev/sde mdadm: Marking array /dev/md3 as 'clean' mdadm: /dev/md3 has been started with 4 drives (out of 5). root@myserver:/root# mdadm /dev/md3 --add /dev/sdb mdadm: added /dev/sdb root@myserver:/root# cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md3 : active raid6 sdb[5] sda[0] sde[4] sdf[2] sdg[1] 5860543488 blocks level 6, 64k chunk, algorithm 2 [5/4] [UUU_U] [>....................] recovery = 0.0% (314448/1953514496) finish=414.0min speed=78612K/sec md1 : active (auto-read-only) raid1 sdc3[0] sdd3[1] 1460253560 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdc1[0] sdd1[1] 3904500 blocks super 1.2 [2/2] [UU] unused devices: