I recently had a problem with a server running a RAID-6 array (via mdadm). When I looked at /proc/mdstat, the array in question had only one member.
I found the elements by running an fdisk and determining which drives were supposed to be used by the md set. When I ran an examine on them individually, they all checked out just fine. I also ran smartctl on the drives and, again, they were just fine (it was a controller issue).
The error message was:
mdadm: /dev/md3 assembled from 1 drive - not enough to start the array.
I searched the web and found some answers, which were not as helpful as I would have hoped since it came down to running mdadm --assemble --force
and I was unclear what would happen if I did that and it failed..
The answer is, it just worked. When I forced the assemble (I had nothing to lose at this point), if found one drive that had "something" messed up on it and it could not join the array. However, it got the array up and running, then I added the drive that could not be used back in. After 8 hours of syncing, the array was back up.
What basically happened was, for some reason, all the drives except one were marked Failed (though this did not show up on the mdadm --examine output). When I used the --force parameter, it attempted to clear the Fail flag on the drives and then mount them.
I am including a transcript of the session. The drives which were part of md3 were sda, sdb, sde, sdf and sdg, the shorthand for which is sd[abefg] when using bash. I've included a full transcript, only changing the server name to "myserver".
Bottom line is, it appears --force is ok to use if there is nothing horribly wrong. It appears to clear some flags, etc. Note, however, how the mdadm --examine differs for the devices. With /dev/sda, it shows all the other drives faulty, yet when I --examine the other drives, they show them all good. There was nothing I could see that would indicate that /dev/sdb was the one with the problem (and, nothing to prove that was the case).
Transcript. Some redundant lines removed.
root@myserver:/root# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active (auto-read-only) raid1 sdc3[0] sdd3[1]
1460253560 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdc1[0] sdd1[1]
3904500 blocks super 1.2 [2/2] [UU]
unused devices:
root@myserver:/root# mdadm --examine /dev/sd[abefg]
/dev/sda:
Magic : a92b4efc
Version : 0.90.00
UUID : 46309546:32ea358a:811bc394:cc8bee49
Creation Time : Wed Jul 21 14:01:53 2010
Raid Level : raid6
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Array Size : 5860543488 (5589.05 GiB 6001.20 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 3
Update Time : Sat Jan 18 03:42:41 2014
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 4
Spare Devices : 0
Checksum : 83ad5f04 - correct
Events : 1465922
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 0 0 active sync /dev/sda
0 0 8 0 0 active sync /dev/sda
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
/dev/sdb:
Magic : a92b4efc
Version : 0.90.00
UUID : 46309546:32ea358a:811bc394:cc8bee49
Creation Time : Wed Jul 21 14:01:53 2010
Raid Level : raid6
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Array Size : 5860543488 (5589.05 GiB 6001.20 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 3
Update Time : Sat Jan 18 03:39:13 2014
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Checksum : 83ad5e02 - correct
Events : 1465918
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 16 3 active sync /dev/sdb
0 0 8 0 0 active sync /dev/sda
1 1 8 96 1 active sync /dev/sdg
2 2 8 80 2 active sync /dev/sdf
3 3 8 16 3 active sync /dev/sdb
4 4 8 64 4 active sync /dev/sde
/dev/sde:
Magic : a92b4efc
Version : 0.90.00
UUID : 46309546:32ea358a:811bc394:cc8bee49
Creation Time : Wed Jul 21 14:01:53 2010
Raid Level : raid6
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Array Size : 5860543488 (5589.05 GiB 6001.20 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 3
Update Time : Sat Jan 18 03:39:20 2014
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Checksum : 8396fffd - correct
Events : 1465919
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 64 4 active sync /dev/sde
0 0 8 0 0 active sync /dev/sda
1 1 8 96 1 active sync /dev/sdg
2 2 8 80 2 active sync /dev/sdf
3 3 8 16 3 active sync /dev/sdb
4 4 8 64 4 active sync /dev/sde
/dev/sdf:
Magic : a92b4efc
Version : 0.90.00
UUID : 46309546:32ea358a:811bc394:cc8bee49
Creation Time : Wed Jul 21 14:01:53 2010
Raid Level : raid6
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Array Size : 5860543488 (5589.05 GiB 6001.20 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 3
Update Time : Sat Jan 18 03:39:20 2014
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Checksum : 83970009 - correct
Events : 1465919
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 80 2 active sync /dev/sdf
0 0 8 0 0 active sync /dev/sda
1 1 8 96 1 active sync /dev/sdg
2 2 8 80 2 active sync /dev/sdf
3 3 8 16 3 active sync /dev/sdb
4 4 8 64 4 active sync /dev/sde
/dev/sdg:
Magic : a92b4efc
Version : 0.90.00
UUID : 46309546:32ea358a:811bc394:cc8bee49
Creation Time : Wed Jul 21 14:01:53 2010
Raid Level : raid6
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Array Size : 5860543488 (5589.05 GiB 6001.20 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 3
Update Time : Sat Jan 18 03:39:20 2014
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Checksum : 83970017 - correct
Events : 1465919
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 96 1 active sync /dev/sdg
0 0 8 0 0 active sync /dev/sda
1 1 8 96 1 active sync /dev/sdg
2 2 8 80 2 active sync /dev/sdf
3 3 8 16 3 active sync /dev/sdb
4 4 8 64 4 active sync /dev/sde
root@myserver:/root# mdadm --assemble /dev/md3 /dev/sd[abefg]
mdadm: /dev/md3 assembled from 1 drive - not enough to start the array.
root@myserver:/root# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md3 : inactive sda[0](S) sde[4](S) sdb[3](S) sdf[2](S) sdg[1](S)
9767572480 blocks
md1 : active (auto-read-only) raid1 sdc3[0] sdd3[1]
1460253560 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdc1[0] sdd1[1]
3904500 blocks super 1.2 [2/2] [UU]
unused devices:
root@myserver:/root# mdadm --stop /dev/md3
mdadm: stopped /dev/md3
root@myserver:/root# mdadm --assemble --force /dev/md3 /dev/sd[abefg]
mdadm: forcing event count in /dev/sdg(1) from 1465919 upto 1465922
mdadm: forcing event count in /dev/sdf(2) from 1465919 upto 1465922
mdadm: forcing event count in /dev/sde(4) from 1465919 upto 1465922
mdadm: clearing FAULTY flag for device 4 in /dev/md3 for /dev/sdg
mdadm: clearing FAULTY flag for device 3 in /dev/md3 for /dev/sdf
mdadm: clearing FAULTY flag for device 2 in /dev/md3 for /dev/sde
mdadm: Marking array /dev/md3 as 'clean'
mdadm: /dev/md3 has been started with 4 drives (out of 5).
root@myserver:/root# mdadm /dev/md3 --add /dev/sdb
mdadm: added /dev/sdb
root@myserver:/root# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md3 : active raid6 sdb[5] sda[0] sde[4] sdf[2] sdg[1]
5860543488 blocks level 6, 64k chunk, algorithm 2 [5/4] [UUU_U]
[>....................] recovery = 0.0% (314448/1953514496) finish=414.0min speed=78612K/sec
md1 : active (auto-read-only) raid1 sdc3[0] sdd3[1]
1460253560 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdc1[0] sdd1[1]
3904500 blocks super 1.2 [2/2] [UU]
unused devices:
