Mail Server Issues

Bill · Jan 20, 2012, 15:27:37

Well, the restore seems to have worked... I've just had 135 emails from the last few days downloaded again!

So unlikely to be much (if any) loss, pleased to say.

Technical Ben · Jan 20, 2012, 16:35:55

Always remember. If the failed disk is still mechanically sound (IE it reads data through windows etc), then FIRST do a complete image clone (including "blank" space) before changing anything. Else you could end up loosing the data in the process of recovery.

If the disk is mechanically borked. I hope IDNet have lots of patience.

Simon_idnet · Jan 20, 2012, 17:11:38

At first we were trying to resurrect the failed server so that we could get the mail spools online again. However, the OS is on a mirrored array and the array controller refused to play with it. We transferred all the disks (OS and mail spool arrays) to a spare machine but the array controller in that machine wouldn't do it either. We tried booting the server from a CD as well. At that point the decision was made to transfer the services to a new server installation, restore the most recent backup and allow the store-and-forward queues to be delivered. In all, 6 machines comprise this service, but it was the POP3/IMAP machine that had failed.

The failed array is now with a Data Recovery company. The expert there thinks that one of the disks blew its voltage suppressor which then caused a corruption on the mirror disk. They are going to work over the weekend to try to retrieve the missing 23 hours of mail. The mail server is backed up a 5am every day but failed at around 4am this morning.

Apologies for the inconvenience. Please be assured that we are doing everything that we can to restore the full mail history.

Simon · Jan 20, 2012, 18:23:07

Thanks Simon. I think that epitomises the phrase, "sh*t happens".

esh · Jan 20, 2012, 18:28:27

I'm not super technical on RAID systems, but I'm guessing that a drive died and in the process corrupted another so it wouldn't rebuild? I guess it's time for RAID-6...or more. One of my worst fears is that my hardware RAID array wouldn't rebuild... Supposedly very very picky, sometimes even another identical controller won't rebuild the array for some reasons. For that reason some people swear by software RAID... and then others will incinerate you with a glare should you even consider it!

Back on topic, the server is probably under load. I still can't access it as of this post time.

Steve · Jan 20, 2012, 18:52:56

I'm having no success via SSL on port 993. mail.idnet.com on 143 seems fine. The imap.example.com certificate is not welcomed on Mac OS X Lion

sunray2 · Jan 20, 2012, 19:00:36

Unlike Bill I am still awaiting the arrival of 158 emails - over a weeks worth! Sent myself a message some 10 minutes ago ago and this came back to me at the double! Nothing else received since 22.58 last night.

Gavin

Simon · Jan 20, 2012, 19:06:21

How can you be waiting for over a weeks worth? The server only went down last night. Apologies if I've missed something, but...

David · Jan 20, 2012, 19:16:10

Thanks Simon got all mine now and I thought you managed it quite well

Care for a hobnob ?

Simon · Jan 20, 2012, 19:17:27

I'd prefer a chocolate digestive.

Baz · Jan 20, 2012, 19:20:24

We seem to be missing lots that were viewed via webmail and left on the server.

are these gone for ever

Bill · Jan 20, 2012, 19:44:24

Quote from: Simon on Jan 20, 2012, 19:06:21
How can you be waiting for over a weeks worth? The server only went down last night. Apologies if I've missed something, but...

When I read the mail via POP3 the client is set to leave it on the server for a week. It's saved my backside once or twice, but it does mean that when something goes pear-shaped there's a good chance I'll get that week's worth again!

Simon · Jan 20, 2012, 19:45:46

Ah, I see. I always download mine, so I tend to assume everyone else does.

cavillas · Jan 20, 2012, 19:52:06

Many thanks to the team at Idnet for working so hard and doing their best. Sometimes these machines have minds of their own and decide to upset everyone because they are just left to run and have no meaningful human companionship.

pctech · Jan 20, 2012, 20:42:18

So you reckon it might not have happened if the desks of the IDNet staff were situated next to the racks holding the servers and networking gear Alf?

Not sure whether you've ever been in a data centre but they tend to be quite cold (particularly in the winter) and noisy places as most servers, storage arrays, large routers and switches have large variable speed fans on them that can go up to 8000 rpm or more when the device is warm because its under heavy load, this creates one hell of a racket, here's an example http://www.youtube.com/watch?v=b33i7cmjhtc

Multiply this by the number of devices required to run services like IDNet's and you could hardly have a conversation over the phone or between staff as you also have the noise of the powerful air conditioning required to keep all the stuff cool, might prove an interesting challenge though.

Joking aside, fair play to IDNet for doing all they can to recover the missing data, I know several ISPs that would simply replace the disk and say tough as its a bundled service, you just pay us for your connection, not a mail service.

Simon · Jan 20, 2012, 21:22:25

Quote from: pctech on Jan 20, 2012, 20:42:18Joking aside, fair play to IDNet for doing all they can to recover the missing data, I know several ISPs that would simply replace the disk and say tough as its a bundled service, you just pay us for your connection, not a mail service.

Indeed, and I don't suppose data recovery comes cheap.

Steve · Jan 20, 2012, 21:58:22

The incoming mail server securemail.idnet.com via ssl port 993 now seems to be functioning plus mail forwarding seems to have returned.

Steve · Jan 20, 2012, 22:08:01

Quote from: sunray2 on Jan 20, 2012, 19:00:36
Unlike Bill I am still awaiting the arrival of 158 emails - over a weeks worth! Sent myself a message some 10 minutes ago ago and this came back to me at the double! Nothing else received since 22.58 last night.

Gavin

It took awhile but

Simon · Jan 20, 2012, 22:15:11

Gavin!

David · Jan 20, 2012, 22:16:02

oops where are my manners

welcome Gavin

esh · Jan 20, 2012, 22:23:36

Ah, victory. I'm up and running again here.

Gary · Jan 20, 2012, 23:02:01

Quote from: Steve on Jan 20, 2012, 18:52:56
I'm having no success via SSL on port 993. mail.idnet.com on 143 seems fine. The imap.example.com certificate is not welcomed on Mac OS X Lion

I could send and receive using that certificate using Lion on pop3, still no SSL on port 995 though using securemail.idnet.com with pop3

sunray2 · Jan 21, 2012, 00:20:14

Hi Guys

Thanks for the great welcome one and all, most unexpected but a pleasant surprise indeed! Simon, as for the 158 emails they have now been downloaded, they are ones that I had already received. All now appears to be well!

Gavin

MarlyM · Jan 21, 2012, 07:08:49

Quote from: Bill on Jan 20, 2012, 19:44:24
When I read the mail via POP3 the client is set to leave it on the server for a week. It's saved my backside once or twice, but it does mean that when something goes pear-shaped there's a good chance I'll get that week's worth again!

Same here, but I do it for fortnight.

I had fun last night removing hundreds of duplicates from my inbox!

Oh well, it's better to have two than none at all I guess!

Technical Ben · Jan 21, 2012, 09:18:31

Nice to hear the fix is on the way. Sad to hear the controller blew + the mirror is corrupted. Hopefully, it's just a basic corruption (MBT error?), and no data was lost. I've also heard of times where when plugging in a new empty disk to a raid system, the computer promptly "backed up" and mirrored the empty disk over the full one.

So I'm glad IDNet are professional. It takes away some of the fear you get when you hear BT/O2/Sky/Talktalk are playing with some hardware!