Results 1 to 25 of 77
Thread: LayeredTech Problems [MERGED]
-
12-10-2006, 07:48 AM #1Junior Guru
- Join Date
- Apr 2006
- Location
- MA
- Posts
- 193
Layered Technologies Nightmare - Unbelievable
Layered Technologies Nightmare:
Oh Thursday December 7th 2006, at 4:03 PM I submit a Layered Tech Support Ticket asking them if they could check my server urgently for rootkits and SSH backdoors as a person was telling me my server was backdoored although he offered no proof, but I figured I better be safe then sorry and invest some money to check the server over because the person threatening me was a very intelligent individual. I submitted the ticket with my server, IP address, root password, and other information. They contact me back by 4:32 PM and let me know that they are going to direct me to a level two security "specialest" (not to mention the spelling errors - I don't mind spelling errors I make them myself but I just had to point those out) At 4:52 PM they contact me telling me it will cost $75/hr to check the server for backdoors. They tell me I have a FreeBSD system when I submitted the ticket stating that I have a CentOS system, which they did not read. I was still rather happy with LayeredTech at the time, as I have been with them for around a year and a half, I think. I reply with the IP address and SSH port at 4:57 PM, and they reply back at 8:40 PM (That's almost three hours for a urgent critical security hole/backdoor possibility, weird huh?) They come back and just tell me what they can do to secure my server including doing some security stuff to recover it after. I was out at the time so I replied back at 12:19 AM AM asking to proceed. In the mean time my server is still up, and I called them 6:28 PM trying to get my server either shut down or looked at because I was not getting any responses via my ticket and I was just really worried that my server was hacked into. I guess next time I shouldn't believe the person but I just didn't know what to think and this was my primary server where I store stuff on and by no means am I a linux expert but I know what I am doing.
I sent three responses back at 12:19 AM (Friday December 8th), one at 1:36 AM, and one at 1:39 AM (Morning), and finally I get a response back at 3:27 ** PM ** stating that they can do this for me in an hour, which they already told me earlier. There was some minor conversation back and forth through the ticket my last response being at 4:03 PM which I tell them to actually secure my server. (The security has still not been checked). I think I called once in the middle of ask to ask them what they were doing and why nothing was happen, which didn't really go anywhere. So at 5:11 PM I ask if they have any updates for this entire issue and he replies back, still thinking is a FreeBSD system after he has been instructed twice that it is not, telling me that he needs my root password, which was already provided twiec. I provide the root password at 5:33 PM after calling to complain again before that. They send me an email at 5:49 asking me to approve the fees affiliated with this security check, which I gladly approve. At this time my server IS STILL NOT TURNED OFF as I have asked multiple times to have done. I send a reply at 5:54 PM on Friday telling them I want the server down now. at 9:36 PM (Four hours later) They tell me they have turned the server off and that they are going to wipe out the drive and restore the user accounts and everything. I reply the next minute, 9:37 PM, asking for an ETA. They reply back at 9:58 PM telling me it should be up by midday.
At 10:55 PM Friday the reload of the OS starts and by 12:59 AM on Sat the reload is done. I then send requests in at 10:14 AM, 10:29 AM, 12:15 PM (all on Saturday), and at 1:11 PM. Now, let’s take a second to look at this. I have not gotten a response yet from 1am and now it is already 1pm the same day. THAT IS TWLEVE HOURS THAT MY CLIENTS HAVE NOT HAD A SERVER. I called them and complained at around 2pm and they state that the professional services manager has been contacted about the issue and will be giving me an update. This is at 2:08 pm on Saturday. Oh, but wait, it is not even close to being over. At 2:09 pm I respond stating my disbelief at the duration of this downtime. Matt, the Individual working on the server finally replies at 3:46 PM on Saturday. By 5:42 PM he installs the cpanel and migrates my accounts back over for me. By 6:26 PM the accounts are restored and I think everything is going alright. Finally by 8:45 PM everything is done and I was sleeping because I had not slept since this entire thing began.
I wake up at 11:13 PM still on Sat asking a few questions about APF and BFD and other things. He never gave me my root password to log into WHM and only gave me a user that I can log in with and has sudo privileges, nothing more. All the mail on the server is not working, FTP is not working, I can't get into WHM and do anything, I can’t log in as root to figure out anything, and I keep sending responses my last one being at 12:07 AM on Sunday. By this point I am so mad with LayeredTech I don't even know what to do. My clients are calling me asking me what is going on I am losing customers people need emails, my personal email is down and stuff is getting bounced so it doesn’t even go to my secondarymx server because it's being bounced back. I can't log in and do anything; my life is pretty much destroyed at that time. By 2:19 AM Sunday I submit a request starting that the email is not working. He contacts me back at 2:23 telling me that the next technician will look at it and I reply asking them to mark it as the highest priority possible because of all my customers complaining and ask him to get a hold of matt to see if he can do anything. At 3:18 AM they begin to take a look at it and by 3:24 the email is not bouncing anymore but SMPT/POP3 does not work - I can only get in though the cpanel email login thing. I reply back at 3:24 AM, 3:47 AM, and 3:48 AM explaining what is going on and have yet to receive a response on that ticket and it is 5:41. So the only time I can get a response is when I open a new ticket so I open another ticket regarding all my problems and stating that FTP doesn’t work at 4:16AM and reply to it again at 5:00 AM.
At 5:17 AM he replies with my request for help and fixes a few things which makes me happy - however I am still pretty pissed about this entire thing. I reply back at 5:44 AM (which was just one second ago while I’m writing this stating thank you.) The FTP and email problems still persist, and I have no idea why. So there's the story so far I will keep you posted and hopefully LT can turn this around. I have had great experiences with them in the past their hardware is stable and the network has never dropped on me but these past few days have been amazingly bad.
Thanks,
Michael
Last edited by Glexia; 12-10-2006 at 08:01 AM.
0
-
12-10-2006, 07:53 AM #2Web Hosting Evangelist
- Join Date
- Nov 2003
- Location
- Israel
- Posts
- 515
I'm sorry, but this is impossible to read!
Can you edit and add some paragraphs?Best regards,
Gil - ZoneServ.com.0
-
12-10-2006, 08:02 AM #3Web Hosting Master
- Join Date
- Sep 2004
- Posts
- 760
Agree with Zone
Sorry to hear that what happened to you,In that case you should go for managed service, $75/hr services is no help for your case.0
-
12-10-2006, 08:03 AM #4Aspiring Evangelist
- Join Date
- Jul 2005
- Posts
- 369
IMHO, You should of gotten some 3rd party company to check for rootkits. It would of costed you about half as much and they would of gotten the job done faster.
If you need another rootkit check in future, I can recommend Steve from Rack911.com.
Mini0
-
12-10-2006, 08:07 AM #5Junior Guru
- Join Date
- Apr 2006
- Location
- MA
- Posts
- 193
Okay thanks is it still not easy to follow I can split it up more and provid a more detailed timeline....
0
-
12-10-2006, 08:21 AM #6Junior Guru
- Join Date
- Apr 2006
- Location
- MA
- Posts
- 193
Rodney Howard from LT has been helping me out and resolving problems. He's doing a good job after this entire mess - I'll get back to you with more updates.
0
-
12-10-2006, 09:01 AM #7Too smart for her own good.
- Join Date
- Feb 2004
- Location
- Your Screen
- Posts
- 3,999
Michael,
I fail to see the need for a reload. Reloads are LT's answer to everything... false-alarm S.M.A.R.T. errors, "possible" exploits, sunspots, the coffee pot being empty. What evidence were you given that a reload was necessary?
Two, never, ever, EVER ask a self-managed provider to do management. They think they know how to do it, but they don't. There is a LOT of evidence on this very board that this is the case with LT. LT knows how to put parts in a chassis, and put an OS on a drive. That's it. Do *NOT* ask them for more, they will screw up your box.
Three, if some random stranger can send you in this kind of tizzy, you're pretty much in over your head. Pony up for the $29/month service at Platinum Server Management. They could have easily gone through and secured your box within that first few hours. At least you would have had an indication pretty quickly whether you should have been so crazy-worried or not. Then if you want to step it up and go with a pricier service like rack911 for account recovery or OS securing or whatever, you could cross that bridge if you really felt the need to go that direction.
I can't think of any testing here that LT would have done that I couldn't have done myself. You should have known how to do this too, and if you didn't you absolutely NEED a server management company.
I'm not saying that it doesn't sound like LT screwed this up. It does. But to me the core problem was giving them the access to the task to begin with. They were the wrong tool for the job, and now you're stuck cleaning up the mess.
BaileyLet's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.0
-
12-10-2006, 09:16 AM #8Junior Guru
- Join Date
- Apr 2006
- Location
- MA
- Posts
- 193
True
Yeah I know what you are saying -- next time this wont happen.
0
-
12-10-2006, 09:35 AM #9Junior Guru
- Join Date
- Apr 2006
- Location
- MA
- Posts
- 193
Email problem still not fixed.
0
-
12-10-2006, 10:11 AM #10Junior Guru
- Join Date
- Apr 2006
- Location
- MA
- Posts
- 193
Ended up spending a few hours on google and found the solution and fixed the problem myself.
Wow..?0
-
12-10-2006, 10:50 AM #11Web Hosting Master
- Join Date
- May 2003
- Location
- Florida
- Posts
- 902
Originally Posted by Glexia
Glad you have the problem resolved. I can understand the frustration. I am pretty happy with LayeredTech for most things, but I agree with two of the previous suggestions. Platinum Server Management is great in helping with securing your server and also helping to resolve the email problems. While they have a monthly program, they are willing to help in these one-time matters.
If you believe the server had been rooted, I think Steven at Rack911 is one of the best.0
-
12-10-2006, 08:26 PM #12Web Hosting Master
- Join Date
- Nov 2002
- Posts
- 1,467
I think a typo and a spelling mistake are two different things. (A typo qualifying as all proper letters, in the wrong order)
But yeah, I've been thinking of going layeredtech or softlayer. From what I can tell, if I get layered tech and want management, get someone else.All life is an experiment. The more experiments you make the better.0
-
12-10-2006, 09:43 PM #13Retired Moderator
- Join Date
- Apr 2005
- Posts
- 3,499
This is pretty much on par from what LT always suggest: OS Reload or Chassis swap. It's much easier to swap the server components or reload the OS (you don't need very experienced techs to do that, so they can save money on hiring), than to go deep down into the server and look for all the individual problems. LT is fine for anything unless you need management.
Alex0
-
12-10-2006, 09:57 PM #14Web Hosting Master
- Join Date
- Sep 2002
- Posts
- 1,017
Originally Posted by GlexiaCaro.Net: Support is everything
Offering High Quality Dedicated Servers.0
-
12-11-2006, 09:00 AM #15Web Hosting Guru
- Join Date
- Apr 2006
- Location
- Mandaluyong, Philippines
- Posts
- 316
Layered tech used to be good.
Their front end support is now clueless, they leave you hang for hours, and they forget the fact that you pay your bills from the income you generate leasing their gear.
What is really, really pitiful is you PAID FOR their extra services and nobody even bothered to see if all services were working.
The "Yep its up close the ticket" mentality there has GOT to STOP. At least TEST your solution before you release it to a client, especially if the client PAID EXTRA for it.
I moved every box I had out of their place over to The New York NOC ( http://www.thenynoc.com ).
Andy has worked with me swapping drives tweaking CMOS / etc when things go wrong. I get the feeling that he wants my business so I continue to bring it there. He even made special packages and offerings for some of my new sites and has taken a genuine interest in them. They aren't a tiny DC, but it feels like your dealing with another small business, the way it used to feel at LT.
Layered Tech treats people like cattle lately. Until they change, avoid them at all cost. You're just something else 'irritating' the tech on duty who seems to want to be doing other things. That mentality cost my business a great deal of money, that's why I moved.
Sad. They [LT] used to be *really* good. I guess once you get to a certain size something changes. I hope I never get that big, I like keeping customers.0
-
12-11-2006, 09:04 AM #16Web Hosting Guru
- Join Date
- Apr 2006
- Location
- Mandaluyong, Philippines
- Posts
- 316
Originally Posted by bithost(NET)
http://www.layeredtech.com/careers/index.php
Things will improve.
What kills me is there are perfectly competent people there, Jeremy / Jack are some .. but issues never get put in front of them. Well, what happens when they go home for the day? Or its a weekend?
Ugh. What a shame. I built my whole business around them and it was a painful (but needed) thing to do when I moved.0
-
12-11-2006, 12:43 PM #17Junior Guru Wannabe
- Join Date
- Aug 2002
- Location
- Mumbai
- Posts
- 34
I stopped reading your post after these 2 lines.
"Oh Thursday December 7th 2006, at 4:03 PM I submit a Layered Tech Support Ticket asking them if they could check my server urgently for rootkits and SSH backdoors "
LT are NOT managed service provider. Their only job is to provide stable hardware and to provide you SSH prompt.
Dont feel offended but if you need anything beyond SSH prompt then you should look for Managed Server Providers...0
-
12-11-2006, 12:48 PM #18Aspiring Evangelist
- Join Date
- Oct 2006
- Location
- Tipton, IN
- Posts
- 417
Originally Posted by Tarak
The only reason why they logged in to do it is to make that $75/hr.
Of course, one could objectively look at it as "He accepted the $75/hr, so he should accept the consequences." But, in all reality, if they couldn't fix his issue properly even if he did accept the charge for it to be done, then they should have told him instead of going ahead and messing things up.0
-
12-11-2006, 01:20 PM #19Web Hosting Guru
- Join Date
- Apr 2006
- Location
- Mandaluyong, Philippines
- Posts
- 316
Originally Posted by Tarak
That's $30 above industry standard. (Typically $40 - $45 / hourly)
... and they fell *flat* on their faces helping him.0
-
12-11-2006, 03:22 PM #20Aspiring Evangelist
- Join Date
- Jun 2006
- Posts
- 407
I had servers with LT and I had some major issues with them and Matt was also the tech the runied my server with them.
I am not going to go into the details because it is not my thread but however the techs at LT messed the server up and then they told me that it had hardware failures and asked me I wanted my drive with all of my data moved to a new server which I already had setup with them so I could get my clients back online.
I said yes and then they moved the drive and said it is done.
Before they touched my server it worked fine. I was just doing a Virtuozzo migration and since they get the licenses from SW Soft they have to support the software.
So now I own my hardware and moved to a new data center and everyone is happy again.0
-
12-11-2006, 10:24 PM #21Junior Guru
- Join Date
- Apr 2006
- Location
- MA
- Posts
- 193
Sorry to see that other people had the same experiences especially with the same individuals. I appreciate your comments and I think I will be moving my server to SoftLayer soon and hiring a server management company.
0
-
12-15-2006, 11:04 AM #22Junior Guru
- Join Date
- Apr 2006
- Location
- MA
- Posts
- 193
LayeredTech Problems [MERGED]
After being with Layered Tech for over a year and a half I have finally cancelled my server with them due to their absolutely horrible support team and company ethics. LayeredTech at first was a great provider; however, as they grow they treat clients more like cattle. Their support tickets are never read fully, the technicians take over twelve hours (in my case) to respond to a ticket. I submitted to have a reload done and have my files transferred, so they removed my backup drive from the server and never put the drive back in the server (of course they are going to come here now and somehow make the drive magically appear).
Then, if you come to this site and post about their company they contact you and state stuff such as: “Nice to see you bashing us on WHT about this Mike. I really do not feel your comments about us are deserved.” I think customers should have freedom of speech and not have their company threatening to cancel their servers if they talk about what happened on Webhosting talk. I told them I was going to post this all on WHT and they said then you are no longer going to be a customer of ours.
<<removed self promotion>>
I really hope people can read this thread and understand how the practices of this firm are completely out of line and their support is completely absurd and horrible. I will be posting more information regarding this entire incident here if they let me get back into my support tickets to give you guys proof of what happened here. (They changed my password so I can’t even submit support tickets right now)
[FONT='Verdana','sans-serif']Oh and they expected me to pay 200$ for the services that they did not even render completely, efficiently, and effectively. They sent me the invoices stating the work was completed in full, which of course it still isn't. LayeredTech just doesn’t know how to treat their customers.
[/FONT]
Anyone else have anything to add to this?Last edited by sirius; 12-24-2006 at 05:24 PM.
0
-
12-15-2006, 11:19 AM #23renegade
- Join Date
- Feb 2006
- Posts
- 1,044
Ok so you had a problem.. Let's move on and not dwell on this, much less create a website dedicated to it?
Nothing is perfect, and I'm sure if you stopped being angry and listened to them, this could be solved in a much more peaceful manner..
I dunno, just my 2 cents..0
-
12-15-2006, 11:22 AM #24Randy
- Join Date
- Aug 2006
- Location
- Ashburn VA, San Diego CA
- Posts
- 4,615
Do you have any off-site or otherwise external backups? Were your only backups on the server itself?
Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
Since 2003 - Ashburn VA + San Diego CA Datacenters0
-
12-15-2006, 11:29 AM #25Aspiring Evangelist
- Join Date
- Oct 2006
- Location
- Tipton, IN
- Posts
- 417
Originally Posted by glexiaOriginally Posted by glexia
Due to this entire incident, I am going to be starting the website LayeredTechSUCKS.com for everyone who has a problem with LayeredTech to post their reviews here. I will post my entire story as well as other opinions on this website.
My advice to you would be, if there is a problem with the service. State the problem in a mature and constructive manor, and just leave it at that. If LT cannot bare to make things right, then that's their problem. But, I would definantly recommend that you keep your problem with LT as yours, and not get into anyone elses problems with LT. Let them handle their own problems.0