Page 1 of 4 1234 LastLast
Results 1 to 25 of 77
  1. #1
    Join Date
    Apr 2006
    Location
    MA
    Posts
    193

    Layered Technologies Nightmare - Unbelievable

    Layered Technologies Nightmare:

    Oh Thursday December 7th 2006, at 4:03 PM I submit a Layered Tech Support Ticket asking them if they could check my server urgently for rootkits and SSH backdoors as a person was telling me my server was backdoored although he offered no proof, but I figured I better be safe then sorry and invest some money to check the server over because the person threatening me was a very intelligent individual. I submitted the ticket with my server, IP address, root password, and other information. They contact me back by 4:32 PM and let me know that they are going to direct me to a level two security "specialest" (not to mention the spelling errors - I don't mind spelling errors I make them myself but I just had to point those out) At 4:52 PM they contact me telling me it will cost $75/hr to check the server for backdoors. They tell me I have a FreeBSD system when I submitted the ticket stating that I have a CentOS system, which they did not read. I was still rather happy with LayeredTech at the time, as I have been with them for around a year and a half, I think. I reply with the IP address and SSH port at 4:57 PM, and they reply back at 8:40 PM (That's almost three hours for a urgent critical security hole/backdoor possibility, weird huh?) They come back and just tell me what they can do to secure my server including doing some security stuff to recover it after. I was out at the time so I replied back at 12:19 AM AM asking to proceed. In the mean time my server is still up, and I called them 6:28 PM trying to get my server either shut down or looked at because I was not getting any responses via my ticket and I was just really worried that my server was hacked into. I guess next time I shouldn't believe the person but I just didn't know what to think and this was my primary server where I store stuff on and by no means am I a linux expert but I know what I am doing.

    I sent three responses back at 12:19 AM (Friday December 8th), one at 1:36 AM, and one at 1:39 AM (Morning), and finally I get a response back at 3:27 ** PM ** stating that they can do this for me in an hour, which they already told me earlier. There was some minor conversation back and forth through the ticket my last response being at 4:03 PM which I tell them to actually secure my server. (The security has still not been checked). I think I called once in the middle of ask to ask them what they were doing and why nothing was happen, which didn't really go anywhere. So at 5:11 PM I ask if they have any updates for this entire issue and he replies back, still thinking is a FreeBSD system after he has been instructed twice that it is not, telling me that he needs my root password, which was already provided twiec. I provide the root password at 5:33 PM after calling to complain again before that. They send me an email at 5:49 asking me to approve the fees affiliated with this security check, which I gladly approve. At this time my server IS STILL NOT TURNED OFF as I have asked multiple times to have done. I send a reply at 5:54 PM on Friday telling them I want the server down now. at 9:36 PM (Four hours later) They tell me they have turned the server off and that they are going to wipe out the drive and restore the user accounts and everything. I reply the next minute, 9:37 PM, asking for an ETA. They reply back at 9:58 PM telling me it should be up by midday.

    At 10:55 PM Friday the reload of the OS starts and by 12:59 AM on Sat the reload is done. I then send requests in at 10:14 AM, 10:29 AM, 12:15 PM (all on Saturday), and at 1:11 PM. Now, let’s take a second to look at this. I have not gotten a response yet from 1am and now it is already 1pm the same day. THAT IS TWLEVE HOURS THAT MY CLIENTS HAVE NOT HAD A SERVER. I called them and complained at around 2pm and they state that the professional services manager has been contacted about the issue and will be giving me an update. This is at 2:08 pm on Saturday. Oh, but wait, it is not even close to being over. At 2:09 pm I respond stating my disbelief at the duration of this downtime. Matt, the Individual working on the server finally replies at 3:46 PM on Saturday. By 5:42 PM he installs the cpanel and migrates my accounts back over for me. By 6:26 PM the accounts are restored and I think everything is going alright. Finally by 8:45 PM everything is done and I was sleeping because I had not slept since this entire thing began.

    I wake up at 11:13 PM still on Sat asking a few questions about APF and BFD and other things. He never gave me my root password to log into WHM and only gave me a user that I can log in with and has sudo privileges, nothing more. All the mail on the server is not working, FTP is not working, I can't get into WHM and do anything, I can’t log in as root to figure out anything, and I keep sending responses my last one being at 12:07 AM on Sunday. By this point I am so mad with LayeredTech I don't even know what to do. My clients are calling me asking me what is going on I am losing customers people need emails, my personal email is down and stuff is getting bounced so it doesn’t even go to my secondarymx server because it's being bounced back. I can't log in and do anything; my life is pretty much destroyed at that time. By 2:19 AM Sunday I submit a request starting that the email is not working. He contacts me back at 2:23 telling me that the next technician will look at it and I reply asking them to mark it as the highest priority possible because of all my customers complaining and ask him to get a hold of matt to see if he can do anything. At 3:18 AM they begin to take a look at it and by 3:24 the email is not bouncing anymore but SMPT/POP3 does not work - I can only get in though the cpanel email login thing. I reply back at 3:24 AM, 3:47 AM, and 3:48 AM explaining what is going on and have yet to receive a response on that ticket and it is 5:41. So the only time I can get a response is when I open a new ticket so I open another ticket regarding all my problems and stating that FTP doesn’t work at 4:16AM and reply to it again at 5:00 AM.

    At 5:17 AM he replies with my request for help and fixes a few things which makes me happy - however I am still pretty pissed about this entire thing. I reply back at 5:44 AM (which was just one second ago while I’m writing this stating thank you.) The FTP and email problems still persist, and I have no idea why. So there's the story so far I will keep you posted and hopefully LT can turn this around. I have had great experiences with them in the past their hardware is stable and the network has never dropped on me but these past few days have been amazingly bad.
    Thanks,

    Michael
    Last edited by Glexia; 12-10-2006 at 08:01 AM.
      0 Not allowed!

  2. #2
    Join Date
    Nov 2003
    Location
    Israel
    Posts
    515
    I'm sorry, but this is impossible to read!
    Can you edit and add some paragraphs?
    Best regards,
    Gil - ZoneServ.com.
      0 Not allowed!

  3. #3
    Agree with Zone
    Sorry to hear that what happened to you,In that case you should go for managed service, $75/hr services is no help for your case.
      0 Not allowed!

  4. #4
    Join Date
    Jul 2005
    Posts
    369
    IMHO, You should of gotten some 3rd party company to check for rootkits. It would of costed you about half as much and they would of gotten the job done faster.

    If you need another rootkit check in future, I can recommend Steve from Rack911.com.

    Mini
      0 Not allowed!

  5. #5
    Join Date
    Apr 2006
    Location
    MA
    Posts
    193
    Okay thanks is it still not easy to follow I can split it up more and provid a more detailed timeline....
      0 Not allowed!

  6. #6
    Join Date
    Apr 2006
    Location
    MA
    Posts
    193
    Rodney Howard from LT has been helping me out and resolving problems. He's doing a good job after this entire mess - I'll get back to you with more updates.
      0 Not allowed!

  7. #7
    Join Date
    Feb 2004
    Location
    Your Screen
    Posts
    3,999
    Michael,

    I fail to see the need for a reload. Reloads are LT's answer to everything... false-alarm S.M.A.R.T. errors, "possible" exploits, sunspots, the coffee pot being empty. What evidence were you given that a reload was necessary?

    Two, never, ever, EVER ask a self-managed provider to do management. They think they know how to do it, but they don't. There is a LOT of evidence on this very board that this is the case with LT. LT knows how to put parts in a chassis, and put an OS on a drive. That's it. Do *NOT* ask them for more, they will screw up your box.

    Three, if some random stranger can send you in this kind of tizzy, you're pretty much in over your head. Pony up for the $29/month service at Platinum Server Management. They could have easily gone through and secured your box within that first few hours. At least you would have had an indication pretty quickly whether you should have been so crazy-worried or not. Then if you want to step it up and go with a pricier service like rack911 for account recovery or OS securing or whatever, you could cross that bridge if you really felt the need to go that direction.

    I can't think of any testing here that LT would have done that I couldn't have done myself. You should have known how to do this too, and if you didn't you absolutely NEED a server management company.

    I'm not saying that it doesn't sound like LT screwed this up. It does. But to me the core problem was giving them the access to the task to begin with. They were the wrong tool for the job, and now you're stuck cleaning up the mess.

    Bailey
    Let's Connect on Twitter! @thatsmsgeek2u || Fighting mediocrity one thread at a time.
      0 Not allowed!

  8. #8
    Join Date
    Apr 2006
    Location
    MA
    Posts
    193

    True

    Yeah I know what you are saying -- next time this wont happen.
      0 Not allowed!

  9. #9
    Join Date
    Apr 2006
    Location
    MA
    Posts
    193
    Email problem still not fixed.
      0 Not allowed!

  10. #10
    Join Date
    Apr 2006
    Location
    MA
    Posts
    193
    Ended up spending a few hours on google and found the solution and fixed the problem myself.

    Wow..?
      0 Not allowed!

  11. #11
    Join Date
    May 2003
    Location
    Florida
    Posts
    902
    Quote Originally Posted by Glexia
    ... level two security "specialest" (not to mention the spelling errors - I don't mind spelling errors I make them myself but I just had to point those out)
    ... which was already provided twiec.
    Sorry, I just had to point that out. Like you said, we all make spelling mistakes.


    Glad you have the problem resolved. I can understand the frustration. I am pretty happy with LayeredTech for most things, but I agree with two of the previous suggestions. Platinum Server Management is great in helping with securing your server and also helping to resolve the email problems. While they have a monthly program, they are willing to help in these one-time matters.

    If you believe the server had been rooted, I think Steven at Rack911 is one of the best.
      0 Not allowed!

  12. #12
    Join Date
    Nov 2002
    Posts
    1,467
    I think a typo and a spelling mistake are two different things. (A typo qualifying as all proper letters, in the wrong order)

    But yeah, I've been thinking of going layeredtech or softlayer. From what I can tell, if I get layered tech and want management, get someone else.
    All life is an experiment. The more experiments you make the better.
      0 Not allowed!

  13. #13
    This is pretty much on par from what LT always suggest: OS Reload or Chassis swap. It's much easier to swap the server components or reload the OS (you don't need very experienced techs to do that, so they can save money on hiring), than to go deep down into the server and look for all the individual problems. LT is fine for anything unless you need management.


    Alex
      0 Not allowed!

  14. #14
    Quote Originally Posted by Glexia
    Ended up spending a few hours on google and found the solution and fixed the problem myself.

    Wow..?
    At least it's fixed now which is the important thing and better yet, you've learnt something about servers that'll help you in similar situations after that
    Caro.Net: Support is everything
    Offering High Quality Dedicated Servers.
      0 Not allowed!

  15. #15
    Join Date
    Apr 2006
    Location
    Mandaluyong, Philippines
    Posts
    316
    Layered tech used to be good.

    Their front end support is now clueless, they leave you hang for hours, and they forget the fact that you pay your bills from the income you generate leasing their gear.

    What is really, really pitiful is you PAID FOR their extra services and nobody even bothered to see if all services were working.

    The "Yep its up close the ticket" mentality there has GOT to STOP. At least TEST your solution before you release it to a client, especially if the client PAID EXTRA for it.

    I moved every box I had out of their place over to The New York NOC ( http://www.thenynoc.com ).

    Andy has worked with me swapping drives tweaking CMOS / etc when things go wrong. I get the feeling that he wants my business so I continue to bring it there. He even made special packages and offerings for some of my new sites and has taken a genuine interest in them. They aren't a tiny DC, but it feels like your dealing with another small business, the way it used to feel at LT.

    Layered Tech treats people like cattle lately. Until they change, avoid them at all cost. You're just something else 'irritating' the tech on duty who seems to want to be doing other things. That mentality cost my business a great deal of money, that's why I moved.

    Sad. They [LT] used to be *really* good. I guess once you get to a certain size something changes. I hope I never get that big, I like keeping customers.
      0 Not allowed!

  16. #16
    Join Date
    Apr 2006
    Location
    Mandaluyong, Philippines
    Posts
    316
    Quote Originally Posted by bithost(NET)
    Michael,

    I fail to see the need for a reload. Reloads are LT's answer to everything... false-alarm S.M.A.R.T. errors, "possible" exploits, sunspots, the coffee pot being empty. What evidence were you given that a reload was necessary?

    Two, never, ever, EVER ask a self-managed provider to do management. They think they know how to do it, but they don't. There is a LOT of evidence on this very board that this is the case with LT. LT knows how to put parts in a chassis, and put an OS on a drive. That's it. Do *NOT* ask them for more, they will screw up your box.

    Three, if some random stranger can send you in this kind of tizzy, you're pretty much in over your head. Pony up for the $29/month service at Platinum Server Management. They could have easily gone through and secured your box within that first few hours. At least you would have had an indication pretty quickly whether you should have been so crazy-worried or not. Then if you want to step it up and go with a pricier service like rack911 for account recovery or OS securing or whatever, you could cross that bridge if you really felt the need to go that direction.

    I can't think of any testing here that LT would have done that I couldn't have done myself. You should have known how to do this too, and if you didn't you absolutely NEED a server management company.

    I'm not saying that it doesn't sound like LT screwed this up. It does. But to me the core problem was giving them the access to the task to begin with. They were the wrong tool for the job, and now you're stuck cleaning up the mess.

    Bailey
    Its their front end staff that need a reload. Perhaps when they fill some of these vacancies :

    http://www.layeredtech.com/careers/index.php

    Things will improve.

    What kills me is there are perfectly competent people there, Jeremy / Jack are some .. but issues never get put in front of them. Well, what happens when they go home for the day? Or its a weekend?

    Ugh. What a shame. I built my whole business around them and it was a painful (but needed) thing to do when I moved.
      0 Not allowed!

  17. #17
    Join Date
    Aug 2002
    Location
    Mumbai
    Posts
    34
    I stopped reading your post after these 2 lines.

    "Oh Thursday December 7th 2006, at 4:03 PM I submit a Layered Tech Support Ticket asking them if they could check my server urgently for rootkits and SSH backdoors "

    LT are NOT managed service provider. Their only job is to provide stable hardware and to provide you SSH prompt.

    Dont feel offended but if you need anything beyond SSH prompt then you should look for Managed Server Providers...
      0 Not allowed!

  18. #18
    Join Date
    Oct 2006
    Location
    Tipton, IN
    Posts
    417
    Quote Originally Posted by Tarak
    I stopped reading your post after these 2 lines.

    "Oh Thursday December 7th 2006, at 4:03 PM I submit a Layered Tech Support Ticket asking them if they could check my server urgently for rootkits and SSH backdoors "

    LT are NOT managed service provider. Their only job is to provide stable hardware and to provide you SSH prompt.

    Dont feel offended but if you need anything beyond SSH prompt then you should look for Managed Server Providers...
    Then they should have turned him away instead of logging in.

    The only reason why they logged in to do it is to make that $75/hr.

    Of course, one could objectively look at it as "He accepted the $75/hr, so he should accept the consequences." But, in all reality, if they couldn't fix his issue properly even if he did accept the charge for it to be done, then they should have told him instead of going ahead and messing things up.
      0 Not allowed!

  19. #19
    Join Date
    Apr 2006
    Location
    Mandaluyong, Philippines
    Posts
    316
    Quote Originally Posted by Tarak
    I stopped reading your post after these 2 lines.

    "Oh Thursday December 7th 2006, at 4:03 PM I submit a Layered Tech Support Ticket asking them if they could check my server urgently for rootkits and SSH backdoors "

    LT are NOT managed service provider. Their only job is to provide stable hardware and to provide you SSH prompt.

    Dont feel offended but if you need anything beyond SSH prompt then you should look for Managed Server Providers...
    While I agree the post is structured so its rather painful to read, if you had gotten further (or read any of the replies) you'd realize he paid them $75 an hour to correct an issue.

    That's $30 above industry standard. (Typically $40 - $45 / hourly)

    ... and they fell *flat* on their faces helping him.
      0 Not allowed!

  20. #20
    Join Date
    Jun 2006
    Posts
    407
    I had servers with LT and I had some major issues with them and Matt was also the tech the runied my server with them.

    I am not going to go into the details because it is not my thread but however the techs at LT messed the server up and then they told me that it had hardware failures and asked me I wanted my drive with all of my data moved to a new server which I already had setup with them so I could get my clients back online.

    I said yes and then they moved the drive and said it is done.

    Before they touched my server it worked fine. I was just doing a Virtuozzo migration and since they get the licenses from SW Soft they have to support the software.

    So now I own my hardware and moved to a new data center and everyone is happy again.
      0 Not allowed!

  21. #21
    Join Date
    Apr 2006
    Location
    MA
    Posts
    193
    Sorry to see that other people had the same experiences especially with the same individuals. I appreciate your comments and I think I will be moving my server to SoftLayer soon and hiring a server management company.
      0 Not allowed!

  22. #22
    Join Date
    Apr 2006
    Location
    MA
    Posts
    193

    LayeredTech Problems [MERGED]

    After being with Layered Tech for over a year and a half I have finally cancelled my server with them due to their absolutely horrible support team and company ethics. LayeredTech at first was a great provider; however, as they grow they treat clients more like cattle. Their support tickets are never read fully, the technicians take over twelve hours (in my case) to respond to a ticket. I submitted to have a reload done and have my files transferred, so they removed my backup drive from the server and never put the drive back in the server (of course they are going to come here now and somehow make the drive magically appear).

    Then, if you come to this site and post about their company they contact you and state stuff such as: “Nice to see you bashing us on WHT about this Mike. I really do not feel your comments about us are deserved.” I think customers should have freedom of speech and not have their company threatening to cancel their servers if they talk about what happened on Webhosting talk. I told them I was going to post this all on WHT and they said then you are no longer going to be a customer of ours.

    <<removed self promotion>>

    I really hope people can read this thread and understand how the practices of this firm are completely out of line and their support is completely absurd and horrible. I will be posting more information regarding this entire incident here if they let me get back into my support tickets to give you guys proof of what happened here. (They changed my password so I can’t even submit support tickets right now)

    [FONT='Verdana','sans-serif']Oh and they expected me to pay 200$ for the services that they did not even render completely, efficiently, and effectively. They sent me the invoices stating the work was completed in full, which of course it still isn't. LayeredTech just doesn’t know how to treat their customers.
    [/FONT]

    Anyone else have anything to add to this?
    Last edited by sirius; 12-24-2006 at 05:24 PM.
      0 Not allowed!

  23. #23
    Ok so you had a problem.. Let's move on and not dwell on this, much less create a website dedicated to it?
    Nothing is perfect, and I'm sure if you stopped being angry and listened to them, this could be solved in a much more peaceful manner..

    I dunno, just my 2 cents..
      0 Not allowed!

  24. #24
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Do you have any off-site or otherwise external backups? Were your only backups on the server itself?
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters
      0 Not allowed!

  25. #25
    Join Date
    Oct 2006
    Location
    Tipton, IN
    Posts
    417
    Quote Originally Posted by glexia
    I think customers should have freedom of speech and not have their company threatening to cancel their servers if they talk about what happened on Webhosting talk.
    vs
    Quote Originally Posted by glexia
    I told them I was going to post this all on WHT and they said then you are no longer going to be a customer of ours.
    There is a right way and a wrong way to post about a company that would benefit people more in the long run, and also give said company a look into what they can do to make things better.

    Due to this entire incident, I am going to be starting the website LayeredTechSUCKS.com for everyone who has a problem with LayeredTech to post their reviews here. I will post my entire story as well as other opinions on this website.
    While I agree there are some major problems at LayeredTech that need ironed out real fast. I do not think this would be the mature avenue for you to be taking.

    My advice to you would be, if there is a problem with the service. State the problem in a mature and constructive manor, and just leave it at that. If LT cannot bare to make things right, then that's their problem. But, I would definantly recommend that you keep your problem with LT as yours, and not get into anyone elses problems with LT. Let them handle their own problems.
      0 Not allowed!

Page 1 of 4 1234 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •