Our network provider, Global Netoptex, has been experiencing problems with their networking equipment. This is affecting the ability of users from certain geographic areas from connecting to TeamWork Live and to other sites that are using Global Netoptex as their network provider. Global Netoptex has been working to resolve the issue by replacing all affected hardware. We apologize to anyone that has had problems connecting to TeamWork Live. We will keep you posted as to the status of the issue as we learn more.

I want to let everyone know that we have been experiencing some site instability since April 9, when we had a hardware failure on one of our servers. The resulting instability is being caused by our automated recovery system which seems to need some tuning as it is now causing service problems instead of preventing it.

I want to apologize for the instability. We are working around the clock to get it fixed and want you to know that this is not the norm. If you have been materially affected by the instability, please contact me and we will try to make things right for you.

UPDATE: We believe that we have resolved all the stability issues. Thank you for your patience while we address this. We will be crediting everyone with a free week to make up for the disruptions.

Regards,
Tuyen Truong, CEO

We experienced a site outage from 10:36AM - 11:27AM PST (5:36PM - 6:27PM GMT). We are still looking into the cause of the outage and will report it as we learn more. We apologize for any inconvenience this has caused you. Please contact us if you were materially affected by the outage and we will try to make it right for you.

UPDATE: The problem was caused a hardware failure. Our automatic recovery programs were supposed to switch over to another server automatically but it did not do so. We will be working to fix it so that there is no downtime for such failures in the future.

One of our database servers became unresponsive and had to be restarted. We are looking into what caused the problem now. We apologize for any inconvenience this may have caused and will update you when we have more information.

UPDATE: We have determined that a required backup area for one of the databases had become full. The database automatically stopped as a precaution because backups of your data couldn't be made. We have updated our automated backup system to prevent the area from filling up in the future.

TeamWork Live was down for about 30 minutes today due to problems with our servers. We are investigating the cause now and will update you with more information when it's available. We apologize sincerely for the interruption.

UPDATE: We just learned that there was a major power outage at 365Main, our server colocation facility. I guess we are in good company as Craigslist, Yelp, and Technorati were also among the many companies affected. Despite 365Main's claim of having redundant power supply, it seems that it is not redundant enough. We will look into adding servers at a second location (in addition to our backup servers which are already at another colocation facility) to ensure better reliability in the future. We apologize again for the inconvenience.

We are experiencing some site issues with sending emails. We are working to resolve the issue now.

We want to apologize for the glitches you may have experienced with TeamWork Live the last few days. They were caused by:
1. We upgraded some of our software to make the site run faster. The upgraded software made our site run faster. Unfortunately, we found out yesterday that the software also caused the server to crash when certain types of image files were uploaded. We have reverted to the original version until the new version is more stable.
2. We upgraded a JavaScript library used throughout the site for our user interface. The library created some issues with sending messages and formatting comments. Those issues have now been addressed.
3. We are still working through an issue where one of our servers is hanging when it tries to send out an email. You may experience this problem when trying to notify other users by email when adding a comment or sending a message.