Когда в 2013 году закрыли Google Reader, у него довольно быстро появилась альтернатива – The Old Reader. Сервис аккуратно скопировал всё хорошее, довольно быстро сам собой распиарился и привлек много народа.

Делали его какие-то русские разработчики, которые, видимо, не рассчитали силы. После массового наплыва сайт начал тормозить, а команда затеяла переезд, который прошёл не очень гладко. В итоге сервис несколько дней лежал, а пользователи бухтели:

On Saturday (July 20) we moved over a terabyte of data from one storage system to another. We made the move because the amount of data we have to store simply did not fit on our servers, and our preliminary tests showed that the new system will only be using about ⅓ of the disk space.

Migration went through fine, however we started seeing a higher I/O load after we finished, and suddenly one SSD drive in one of our database servers stopped working. Not a big deal, we thought — we obviously store data in mirrored mode on several servers — so we asked our hosting provider to swap the drive. We had to re-sync the data to the new disk, so you might have noticed the site being flacky.

However while data was syncing, another SSD drive went down in a different server. We got it replaced, and started syncing two database shards at the same time. At this point we lost 2 more drives, one of which unfortunately was on the server in one of the shards that were repairing.

While you can fly a plane with an engine off, unfortunately when all your engines stop your flight is over. Right now we have to restore our last pre-migration full database backup, and apply incremental updates to bring database to a fresh and (hopefully) consistent state.

Here comes the worst news - this will probably take a day or two. Sorry about that.

This is a tough and incredibly stressful situation, but it looks like we have no other choice. We understand your frustration (actually, we are in the same boat: we are RSS junkies and built The Old Reader for ourselves and friends) and we are doing everything to make it as fast and painless as possible and live happily ever after.

After that, we will deploy bug fixes along with new things and improvements we have already developed. During last year we adapted and successfully expanded first from 2000 to 5000 users practically overnight, then from 10 000 to 160 000 in several weeks and from 200 000 to 400 000 in four months, so we are considering this as a new level-up for the project (although bumpy and painful one).

It’s 5 AM right now and backup restoration has already begun. We are monitoring and working on The Old Reader nearly 24 hours a day. We will keep frequent (but not annoying updates) in Twitter and will answer all your questions.

We deeply apologize for what has happened but we intend to come back in a much better way. Thank you very much for your patience, support and understanding,

The Old Reader team.

UPD:

July 25, 21:07 UTC

Back online! We hope this outage lasting July 25 19:12 UTC If everything goes as planned, we should be back in 4-5 hours.

July 25 15:50 UTC

Import — check; Indexes — check; Balancing data between shards and configuring replicasets — in progress.

July 25 08:45 UTC

It looks like we have managed to upload the data. If indexes get generated correctly, we might be back > online later today.

July 25 2:00 UTC

Continuing the upload, hoping it goes as planned, counting hours.

July 24 14:00 UTC

Proceeding with restore. More details hopefully in the evening.

July 23 18:00 UTC

We have managed to create a consistent dump of our database and started uploading it to the database servers.

В какой-то момент у разработчиков не выдержали нервы и (за давностью лет уже точно не помню) они выкатили какое-то эмоциональное письмо про то, что очень устали, а пользователи этого не ценят.

Примерно в это же время то же самое, но в чуть более сдержанных тонах они опубликовали в официальном блоге. Доступ к сервису собирались закрыть, оставив его для семьи, друзей и ранних пользователей:

Since we launched first public version almost a year ago up until March 2013 we have been working on The Old Reader in “normal” mode. In March things became “nightmare”, but we kept working hard and got things done. First, we were out of evenings, then out of weekends and holidays, and then The Old Reader was the only thing left besides our jobs. Last week difficulty level was changed to “hell” in every possible aspect we could imagine, we have been sleep deprived for 10 days and this impacts us way too much. We have to look back.

The truth is, during last 5 months we have had no work life balance at all. The “life” variable was out of equation: you can limit hours, make up rules on time management, but this isn’t going to work if you’re running a project for hundreds of thousands of people. Let me tell you why: it tears us to bits if something is not working right, and we are doing everything we can to fix that. We can’t ignore an error message, a broken RAID array, or unanswered email. I personally spent my own first wedding anniversary fixing the migration last Sunday. Talk about “laid back” attitude now. And I won’t even start describing enormous sentimental attachment to The Old Reader that we have.

We would really like to switch the difficulty level back to “normal”. Not to be dreaded of a vacation. Do something else besides The Old Reader. Stop neglecting ourselves. Think of other projects. Get less distant from families and loved ones. The last part it’s the worst: when you are with your family, you can’t fall out of dialogues, nodding, smiling and responding something irrelevant while thinking of refactoring the backend, checking Graphite dashboard, glancing onto a Skype chat and replying on Twitter. You really need to be there, you need to be completely involved. We want to have this experience again.

That’s why The Old Reader has to change. We have closed user registration, and we plan to shut the public site down in two weeks. We started working on this project for ourselves and our friends, and we use The Old Reader on a daily basis, so we will launch a separate private site that will keep running. It will have faster refresh rate, more posts per feed, and properly working full-text search — we are sure that we can provide all this at a smaller scale without that much drama, just like we were doing before March.

Примерно в тот момент я понял, что ну его на фиг, и пересел на Feedly, котором пользовался без малого 10 лет до тех пор, пока Telegram окончательно не поглотил все rss-фиды, ставшие каналами.

На днях с удивлением обнаружил, что сайт еще работает, а сервис после факапов 2013 года не закрылся, а существовал довольно долго – на blog.theoldreader.com есть новости аж от 2021 года.