Can you imagine trying to sell an email service that guarantees you’ll miss some emails? Yet that is what Twitter promises. There is no guarantee you’re seeing all tweets with your username. Twitter provides three primary ways to view tweets:
- Mentions: All tweets that includes your @username, e.g. in our case @TweetSmarter
- Timeline: All the tweets of all the people you follow (your “followees”—with the exception of certain @ mentions)
- Search: This requires more elaboration.
The way most people think of Search (#3) is by going to Search.Twitter.com and entering keywords to find tweets that match. But Twitter.com and many apps also allow you to save searches, and additionally, many apps allow you to “filter” methods #1 and #2, converting them into more complex searches.
What’s the problem?
Whenever you are looking at a filtered stream of tweets, whether your app technically calls it a search or not, you are looking at tweets from the Twitter search API. The problem is this:
Twitter search doesn’t guarantee it will show all tweets, and fails EVERY DAY at least a little bit. And the more tweets there are, the more likely it is that some tweets won’t be shown (a serious problem for popular users).
Plus, there are often issues with Twitter search that Twitter is working on, small errors that cause some searches a small amount of the time to work improperly, beyond the general failure of Twitter search to work perfectly.
What does Twitter say about this?
Mention a specific search that repeatedly fails to Twitter support, and they will repond “Unfortunately, we are unable to provide a specific timeline for this issue.” Press them, and they will a link to this Twitter support page:
“Some users may not be seeing their Tweets because of resource constraints….Our search engineers are working on this known issue.” While on that page it suggests this as a reason that you might not be seeing ANY tweets, Twitter support employees link to that page to say, in essence, “Sorry, whether you’re missing a random tweet or a lot of tweets, we’ve publicly pointed this out and are working on it.”
And Twitter sometimes has specific issues with main timelines (tweets from your followers) and mentions timeline (tweets with your username). Some of these issues are never listed in Twitter’s help section, but there are listed as known issues within Twitter.
When Twitter intentionally removes your tweets
Twitter will also remove many tweets on purpose, from accounts that it feels don’t have anything worth adding to search results. In this case, it describes your tweets as “being filtered out of search due to a quality issue.” “Frequent and repetitious status updates may, in some instances, and in combination with other metrics, result in a [low] status quality score for a given account” causing it’s tweets to be removed from “Both the Streaming API and the Search API filter.”
Specific Problems that Twitter does address
Sometimes certain issues with missing tweets get out of hand, and Twitter lists them on their help page, in part to collect comments from users to see how many are affected. But just because an issue is NOT listed here doesn’t mean there is no problem.
How bad is it? Twitter’s “quality” promise
Here are a few highlights from Twitter’s Streaming API Quality of Service section:
- Without notice, statuses may be missing from the delivered stream;
- Statuses will arrive in any order (not always completely chronological—newer tweets may appear after older tweets);
- Duplicate and [unsorted] statuses may be delivered;
- The Streaming API may disconnect your connection at any time, perhaps multiple times per day;
Twitter goes on to state that if you are making an app to read tweets for people, “Applications must tolerate duplicate statuses, out-of-order statuses and non-status messages … Proper client coding will prevent data loss due to occasional disconnections.”
Obviously, if tweets are delivered without being too late, apps can mostly sort things out for you. But when they’re very late or completely missing, no such luck. And Twitter does not provide any such “quality” information for the Twitter search API. Of course, if you missed something that is over about a week old, you won’t find it via search.twitter.com anymore. And if it’s too many tweets ago, you can find it through special API searches, such as what Searchtastic.com can do for you, because although the one week limit isn’t used, there are other search limits. Not to mention that API searches can have their own problems.
Does Twitter promise to show all Tweets to anyone?
Twitter says if you buy access to the stream of tweets, you still won’t see them all—they say you will see a “subset”—unless you pay for full access. Yup: you have to pay for special access to see ALL tweets. So although for average users using free tools, Twitter says “resource constraints” may cause you not to see all tweets, if you’ve got the bucks, they will show them all to you, whether the full stream or any search filtered results you wish. This doesn’t make me hopeful that those “resource constraints” are going to go away for average users any time soon. Maybe this is something Twitter would take care of if Google were to buy them out
How does this affect you?
In my experience, the most malfunctioning types of searches are
- “to:myusername” searches;
- Complex, multi-word searches;
- Searches that return a lot of results.
This is really, really unfortunate, because these are often the most popular kinds of searches. For example, if you are a popular user trying to see the most relevant tweets with your username in them:
- Search #1 doesn’t work consistently;
- Doing a search for your username and then filtering to remove retweets (e.g. @username -RT -via) is a type #2 search;
- And the fact that you’re popular and get a lot of results to any search for your username makes your searches type #3
However, not all users are always affected equally. Believe it or not, Twitter admits that specific users can have problems while other users remain problem free.
If you are using an app that shows you a search results column, sometimes old tweets will appear suddenly further down the timeline, sandwiched in between other old tweets. That’s right, one moment a tweet from yesterday might now show, and on the next refresh of the column, there it is.
What can you do?
What I do on my @TweetSmarter account is I first reply to all tweets found by filtered searches for mentions of @TweetSmarter.
For example, I’ll do a search that removes common automated tweet types, such as daily twitter “newspapers” and whatever the last spammy game is, filter out follow friday and retweets, and then read the remaining tweets. An example search would look something like this:
@TweetSmarter -#EAv -buzztweetbingo -RT -via -#followfriday -#ff -daily -snowball
And sometimes, I might simply look for users with questions, e.g.
Then 1-3 times a day, I also check a search for simply @TweetSmarter which returns up to a thousand tweets a day (that’s why I don’t check it first!) I would like to use to:tweetsmarter as a search, but it’s worked so badly for me so many times I mostly just ignore it now.
And just using the “mentions” column won’t protect you from missing Tweets. For example, I just shared this info with a user, who said “I use main Mentions column…just added a search column & instantly see 2 tweets this morning that I didn’t get. Yikes.”
The browser’s cache bypass trick
Twitter will often suggest this, as browser problems are common sources of errors for Twitter.com. This is a common solution for missing DMs if you are viewing them on the Twitter.com web interface.
How bad is it in reality? “Each day is a nightmare.”
For the searches outlined above, I find errors every single day in the results Twitter search gives, e.g. Tweets that should show up in my filtered searches…don’t. That’s right, tweets are missing every DAY. And there are probably tweets that I never see, given Twitter’s “resource constraints” and the fact that I don’t pay to see tweets.
Many users, having never tested for this issue, may feel they are seeing all tweets. And they may be. But also, some large users have had communications with Twitter and been misinformed about the problem, in my opinion. For example, on the same day I last tested and found to:username searches to be NOT working (and filed another ticket with Twitter to see what they would say), Twitter employee @Jess was telling Om Malik that it WAS working.
Om’s reply when I informed him? “Each day is a nightmare of sorts now when it comes to Twitter/at replies” Not surprising that @Om would be intimately familiar with the problem, as he is a popular and engaged Twitter user—exactly the kind of user Twitter most wants, and exactly the kind of Twitter user that gets the worst experience in trying to find tweets.