Follow

Working on a wildly hacky Tweet importer for preserving/continuing conversations. It writes directly into the backing Mastodon database so it's really only an option for self-hosted users.

My first import: mastodon.jakewharton.com/@jw/8

Had all kinds of problems inserting directly into the Mastodon DB. For some reason links weren't becoming linkified (this is separate from the preview card).

I switched to patching the Mastodon app to allow creating statuses with a timestamp through the API: github.com/JakeWharton/dumbo#m. The idea came from someone else's import tool: github.com/lucahammer/fediport.

Links now work, and we get all kinds of things for free like those link preview cards. Support for media and threaded replies are up next.

@jw one could host an instance that lets people import their old tweets as an archive.

@henriquesilva Yes "self-hosted" isn't the right term. It will work on any Mastodon instance so long as the host provided access to the backing database.

@jw 🤯
Now it feels complete 🧘🏽‍♂️

However, I'd imagine that Mastedon has Twitter importer plugin that would do something like that. I guess I'm wrong. Or, is Dumbo the plugin you made?

@hossainkhan Didn't find one. Plus mine works on archives rather than the live Twitter website.

@jw 👍🏽

While looking at the timeline, it made me realize that none of these platforms have a great way to revisit old posts.

I had to press "Show older" 4x+ times to get to the last item out of 160 posts here.

Imagine importing all 15K tweets! 🙀

Would be great to jump to a specific year/month/day!

Ref: The last page was at `@jw?max_id=85165291339829460`

@jw Is Mastodon clever enough to publish everything manually inserted into its DB? I'd have thought there needs to be some sort of 'publish' which happens?

@chris Since they're all very old, I don't expect them to be sent out. And actually I don't want them to be. It should behave as if I had been tweeting for a decade and then you followed me. You'll have to go to my profile and scroll and scroll to find them.

@jw I’ve been tooting since birth.

I’ll happily test this out if you need any volunteers!

@chris @jw I'm curious how this interacts with other instances, yeah...

One bit of anecdata: fedilab doesn't seem to find these 2011 toots when browsing your profile.

@jw @chris someone tried it through the api and wrote some notes about the behaviour here github.com/lucahammer/fediport

@Memorion @chris Oh, nice! I thought about a similar approach but I'm so terrible at Ruby I didn't even want to attempt it.

@bencord0 Woah, very cool! Keeping the same status IDs is a neat idea. I replicated the Mastodon snowflake ID algorithm in code so I could get the IDs that would have been generated historically.

I've noticed that links don't seem work when I insert directly into the DB, and it looks like yours don't either (ex: nfra.club/@bencord0/1027568398). I can't find anything in the DB that I'm doing wrong or missing when comparing to regular statuses I post. Did you ever look into that?

@jw I think I need to run the LinkCrawlerWorker on those entries.

It's not been a high priority for me to make all previews work, especially since many of them now refer to (dead) Twitter links.

Tempted to see if I can use the Internet Archive's API for previews for those cases. That may need modifying the LinkCrawler code

@bencord0 I looked at that, but it seems to populate only the preview_cards DB table. I didn't see anything that would cause links to become clickable.

I think I may try to modify the mastodon code like this: github.com/lucahammer/fediport. They added a way to specify a timestamp in the regular API call. And since it's all in a container I can just restart once I'm done with the import.

@jw can you also theoretically add posts into the future?

> The Mastodon API does not allow inserting posts in the past. In order to write statuses which occur in the past, a created_at parameter is needed.

Could something like that also then share to other servers? I would hope there's a failsafe for that :partyparrot: . That could really weird up some timelines.

@knossos Yes, absolutely it could. I'm not sure what would happen. Would it stick to the top of everyone's feed? I might have to try it!

@jw That was my thought too. I would expect it to behave like some kind of ultra sticky toot.

And likely other instances couldn't easily deal with it either, since only you can delete your toots (being the instance owner).

People would need to patch their instances and or ban misbehaving instances.

A fun thought experiment in any case. I look forward to your findings :dumpsterfire: :mastolaughing:

Sign in to participate in the conversation
Mastodon

This server is a place for Jake Wharton. Are you Jake Wharton? This is your place. Are you not Jake Wharton? Well, at least you can find him here.